FAQ

This page includes the more commonly asked questions. For less frequently asked questions, or to ask your own question, see the shake-build-system tag on StackOverflow.

Q: What else is on this website?

Q: Any more documentation?

There is a complete list of every function in Shake which can be searched. Each function comes with documentation and examples.

Much of the theory behind Shake is covered in a conference paper which was accompanied by this video (slides). Since then I've given videoed talks on small worked examples (slides) and how to structure large Shake systems (slides).

I sometimes write about ongoing development work or other Shake-related things on my blog.

If you have any further questions:

Q: Is Shake limited to building Haskell?

Not at all – Shake can build any project in any combination of languages. In fact, Shake isn't typically necessary for vanilla Haskell projects, as you can use cabal or stack. Shake is often used for building C/C++, Docker containers and JavaScript/HTML/CSS projects.

Q: Where are functions for string manipulation?

Shake is a Haskell package focused on providing build-system functionality. Since Shake scripts are written in Haskell, they can easily access other Haskell packages. Most general needs are met by the standard base library, but a few other useful general functions can be found in the extra library (e.g. trim and replace). For more specific functionality (e.g. parsing, databases, JSON) find a suitable Haskell library and use that.

Q: Why is there a shake executable?

Most users will write their own Haskell file and compile it to produce an executable that is their build tool. The shake executable is there to run the demo, run Ninja build files and will also run a Shakefile.hs if present.

Q: Can file patterns overlap?

No. If two patterns overlap for a file being built it will result in a runtime error – you cannot have a pattern for *.txt, and another for foo.*, and then build a file named foo.txt. For objects that typically share the same extension (e.g. C and Haskell both produce .o objects), either disambiguate with a different extension (e.g. .c.o and .hs.o), or different directory (e.g. obj/c/**/.o and obj/hs/**/.o). For more information, including ways to enable overlap and set priorities, see %>.

Q: Do multiple calls to need run sequentially? Are Applicative actions run in parallel?

In Shake, need xs >> need ys will build xs and ys in parallel, since version 0.17.10, in a similar manner to Haxl. As a consequence, enabling the ApplicativeDo extension may cause more of the build run in parallel. However, this implicit parallelisation of adjacent dependencies is easy to loose, for example by defining need in terms of >>=. Users are encouraged to merge adjacent need operations (e.g. need (xs++ys)), and where that is not possible use parallel explicitly.

Q: Should file names be relative or absolute?

We recommend using only relative file names in Shake.

Shake can use file names that are either absolute (C:\file.txt, /file.txt) or relative (file.txt). However, Shake compares filenames by value, so if you need both file.txt and /file.txt it will attempt to build both, likely resulting in failure - within a single Shake project you must stick to either relative or absolute file names.

Q: How can I depend on directories?

Think of directories as containers for files. They exist or don't pretty randomly, but if they have files, they must exist. In particular, you can't depend on a directory with need or write a rule to create a directory. Directories are created as needed – the rule for bar/baz.exe will create the bar directory if necessary. As some examples of "depending" on directories:

There is a tracked function doesDirectoryExist, to depend on the presence or absence of a directory, but you should not call it on directories which might be created by the build system.

Q: Does Shake work with Continuous Integration?

Shake works well when run by Continuous Integration (CI) systems. There are a few tweaks that sometimes make Shake work even better for CI:

Q: What GHC bugs has Shake found?

For some reason, Shake tends to find a reasonable number of serious bugs in GHC, given how few bugs there are generally in GHC. I suspect the reason is a combination of thorough testing including with GHC pre-releases. Some of the best bugs found by Shake are:

Q: What's the history of Shake?

I (Neil Mitchell) was one of the people behind the Yhc project, a Haskell compiler that died in a large part because of its build system. To quote from the final blog post:

The biggest challenge for Yhc was the build system – we ended up with 10,000 lines of Python Scons scripts. Without a robust build system nothing else matters. When our sole Python hacker left the team that was the beginning of the end.

A Haskell compiler is a big undertaking, but the build system for a simple Haskell compiler shouldn't be that complicated.

When writing my thesis I needed a build system, and decided to try writing a simple Haskell DSL, which is still online here. I defined a single operator <== which let me express a relationship between an output and its dependencies – very simple, but it worked.

Later I moved to Standard Chartered, where the build system was a mass of Makefiles, and it quickly became apparent that the project had outgrown the current approach. Without really surveying the alternatives, I decided that a Haskell DSL would be easiest to fit in with the existing infrastructure, so started writing some code. The first version of the build library took under a week, followed by a month of reimplementing the existing system. It wasn't until many months later I realised that the reason everything was suddenly so much easier was because we had monadic dependencies.

While people at Standard Chartered wanted to open source Shake, that turned out not to be possible. A number of people in the Haskell community implemented their own versions of Shake, but none were as polished or as strong as our internal one. Eventually, I reimplemented Shake, from scratch, in my spare time. Writing Shake from scratch, without the original source code or documentation, it naturally turned out better than the first attempt. A while later Standard Chartered migrated to the open-source version.

I still maintain Shake, but am fortunate to have other contributors extending and improving Shake. If you want to join in, see notes for developers.