This page includes the more commonly asked questions. For less frequently asked questions, or to ask your own question, see the
shake-build-system tag on StackOverflow.
- User manual – the place to start, which also serves as a tutorial.
- Why – details on why you should use Shake, for those who are easily influenced.
- Includes – how to deal with
#includefiles, import statements and other dependencies between files.
- Debugging – how to determine what a Shake build system is doing.
- Profiling and optimisation – how to speed up an existing Shake build system.
- Command line flags – the flags and settings supported by Shake, a better version of
- Developing Shake – notes for people who want to contribute to Shake itself.
- Ninja – features of Shake for those people who use Ninja.
Much of the theory behind Shake is covered in a conference paper which was accompanied by this video (slides). Since then I've given videoed talks on small worked examples (slides) and how to structure large Shake systems (slides).
I sometimes write about ongoing development work or other Shake-related things on my blog.
If you have any further questions:
- Ask on StackOverflow, using the tag
- Email us for any questions/bugs/thoughts on Shake. If you need more information and aren't sure where to start, use the mailing list.
Not at all – Shake can build any project in any combination of languages. In fact, Shake isn't typically necessary for vanilla Haskell projects, as you can use
Shake is a Haskell package focused on providing build-system functionality. Since Shake scripts are written in Haskell, they can easily access other Haskell packages. Most general needs are met by the standard
base library, but a few other useful general functions can be found in the
extra library (e.g.
replace). For more specific functionality (e.g. parsing, databases, JSON) find a suitable Haskell library and use that.
Most users will write their own Haskell file and compile it to produce an executable that is their build tool. The
shake executable is there to run the demo, run Ninja build files and will also run a
Shakefile.hs if present.
No. If two patterns overlap for a file being built it will result in a runtime error – you cannot have a pattern for
*.txt, and another for
foo.*, and then build a file named
foo.txt. For objects that typically share the same extension (e.g. C and Haskell both produce
.o objects), either disambiguate with a different extension (e.g.
.hs.o), or different directory (e.g.
obj/hs/**/.o). For more information, including ways to enable overlap and set priorities, see
need xs >> need ys will build
xs in parallel, then afterwards build
ys in parallel. The same is true of
need xs *> need ys, where
*> is the applicative equivalent of
>>. In contrast, Haxl will execute both arguments to
*> in parallel. For Shake, you are encouraged to merge adjacent
need operations (e.g.
need (xs++ys)), and where that is not possible (e.g. when using
Shake could follow the Haxl approach, but does not, mainly because they are targeting different problems. In Haxl, the operations are typically read-only, and any single step is likely to involve lots of operations. In contrast, with Shake the operations definitely change the file system, and there are typically only one or two per rule. Consequently, Shake opts for an explicit approach, rather than allow users to use
*> (and then inevitably add a comment because its an unusual thing to do).
We recommend using only relative file names in Shake.
Shake can use file names that are either absolute (
/file.txt) or relative (
file.txt). However, Shake compares filenames by value, so if you
/file.txt it will attempt to build both, likely resulting in failure - within a single Shake project you must stick to either relative or absolute file names.
Think of directories as containers for files. They exist or don't pretty randomly, but if they have files, they must exist. In particular, you can't depend on a directory with
need or write a rule to create a directory. Directories are created as needed – the rule for
bar/baz.exe will create the
bar directory if necessary. As some examples of "depending" on directories:
- If you want to depend on a
git clonehaving being performed, depend on a particular checked-out file instead (e.g.
README.md), with the action in the rule to create it being
- If you want to depend on a
node_moduleshaving been updated after changing
package.json, create a stamp file by copying
npm update(see this detailed answer).
There is a tracked function
doesDirectoryExist, to depend on the presence or absence of a directory, but you should not call it on directories which might be created by the build system.
Shake works well when run by Continuous Integration (CI) systems. There are a few tweaks that sometimes make Shake work even better for CI:
progressDisplay 5 putStrLnwill print the progress information to stdout every 5 seconds. In many CI systems, you can tag this information to display it through the UI - e.g. TeamCity spots messages of the form
##teamcity[progressMessage 'message goes here'].
EchoStdout Truewill ensure that the standard output of all failing commands is captured in the final error messages.
Truewill cause the CI to find as many errors as it can - taking longer to fail, but producing a great number of errors when it does.
shakeErrorsDatabaseyou can collect the exceptions for many build rules, allowing them to be presented in a CI dashbaord.
For some reason, Shake tends to find a reasonable number of serious bugs in GHC, given how few bugs there are generally in GHC. I suspect the reason is a combination of thorough testing including with GHC pre-releases. Some of the best bugs found by Shake are:
- GHC bug 7646, a race condition when closing file handles that had been in several releases.
- GHC bug 10830,
maximumByhad a space leak in a released version.
- GHC bug 11555,
catchwouldn't catch an
undefinedargument passed to it in a pre-release.
- GHC bug 11830, livelock when disabling idle GC in a pre-release.
- GHC bug 11458 (originally from GHC bug 11379), serious issue with type applications producing dodgy programs in a pre-release.
- GHC bug 11978, segfaults when running certain profiling modes that weren't multithread safe.
- GHC bug 10553,
getEnvironmentwas blank when run on PowerPC in
- GHC bug 10549, inconsistent optimisation flags leading to fatal errors in a pre-release.
- GHC bug 10176, invalid optimisations caused by a part of GHC that had been formally proved to be correct, in a pre-release.
- GHC bug 10793,
BlockedIndefinitelyOnMVarcan be raised even if the thread isn't indefinitely blocked.
- GHC bug 15595,
withArgson a limited stack loops forever consuming memory.
The biggest challenge for Yhc was the build system – we ended up with 10,000 lines of Python Scons scripts. Without a robust build system nothing else matters. When our sole Python hacker left the team that was the beginning of the end.
A Haskell compiler is a big undertaking, but the build system for a simple Haskell compiler shouldn't be that complicated.
When writing my thesis I needed a build system, and decided to try writing a simple Haskell DSL, which is still online here. I defined a single operator
<== which let me express a relationship between an output and its dependencies – very simple, but it worked.
Later I moved to Standard Chartered, where the build system was a mass of Makefiles, and it quickly became apparent that the project had outgrown the current approach. Without really surveying the alternatives, I decided that a Haskell DSL would be easiest to fit in with the existing infrastructure, so started writing some code. The first version of the build library took under a week, followed by a month of reimplementing the existing system. It wasn't until many months later I realised that the reason everything was suddenly so much easier was because we had monadic dependencies.
While people at Standard Chartered wanted to open source Shake, that turned out not to be possible. A number of people in the Haskell community implemented their own versions of Shake, but none were as polished or as strong as our internal one. Eventually, I reimplemented Shake, from scratch, in my spare time. Writing Shake from scratch, without the original source code or documentation, it naturally turned out better than the first attempt. A while later Standard Chartered migrated to the open-source version.