FAQ
This page includes the more commonly asked questions. For less frequently asked questions, or to ask your own question, see the shake-build-system
tag on StackOverflow.
Q: What else is on this website?
- User manual – the place to start, which also serves as a tutorial.
- Why – details on why you should use Shake, for those who are easily influenced.
- Includes – how to deal with
#include
files, import statements and other dependencies between files. - Debugging – how to determine what a Shake build system is doing.
- Profiling and optimisation – how to speed up an existing Shake build system.
- Command line flags – the flags and settings supported by Shake, a better version of
--help
. - Developing Shake – notes for people who want to contribute to Shake itself.
- Ninja – features of Shake for those people who use Ninja.
- Lint – how to use the linting features.
- Controlling rebuilds – how to use the linting features.
Q: Any more documentation?
There is a complete list of every function in Shake which can be searched. Each function comes with documentation and examples.
Much of the theory behind Shake is covered in a conference paper which was accompanied by this video (slides). Since then I've given videoed talks on small worked examples (slides) and how to structure large Shake systems (slides).
I sometimes write about ongoing development work or other Shake-related things on my blog.
If you have any further questions:
- Ask on StackOverflow, using the tag
shake-build-system
. - Email us for any questions/bugs/thoughts on Shake. If you need more information and aren't sure where to start, use the mailing list.
Q: Is Shake limited to building Haskell?
Not at all – Shake can build any project in any combination of languages. In fact, Shake isn't typically necessary for vanilla Haskell projects, as you can use cabal
or stack
. Shake is often used for building C/C++, Docker containers and JavaScript/HTML/CSS projects.
Q: Where are functions for string manipulation?
Shake is a Haskell package focused on providing build-system functionality. Since Shake scripts are written in Haskell, they can easily access other Haskell packages. Most general needs are met by the standard base
library, but a few other useful general functions can be found in the extra
library (e.g. trim
and replace
). For more specific functionality (e.g. parsing, databases, JSON) find a suitable Haskell library and use that.
Q: Why is there a shake
executable?
Most users will write their own Haskell file and compile it to produce an executable that is their build tool. The shake
executable is there to run the demo, run Ninja build files and will also run a Shakefile.hs
if present.
Q: Can file patterns overlap?
No. If two patterns overlap for a file being built it will result in a runtime error – you cannot have a pattern for *.txt
, and another for foo.*
, and then build a file named foo.txt
. For objects that typically share the same extension (e.g. C and Haskell both produce .o
objects), either disambiguate with a different extension (e.g. .c.o
and .hs.o
), or different directory (e.g. obj/c/**/.o
and obj/hs/**/.o
). For more information, including ways to enable overlap and set priorities, see %>
.
Q: Do multiple calls to need
run sequentially? Are Applicative
actions run in parallel?
In Shake, need xs >> need ys
will build xs
and ys
in parallel, since version 0.17.10, in a similar manner to Haxl. As a consequence, enabling the ApplicativeDo
extension may cause more of the build run in parallel. However, this implicit parallelisation of adjacent dependencies is easy to loose, for example by defining need
in terms of >>=
. Users are encouraged to merge adjacent need
operations (e.g. need (xs++ys)
), and where that is not possible use parallel
explicitly.
Q: Should file names be relative or absolute?
We recommend using only relative file names in Shake.
Shake can use file names that are either absolute (C:\file.txt
, /file.txt
) or relative (file.txt
). However, Shake compares filenames by value, so if you need
both file.txt
and /file.txt
it will attempt to build both, likely resulting in failure - within a single Shake project you must stick to either relative or absolute file names.
Q: How can I depend on directories?
Think of directories as containers for files. They exist or don't pretty randomly, but if they have files, they must exist. In particular, you can't depend on a directory with need
or write a rule to create a directory. Directories are created as needed – the rule for bar/baz.exe
will create the bar
directory if necessary. As some examples of "depending" on directories:
- If you want to depend on a
git clone
having being performed, depend on a particular checked-out file instead (e.g.README.md
), with the action in the rule to create it beinggit clone
. - If you want to depend on a
node_modules
having been updated after changingpackage.json
, create a stamp file by copyingpackage.json
after runningnpm update
(see this detailed answer).
There is a tracked function doesDirectoryExist
, to depend on the presence or absence of a directory, but you should not call it on directories which might be created by the build system.
Q: Does Shake work with Continuous Integration?
Shake works well when run by Continuous Integration (CI) systems. There are a few tweaks that sometimes make Shake work even better for CI:
- Setting
shakeProgress
toprogressDisplay 5 putStrLn
will print the progress information to stdout every 5 seconds. In many CI systems, you can tag this information to display it through the UI - e.g. TeamCity spots messages of the form##teamcity[progressMessage 'message goes here']
. - Setting
shakeCommandOptions
toEchoStdout True
will ensure that the standard output of all failing commands is captured in the final error messages. - Setting
shakeStaunch
toTrue
will cause the CI to find as many errors as it can - taking longer to fail, but producing a great number of errors when it does. - Using
shakeErrorsDatabase
you can collect the exceptions for many build rules, allowing them to be presented in a CI dashbaord.
Q: What GHC bugs has Shake found?
For some reason, Shake tends to find a reasonable number of serious bugs in GHC, given how few bugs there are generally in GHC. I suspect the reason is a combination of thorough testing including with GHC pre-releases. Some of the best bugs found by Shake are:
- GHC bug 7646, a race condition when closing file handles that had been in several releases.
- GHC bug 10830,
maximumBy
had a space leak in a released version. - GHC bug 11555,
catch
wouldn't catch anundefined
argument passed to it in a pre-release. - GHC bug 11830, livelock when disabling idle GC in a pre-release.
- GHC bug 11458 (originally from GHC bug 11379), serious issue with type applications producing dodgy programs in a pre-release.
- GHC bug 11978, segfaults when running certain profiling modes that weren't multithread safe.
- GHC bug 10553,
getEnvironment
was blank when run on PowerPC inghci
. - GHC bug 10549, inconsistent optimisation flags leading to fatal errors in a pre-release.
- GHC bug 10176, invalid optimisations caused by a part of GHC that had been formally proved to be correct, in a pre-release.
- GHC bug 10793,
BlockedIndefinitelyOnMVar
can be raised even if the thread isn't indefinitely blocked. - GHC bug 15595,
withArgs
on a limited stack loops forever consuming memory. - GHC bug 17575,
runhaskell -package=Cabal
causes an error. - GHC bug 18221,
forkOn
has complexity O(n^2).
Q: What's the history of Shake?
I (Neil Mitchell) was one of the people behind the Yhc project, a Haskell compiler that died in a large part because of its build system. To quote from the final blog post:
The biggest challenge for Yhc was the build system – we ended up with 10,000 lines of Python Scons scripts. Without a robust build system nothing else matters. When our sole Python hacker left the team that was the beginning of the end.
A Haskell compiler is a big undertaking, but the build system for a simple Haskell compiler shouldn't be that complicated.
When writing my thesis I needed a build system, and decided to try writing a simple Haskell DSL, which is still online here. I defined a single operator <==
which let me express a relationship between an output and its dependencies – very simple, but it worked.
Later I moved to Standard Chartered, where the build system was a mass of Makefiles, and it quickly became apparent that the project had outgrown the current approach. Without really surveying the alternatives, I decided that a Haskell DSL would be easiest to fit in with the existing infrastructure, so started writing some code. The first version of the build library took under a week, followed by a month of reimplementing the existing system. It wasn't until many months later I realised that the reason everything was suddenly so much easier was because we had monadic dependencies.
While people at Standard Chartered wanted to open source Shake, that turned out not to be possible. A number of people in the Haskell community implemented their own versions of Shake, but none were as polished or as strong as our internal one. Eventually, I reimplemented Shake, from scratch, in my spare time. Writing Shake from scratch, without the original source code or documentation, it naturally turned out better than the first attempt. A while later Standard Chartered migrated to the open-source version.
I still maintain Shake, but am fortunate to have other contributors extending and improving Shake. If you want to join in, see notes for developers.