FBU: My First Build Tool™
EDIT 2017-04-04: I have since renamed the project to MBU: Mix Build Utilities and published it on Hex.pm: hex.pm/packages/mbu. I have edited the links and code examples in this post to reflect that.
tl;dr I wrote my own build tool using Elixir’s Mix: Nicd/mbu.
A lot of text about my motivations ahead, if you just want to see the end result, click here to jump to the code examples.
Things that irk me the most in the build tools I have used – those being Grunt, Gulp, Brunch, and Webpack (though it’s not meant to really be one, it can be used as one):
- Too many packages. When using the usual build systems, in addition to searching for package $THING, you also need to search for package $BUILDTOOL-THING that binds the thing and the build tool together. And you need to hope it’s updated to the latest version of $THING and exposes all of the features you need. Many of these are unstable 0.x versions that you need to keep updated. Sometimes the correct combination doesn’t even exist and you have to decide if it’s worth writing yourself.
- Duplicated APIs. Most of the available compilers and other tools have good command-line interfaces for controlling all of their features. Some also have configuration files. But when using them in a build tool, there is often another API provided by the glue package described above. Now you need to learn both the APIs: first how to use a feature and then how to use it in this specific build tool. If it is exposed in the glue package, that is.
- Hidden states. Most build tools have these cool concepts of streams or pipes or something similar, where each task feeds the next one and the end result is written to a file. The problem with this is that it makes it easy to lose track of what is happening in the build system at any given moment. If you want to track where a specific thing happened to a file (where does it lose that damn source map?), you have to insert debug statements to inspect the stream contents, if that is even possible in the specific build tool. And of course for that you need to install $BUILDTOOL-debug.
- Magic. In tools like Brunch where you don’t have code that is run, but only a configuration file, what happens feels like sheer magic in both good and bad. It’s cool that it works just like that, but finding out the correct combinations and places of keys and values to put in the configuration can be a daunting task. When the magic stops working, you’re in trouble. A while back I tried to get RiotJS to compile using Brunch, but even with the correct versions of RiotJS and the Brunch glue package, it didn’t work. I didn’t want to learn how to write Brunch plugins so all I could do was change to another JS library.
So, what is the answer? Writing my own build tool, of course!
I took a look at some alternatives like NPM scripts, Make, and a couple of Python based build tools, but none of them really got me interested. I also wanted to avoid adding dependencies in another language, since I was writing in Elixir. Suddenly I had a thought: doesn’t Elixir have its own build tool already? Elixir comes with a swiss army knife called Mix that is used for compiling, testing, running tasks, and managing dependencies. So I thought, maybe I’ll write my own build tool using custom Mix tasks, just to see what it would look like. Because I’m not an imaginative person, I call it Frontend Build Utilities, or FBU for short.
From the start I knew it would not be a silver bullet that solves everyone’s problems, but rather a tool very tailored to my own needs. I knew it would be bad in some ways, but at least I would have more control over the ways it was bad in. And most of all, I just wanted to write it for the heck of it, to see what it would be like.
How it Works
Here’s how I set out to fix the problems above:
- Minimize JS packages. Forget about installing glue packages. Just installing $TOOL should be enough to use that tool. This is accomplished by…
- using the CLIs. If a tool offers a CLI, use it directly. Just like in Make, write simple tasks that call the required programs with the required arguments. Don’t hide implementations behind duplicated APIs.
- Make task outputs visible. I chose to not implement any kind of streaming features to avoid the problem of opaque pipes (and to make my job easier). Build tasks should take input from a specific folder and put their output into the next folder. With modern SSDs, the filesystem is fast enough and after running tasks you can check all the output folders to see what files they created.
- No magic. Build tasks would just be custom Mix tasks that would call each other. There’s nothing hidden and everything that is done is directly traceable to some source code in the tasks. I wanted to keep the things the build tool does at a minimum, since looking at the build tool sources is not what I would like to do when debugging issues.
- Use Elixir in tasks. I like Elixir and was happy that I could now use it in my build system too.
FBU is basically a collection of utilities to help with writing Mix tasks that do the building. A basic task looks like this:
defmodule Mix.Tasks.Frontend.Build.Css.Copy do use MBU.BuildTask import MebeWeb.FrontendConfs @shortdoc "Copy compiled CSS to target dir" @deps [ "frontend.build.css.compile" ] task _ do # Ensure target path exists out_path = Path.join([dist_path(), "css"]) File.mkdir_p!(out_path) File.cp_r!(Mix.Tasks.Frontend.Build.Css.Compile.out_path(), out_path) end end
The relevant parts above are the
@deps attribute and the
a list of tasks this task depends on. They will be automatically run in parallel before the
task itself is run. This allows for easily building hierarchies of tasks that run when one
target is executed, like in Gulp or Make.
task block is a macro that is converted to the usual
run/1 function of a Mix task.
It takes in an optional list of arguments, prints out logging information and runs the dependencies
deps: false was given in the argument list. Inside the task there’s just regular
Elixir code with nothing special about it. This task creates the output folder and copies
files into it. Simple.
For the tasks themselves, FBU contains a few utility functions. The most important are
listen. Here is an example of using
listen (with only
the relevant parts):
def bin(), do: node_bin("babel") def out_path(), do: Path.join([tmp_path(), "transpiled", "js"]) def args(), do: [ Path.join([src_path(), "js"]), "--out-dir", out_path(), "--source-maps", "inline" ] task _ do bin() |> exec(args()) |> listen() end
exec takes in a path to a binary (given here by
bin) and the
command line arguments to give to that binary as a list. It executes the binary and returns
%ProgramSpec struct that can be passed to
listen takes in a single spec (can be a program spec or a watch spec that you will see
later) or a list of them and listens to their output. Output is logged to the
console. By default the program name is used for prefixing the logs, but it can be changed
by passing an additional argument
name: "foo" to
exec. When the last program has
listen will return.
Watching source files is an important feature of build tools. Many tools also have builtin
watch capabilities. With FBU you can leverage them by calling
exec and giving the right
command line switches to make the tool enter watch mode, usually by giving a switch
If the tool does not have watch capabilities, or you wish to run custom Elixir code when
watching, you can use the
watch function. It starts a custom filesystem watcher that
can call any callback, and returns a
%WatchSpec to put into
Here’s this blog’s watch task:
defmodule Mix.Tasks.Frontend.Watch do use MBU.BuildTask import MBU.TaskUtils alias Mix.Tasks.Frontend.Build.Js.Transpile, as: TranspileJS alias Mix.Tasks.Frontend.Build.Js.Bundle, as: BundleJS alias Mix.Tasks.Frontend.Build.Js.Copy, as: CopyJS alias Mix.Tasks.Frontend.Build.Css.Compile, as: CompileCSS alias Mix.Tasks.Frontend.Build.Css.Copy, as: CopyCSS @shortdoc "Watch frontend and rebuild when necessary" @deps [ "frontend.build" ] task _ do [ exec( TranspileJS.bin(), TranspileJS.args() ++ ["-w"] ), watch( "JSBundle", TranspileJS.out_path(), BundleJS ), watch( "JSCopy", BundleJS.out_path(), CopyJS ), exec( CompileCSS.bin(), CompileCSS.args() ++ [ "-w", CompileCSS.scss_file() ] ), watch( "CSSCopy", CompileCSS.out_path(), fn events -> IO.inspect(events); run_task(CopyCSS, deps: false) end ) ] |> listen(watch: true) end end
As you can see above, the watch task consists of two normal executions (babel and
node-sass have their own builtin watchers) and three custom watchers. The
take the name of the watcher as the first argument (for logging purposes), the path or
paths to watch as the second and a callback function or task module that is called when file events
happen. The callback receives the a list of events as the
argument, here we inspect it (for debugging purposes) and then run the CopyCSS task manually. Note that we
deps: false to the tasks in the CopyCSS watcher, to avoid it running its dependencies,
which would not be useful when watching. This is done automatically if you just provide the
task module instead of a callback function.
After the specs, again, the
listen function is called, but this time with an argument
watch: true. This instructs the function to start a key listener that will stop the
started programs when the user presses the enter key. That way the watch can be stopped
So I built a frontend build tool and all I got was this crappy blog post. I’m withholding judgement over whether the tool is good or not before I have used it for a while. By my first impressions, it turned out to be magic free and explicit, but at the same time very verbose and minimal. I kind of like how it looks, but only time will tell if this whole thing made any sense.
If you’d like to check it out, it’s open source and can be found on Github at Nicd/mbu or on Hex.pm. For example usage, see my blog’s build setup on BitBucket.