FBU: My First Build Tool™
EDIT 2017-04-04: I have since renamed the project to MBU: Mix Build Utilities and published it on Hex.pm: hex.pm/packages/mbu. I have edited the links and code examples in this post to reflect that.
tl;dr I wrote my own build tool using Elixir’s Mix: Nicd/mbu.
It’s no secret that I somewhat dislike the state of modern web development. JavaScript is its own terrible world, but one of the sad parts of it is the ecosystem and tooling around it. There’s a lot of innovation and hard work going on in very many fragmented projects, resulting in reimplementations of already solved problems and a ton of half working, alpha quality, 0.x versioned packages with unknown support status. With these packages, you start your project by building an elaborate house of cards that is the build system. And you dread the day when you need to touch it again.
Motivations
A lot of text about my motivations ahead, if you just want to see the end result, click here to jump to the code examples.
Things that irk me the most in the build tools I have used – those being Grunt, Gulp, Brunch, and Webpack (though it’s not meant to really be one, it can be used as one):
- Too many packages. When using the usual build systems, in addition to searching for package $THING, you also need to search for package $BUILDTOOL-THING that binds the thing and the build tool together. And you need to hope it’s updated to the latest version of $THING and exposes all of the features you need. Many of these are unstable 0.x versions that you need to keep updated. Sometimes the correct combination doesn’t even exist and you have to decide if it’s worth writing yourself.
- Duplicated APIs. Most of the available compilers and other tools have good command-line interfaces for controlling all of their features. Some also have configuration files. But when using them in a build tool, there is often another API provided by the glue package described above. Now you need to learn both the APIs: first how to use a feature and then how to use it in this specific build tool. If it is exposed in the glue package, that is.
- Hidden states. Most build tools have these cool concepts of streams or pipes or something similar, where each task feeds the next one and the end result is written to a file. The problem with this is that it makes it easy to lose track of what is happening in the build system at any given moment. If you want to track where a specific thing happened to a file (where does it lose that damn source map?), you have to insert debug statements to inspect the stream contents, if that is even possible in the specific build tool. And of course for that you need to install $BUILDTOOL-debug.
- Magic. In tools like Brunch where you don’t have code that is run, but only a configuration file, what happens feels like sheer magic in both good and bad. It’s cool that it works just like that, but finding out the correct combinations and places of keys and values to put in the configuration can be a daunting task. When the magic stops working, you’re in trouble. A while back I tried to get RiotJS to compile using Brunch, but even with the correct versions of RiotJS and the Brunch glue package, it didn’t work. I didn’t want to learn how to write Brunch plugins so all I could do was change to another JS library.
- JavaScript. The less of it I have to write, the better. I’ll fully admit this one is just personal preference, but JavaScript isn’t a language I enjoy working in too much.
So, what is the answer? Writing my own build tool, of course!
I took a look at some alternatives like NPM scripts, Make, and a couple of Python based build tools, but none of them really got me interested. I also wanted to avoid adding dependencies in another language, since I was writing in Elixir. Suddenly I had a thought: doesn’t Elixir have its own build tool already? Elixir comes with a swiss army knife called Mix that is used for compiling, testing, running tasks, and managing dependencies. So I thought, maybe I’ll write my own build tool using custom Mix tasks, just to see what it would look like. Because I’m not an imaginative person, I call it Frontend Build Utilities, or FBU for short.
From the start I knew it would not be a silver bullet that solves everyone’s problems, but rather a tool very tailored to my own needs. I knew it would be bad in some ways, but at least I would have more control over the ways it was bad in. And most of all, I just wanted to write it for the heck of it, to see what it would be like.
How it Works
Here’s how I set out to fix the problems above:
- Minimize JS packages. Forget about installing glue packages. Just installing $TOOL should be enough to use that tool. This is accomplished by…
- using the CLIs. If a tool offers a CLI, use it directly. Just like in Make, write simple tasks that call the required programs with the required arguments. Don’t hide implementations behind duplicated APIs.
- Make task outputs visible. I chose to not implement any kind of streaming features to avoid the problem of opaque pipes (and to make my job easier). Build tasks should take input from a specific folder and put their output into the next folder. With modern SSDs, the filesystem is fast enough and after running tasks you can check all the output folders to see what files they created.
- No magic. Build tasks would just be custom Mix tasks that would call each other. There’s nothing hidden and everything that is done is directly traceable to some source code in the tasks. I wanted to keep the things the build tool does at a minimum, since looking at the build tool sources is not what I would like to do when debugging issues.
- Use Elixir in tasks. I like Elixir and was happy that I could now use it in my build system too.
Example Usage
FBU is basically a collection of utilities to help with writing Mix tasks that do the building. A basic task looks like this:
defmodule Mix.Tasks.Frontend.Build.Css.Copy do
use MBU.BuildTask
import MebeWeb.FrontendConfs
@shortdoc "Copy compiled CSS to target dir"
@deps [
"frontend.build.css.compile"
]
task _ do
# Ensure target path exists
out_path = Path.join([dist_path(), "css"])
File.mkdir_p!(out_path)
File.cp_r!(Mix.Tasks.Frontend.Build.Css.Compile.out_path(), out_path)
end
end
The relevant parts above are the @deps
attribute and the task
block. @deps
defines
a list of tasks this task depends on. They will be automatically run in parallel before the
task itself is run. This allows for easily building hierarchies of tasks that run when one
target is executed, like in Gulp or Make.
The task
block is a macro that is converted to the usual run/1
function of a Mix task.
It takes in an optional list of arguments, prints out logging information and runs the dependencies
unless deps: false
was given in the argument list. Inside the task there’s just regular
Elixir code with nothing special about it. This task creates the output folder and copies
files into it. Simple.
For the tasks themselves, FBU contains a few utility functions. The most important are
exec
, watch
and listen
. Here is an example of using exec
and listen
(with only
the relevant parts):
def bin(), do: node_bin("babel")
def out_path(), do: Path.join([tmp_path(), "transpiled", "js"])
def args(), do: [
Path.join([src_path(), "js"]),
"--out-dir",
out_path(),
"--source-maps",
"inline"
]
task _ do
bin() |> exec(args()) |> listen()
end
exec
takes in a path to a binary (given here by node_bin
inside bin
) and the
command line arguments to give to that binary as a list. It executes the binary and returns
a %ProgramSpec
struct that can be passed to listen
.
listen
takes in a single spec (can be a program spec or a watch spec that you will see
later) or a list of them and listens to their output. Output is logged to the
console. By default the program name is used for prefixing the logs, but it can be changed
by passing an additional argument name: "foo"
to exec
. When the last program has
stopped, listen
will return.
Watching source files is an important feature of build tools. Many tools also have builtin
watch capabilities. With FBU you can leverage them by calling exec
and giving the right
command line switches to make the tool enter watch mode, usually by giving a switch -w
.
If the tool does not have watch capabilities, or you wish to run custom Elixir code when
watching, you can use the watch
function. It starts a custom filesystem watcher that
can call any callback, and returns a %WatchSpec
to put into listen
.
Here’s this blog’s watch task:
defmodule Mix.Tasks.Frontend.Watch do
use MBU.BuildTask
import MBU.TaskUtils
alias Mix.Tasks.Frontend.Build.Js.Transpile, as: TranspileJS
alias Mix.Tasks.Frontend.Build.Js.Bundle, as: BundleJS
alias Mix.Tasks.Frontend.Build.Js.Copy, as: CopyJS
alias Mix.Tasks.Frontend.Build.Css.Compile, as: CompileCSS
alias Mix.Tasks.Frontend.Build.Css.Copy, as: CopyCSS
@shortdoc "Watch frontend and rebuild when necessary"
@deps [
"frontend.build"
]
task _ do
[
exec(
TranspileJS.bin(),
TranspileJS.args() ++ ["-w"]
),
watch(
"JSBundle",
TranspileJS.out_path(),
BundleJS
),
watch(
"JSCopy",
BundleJS.out_path(),
CopyJS
),
exec(
CompileCSS.bin(),
CompileCSS.args() ++ [
"-w",
CompileCSS.scss_file()
]
),
watch(
"CSSCopy",
CompileCSS.out_path(),
fn events -> IO.inspect(events); run_task(CopyCSS, deps: false) end
)
]
|> listen(watch: true)
end
end
As you can see above, the watch task consists of two normal executions (babel and
node-sass have their own builtin watchers) and three custom watchers. The watch
calls
take the name of the watcher as the first argument (for logging purposes), the path or
paths to watch as the second and a callback function or task module that is called when file events
happen. The callback receives the a list of events as the
argument, here we inspect it (for debugging purposes) and then run the CopyCSS task manually. Note that we
give deps: false
to the tasks in the CopyCSS watcher, to avoid it running its dependencies,
which would not be useful when watching. This is done automatically if you just provide the
task module instead of a callback function.
After the specs, again, the listen
function is called, but this time with an argument
watch: true
. This instructs the function to start a key listener that will stop the
started programs when the user presses the enter key. That way the watch can be stopped
gracefully.
Conclusion
So I built a frontend build tool and all I got was this crappy blog post. I’m withholding judgement over whether the tool is good or not before I have used it for a while. By my first impressions, it turned out to be magic free and explicit, but at the same time very verbose and minimal. I kind of like how it looks, but only time will tell if this whole thing made any sense.
If you’d like to check it out, it’s open source and can be found on Github at Nicd/mbu or on Hex.pm. For example usage, see my blog’s build setup on BitBucket.