FBU: My First Build Tool™

Posted on .

EDIT 2017-04-04: I have since renamed the project to MBU: Mix Build Utilities and published it on Hex.pm: hex.pm/packages/mbu. I have edited the links and code examples in this post to reflect that.

tl;dr I wrote my own build tool using Elixir's Mix: Nicd/mbu.

It's no secret that I somewhat dislike the state of modern web development. JavaScript is its own terrible world, but one of the sad parts of it is the ecosystem and tooling around it. There's a lot of innovation and hard work going on in very many fragmented projects, resulting in reimplementations of already solved problems and a ton of half working, alpha quality, 0.x versioned packages with unknown support status. With these packages, you start your project by building an elaborate house of cards that is the build system. And you dread the day when you need to touch it again.

Motivations

A lot of text about my motivations ahead, if you just want to see the end result, click here to jump to the code examples.

Things that irk me the most in the build tools I have used – those being Grunt, Gulp, Brunch, and Webpack (though it's not meant to really be one, it can be used as one):

  • Too many packages. When using the usual build systems, in addition to searching for package $THING, you also need to search for package $BUILDTOOL-THING that binds the thing and the build tool together. And you need to hope it's updated to the latest version of $THING and exposes all of the features you need. Many of these are unstable 0.x versions that you need to keep updated. Sometimes the correct combination doesn't even exist and you have to decide if it's worth writing yourself.
  • Duplicated APIs. Most of the available compilers and other tools have good command-line interfaces for controlling all of their features. Some also have configuration files. But when using them in a build tool, there is often another API provided by the glue package described above. Now you need to learn both the APIs: first how to use a feature and then how to use it in this specific build tool. If it is exposed in the glue package, that is.
  • Hidden states. Most build tools have these cool concepts of streams or pipes or something similar, where each task feeds the next one and the end result is written to a file. The problem with this is that it makes it easy to lose track of what is happening in the build system at any given moment. If you want to track where a specific thing happened to a file (where does it lose that damn source map?), you have to insert debug statements to inspect the stream contents, if that is even possible in the specific build tool. And of course for that you need to install $BUILDTOOL-debug.
  • Magic. In tools like Brunch where you don't have code that is run, but only a configuration file, what happens feels like sheer magic in both good and bad. It's cool that it works just like that, but finding out the correct combinations and places of keys and values to put in the configuration can be a daunting task. When the magic stops working, you're in trouble. A while back I tried to get RiotJS to compile using Brunch, but even with the correct versions of RiotJS and the Brunch glue package, it didn't work. I didn't want to learn how to write Brunch plugins so all I could do was change to another JS library.
  • JavaScript. The less of it I have to write, the better. I'll fully admit this one is just personal preference, but JavaScript isn't a language I enjoy working in too much.

So, what is the answer? Writing my own build tool, of course!

Standards
Somewhat relevant xkcd. © Randall Munroe, licensed under CC-BY-NC 2.5.

I took a look at some alternatives like NPM scripts, Make, and a couple of Python based build tools, but none of them really got me interested. I also wanted to avoid adding dependencies in another language, since I was writing in Elixir. Suddenly I had a thought: doesn't Elixir have its own build tool already? Elixir comes with a swiss army knife called Mix that is used for compiling, testing, running tasks, and managing dependencies. So I thought, maybe I'll write my own build tool using custom Mix tasks, just to see what it would look like. Because I'm not an imaginative person, I call it Frontend Build Utilities, or FBU for short.

From the start I knew it would not be a silver bullet that solves everyone's problems, but rather a tool very tailored to my own needs. I knew it would be bad in some ways, but at least I would have more control over the ways it was bad in. And most of all, I just wanted to write it for the heck of it, to see what it would be like.

How it Works

Here's how I set out to fix the problems above:

  • Minimize JS packages. Forget about installing glue packages. Just installing $TOOL should be enough to use that tool. This is accomplished by…
  • using the CLIs. If a tool offers a CLI, use it directly. Just like in Make, write simple tasks that call the required programs with the required arguments. Don't hide implementations behind duplicated APIs.
  • Make task outputs visible. I chose to not implement any kind of streaming features to avoid the problem of opaque pipes (and to make my job easier). Build tasks should take input from a specific folder and put their output into the next folder. With modern SSDs, the filesystem is fast enough and after running tasks you can check all the output folders to see what files they created.
  • No magic. Build tasks would just be custom Mix tasks that would call each other. There's nothing hidden and everything that is done is directly traceable to some source code in the tasks. I wanted to keep the things the build tool does at a minimum, since looking at the build tool sources is not what I would like to do when debugging issues.
  • Use Elixir in tasks. I like Elixir and was happy that I could now use it in my build system too.

Example Usage

FBU is basically a collection of utilities to help with writing Mix tasks that do the building. A basic task looks like this:

defmodule Mix.Tasks.Frontend.Build.Css.Copy do
  use MBU.BuildTask
  import MebeWeb.FrontendConfs

  @shortdoc "Copy compiled CSS to target dir"

  @deps [
    "frontend.build.css.compile"
  ]

  task _ do
    # Ensure target path exists
    out_path = Path.join([dist_path(), "css"])
    File.mkdir_p!(out_path)

    File.cp_r!(Mix.Tasks.Frontend.Build.Css.Compile.out_path(), out_path)
  end
end

The relevant parts above are the @deps attribute and the task block. @deps defines a list of tasks this task depends on. They will be automatically run in parallel before the task itself is run. This allows for easily building hierarchies of tasks that run when one target is executed, like in Gulp or Make.

The task block is a macro that is converted to the usual run/1 function of a Mix task. It takes in an optional list of arguments, prints out logging information and runs the dependencies unless deps: false was given in the argument list. Inside the task there's just regular Elixir code with nothing special about it. This task creates the output folder and copies files into it. Simple.

For the tasks themselves, FBU contains a few utility functions. The most important are exec, watch and listen. Here is an example of using exec and listen (with only the relevant parts):

def bin(), do: node_bin("babel")

def out_path(), do: Path.join([tmp_path(), "transpiled", "js"])

def args(), do: [
  Path.join([src_path(), "js"]),
  "--out-dir",
  out_path(),
  "--source-maps",
  "inline"
]

task _ do
  bin() |> exec(args()) |> listen()
end

exec takes in a path to a binary (given here by node_bin inside bin) and the command line arguments to give to that binary as a list. It executes the binary and returns a %ProgramSpec struct that can be passed to listen.

listen takes in a single spec (can be a program spec or a watch spec that you will see later) or a list of them and listens to their output. Output is logged to the console. By default the program name is used for prefixing the logs, but it can be changed by passing an additional argument name: "foo" to exec. When the last program has stopped, listen will return.

Watching source files is an important feature of build tools. Many tools also have builtin watch capabilities. With FBU you can leverage them by calling exec and giving the right command line switches to make the tool enter watch mode, usually by giving a switch -w. If the tool does not have watch capabilities, or you wish to run custom Elixir code when watching, you can use the watch function. It starts a custom filesystem watcher that can call any callback, and returns a %WatchSpec to put into listen.

Here's this blog's watch task:

defmodule Mix.Tasks.Frontend.Watch do
  use MBU.BuildTask
  import MBU.TaskUtils
  alias Mix.Tasks.Frontend.Build.Js.Transpile, as: TranspileJS
  alias Mix.Tasks.Frontend.Build.Js.Bundle, as: BundleJS
  alias Mix.Tasks.Frontend.Build.Js.Copy, as: CopyJS
  alias Mix.Tasks.Frontend.Build.Css.Compile, as: CompileCSS
  alias Mix.Tasks.Frontend.Build.Css.Copy, as: CopyCSS

  @shortdoc "Watch frontend and rebuild when necessary"

  @deps [
    "frontend.build"
  ]

  task _ do
    [
      exec(
        TranspileJS.bin(),
        TranspileJS.args() ++ ["-w"]
      ),

      watch(
        "JSBundle",
        TranspileJS.out_path(),
        BundleJS
      ),

      watch(
        "JSCopy",
        BundleJS.out_path(),
        CopyJS
      ),

      exec(
        CompileCSS.bin(),
        CompileCSS.args() ++ [
          "-w",
          CompileCSS.scss_file()
        ]
      ),

      watch(
        "CSSCopy",
        CompileCSS.out_path(),
        fn events -> IO.inspect(events); run_task(CopyCSS, deps: false) end
      )
    ]
    |> listen(watch: true)
  end
end

As you can see above, the watch task consists of two normal executions (babel and node-sass have their own builtin watchers) and three custom watchers. The watch calls take the name of the watcher as the first argument (for logging purposes), the path or paths to watch as the second and a callback function or task module that is called when file events happen. The callback receives the a list of events as the argument, here we inspect it (for debugging purposes) and then run the CopyCSS task manually. Note that we give deps: false to the tasks in the CopyCSS watcher, to avoid it running its dependencies, which would not be useful when watching. This is done automatically if you just provide the task module instead of a callback function.

After the specs, again, the listen function is called, but this time with an argument watch: true. This instructs the function to start a key listener that will stop the started programs when the user presses the enter key. That way the watch can be stopped gracefully.

Conclusion

So I built a frontend build tool and all I got was this crappy blog post. I'm withholding judgement over whether the tool is good or not before I have used it for a while. By my first impressions, it turned out to be magic free and explicit, but at the same time very verbose and minimal. I kind of like how it looks, but only time will tell if this whole thing made any sense.

If you'd like to check it out, it's open source and can be found on Github at Nicd/mbu or on Hex.pm. For example usage, see my blog's build setup on BitBucket.