Random Notes

Simple Configuration Setup for Elixir Projects (v1.11+)

Posted on 17 Apr 2021, 00:50 EEST.

I’ve written about Elixir configuration in an earlier post, describing the differences in the configuration styles. If you don’t know how Elixir configuration works, I suggest reading it also. In this post, I will demonstrate a system for configuring an Elixir project using the config/runtime.exs system introduced in Elixir 1.11. This system is how I configure my projects, so feel free take it as inspiration, but it’s not a law that you have to follow.

Aim

The aims of this system are the following:

We accomplish these by using these parts:

Do note that I use this method in a relatively simple, single-node deployment that does not use Docker or Kubernetes or the other modern fancy-pants stuff. So maybe those have a different way of doing things. But at least this is one way and maybe food for thought.

The examples in this post are from a Phoenix project I have. Phoenix still generates a project skeleton with the “traditional” config/{config,dev,prod,test}.exs files, so it requires a bit of a cleanup to move to this system.

Compile config

The classic config files in Elixir are config/config.exs, and all the files you import from it, usually things like config/dev.exs, config/dev.secret.exs, and so on. These are for compile time configuration. When developing, the app is constantly recompiled, so putting configuration here can work and is tempting, but it will bite you when you build a release from your project, as the values can no longer be changed after compilation.

Let’s start with config/config.exs. It should only contain configuration required at compile time (such as settings that will affect the compilation of modules), and may also contain configuration that we can also say with certainty will not be needed to change once the project has been deployed. You can import other compile time config files, but personally I think it’s not needed for most projects. The config/*.secret.exs style should generally not be used, instead configuration using secrets should live in config/runtime.exs, which I will talk about later.

Example file:

import Config

# Configures the endpoint
config :code_stats, CodeStatsWeb.Endpoint,
  render_errors: [accepts: ~w(html json)],
  pubsub_server: CodeStats.PubSub,
  code_reloader: Config.config_env() == :dev,
  debug_errors: Config.config_env() == :dev,
  check_origin: Config.config_env() == :prod

config :code_stats,
  ecto_repos: [CodeStats.Repo]

config :geolix,
  init: {CodeStatsWeb.Geolix, :init}

# Store mix env used to compile app, since Mix won't be available in release
config :code_stats, compile_env: Mix.env()

# Store app version and commit hash at compile time
config :code_stats,
  commit_hash: System.cmd("git", ["rev-parse", "--verify", "--short", "HEAD"]) |> elem(0),
  version: Mix.Project.config()[:version]

# Use Jason for JSON parsing in Phoenix
config :phoenix, :json_library, Jason

You can see there are some settings that affect Phoenix compilation, such as the code reloader setting. There are also settings that I don’t expect to change, such as the Ecto repo, Geolix initialisation module, and JSON parsing library. There’s also values that I deliberately want to pick at compile time, such as the Mix environment and git version hash.

Runtime config

Runtime configuration is the meat of this post. With Elixir 1.11, we have the magnificent config/runtime.exs file, that has a couple of useful properties. Firstly, it is evaluated at release startup time, so it can use the system environment as it exists on the target system. And secondly, since all the modules have already been compiled, we can call code from our project and dependencies. This will come in handy later.

Let’s start with a simple example:

import Config

config :code_stats,
  site_name: System.get_env("SITE_NAME", "Code::Stats")

Since this is evaluated at startup time, we can now change the site name without recompiling the project. Just change the environment variable and restart.

As .exs files are just regular Elixir, we can also use code:

with key when is_binary(key) <- System.get_env("BAMBOO_API_KEY") do
  config :code_stats, CodeStats.Mailer,
    adapter: Bamboo.MailgunAdapter,
    api_key: key,
    domain: System.get_env("BAMBOO_DOMAIN")
else
  nil ->
    IO.puts(:stderr, "Mailer key not configured, using LocalAdapter.")
    config :code_stats, CodeStats.Mailer, adapter: Bamboo.LocalAdapter
end

This allows for some handy things, but I would limit it to basic conditionals to avoid going overboard. Don’t make your config Turing complete.

Finally, if some configuration only needs to be applied in certain environments, you can use Config.config_env/0:

case Config.config_env() do
  :prod ->
    config :code_stats, CodeStatsWeb.Endpoint,
      secret_key_base: System.get_env("SECRET_KEY_BASE"),
      server: true
  
  :dev ->
    config :code_stats, CodeStatsWeb.Endpoint,
      watchers: [mix: ["frontend.watch", cd: Path.expand("../", __DIR__)]]
  
  # And so on
end

This is also the weakness with config/runtime.exs: you cannot import other files, so the file may get lengthy. In my projects it has stayed manageable, but your mileage may vary. Additionally, you might have noticed that the watcher configuration above could have just as well been in the compile time config files. It’s true, but personally I like to keep as much of it in the same file as I can, to make things easier to find. Just find your own style. :)

.env and config helpers

Now that we have a proper config/runtime.exs, we have a unified configuration that runs both when we are developing locally, and in our production release. We have two issues still, though.

The first is that setting all of these environment variables when working locally can be tedious. There are different ways to do it, such as having a script file with lots of export FOO=1 statements, and remembering to source .env often enough, but for myself I wanted something that didn’t require extra work. That’s why I wrote DotenvParser, a library to parse a .env file and load its values to the system environment.

With DotenvParser, we can do the following trick at the start of the file:

import Config

if Config.config_env() == :dev do
  DotenvParser.load_file(".env") # Works because the dependency is already compiled
end

# Continue with configuration

Now after this block, all the environment variables defined in .env have been loaded and can be retrieved with System.get_env/2. Add it to .gitignore to be able to play with the values without hassle (just document it elsewhere for other devs).

The other issue that you always hit with environment variables is that they are all strings. It would be nice to have real datatypes in our configuration. Since config/runtime.exs can call our modules, we can just write some helper code to deal with this. This is such a simple module that I haven’t even packaged it, but you can find it in my project’s codebase:

defmodule CodeStats.ConfigHelpers do
  @type config_type :: :string | :integer | :boolean | :json

  @doc """
  Get value from environment variable, converting it to the given type if needed.

  If no default value is given, or `:no_default` is given as the default, an error is raised if the variable is not
  set.
  """
  @spec get_env(String.t(), :no_default | any(), config_type()) :: any()
  def get_env(var, default \\ :no_default, type \\ :string)

  def get_env(var, :no_default, type) do
    System.fetch_env!(var)
    |> get_with_type(type)
  end

  def get_env(var, default, type) do
    with {:ok, val} <- System.fetch_env(var) do
      get_with_type(val, type)
    else
      :error -> default
    end
  end

  @spec get_with_type(String.t(), config_type()) :: any()
  defp get_with_type(val, type)

  defp get_with_type(val, :string), do: val
  defp get_with_type(val, :integer), do: String.to_integer(val)
  defp get_with_type("true", :boolean), do: true
  defp get_with_type("false", :boolean), do: false
  defp get_with_type(val, :json), do: Jason.decode!(val)
  defp get_with_type(val, type), do: raise("Cannot convert to #{inspect(type)}: #{inspect(val)}")
end

The caveat to remember here is that since the code is called at startup time, no applications are running. So we wouldn’t be able to query the database, for example. But we can call regular, non-app-specific code like that.

Now when we want to retrieve configuration in the file, we can do the following:

import Config
import CodeStats.ConfigHelpers, only: [get_env: 3]

if Config.config_env() == :dev do
  DotenvParser.load_file(".env")
end

config :code_stats,
  beta_mode: get_env("BETA_MODE", false, :boolean),
  max_xp_per_pulse: get_env("MAX_XP_PER_PULSE", 1_000, :integer),
  cors_allowed_origins: get_env("CORS_ALLOWED_ORIGINS", ["http://localhost"], :json)

So now we can easily get environment variables and cast them to the correct types. The only thing left to do is to set the environment variables in the production system. I use systemd, so I tend to set them in the unit file as Environment=FOO=val:

[Unit]
After=postgresql.service

[Service]
Type=simple
User=release_user
Restart=always
RestartSec=10

Environment=PORT=51110
Environment=HOST=codestats.net
Environment=HOST_PORT=443
# And so on...

ExecStart=/path/to/bin/code_stats start
WorkingDirectory=/path/to/wrk/dir

[Install]
WantedBy=multi-user.target

Conclusion

What we have set up in this post:

You can see this system in action in my project Code::Stats:

If you have questions or feedback about this system, please add a comment below, or contact me: