Random Notes

Elixir: Time for Some Configuration

Configuring Elixir applications is a common problem point for new developers and I’ve seen many questions about it in the community chats. So I thought to write down my knowledge in case it helps anyone.

tl;dr for those on Elixir v1.11: Use config/runtime.exs for your configuration needs. Read on for more details / nuance.

UPDATED 6th Oct 2020: Elixir 1.11 was released and the post has been updated with information about config/runtime.exs.

The usual way of getting configured values in your application is using Application.get_env/3. Environment variables are fetched with System.get_env/2. Configuration is set using the config functions in Config or Mix.Config (deprecated). But when and where you use these matters. To simplify, I consider there to be three phases where you can configure your application: build time, startup time, and runtime.

Build time Startup time Runtime
  • Regular config (config/config.exs etc.)
  • Code outside functions
  • config/releases.exs
  • config/runtime.exs
  • Config providers
  • Init callbacks
  • Runtime get_env calls

Build time

The usual way of configuring an application is through configuration files in the config directory, starting with config.exs, that may import other files based on the current environment. But configurations set in the these files are set at build time. This means that if you use code such as System.get_env/2, it will use the environment as it was when the project was built. During development when you run the program with mix run or iex -S mix this is fine, as it is constantly recompiled. But for production using Mix releases, you usually don’t want to use the values as they were on the build machine.

This also applies for metaprogramming, i.e. code outside function bodies. See the following example:

defmodule Test do
  @attr Application.get_env(:my_app, :foo) # Build time

  def fun() do
    Application.get_env(:my_app, :foo) # Runtime
  end
end

Macros can muddle the waters somewhat, because they can be evaluated at build or runtime depending on the macro. For example the following in a Phoenix router would be evaluated at build time:

defmodule Router do
  scope "/" do
    get(Application.get_env(:my_app, :index_route), Ctrl, :index)
  end
end

But the following task macro’s body (from MBU) would be runtime instead:

defmodule Mix.Tasks.Foo do
  task _ do
    IO.inspect(System.get_env("FOO"))
  end
end

So with macros you will have to look up the macro’s documentation and/or source code.

Startup time

To avoid the issue of the configuration being compiled in at build time and unchangeable, you can configure the system at startup time (called runtime in Mix’s documentation). Startup time configuration is processed when the Erlang runtime system is booting. In technical terms a small subset of applications in the runtime environment are started (those needed to process the configuration), then the configuration is applied, and finally the system is restarted with the updated configuration.

From Elixir v1.9 to v1.10, the place to do this was config/releases.exs. This was only used when a Mix release was starting up. Since Elixir v1.11, there is a new file called config/runtime.exs that is always processed, whether in dev, test, or production environment. This allows you to have a truly unified configuration file for all environments.

Mix releases also support configuration providers that allow for extending these files and loading configuration from e.g. JSON files on the target machine. You can read more about Mix release configuration and its limitations in the Mix documentation.

NOTE: Since config/releases.exs and config/runtime.exs are also used in releases, where Mix is not available, you cannot call any Mix functions in them. Instead functions from the Config module should be used. Luckily (since v1.11) they include the new config_env/0 and config_target/0 functions to get the configuration environment and target respectively.

Runtime

I tend to separate runtime from startup time as it’s slightly different. At startup time only a minimal set of applications is running to process the configuration. After that, the VM restarts with the rest of the applications and subsequently starts running their init callbacks. As far as the VM is concerned, this is just normal operation, so I call this runtime.

Many libraries offer such init callbacks that are used to configure their operation at runtime. When for example System.get_env/2 is used here, it will get the environment on the target machine at that time. Finally, you can always just call Application.get_env/3 and System.get_env/2 in any regular function at runtime and it will use whatever value is current at that time (both application environment and environment variables can be changed at runtime).

Here’s an example of an init callback from a real project of mine:

defmodule CodeStatsWeb.Geolix do
  @moduledoc """
  Module for initialising Geolix databases at runtime instead of build time.
  """

  @spec init() :: :ok
  def init() do
    db_dir = Application.app_dir(:code_stats, "priv")

    databases = [
      %{
        id: :city,
        adapter: Geolix.Adapter.MMDB2,
        source: Path.join([db_dir, "geoip-cities.gz"])
      },
      %{
        id: :country,
        adapter: Geolix.Adapter.MMDB2,
        source: Path.join([db_dir, "geoip-countries.gz"])
      }
    ]

    Application.put_env(:geolix, :databases, databases)
  end
end

As you can see, I’ve taken care to call functions such as Application.app_dir/2 only inside the init/0 function body so that the values will be evaluated at runtime. This function will be called by the Geolix library when it is starting up. Different libraries have different methods of configuration but this is a popular one.

Note about Application.compile_env/3

In Elixir 1.10, there is a new function to retrieve configured values. Since the mistake of using Application.get_env/3 at build time is so common, there is a new function Application.compile_env/3 that works a bit differently. It is used to explicitly read configuration at build time. When Elixir starts up, it checks if there is a different configuration value available than the one that was compiled, and raises an error if this happens. This is meant to help you avoid mistakes and surprises when you have compiled in one configuration, but set up another configuration in the target environment, and are wondering why you are seeing the wrong value.

You can read more information about the function from its documentation or the announcement blog post.

Additional resources and notes

So I hope that’s a simple explanation into the different configuration methods. Here are some additional links for reading:

The techniques mentioned here are quite recent. Mix releases are available from Elixir 1.9 onwards, in older versions you need to use Distillery releases and its own configuration providers.

Full example and practise

Let’s imagine a project using Mix releases that has been built with the following config/config.exs:

import Config

config :my_app,
  db_user: "Jeff",
  db_name: "devdb",
  db_dir: "/tmp/db"

Now it has been deployed with the following config/runtime.exs:

import Config

config :my_app,
  db_user: "Joan",
  db_name: System.get_env("DB_NAME"),
  db_dir: "/var/lib/postgres"

When starting up, the environment DB_NAME=proddb DB_DIR=/home/kari/db is given to the program.

The project contains this module:

defmodule Foo do
  require Logger

  @db_user Application.get_env(:my_app, :db_user)

  def init(db_name \\ Application.get_env(:my_app, :db_name)) do
    db_dir = Application.get_env(:my_app, :db_dir)

    Logger.debug("DB_DIR=" <> System.get_env("DB_DIR"))

    %{
      user: @db_user,
      name: db_name,
      dir: db_dir
    }
  end
end

Can you figure out what is returned from the init/1 function, assuming it is given no arguments? And what is logged to the debug console?

If you think you’ve got it, or you’re just eager to get the answers, here they are: