Running Ecto Migrations and Other Startup Tasks With Distillery Hooks

When I was trawling the web for an easy way to run database migrations for my side project, PillowSkin, a Phoenix application that I am building written in Elixir, I came across this post by Sophie DeBenedetto on her blog The Great Code Adventure, and gave it a try.

The Method, Briefly

When using Distillery for your OTP releases, it exposes a post-startup hook that you can use to execute shell scripts. What Sophie describes in her blog post is basically to run a shell script that uses Erlang's rpc (i.e. remote produce call) module to call an elixir script that then executes an Elixir module's function for us.

However, we're going to modify this method into a generalized "startup tasks" strategy that we can use to ensure that our database-related tasks are executed in the correct sequence.

Reason why you do not want to put any database reliant code it into the application.ex startup call

When your Elixir app starts up, it starts up each GenServer marked in the dependency tree. However, as these GenServers are started up asyncronously, it is not guaranteed that Ecto will startup when your code calls the YourApplication.Repo module or any database related modules.

Note: When in development/test environments, I suggest that you do include a call to run your startup tasks so that you can automate activities such as inserting demo data. Will be discussed more below.

The Code

First we add in the post startup hook

# in rel/config.exs
environment :prod do
 set(include_erts: true)
 set(include_src: false)
 set(cookie: :"MY-COOKIE-NOM-NOM")
 set(post_start_hooks: "rel/hooks/post_start") #Add this line
end

Then, we add the startup shell script. What this does is to test that the application is up, and if so, we run the init/0 function within the MyApp.StartupTasks module.

# in re/hooks/post_start/startup.sh
set +e
echo "Preparing to run startup tasks"
while true; do
	nodetool ping
	EXIT_CODE=$?
	if [ $EXIT_CODE -eq 0 ]; then
		echo "Application is up!"
	break
	fi
done
set -e
echo "Running startup tasks..."
# Note that this differs from Sophie's example,
# as it does not contain quotes
bin/my_app rpc Elixir.MyApp.StartupTasks init
echo "Startup tasks ran successfully"

Great! Now that's two pieces of the puzzle fixed. All we need now is the final piece, which is the MyApp.StartupTasks module and the related init/0 function:

defmodule MyApp.StartupTasks do
	def init do
		# We check that all necessary components have started
		{:ok, _} = Application.ensure_all_started(:my_app)
		# Execute my startup tasks syncronously
		# Always run your migrations first before any database related things!
		migrate() # My startup task 1
		insert_users() # My startup task 2
	end
	def migrate do
		# Get the path to the migration files
		path = Application.app_dir(:my_app, "priv/repo/migrations")
		# Run the Ecto.Migrator
		Ecto.Migrator.run(MyApp.Repo, path, :up, all: true)
	end
	def insert_users do
		# Some code to insert my demo users
		# ...
	end
end

Can we complicate optimize this further?

We can convert the startup module into a GenServer, though I don't see the point, since this is only execute once on startup.

Using application.ex with startup tasks

When you want to execute your startup tasks when you're in development mode, you can follow this method where we conditionally execute the startup tasks based on environment. First, we pass in the current environment as an application environment variable:

# in config/config.exs
config :my_app, MyApp.Endpoint,
	# ... blah ...
	env: Mix.env()

Note: Why this works even in production mode in a distillery release is because when the OTP app is compiled, all values passed into the config are "frozen". Read more about this behaviour in OTP releases here.

Next, we conditionally execute the startup tasks that we want, which in this case is MyApp.StartupTasks.insert_users/0:

# in lib/MyApp/application.ex
defmodule MyApp.Application do
	use Application
	def start(_type, _args) do
		# List all child processes to be supervised
		children = [MyApp.Repo, MyAppWeb.Endpoint]
		opts = [strategy: :one_for_one, name: MyApp.Supervisor]
		Supervisor.start_link(children, opts)

		# We get the current environment
		env = Application.get_env(:my_app, MyAppWeb.Endpoint)[:env]
		if env == :dev do
			MyApp.StartupTasks.insert_users()
		end
	end
end

Ending Notes

So far, I have been having success with this method running it in production. I have not experimented with using this together with eDeliver, as I am using a docker-compose strategy for my deployments. However, the basic principles should still be the same regardless of how you push your OTP app the your server.

Possible limitations

One issue that I do foresee is when you have multiple servers executing the migration at the same time. For example, if you were using Docker Swarm with multiple nodes running instances of the OTP app, a rolling deployment may result in multiple applications running the startup tasks at the same time.

This may give undesirable results, such as database locking, corruptions, conflicts, etc. I have not reached this stage of scale yet to have a need to test out other strategies, but feel free to email me if you have any thoughts or experience in this area. Perhaps having a way to call the migrations from only one node may be a desirable behaviour...

That's some food for thought for another post for another day.