Wolfmans Howlings

Running Erlang OTP applications as Daemons on Ubuntu Servers

Posted by Jim Morris on Sun Sep 19 17:26:42 -0700 2010

Recently I needed to deploy an Erlang OTP application to a production server, and was surprised at the lack of official documentation on how to do that. My requirements are to have the application run on system startup, and if the application crashes to have it automatically be restarted, some optional requirements are to have rotated logs and to get email notifications if the application does crash.

Erlang itself has several layers of supervision to keep its processes running, however on occasion the VM itself may crash, if it runs out of memory for instance. I would want the entire appliction to be restarted in that case.

The only official mention says to use the -heart option when you run erl, but reading blogs and news posts I see that this does not necessarily work too well when the Erlang VM runs out of memory or crashes for some other reasons.

I found several blogs on how others have managed it, and after taking ideas from several places on how to start and stop my application, I came up with the following scheme to deploy my application.

I used DJB's daemontools for a long time to deploy always up production applications written in Java or other languages, and it works really well, and it takes care of rolling over the logs written to stdout or stderr and adds an accurate timestamp to each line logged. The main power is the ability to restart a process that it is monitoring if it dies unexpectedly. it also allows me to start and stop the daemon from the command line, which I need to do when the application is updated.

The main reason I could not simply use daemontools to mangage my Erlang application is I want my Erlang OTP application to stop gracefully when I issue a stop and daemontools will just send a unix TERM signal when you do svc -d /services/my_application, which for a Java app is fine as you can trap a TERM signal and do the clean shutdown.

Erlang seems to deliberately disallow trapping of OS signals like TERM, and the Erlang VM simply stops abruptly on getting the signal, no chance to capture it and close down gracefully. (Maybe they should add an option to close down on receipt of TERM by issuing an application:stop() or allow trapping of TERM).

With Erlang OTP applications it seems the way to gracefully shutdown is to issue an application:stop() from another Erlang node or shell, or use a simple control application to issue that command to the node you want to shutdown.

So I needed an option that would let me run a shell command when I want to shutdown my Erlang OTP application.

(Side note when I refer to daemon here I actually mean a process running as a service, and not necessarily daemonized in the Linux definition of a daemon. Most of my services do not background or detach from the console and logging info is usually written to stdout).

On Ubuntu I could use the new upstart, which allows me to write scripts to start and stop my services, and control them with start and stop commands, it also will restart a service that dies unexpectedly, (unlike a service/daemon run with an init.d script).

Some use monit, but that just restarts an app when it dies, it does not let me start or stop the app, and it is annoying that if you want to update the daemon, you must remember to shutdown monit first then kill your daemon, then update it, then restart it then restart monit, it also has a rather verbose scripting language.

Out of several other candidates I found the oddly named god utility that is written in ruby and the control scripts are also written in ruby, and being a ruby fan that was a big plus. Basically god provides a Ruby based DSL to control daemons, it will allow you to log the stdin/stderr of the daemon and start and stop the daemon from the command line, all the features I wanted.

So here is how I use god to control my Erlang OTP applications, and how I use daemontools to keep god running, which is kinda odd, but was easier than writing an upstart script which would probably be more appropriate. However this solution will work on any Linux distribution that you can get daemontools and ruby on.

Presuming you have built a compliant OTP application and packaged it up for distribution, (although you don't have to do that as the command line to run the app is arbitrary).

Here is the god script, called something like myservice.god

First I define some constants that could change on a production server, and are used further on in the script.

Then I setup the email options so I get notified whenever it starts the app.

Then comes the main stanza that tells god whet to monitor, I set a directory for it to run from, how often to poll it to see if it is up, and the UID I want it to run as, I also set any environment variables, and as erl needs HOME to be set we set it here.

Then comes the command line that starts the application, a pretty standard way of running an Erlang OTP application from a packaged system.

Then comes the all important command that will gracefully shutdown the application, in my case the whole reason I cannot simply use daemontools. I have a very simple erlang module which when run will send an rpc to the application to tell it to stop, basically it does...

I found that this gracefully stops my applications main supervisor then shuts down the Erlang VM. You could also write a simple escript batch file to do the same thing.

NOTE because my applications are not real daemons and do not run in the background I allow god to handle the pids etc by not specifying pid_file, which forces god to daemonize my app for me and keep track of its pid.

This all works nicely I can do...

> god stop myservice
> god start myservice

to manually start and stop my service from the command line.

Now because I always output all my logging info to stdout (using log4j or printfs etc) I like to have these neatly logged in rotating logs files, with time stamps. As I am used to running everything with daemontools I like the way it does it, so I use the multilog app from daemontools to handle all the logging. All this does is take stdin and write it to a log file, prepending each line with an accurate timestamp. This line in the god script handles that...

 w.log_cmd = "/opt/daemontools-0.76/command/multilog t #{DIR}/log"

However you can have it log to a file directly with...

w.log = "#{DIR}/mylogfile"

However there will be no timestamps unless you output them, and the file will not be rotated. You could also use the standard logger...

w.log_cmd = '/usr/bin/logger'

The last problem I need to solve is how to run god on startup and keep it running if it crashes, ie who monitors the monitor?

Well as I like daemontools, guess what, I use daemontools to run god, this will keep it alive. (Don't ask who keeps daemontools running! actually it is upstart in my case).

To run god under daemontools I want to tell god to not daemonize itself, so this is the run script I use for daemontools to run god...

#!/bin/sh
cd /opt/god
exec /var/lib/gems/1.8/bin/god -c ./main.god -D --no-syslog 2>&1

I tell it to not fill my syslog files, and the -D keeps it from daemonizing itself and output its logs to stdout, which daemontools will conveniently log for me.

the main.god config file has this stanza...

God.load "/opt/services/*.god"

which will load any god config files found in /opt/services, which means I can add applications for it to monitor by simply adding a symbolic link in /opt/services to the new god config file.

So that is it, a somewhat convoluted way to have Erlang OTP applications run in a production environment, and get restarted if the VM crashes. (The rest of Erlang process monitoring is handled by the Erlang supervisor and OTP stuff, which you can read about elsewhere).