Wolfmans Howlings

Using Erlang with JInterface

Posted by Jim Morris on Sun Apr 12 16:56:50 -0700 2009

Background

I recently undertook a project to rewrite a voice conference server I wrote over 10 years ago. This server was written in C++ running on a Windows Server (originally NT and then win2k). The reason it was written for win2k was to use I/O Completion Ports which were not (and still are not) available on Linux.

Solaris has an equivalent AIO feature but not quite the same. I/O Completion Ports make very efficient use of multiple CPU's, and the conference server had to be very fast and handle a large number of concurrent connections.

The program uses a limited number of threads basically enough to keep all the CPUs busy, it uses the Proactor Pattern to handle the large number of concurrent connections. Each client connected has its own state, and every Asynchronous I/O needs to switch to that context. Needless to say there was a lot of shared memory and locks everywhere. The program was very complex to write, trying to keep all those locks and memory synchronized yet avoid deadlocks. I guess I did a good job as the servers are still running to this day, with barely a reboot in 5 years.

However the code is tortuous, and making any changes risks introducing deadlocks, also I wanted to get off of the windows servers and onto Linux which all my other servers run.

I have written several scalable servers using Java NIO, and even though they use the Reactor Pattern and I avoid creating hundreds of threads, instead I have several threads working through the ready sockets, it is still very hard to write, even with the great new concurrency libraries,

An article comparing the two patterns is here.

The trick is to write servers that you can simply add new servers to increase capacity, and to allow for redundancy so if one server goes down the service continues to run.

So having heard a lot about Erlang and how it auto-magically allowed you to run scalable and redundant servers I had to give it a go.

I bought Joe Armstrong's Book to learn Erlang, it is a good book to learn the basics, but is very short on details about OTP and how to actually use Erlang to write scalable servers. I guess the next book in the series will cover that ;)

I actually found it very hard to wrap my brain around Erlang initially. Having programmed in procedural and object oriented languages for so many years the switch to a functional, non object oriented language was tough.

Once I got over that initial hurdle (but am still struggling with it), I was able to quickly write the basic operations of the voice server in Erlang. I used a number of gen_servers, and tried to follow the OTP principles as closely as possible. I still have not figured out how to achieve the transparent scalability and redundancy that Erlang is touted for and that my old server has. However I do see the possibilities of making that transition.

What made the whole thing very easy is that every client connection can have its own process and state, totally independent of other connections. No locks are needed as there is no shared memory (except for the large ETS tables I use to store the list of users that are online etc). It turns out that not having to use either the Reactor or Proactor pattern makes things very simple, although as I am using the asynchronous mode of gen_tcp it is almost like a Proactor, as I get a callback (actually Erlang messages) everytime a packet comes in. As every client has its own process, and there could be about 100,000 clients concurrently connected, that is more than enough process to keep all the processors busy, also distributing those process over several physical servers should not be a problem (if I can figure out a way to load balance, probably using a H/W load balancer). I won't go into details here of gen_servers and the Erlang language as there is plenty of documentation on the web covering those, one particularly good series I found is here.

I use a binary protocol between the clients and servers and Erlang is particularly good at handling binary protocols, with its Binary type and easy pattern matching to extract parts of the message. Packing and unpacking binary messages is a breeze once you get the binary matching syntax.

Java and Erlang

However (and finally getting to the point of this article) one thing I found Erlang is particularly bad at is database access. It does have ODBC built in, and there is a Postgresql driver too, however the code is basically using raw SQL, and gets very ugly very fast. Now I am used to using Java and Spring and SpringJdbc to do all my database work, and it is a pleasure to work with. I use a common pattern of Dao, Business logic and I/O separation, and it keeps things very clean. Plus I already had all the database access written in Java.

So being one to use the best tool for the given situation, I wanted to continue to use Java/Spring for all the database accesses, and use Erlang as the server frontend.

Turns out Erlang ships with a pretty robust Java to Erlang interface called JInterface.

Basically this allows me to write an Erlang Node in Java, which Erlang can communicate with seamlessly using the normal Erlang message passing systems.

I have structured the database back end as 3 basic packages. The I/O package that contains all the knowledge of the server/server protocol for database requests and actually runs as an Erlang node. It calls methods in a POJO package which handles all business logic for those requests, and the business logic POJOs access the database entirely through calls to methods in the DAO package. The whole system is hooked together using Spring IoC, and the DAO is written using Spring JDBC.

I won't go into details here of how to write Spring based JDBC access as that is also well documented on the Spring site I may write another article on Spring JDBC as it is very elegant and easy to use.

So the focus of this article is the JInterface based part of the Java system, which receives database requests from the Erlang based front-end server and does whatever it needs to do getting data from the database then replies to that request back to the calling Erlang process.

Here is a portion of the java code setting up and handling the erlang requests, only the JInterface portions are shown

That is it, a very simple Erlang node which sits and receives Erlang messages from another Erlang node and processes them then sends its responses back to the requesting Erlang node.

You only need the jinterface package from the Erlang distribution, and import it... import com.ericsson.otp.erlang.*;

To start the node listening you simply do this...

OtpNode self = new OtpNode(getNodeName());
mbox = self.createMbox(getMboxName());

Now you have a node which can be sent to using...

{database, databasenode@localhost} ! message.

The loop has the following...

OtpErlangObject o = mbox.receive();

which receives a message, which you can then process. This is a little wordy but fairly simple...

The worst part is converting back and forth between Erlang objects and Java objects.

You could have several of these nodes running so you can balance the load across several servers if needed.