Using Erlang with JInterface

Jim Morris — Sun, 12 Apr 2009 17:13:17 -0700

Background

I recently undertook a project to rewrite a voice conference server I wrote over 10 years ago. This server was written in C++ running on a Windows Server (originally NT and then win2k). The reason it was written for win2k was to use I/O Completion Ports which were not (and still are not) available on Linux.

Solaris has an equivalent AIO feature but not quite the same. I/O Completion Ports make very efficient use of multiple CPU's, and the conference server had to be very fast and handle a large number of concurrent connections.

The program uses a limited number of threads basically enough to keep all the CPUs busy, it uses the Proactor Pattern to handle the large number of concurrent connections. Each client connected has its own state, and every Asynchronous I/O needs to switch to that context. Needless to say there was a lot of shared memory and locks everywhere. The program was very complex to write, trying to keep all those locks and memory synchronized yet avoid deadlocks. I guess I did a good job as the servers are still running to this day, with barely a reboot in 5 years.

However the code is tortuous, and making any changes risks introducing deadlocks, also I wanted to get off of the windows servers and onto Linux which all my other servers run.

I have written several scalable servers using Java NIO, and even though they use the Reactor Pattern and I avoid creating hundreds of threads, instead I have several threads working through the ready sockets, it is still very hard to write, even with the great new concurrency libraries,

An article comparing the two patterns is here.

The trick is to write servers that you can simply add new servers to increase capacity, and to allow for redundancy so if one server goes down the service continues to run.

So having heard a lot about Erlang and how it auto-magically allowed you to run scalable and redundant servers I had to give it a go.

I bought Joe Armstrong's Book to learn Erlang, it is a good book to learn the basics, but is very short on details about OTP and how to actually use Erlang to write scalable servers. I guess the next book in the series will cover that ;)

I actually found it very hard to wrap my brain around Erlang initially. Having programmed in procedural and object oriented languages for so many years the switch to a functional, non object oriented language was tough.

Once I got over that initial hurdle (but am still struggling with it), I was able to quickly write the basic operations of the voice server in Erlang. I used a number of gen_servers, and tried to follow the OTP principles as closely as possible. I still have not figured out how to achieve the transparent scalability and redundancy that Erlang is touted for and that my old server has. However I do see the possibilities of making that transition.

What made the whole thing very easy is that every client connection can have its own process and state, totally independent of other connections. No locks are needed as there is no shared memory (except for the large ETS tables I use to store the list of users that are online etc). It turns out that not having to use either the Reactor or Proactor pattern makes things very simple, although as I am using the asynchronous mode of gen_tcp it is almost like a Proactor, as I get a callback (actually Erlang messages) everytime a packet comes in. As every client has its own process, and there could be about 100,000 clients concurrently connected, that is more than enough process to keep all the processors busy, also distributing those process over several physical servers should not be a problem (if I can figure out a way to load balance, probably using a H/W load balancer). I won't go into details here of gen_servers and the Erlang language as there is plenty of documentation on the web covering those, one particularly good series I found is here.

I use a binary protocol between the clients and servers and Erlang is particularly good at handling binary protocols, with its Binary type and easy pattern matching to extract parts of the message. Packing and unpacking binary messages is a breeze once you get the binary matching syntax.

Java and Erlang

However (and finally getting to the point of this article) one thing I found Erlang is particularly bad at is database access. It does have ODBC built in, and there is a Postgresql driver too, however the code is basically using raw SQL, and gets very ugly very fast. Now I am used to using Java and Spring and SpringJdbc to do all my database work, and it is a pleasure to work with. I use a common pattern of Dao, Business logic and I/O separation, and it keeps things very clean. Plus I already had all the database access written in Java.

So being one to use the best tool for the given situation, I wanted to continue to use Java/Spring for all the database accesses, and use Erlang as the server frontend.

Turns out Erlang ships with a pretty robust Java to Erlang interface called JInterface.

Basically this allows me to write an Erlang Node in Java, which Erlang can communicate with seamlessly using the normal Erlang message passing systems.

I have structured the database back end as 3 basic packages. The I/O package that contains all the knowledge of the server/server protocol for database requests and actually runs as an Erlang node. It calls methods in a POJO package which handles all business logic for those requests, and the business logic POJOs access the database entirely through calls to methods in the DAO package. The whole system is hooked together using Spring IoC, and the DAO is written using Spring JDBC.

I won't go into details here of how to write Spring based JDBC access as that is also well documented on the Spring site I may write another article on Spring JDBC as it is very elegant and easy to use.

So the focus of this article is the JInterface based part of the Java system, which receives database requests from the Erlang based front-end server and does whatever it needs to do getting data from the database then replies to that request back to the calling Erlang process.

Here is a portion of the java code setting up and handling the erlang requests, only the JInterface portions are shown

import com.ericsson.otp.erlang.*;
        
        public class ServerNodeImpl implements Server {
            private static final Logger jlog = Logger.getLogger(ServerNodeImpl.class);
            private String nodeName = "databasenode@localhost";
            private String mboxName = "database";
        
        ...
        
            private OtpMbox mbox;
        
            // create some convenient atoms
            private final OtpErlangAtom request= new OtpErlangAtom("request");
            private final OtpErlangAtom authenticate= new OtpErlangAtom("authenticate");
            private final OtpErlangAtom failed= new OtpErlangAtom("failed");
            private final OtpErlangAtom ok= new OtpErlangAtom("ok");
        
            // start the node up
            public boolean start() {
                try{
                    init();
                }catch(IOException e){
                    jlog.error("start()", e);
                    return false;
                }
        
                // this will block and the caller will block until the loop exits
                loop();
        
                return true;
            }
        
            // stop the node
            public boolean stop() {
                // to stop it, try sending myself a stop message
                OtpErlangObject[] msg = new OtpErlangObject[2];
                msg[0] = mbox.self();
                msg[1] = new OtpErlangAtom("stop");
                OtpErlangTuple tuple = new OtpErlangTuple(msg);  
                mbox.send(mbox.self(), tuple);
                return true;
            }
        
            private void init() throws IOException {
                jlog.info("Node: " + nodeName + ", MBox: " + mboxName);
                OtpNode self = new OtpNode(getNodeName());
                mbox = self.createMbox(getMboxName());
            }
        
            /**
             * We get requests which are tuples.. {from,cmd,ref,...}
             * where cmd is currently an atom stop|request
             */
            private void loop() {
                jlog.info("loop(): starting...");
                boolean running= true;
                while(running){
                    try{
                        OtpErlangObject o = mbox.receive();
                        jlog.debug("loop(): Received Request: " + o);
                        if(o instanceof OtpErlangTuple){
                            OtpErlangTuple msg = (OtpErlangTuple) o;
                            OtpErlangPid from = (OtpErlangPid) msg.elementAt(0);
                            OtpErlangObject cmd = msg.elementAt(1);
                            OtpErlangRef ref = (OtpErlangRef) msg.elementAt(2);
                            if(cmd instanceof OtpErlangAtom){
                                OtpErlangAtom atom= (OtpErlangAtom)cmd;
                                String c= atom.atomValue();
                                if(c.equals("stop")){
                                    jlog.info("loop(): got stop message");
                                    running= false;
                                }else if(c.equals("request")){
                                    try{
                                        byte [][] reply= handleRequest(from, msg);
                                        sendResponse(from, ref, reply);
                                    }catch(Exception e){
                                        jlog.error("loop(): got exception from handleRequest()", e);
                                    }
                                }else{
                                    jlog.error("loop(): got unknown cmd: " + c);                          
                                }
                            }else{
                                jlog.error("loop(): got unknown cmd type: " + cmd);
                            }
        
                        }else{
                            jlog.error("loop(): Unknown message type: " + o);
                        }
                    }catch(Exception e){
                        jlog.error("loop(): got exception", e);
                        running= false;
                    }
                }
                jlog.info("loop(): ...leaving");
            }
        
            ...
        
            // sends a tuple back to requesting Pid {From, Ref, Data}
            private void sendResponse(OtpErlangPid from, OtpErlangRef ref, byte [] ba) {
                OtpErlangBinary data= new OtpErlangBinary(ba);      
                OtpErlangObject [] eoa= new OtpErlangObject[]{mbox.self(), ref, data};
                OtpErlangTuple tuple= new OtpErlangTuple(eoa);
                mbox.send(from, tuple);
            }

That is it, a very simple Erlang node which sits and receives Erlang messages from another Erlang node and processes them then sends its responses back to the requesting Erlang node.

You only need the jinterface package from the Erlang distribution, and import it... import com.ericsson.otp.erlang.*;

To start the node listening you simply do this...

OtpNode self = new OtpNode(getNodeName());
        mbox = self.createMbox(getMboxName());

Now you have a node which can be sent to using...

{database, databasenode@localhost} ! message.

The loop has the following...

OtpErlangObject o = mbox.receive();

which receives a message, which you can then process. This is a little wordy but fairly simple...

   
        // first check that the message is a tuple
        if(o instanceof OtpErlangTuple){
            // cast it into a tuple object
            OtpErlangTuple msg = (OtpErlangTuple) o;
        
            // extract the first element of the tuple
            OtpErlangPid from = (OtpErlangPid) (msg.elementAt(0));
        
            // extract the second element of the tuple
            OtpErlangObject cmd = msg.elementAt(1);
        
            // extract the ref of the request
            OtpErlangRef ref = (OtpErlangRef) msg.elementAt(2);
        
            // which should be an atom of stop | request
            if(cmd instanceof OtpErlangAtom){
                OtpErlangAtom atom= (OtpErlangAtom)cmd;
        
                // convert the atom to a string
                String c= atom.atomValue();
        
                // if it is stop then we exit the loop
                if(c.equals("stop")){
                    jlog.info("loop(): got stop message");
                    running= false;
        
                // if it is request we pass the rest of the message
                // onto the code to handle the request
                }else if(c.equals("request")){
                    try{
                        byte [][] reply= handleRequest(from, msg);
        
                        // send the rsponse back to the requesting node
                        sendResponse(from, ref, reply);
                    }catch(Exception e){
                        jlog.error("loop(): got exception from handleRequest()", e);
                    }
                }else{
                    jlog.error("loop(): got unknown cmd: " + c);                          
                }
            }else{
                jlog.error("loop(): got unknown cmd type: " + cmd);
            }
        
        
            // sends a tuple back to requesting Pid {From, Ref, Data}
            private void sendResponse(OtpErlangPid from, OtpErlangRef ref, byte [] ba) {
                // convert the binary reply into and Erlang Binary
                OtpErlangBinary data= new OtpErlangBinary(ba);
        
                // make an array of Erlang Objects
                OtpErlangObject [] eoa= new OtpErlangObject[]{mbox.self(), ref, data};
        
                // make an Erlang tuple out of the Object array above
                OtpErlangTuple tuple= new OtpErlangTuple(eoa);
        
                // send the message back to requesting process Pid
                mbox.send(from, tuple);
            }
        
        }

The worst part is converting back and forth between Erlang objects and Java objects.

You could have several of these nodes running so you can balance the load across several servers if needed.

"Using Erlang with JInterface" by jaime.a.flores@gmail.com

Tue, 23 Jun 2009 09:21:54 -0700

Hello, It would have been good to have the complete source code available.

"Using Erlang with JInterface" by Jim Morris

Tue, 23 Jun 2009 15:56:22 -0700

I agree, unfortunately I cannot release the actual database code for various reasons. Also the Java/Spring/Dao code is pretty standard stuff and good examples can be found in the Spring documentation. All the JInterface code should be here though, and this article was more about JInterface rather than Java/Spring

"Using Erlang with JInterface" by alanmckean@mac.com

Fri, 03 Jul 2009 09:45:27 -0700

I am trying to get an example running and I dfon't understand how something works ... how can you instantiate OtperlangObject when it is an abstract class. This question only arises because I am using a Scala shell to develop an example. Here is the problem: When I evaluate this expression: 'new OtpErlangObject(2)' the shell give me an error that says I can't instantiate OtpErlangObject because it is an abstract class (which, according to the javadocs, it is). But when I run the same example by compiling a Java file, it works! To me, the Scala error makes sense and the Java doesn't. I shouldn't be able to compile the Java file if the class is abstract. What gives?

"Using Erlang with JInterface" by Jim Morris

Fri, 03 Jul 2009 14:19:44 -0700

Did you mean 'new OtpErlangObject[2]' to create an array of 2 Erlang Objects? Because new OtpErlangObject(2) would be wrong. The Formet does not instantiate the object which is why it works, the latter would try to instantiate an Object which would be wrong as it is abstract. It is fine to have a variable of an abstract class so long as it is assigned to a concrete sub-class of that class.

"Using Erlang with JInterface" by goldy

Tue, 10 Nov 2009 15:33:17 -0800

i am working on a project which is a bit torrent software... i have created the GUI using netbeans and the complete software is in erlang. Am really cofused on how to get the corrent data(file name, size, peers, seeds etc) from the database can display it in the GUI.... Can anyone help??

Wolfmans Howlings: Using Erlang with JInterface

Using Erlang with JInterface

Background

Java and Erlang

"Using Erlang with JInterface" by jaime.a.flores@gmail.com

"Using Erlang with JInterface" by Jim Morris

"Using Erlang with JInterface" by alanmckean@mac.com

"Using Erlang with JInterface" by Jim Morris

"Using Erlang with JInterface" by goldy