Wolfmans Howlings

Using Cucumber to test Erlang Servers

Posted by Jim Morris on Sat May 02 00:09:36 -0700 2009

Background

I've been using cucumber to do all my integration testing. Usually testing over the wire to a live system, regardless of what the target system is written in (Ruby Rails, Merb, Java etc).

I have been writing a conference server in Erlang (actually rewriting), and have been testing it over the wire using JRuby and Cucumber.

I have a Java library which talks the binary protocol, so using JRuby allows me to use that library. However I need to test a lot of edge conditions in the server, and these cannot easily be tested using the wire protocol.

So after discovering JInterface I realized I could probably use JRuby and Cucumber to directly test my Erlang gen_server based module directly.

It works pretty well, first I wrap the JInterface code in a ruby library (actually JRuby), then I write my steps to talk to that wrapper. I can get pretty deep in my testing, even to the point of directly testing the returned tuplets.

I could use one of the Ruby to Erlang libraries, but the two I found are either no longer supported (Erltricity) or too new to be stable (erlix), so using JInterface under JRuby seems a good alternative for now. I'll be keeping my eye on erlix as it is under development and seems to be written the right way.

The Code

Erlang Wrapper

OK down to business, here is the wrapper, what I have tried to do is put all the JInterface knowledge and knowledge of Erlang types in this wrapper, I would also put any specific calls to the Erlang Node and unpacking of any return data here as well, but for now I'll keep it simple.

Put this in the libs directory of the cucumber test directory, given the standard cucumber directory hierarchy...

workdir
  features
    step_definitions
support
  libs

You need to put the OtpErlang.jar file in the libs directory as well.

So this is the wrapper, you will probably add some convenience methods here, but these are the basic ones for this article.

What I am doing here is monkey patching some of the Ruby types and Erlang types to be able to easily convert to and from each type, and to produce strings etc. You can extend this to handle types that occur commonly in your programs.

In connect I do the JInterface incantations necessary to use the rpc call API. When using rpc to call functions on other nodes, it uses rex, and the return tuple always starts with the atom rex. However I have a few out of band messages coming in from the server under test and these can come in pretty much anytime, so rather than using the receiveRpc call I extract the messages myself, with a timeout, and if it isn't a rex reply then I put them in a fifo for later testing.

To try to make things easier in the steps, I try to convert Ruby types to Erlang types in any arguments I pass in, and I make some assumptions, like a Ruby symbol becomes an Erlang atom, and that Ruby strings become Erlang binaries (as I use binaries to store all my strings in the server I am writing, you can change that to list based strings if you prefer).

env.rb

Next the features/support/env.rb which sets up the environment and provides a few helper functions to talk to the erlang wrapper, it basically creates a new erlang_wrapper instance for each connection and stores it so it can clean up and close the connections when done, I actually create multiple nodes so I can have them talk with each other, but I'll leave that for another article.

The Feature

Here is the feature for the simple server I am testing. It is a gen_server called id_server that has one job and that is to return a an integer every time next_value is called, this is used globally in my system, to return a unique id for anything that calls it.

This is put in features/id_server.feature

Feature: test connection to an Erlang node
In order to test my Erlang code with Cucumber
Cucumber needs to be able to communicate with Erlang nodes
So I can write cool features to test the nodes directly

  Scenario: ask id_server for next value
    Given a connection to node "vs1@localhost"
    When I call id_server:next_value
    Then the result is a tuple of arity 2
    And the tuple has the following elements:
      | idx | result |
      | 0   | value  |
      | 1   | :int   |

    When I call id_server:next_value
    Then the 2nd value will be incremented by 1

The Steps

Lastly the basic steps go in features/step_definitions/erlang_steps

Given /^a connection to node "([^"]*)"$/ do |node|
  @erl= create_node_to(node)
end

When /^I call ([^:]+):([^ ]+)$/ do |mod, fun|
  erl= get_node(1)
  @result= erl.call(mod, fun)
end

Then /^the result is a tuple of arity (d+)$/ do |n|
  erl= get_node(1)
  erl.arity(@result).should == n.to_i
end

Then /^the tuple has the following elements:$/ do |table|
  erl= get_node(1)
  # table is a Cucumber::Ast::Table
  table.hashes.each do |h|
    i = h['idx']
    v = h['result']
    e = @result[i.to_i]
    # remember the value at this index for further tests
    @values ||= []
    @values[i.to_i]= e

    if v[0] == ?:
      case v[1..-1].to_sym
      when :int then e.to_ruby.should  be_kind_of(Numeric)
      when :str then e.to_ruby.should be_kind_of(String)
      when :atom then e.to_ruby.should be_kind_of(Symbol)
      else raise "unknown type: #{v}"
      end
    else
      e.to_s.should == v
    end
  end
end

Then /^the (d+)(st|nd|rd|th) value will be incremented by (d+)$/ do |idx, _, inc|
  e = @result[idx.to_i-1]
  o = @values[idx.to_i-1]
  e.to_ruby.should == (o.to_ruby + inc.to_i)
end

So some BDD'ists in the audience may start to howl at me, when they see I carry state around between steps specifically the @result and @values variables, but to be honest I don't think it is a big deal and I don't really see any way to avoid it in this case (if someone can enlighten me please leave a comment).

What I am doing here is telling my erlang_wrapper to connect to the specified Erlang node, make an RPC call to that node, then analyze the results. The results stored in @results contain a tuple, whose arity I check, then use the neat Cucumber tables feature to test each element in the tuple. I use a convention here where I either test the specific value or the returned type if I prefix a : to the value. I have to do this because I do not know what the initial value will be that the id_server will return. I store each element of the tuple in a variable @values so I can test it later in another step.

I then call next_value again, and then use that stored value to see if the integer has incremented, presuming I am the only one talking to the server at this time that would be the correct result, of course if this were a live system it could have been incremented by someone else in the meantime, but I am not going to worry about that right now.

I hope this goes to show just how powerful Cucumber can be, with a little help from JRuby.