Wolfmans Howlings

A programmers Blog about Programming solutions and a few other issues

Using Cucumber to test Erlang Servers

Posted by Jim Morris on Sat May 02 00:09:36 -0700 2009

Background

I've been using cucumber to do all my integration testing. Usually testing over the wire to a live system, regardless of what the target system is written in (Ruby Rails, Merb, Java etc).

I have been writing a conference server in Erlang (actually rewriting), and have been testing it over the wire using JRuby and Cucumber.

I have a Java library which talks the binary protocol, so using JRuby allows me to use that library. However I need to test a lot of edge conditions in the server, and these cannot easily be tested using the wire protocol.

So after discovering JInterface I realized I could probably use JRuby and Cucumber to directly test my Erlang gen_server based module directly.

It works pretty well, first I wrap the JInterface code in a ruby library (actually JRuby), then I write my steps to talk to that wrapper. I can get pretty deep in my testing, even to the point of directly testing the returned tuplets.

I could use one of the Ruby to Erlang libraries, but the two I found are either no longer supported (Erltricity) or too new to be stable (erlix), so using JInterface under JRuby seems a good alternative for now. I'll be keeping my eye on erlix as it is under development and seems to be written the right way.

The Code

Erlang Wrapper

OK down to business, here is the wrapper, what I have tried to do is put all the JInterface knowledge and knowledge of Erlang types in this wrapper, I would also put any specific calls to the Erlang Node and unpacking of any return data here as well, but for now I'll keep it simple.

Put this in the libs directory of the cucumber test directory, given the standard cucumber directory hierarchy...

workdir
  features
    step_definitions
support
  libs

You need to put the OtpErlang.jar file in the libs directory as well.

So this is the wrapper, you will probably add some convenience methods here, but these are the basic ones for this article.

What I am doing here is monkey patching some of the Ruby types and Erlang types to be able to easily convert to and from each type, and to produce strings etc. You can extend this to handle types that occur commonly in your programs.

In connect I do the JInterface incantations necessary to use the rpc call API. When using rpc to call functions on other nodes, it uses rex, and the return tuple always starts with the atom rex. However I have a few out of band messages coming in from the server under test and these can come in pretty much anytime, so rather than using the receiveRpc call I extract the messages myself, with a timeout, and if it isn't a rex reply then I put them in a fifo for later testing.

To try to make things easier in the steps, I try to convert Ruby types to Erlang types in any arguments I pass in, and I make some assumptions, like a Ruby symbol becomes an Erlang atom, and that Ruby strings become Erlang binaries (as I use binaries to store all my strings in the server I am writing, you can change that to list based strings if you prefer).

env.rb

Next the features/support/env.rb which sets up the environment and provides a few helper functions to talk to the erlang wrapper, it basically creates a new erlang_wrapper instance for each connection and stores it so it can clean up and close the connections when done, I actually create multiple nodes so I can have them talk with each other, but I'll leave that for another article.

The Feature

Here is the feature for the simple server I am testing. It is a gen_server called id_server that has one job and that is to return a an integer every time next_value is called, this is used globally in my system, to return a unique id for anything that calls it.

This is put in features/id_server.feature

Feature: test connection to an Erlang node
In order to test my Erlang code with Cucumber
Cucumber needs to be able to communicate with Erlang nodes
So I can write cool features to test the nodes directly

  Scenario: ask id_server for next value
    Given a connection to node "vs1@localhost"
    When I call id_server:next_value
    Then the result is a tuple of arity 2
    And the tuple has the following elements:
      | idx | result |
      | 0   | value  |
      | 1   | :int   |

    When I call id_server:next_value
    Then the 2nd value will be incremented by 1

The Steps

Lastly the basic steps go in features/step_definitions/erlang_steps

Given /^a connection to node "([^"]*)"$/ do |node|
  @erl= create_node_to(node)
end

When /^I call ([^:]+):([^ ]+)$/ do |mod, fun|
  erl= get_node(1)
  @result= erl.call(mod, fun)
end

Then /^the result is a tuple of arity (d+)$/ do |n|
  erl= get_node(1)
  erl.arity(@result).should == n.to_i
end

Then /^the tuple has the following elements:$/ do |table|
  erl= get_node(1)
  # table is a Cucumber::Ast::Table
  table.hashes.each do |h|
    i = h['idx']
    v = h['result']
    e = @result[i.to_i]
    # remember the value at this index for further tests
    @values ||= []
    @values[i.to_i]= e

    if v[0] == ?:
      case v[1..-1].to_sym
      when :int then e.to_ruby.should  be_kind_of(Numeric)
      when :str then e.to_ruby.should be_kind_of(String)
      when :atom then e.to_ruby.should be_kind_of(Symbol)
      else raise "unknown type: #{v}"
      end
    else
      e.to_s.should == v
    end
  end
end

Then /^the (d+)(st|nd|rd|th) value will be incremented by (d+)$/ do |idx, _, inc|
  e = @result[idx.to_i-1]
  o = @values[idx.to_i-1]
  e.to_ruby.should == (o.to_ruby + inc.to_i)
end

So some BDD'ists in the audience may start to howl at me, when they see I carry state around between steps specifically the @result and @values variables, but to be honest I don't think it is a big deal and I don't really see any way to avoid it in this case (if someone can enlighten me please leave a comment).

What I am doing here is telling my erlang_wrapper to connect to the specified Erlang node, make an RPC call to that node, then analyze the results. The results stored in @results contain a tuple, whose arity I check, then use the neat Cucumber tables feature to test each element in the tuple. I use a convention here where I either test the specific value or the returned type if I prefix a : to the value. I have to do this because I do not know what the initial value will be that the id_server will return. I store each element of the tuple in a variable @values so I can test it later in another step.

I then call next_value again, and then use that stored value to see if the integer has incremented, presuming I am the only one talking to the server at this time that would be the correct result, of course if this were a live system it could have been incremented by someone else in the meantime, but I am not going to worry about that right now.

I hope this goes to show just how powerful Cucumber can be, with a little help from JRuby.

Posted in Erlang,Cucumber  |  Tags erlang,cucumber,bdd  |  8 comments

Comments

  1. Aslak Hellesøy said on Sat May 02 19:46:45 -0700 2009
    This is huge. For me at least. I want to learn Erlang, and now I can be led by Cucumber :-)
  2. wolfmanjm said on Sun May 03 14:39:54 -0700 2009
    I added the missing parse_oobd_message, although it is not used in this feature It will store any messages from the erlang node that are not rex messages
  3. Aslak Hellesøy said on Wed Oct 07 15:51:43 -0700 2009
    Jim,

    If you could write Step Definitions in Erlang, would you? And if you would, how could that look?
    I'm considering adding first-class Erlang support in cuke4duke: http://wiki.github.com/aslakhellesoy/cuke4duke
  4. Jim Morris said on Wed Oct 07 19:31:32 -0700 2009
    No I wouldn't write them in erlang, necessarily, as Erlang is such an ugly language! try writing unit tests in EUNIT, its horrible.

    However it may make some things easier as the type conversion can be a pain to handle the way I currently do it.
  5. Steve Yen said on Tue Dec 01 18:58:29 -0800 2009
    I created a 100% erlang implementation of Cucumber, called cucumberl, for my OTP server testing needs, which others might find useful. A step definition in erlang looks like: "step([given, i, have, entered, N, into, the, calculator], _) -> ..."

    The repo for cucumberl is at http://github.com/northscale/cucumberl

    Cheers.
  6. Jim Morris said on Tue Dec 01 19:24:43 -0800 2009
    Steve that is very cool (although the syntax is not as nice as in Ruby ;)
    You should announce this on the cucumber and erlang groups.

    I'll try playing with it, thanks
  7. Aslak Hellesøy said on Mon Mar 15 08:47:05 -0700 2010
    FYI - since this blog post was written there is now Cuke4Duke. Cuke4Duke lets you write Cucumber Step Definitions in Java instead of Ruby. If you only want to talk to JInterface I'd use that instead of Ruby I think.
  8. Carl said on Fri Apr 19 03:30:57 -0700 2013
    Thank you, this was very helpful.

(leave email »)