Fixtures VS Factories, or how I do fixtures

Jim Morris — Fri, 06 Feb 2009 20:40:55 -0800

There is a raging debate in many forums about how to do fixture-like things. Basically how do you populate a database with test data so you can run your Specs/Tests/Features.

There are several libraries out there to do this like FactoryGirl, FixtureReplacement, Machinist, Fixjour etc etc. If you use Rails and ActiveRecord pick the one you like and be happy ;)

However I am focusing on how to do this for Integration tests that test the entire stack, using Cucumber/Webrat in Merb and using the Sequel ORM. I also use these techniques to test Java based web services directly over HTTP.

I can't use the above mentioned libraries as they rely on ActiveRecord, and I don't use that ORM in my Merb Apps (and obviously not in my Java apps).

I also don't have a preference for fixtures over factories, in fact I like bits of both paradigms.

My problem with fixtures is that they are really hard to maintain, and can be slow to load the entire database test set if run for every Scenario. My problem with factories is they use the Models defined in ActiveRecord to setup the test data set, which can be very slow as well. What I like about Factories is keeping the data setup close to the actual tests, and ease of use and maintenance. What I like about Fixtures is directly loading the database, and bypassing the models and associations.

I have solved the problem many times, and in different ways for each project I do. For my Java projects I have written a typical fixture loader in Java reading from a standard YAML file, so this is no different from Rails fixtures, just written in Java. I also wrote a Sequel based ruby script which loaded the database with test data, and needed to be manually run every time my integration tests were run. This was acceptable as it was a final test against a remote staging server before deployment to production. I needed to be able to load the staging database over an SSH pipe and test over an SSH pipe.

For my Merb/Sequel based wolfmanblog project I use a hybrid solution which so far seems to work well for me, as it keeps the test data really close to the testing code, it is relatively fast as it loads the database directly rather than through the models, and is maintainable as it is close to the testing code.

Using Cucumber for the integration tests, and loading the test data in Givens with steps specifically setup to load data, and writing a Class that talks directly to the database via Sequel seems to work very nicely for me.

Basically I write a little helper in a DBHelper Class, that loads specific types of data for a given test or set of tests, these helpers are called from the steps called by the Givens. i use a pretty high level abstract in the givens rather than defining low level data, so I say Given a valid user or Given there are 20 Articles, I let the step specify the actual data written, because the step also has the Then clauses so I can keep the data in one place so if the user name is testuser1, I can test for testuser1 in the same place further down the file in the Then clause for testing the results. A good example is in wolfmanblog/features/posts/...

   # index.feature
           Scenario: GET /
             Given 8 posts exist
             When I go to /
             Then the request should succeed
             And I should see post 1
             And I should see post 2
             And I should see post 3
             And I should see post 4
             And I should not see post 5

#
        # steps/post_steps.rb
        #
        
        # creates n posts
        Given /^(\d+) posts exist$/ do |n|
          @dbhelper.truncate(:posts)
          for i in (1..n.to_i) do
            @dbhelper.add_post(:id => i, :title => "post #{i}", :body => "body of post #{i}")
          end
        end
        
        # tests if a specific post is visible
        Then /^I should see post (\d+)$/ do |n|
          @response.should have_xpath("//h2/a[@href='/posts/#{n}']['post #{n}']")
          @response.should have_selector("p:contains('body of post #{n}')")
        end
        
        Then /^I should not see post (\d+)$/ do |n|
          @response.should_not have_xpath("//h2/a[@href='/posts/#{n}']")
        end

The dbhelper is defined in ./features/support/db_helper.rb and @dbhelper is set in ./features/support/env.rb.

Notice I truncate the database before loading the new data, and I set the id of the record so I can test later on. I do this rather than use transactions as I like to be able to look in the database later for debugging purposes. Also note I use Postgresql which supports TRUNCATE .. CASCADE which makes it easier to clean up when you have foreign key constraints all over the place as I do.

The DBHelper class...

#
        # Helper class to manipulate database directly
        #
        class DBHelper
          attr_reader :db
        
          # setup database access
          def initialize(target=nil, debug=nil)
            if target.nil?
              @@target= ENV['testtarget'].nil? ? "test" : ENV['testtarget']
            else
              @@target= target
            end
            dburl= case @@target
              when 'test'
                "postgres://morris:test@localhost:5432/sample1_test"
              else
                raise 'Bad target'
            end
        
            @db= Sequel.open dburl
            dblog= Logger.new($stdout)
            dblog.level= debug ? Logger::INFO : Logger::WARN
            @db.logger= dblog
          end
        
          def close
            @db.disconnect
          end
        
          def truncate(table)
            @db.execute("TRUNCATE #{table.to_s} CASCADE")
          end
        
          def add_user(name, password, salt)
            @db[:users] << {:name => name, :crypted_password => password, :salt => salt}
          end
        
          def add_post(h)
            @db[:posts].insert(h.merge(:created_at => Time.now.iso8601, :updated_at => Time.now.iso8601))
          end
        end
        
        # in ./features/support/env.rb
        # runs before each Scenario
        Before do
          @dbhelper= DBHelper.new('test', false)
        end
        
        After do
          @dbhelper.close
        end

I open a new connection to the database in this case, but I could use the one used by Merb using @db= Sequel::DATABASES.first.

I try to keep the helpers in this file as generic as possible so they can be used in different Givens, but sometimes you have to be very specific. Try to encapsulate all database knowledge in this file.

To summarize...

I put very high level setup commands in the features Given.
I put the detailed contents in the associated step.
I actually load the database in the helper.
I keep the Then tests close to the Given that sets up the data

"Fixtures VS Factories, or how I do fixtures" by Phlip

Sun, 15 Feb 2009 16:55:07 -0800

I have a few tips about tests that resist fixture drift - of whatever type - in this post here: http://broadcast.oreilly.com/2009/02/merb-mind-maps.html And tx for this blog - I used it in that post!

Wolfmans Howlings: Fixtures VS Factories, or how I do fixtures

Fixtures VS Factories, or how I do fixtures

"Fixtures VS Factories, or how I do fixtures" by Phlip