Posted by Jim Morris
on Fri Feb 06 15:04:54 -0800 2009
There is a raging debate in many forums about how to do fixture-like
things. Basically how do you populate a database with test data so you
can run your Specs/Tests/Features.
There are several libraries out there to do this like FactoryGirl,
FixtureReplacement, Machinist, Fixjour etc etc. If you use Rails and
ActiveRecord pick the one you like and be happy ;)
However I am focusing on how to do this for Integration tests that
test the entire stack, using Cucumber/Webrat in Merb and using the Sequel
ORM. I also use these techniques to test Java based web services
directly over HTTP.
I can't use the above mentioned libraries as they rely on
ActiveRecord, and I don't use that ORM in my Merb Apps (and obviously
not in my Java apps).
I also don't have a preference for fixtures over factories, in
fact I like bits of both paradigms.
My problem with fixtures is that they are really hard to maintain, and
can be slow to load the entire database test set if run for every
Scenario. My problem with factories is they use the Models defined in
ActiveRecord to setup the test data set, which can be very slow as
well. What I like about Factories is keeping the data setup close to
the actual tests, and ease of use and maintenance. What I like about
Fixtures is directly loading the database, and bypassing the models
and associations.
I have solved the problem many times, and in different ways for each
project I do. For my Java projects I have written a typical fixture
loader in Java reading from a standard YAML file, so this is no
different from Rails fixtures, just written in Java. I also wrote a
Sequel based ruby script which loaded the database with test data, and
needed to be manually run every time my integration tests were run.
This was acceptable as it was a final test against a remote staging
server before deployment to production. I needed to be able to load
the staging database over an SSH pipe and test over an SSH pipe.
For my Merb/Sequel based
wolfmanblog
project I use a hybrid solution which so far seems to work well for
me, as it keeps the test data really close to the testing code, it is
relatively fast as it loads the database directly rather than through
the models, and is maintainable as it is close to the testing code.
Using Cucumber for
the integration tests, and loading the test data in Givens with steps
specifically setup to load data, and writing a Class that talks
directly to the database via Sequel
seems to work very nicely for me.
Basically I write a little helper in a DBHelper Class, that loads
specific types of data for a given test or set of tests, these helpers
are called from the steps called by the Givens. i use a pretty high
level abstract in the givens rather than defining low level data, so
I say Given a valid user or Given there are 20 Articles, I let the
step specify the actual data written, because the step also has the
Then clauses so I can keep the data in one place so if the user name
is testuser1, I can test for testuser1 in the same place further down
the file in the Then clause for testing the results. A good example
is in wolfmanblog/features/posts/...
# index.feature
Scenario: GET /
Given 8 posts exist
When I go to /
Then the request should succeed
And I should see post 1
And I should see post 2
And I should see post 3
And I should see post 4
And I should not see post 5
Given /^(\d+) posts exist$/ do |n|
@dbhelper.truncate(:posts)
for i in (1..n.to_i) do
@dbhelper.add_post(:id => i, :title => "post #{i}", :body => "body of post #{i}")
end
end
Then /^I should see post (\d+)$/ do |n|
@response.should have_xpath("//h2/a[@href='/posts/#{n}']['post #{n}']")
@response.should have_selector("p:contains('body of post #{n}')")
end
Then /^I should not see post (\d+)$/ do |n|
@response.should_not have_xpath("//h2/a[@href='/posts/#{n}']")
end
The dbhelper is defined in ./features/support/db_helper.rb and
@dbhelper is set in ./features/support/env.rb.
Notice I truncate the database before loading the new data, and I set
the id of the record so I can test later on. I do this rather than use
transactions as I like to be able to look in the database later for
debugging purposes. Also note I use Postgresql which supports
TRUNCATE .. CASCADE which makes it easier to clean up when you have
foreign key constraints all over the place as I do.
The DBHelper class...
class DBHelper
attr_reader :db
def initialize(target=nil, debug=nil)
if target.nil?
@@target= ENV['testtarget'].nil? ? "test" : ENV['testtarget']
else
@@target= target
end
dburl= case @@target
when 'test'
"postgres://morris:test@localhost:5432/sample1_test"
else
raise 'Bad target'
end
@db= Sequel.open dburl
dblog= Logger.new($stdout)
dblog.level= debug ? Logger::INFO : Logger::WARN
@db.logger= dblog
end
def close
@db.disconnect
end
def truncate(table)
@db.execute("TRUNCATE #{table.to_s} CASCADE")
end
def add_user(name, password, salt)
@db[:users] << {:name => name, :crypted_password => password, :salt => salt}
end
def add_post(h)
@db[:posts].insert(h.merge(:created_at => Time.now.iso8601, :updated_at => Time.now.iso8601))
end
end
Before do
@dbhelper= DBHelper.new('test', false)
end
After do
@dbhelper.close
end
I open a new connection to the database in this case, but I could use
the one used by Merb using @db= Sequel::DATABASES.first.
I try to keep the helpers in this file as generic as possible so they
can be used in different Givens, but sometimes you have to be very
specific. Try to encapsulate all database knowledge in this file.
To summarize...
- I put very high level setup commands in the features Given.
- I put the detailed contents in the associated step.
- I actually load the database in the helper.
- I keep the Then tests close to the Given that sets up the data
Posted in
Rails,Merb,Sequel
|
Tags
merb,sequel,fixtures
|
1 comments
Show
Posted by Jim Morris
on Wed Aug 27 14:46:34 -0700 2008
Background
I was getting bored waiting for Trolltech to release the next version
of Qtopia for the Freerunner, so I ported the xgps client from gpsd's
distribution to Qtopia.
As you may know by now, Qtopia does not have X11 so none of the
existing X11 based or GTK based GPS clients work. I was exploring GPSD
because I wanted to to be able to get a one time position for my sunset
calculator which requires your current latitude and longitude.
Although GPSD is not well suited for that (thats a whole other blog
entry), I did notice you can connect to it over the ethernet, so I was
playing with cgps and xgps that you find in the GPSD tar file, running
on my desktop, talking to gpsd running on my Freerunner.
So wanting to dust of my Qt skills (which were so ancient I barely
even recognize Qt4) I decided to port xgps to Qt.
I first did a pretty straight forward port to my Desktop Qt4/X11.
I designed the form using designer, then plugged in the code taken as
closely as I could from xgps. This was not 1:1 by any means as XLib
and Qt are not similar ;) I ended up just taking the ideas and
algorithms from the xgps program and recoding.
So that took a day or so, as I was relearning Qt as I went, I have to
say the documentation for Qt is excellent, unlike another graphics
library I could mention (I'm looking at you GTK).
I got the basics working, and as my GPS fix inside was not very good
(works about 50/50), I found you could also point any gpsd based
client at gpsd.rellim.com and always get good data from there, this is
great for testing.
So I had qtgps (for want of a better name) running on my desktop,
looking pretty much the same as xgps, with the satellite list the sky
view and of course the basic Fix data.
Now the trick to port it to FR, so I could run this on my FR.
The UI wasn't going to fit on the FR screen so I revamped the UI in
Designer, by basically dragging all the views into a tabbed frame. So
the satellite list, the sky view, the position information and the
movement information all had their own tab, this fits pretty well on a
tiny screen.
Other than that I didn't have to do much. I had to create a new
qtgps.pro file for Qtopia, as the qmake qt one is somewhat different.
I had to use the skeleton main.cpp that all Qtopia apps seem to use,
but the bulk of the code remains the same.
Sending signals from callbacks
I did have an interesting time trying to figure out how to emit a
signal from the gps callback thread to the main window, not sure if I
chose the right way to do it but it worked... I created a singleton
QObject based class, and allowed it to be called statically, then that
does the emit. This is used to signal the main display thread that new
gps data has arrived. (Note this won't work if the singleton can be
called concurrently from multiple threads, Google c++ singleton
pattern to see how to do that).
// Singleton for sending signal from a callback
class ExtSig : public QObject
{
Q_OBJECT
public:
static ExtSig *self();
void send(struct gps_data_t* p, char* buf);
signals:
void sendit(struct gps_data_t* p, char* buf);
private:
ExtSig();
};
// Singleton defns for emiting a signal from a callback
ExtSig::ExtSig(){}
ExtSig *ExtSig::self()
{
static ExtSig inst;
return &inst;
}
void ExtSig::send(struct gps_data_t* p, char* buf)
{
emit sendit(p, buf);
}
// in callbeck call...
// signal the GUI thread we have new gps data
ExtSig::self()->send(p, buf);
Howto build your own applications
Ok so how can you write your own Qtopia applications? There are at
least two ways, 1) use the VMPlayer based Toolchain that Trolltech
provide, or 2) setup your Linux workstation to use the toolkit. 1) is
easier, and just works on pretty much anything that runs VMPlayer and
has a lot of memory. 2) is faster and used less memory. I chose 2) but
I did play with 1).
I won't go into how to do 1) as it is pretty much straight forward,
just download the
iso, and
install it (either from a loopback device or from a burned CD).
The second method is not so well documented (if at all) so here is
what I did to make it work on a Ubuntu Hardy i686 32-bit workstation
with KDE installed..
Download the
toolchain
and install it into /opt/toolchains, it seems it must be there
otherwise it doesn't quite work.
Then download the latest
qtopia snapshot and put that in a
working directory and set QTOPIA_DEPOT_PATH to that directory.
Then create a build directory for the qtopia build and set the QPEDIR
environment variable to that path.
In my case I have
QTOPIA_DEPOT_PATH=/opt/QtopiaSDK/qtopia-opensource-src-4.3.2-snapshot-20080815
and
QPEDIR=/opt/QtopiaSDK/build/qtopia-4.3.2
Now we need to build the qtopia snapshot, you can use the result to
copy to /opt/Nokia/Qtopia on your FR, or not, but you need to do this to get
the tools to build your own apps.
Now follow these steps to build Qtopia:-
> cd $QPEDIR
> $QTOPIA_DEPOT_PATH/configure -device ficgta01
> make
> make install
Go get some coffee or dinner :) this will take a while.....
NOTE You may need to set -prefix /opt/Nokia/Qtopia in the
./configure stage for newer versions, as Trolltech changed where Qtopia
runs from.
Now you will have a full Qtopia build in $QPEDIR/image
You can use the following script to copy this to your FR, it is
modified from Trolltechs update script. It will keep your settings.
However you will have needed to have flashed to the 0808 version of
Qtopia and kernel for this to work.
#!/bin/sh
QTOPIA_DIR=$QPEDIR/image
QTOPIA_IMAGE=qtopia.tar.gz
PHONEIP=your_fr_ip_address
# sudo ifconfig usb0 your_fr_ip_address up
tar -C $QTOPIA_DIR czvf $QTOPIA_IMAGE
cat $QTOPIA_IMAGE | ssh "root@$PHONEIP" '(set -x;rm -f /tmp/restart-qtopia;killall qpe; mkdir -p /opt/Nokia/Qtopia;rm -rf /opt/Nokia/Qtopia/*; cd /opt/Nokia/Qtopia;gunzip |tar xvf -;/etc/init.d/qpe start &)'
Whether you do that or just keep the Trolltech released image you are
now ready to write your own apps. I recommend you study the
Qt docs Qtopia is pretty much Qt4.3 at
the moment.
As an example you can use the
qtgps.tar.gz as an
example app or use $QTOPIA_DEPOT_PATH/examples/application
I created a build/myapps directory, and then tar xvfz qtgps.tar.gz
into that directory. Then...
> cd build/myapps/qtgps
> $QPEDIR/bin/qtopiamake
> make
This will create a qtgps excutable that you can copy to your FR and
run.
A nice trick I found is to run a GUI app from the ssh into FR you can
do this...
> ssh your_fr_ip
fr# . /opt/Nokia/Qtopia/qpe.env
fr# ./qtgps
The app will popup on the FR screen, you could also run it from the
terminal on the FR or from the file manager on the FR.
To create a new application you can do this..
> cd build/myapps/mynewapplication
> ... edit some .cpp and .h files
> $QPEDIR/bin/qtopiamake -project
> ... edit mynewapplication.pro appropriately
> $QPEDIR/bin/qtopiamake
> make
The skeleton main.cpp should look like this...
#include "mynewapp.h"
#include <qtopiaapplication.h>
QTOPIA_ADD_APPLICATION(QTOPIA_TARGET,MyNewApp)
QTOPIA_MAIN
The rest of the files are up to you. I like to create the UI using
designer...
> $QPEDIR/bin/designer mynewapp.ui
look in either example.cpp or qtgps.cpp and qtgps.h on how to
initialize a designer based UI.
There are lots of example apps on the Trolltech docs site.
Running app on the workstation
If you want to run the app locally rather than on your FR, you can use
$QPEDIR/bin/qvfb
You will need to build a native version of qtopia to do that, which
can be done by repeating the above steps but in a new build directory
do:-
> export QPEDIR=build/qtopia-native
> mkdir $QPEDIR
> cd $QPEDIR
> $QTOPIA_DEPOT_PATH/configure -device i686fb
> make
> make install
and then rebuild your app for the native machine...
> cd build/myapps/qtgps
> make clean
> $QPEDIR/bin/qtopiamake
> make
Then run the native version of your app, but you will need to do the following to make the size
correct...
> export QWS_DISPLAY=QVFb:mmWidth43:mmHeight58:0
> $QPEDIR/bin/qvfb -width 480 -height 640 &
> sleep 2
> ./qtgps -qws gpsd.rellim.com
This will run qtgps on the local workstation in a simulation of the FR
screen (without qtopia), using gps data from gpsd.rellim.com.
Packaging
UPDATED I figured out how to install.
This document
explains how to install a package so it shows up in the applications
list, here is how I actually did it.
I created a directory called feed in my work directory, and set the
path to $HTTPDIR. Then I ran a simple web server (I used boa but any
will do), that points to $HTTPDIR to serve files.
Then I built the qpk package...
> cd $WORKDIR/qtgps
> make packages
> $QPEDIR/bin/mkPackages $HTTPDIR
Then make sure the web server is running, on the FR goto the Software
Packages App, add the following server to the server lists
http://192.168.0.200:8000, this presumes you are connected via USB and
you ran the webserver at port 8000. Then click download, and you
should see qtgps in the list, select it for install, and it will get
installed. From then on you can replace the qtgps binary it installs
with any new builds. It will be under
/home/root/packages/somelongrandomstring/bin/qtgps
If you are using the vmplayer version of the SDK, then you can simply
do this...
> sdk -b
> sdk -p
> sdk -i
The package for this is available
here,
(Right click to download). (Remember to generate the packages.list
file using mkPackages). Setup a local web server and install as
explained above.
A simple ruby script like this could also be used as a web server:-
--------- webserver.rb ---------
#!/usr/local/bin/ruby
require 'webrick'
include WEBrick
s = HTTPServer.new(
:Port => 8000,
:DocumentRoot => Dir::pwd
)
trap("INT"){ s.shutdown }
s.start
--------------------------------
> ruby webserver.rb
Then point your feed to http://192.168.0.200:8000/feed if using USB
networking and the .qpk file is in the feed directory.
Where to get the source
The source code to this project is now on github
http://github.com/wolfmanjm/qtopiagps/tree/master
Installing required libraries for the toolchain
UPDATE I forgot to mention how I got the libgps stuff into the
toolchain. I found a hack, first I built gpsd-dev using mokomakefile
then I did this...
dpkg-deb -X build/tmp/deploy/glibc/ipk/armv4t/gpsd-dev_2.34-r9_armv4t.ipk /opt/QtopiaSDK/opt/toolchains/arm920t-eabi/arm-angstrom-linux-gnueabi
This worked for this case and setup the .so links correctly. You could
also copy the libgps.so from your FR.
Another user (Radek Barton) did it this way, but I haven't tested
this...
1. Download lastest gpsd source code at http://download.berlios.de/gpsd/gpsd-2.37.tar.gz and unpack it somewhere.
2. Add line #include <linux/limits.h> to gpsd.h-head file.
3. Modify line 15 of gps.h file from <limits.h> to <linux/limits.h>
4. configure with command: PATH=/opt/toolchains/arm920t-eabi/bin:$PATH ./configure --host=arm-angstrom-linux-gnueabi
5. Then make, make install, etc.
For those that cannot do that, then download this tar file to /pathtodownload/...
and...
> cd /
> tar xzvf /pathtodownload/libgps.tar.gz
This presumes that your Qtopia toolchain is in /opt/toolchains
Also note that the following packages need to be installed on the FR
libgps16, gps-utils and gpsd.
Posted in
Openmoko
|
Tags
openmoko,freerunner,gps
|
32 comments
Show
Posted by Jim Morris
on Sun Aug 10 23:09:54 -0700 2008
I finally got around to building ruby 1.8.6 for my FR. I modified the
ruby bitbake files that I found in the Mokomakefile openembedded
directory. I am not sure how one actually is meant to do this, as the
OE site is down and the docs don't explain it. So I just replaced the
1.8.5 ones with the 1.8.6 ones. I also managed to fix a bug in the
1.8.5 BB recipe that was causing socket to not build.
I also got the ruby dbus library to work, although I don't know what to
do with it yet :)
So until I figure out how you are meant to add a new version to OE,
I have put the ipk here.
If you want ruby 1.9 you can find it
here
To install it just ssh into your FR and type
> wget http://blog.wolfman.com/files/ruby_1.8.6-p287-r1_armv4t.ipk
> opkg install ruby_1.8.6-p287-r1_armv4t.ipk
If you want to install gem then you need to ssh into the FR and do
this...
> wget http://rubyforge.org/frs/download.php/38646/rubygems-1.2.0.tgz
> tar xzvf rubygems-1.2.0.tgz
> cd rubygems-1.2.0
> ruby setup.rb --no-rdoc --no-ri
I recommend you do the no doc and no ri otherwise it takes forever.
As for dbus...
> wget https://trac.luon.net/data/ruby-dbus/releases/ruby-dbus-0.2.1.tar.gz
> tar xzvf ruby-dbus-0.2.1.tar.gz
> cd ruby-dbus-0.2.1
> ruby setup.rb config
> ruby setup.rb setup
> ruby setup.rb install
look in the examples to see how to use it, I tries this...
> ruby examples/utils/listnames.rb --system
org.freedesktop.DBus
:1.3
:1.4
org.freedesktop.Avahi
:1.0
org.bluez
:1.1
:1.6
:1.2
org.freedesktop.Hal
So it seems to work.
Let me know what you do with it.
I also built jove, my favorite mini-emacs editor, this is just an
executable just copy to /usr/bin Use instead of vi or nano.
UPDATE The bitbake recipe I used to build ruby 1.8.6 is here
Posted in
Openmoko
|
Tags
ruby,freerunner
|
3 comments
Show
