An Architect's View

CFML, Clojure, Software Design, Frameworks and more...

An Architect's View

Automated Testing with Selenium

December 30, 2009 ·

A few weeks ago, I blogged about a fix for running Selenium with ant on Snow Leopard and several people have asked me to blog about the basic setup for which that fix applies since it is of much broader interest to developers... Yes, good point! A bit of background first. I'm working on a large ColdBox project and one of the first things I did when I joined the project was to start creating some unit tests and integration tests. ColdBox has integration with MXUnit so you can fairly easily write unit tests for parts of your model and integration tests for your event handlers. Over time we've built a suite of over 100 MXUnit-based tests. If I had my way, we'd have a lot more by now but test-infecting your colleagues isn't always easy!Once we had enough of the system working, testing some of the complex flows through the application became pretty tedious and some sort of automation became necessary. We picked Selenium since it offers a plugin for Firefox that lets you record a click-thru session for your application and play it back. Selenium tests are saved as simple HTML documents with a table listing actions, targets and values in three columns. You turn this into a test by adding rows into the table that assert text is present (in the rendered page). You might end up with rows like this:
<tr>
   <td>open</td>
   <td>/</td>
   <td></td>
</tr>
<tr>
   <td>assertTextPresent</td>
   <td>Home Page</td>
   <td></td>
</tr>
<tr>
   <td>type</td>
   <td>username</td>
   <td>testuser</td>
</tr>
<tr>
   <td>type</td>
   <td>password</td>
   <td>secret</td>
</tr>
<tr>
   <td>clickAndWait</td>
   <td>FONT COLOR=GRAY<>I<>//input[@value='login']</td>    <td></td>
</tr>
<tr>
   <td>assertTextPresent</td>
   <td>Welcome, testuser!</td>
   <td></td>
</tr>
The next step is to assemble these test scripts together in test suites. Again, these are simple HTML documents with tables. This time tho' the tables have a single column that contains a link to a test script in each row. The scripts and suites can be run in pretty much any web browser using the Selenium core/TestRunner.html test harness. The final step is to fully automate these suites by running them automatically as part of your ant build script. You have an ant build script, right? When you download Selenium, you get an IDE to install into Firefox (an XPI file) and a JAR file, which is what you need for use with ant. A good idea is to create an "init-selenium" target in your ant script that looks something like this:
<target name="init-selenium" depends="clean" description="Set up the environment for Selenese tests.">
   <property name="selenium-library.jar" value="${selenium.home}/selenium-server.jar"/>
   <taskdef resource="selenium-ant.properties">
      <classpath>
         <pathelement location="${selenium-library.jar}" />
      </classpath>
   </taskdef>
</target>
This assumes the property selenium.home is already set up to point to the folder where you installed Selenium. This ant target can be used as a dependency for other ant targets that actually run Selenium tests. It defines the <selenese> ant task (by pulling its definition out of the properties file in the Selenium JAR file). After this target has run, you can execute the following ant task:
<selenese
      suite="${tests}/selenium/suites/${suite.name}.html"
      results="${tests}/results/selenium/${suite.name}-results.html"
      browser="*${browser}"
      multiWindow="false"
      timeoutInSeconds="300"
      startURL="${selenium.host}" />
Again, this has a number of ant properties predefined just to make the script easier to customize. The browser property is normally going to be "firefox" but could be "safari" or any other browser supported by Selenium (such as "iexplore" if you must). The selenium.host property specifies the base URL of the web site that should be tested and the tests property specifies the root folder for your Selenium tests (I have lots of different types of tests in that directory, including the Selenium tests). I hope that gives you a taste of how to get automated testing going with Selenium. In addition to our 100+ MXUnit tests, we now have three suites of automated Selenium tests covering several areas of our application. Our ant build script resets the entire state of the server and database and runs our Scala test suite (I blogged about that a bit before), our MXUnit test suites and our Selenium test suites in anything between three and eight minutes depending on the speed of each developer's machine.

Tags: coldfusion · tdd

5 responses

  • 1 Jamie Krug // Dec 30, 2009 at 10:00 AM

    Good stuff!

    Can you elaborate on the &quot;resets the entire state of the server and database&quot; part?

    The database part makes sense -- any data-dependent tests need to start with a clean (maybe empty or nearly empty) database. Are there ANT components that specifically assist with say MySQL resets, or do you just invoke a command line call to execute a SQL script?

    The server part I'm more curious about. Are you referring to the CFML engine and/or servlet container? What gets reset?

    Thanks!
  • 2 Sean Corfield // Dec 30, 2009 at 10:15 PM

    In this particular case we are restarting Railo using the cfadmin tag in a script that we run with the ant get task. It's nice because we don't have to restart the container.
    We also have a script that drops and recreates the database into a known state (we use a CFML script and XML files but any similar setup would work just fine).

    The overall ant script does a lot of stuff :) It builds the Scala code, runs the ScalaTest suite, auto-generates ScalaDocs, restarts Railo, reloads the database with seed data, repopulates the search engine, runs the unit test suite (component-based tests via MXUnit) and the integration test suite (controller-level tests via ColdBox/MXUnit) and then the Selenium test suites.

    Takes about 7-8 minutes on most people's machines (but only 2-3 minutes on my 2.8GHz quad core iMac :)
  • 3 Sean Corfield // Dec 31, 2009 at 11:10 PM

    @Jamie, BTW, I blogged this back in November:

    http://corfield.org/entry/Restarting_Railo_with_Ant
  • 4 Todd // Jan 8, 2010 at 9:40 AM

    How large or long should you be working on a project before automation testing with Selenium &quot;makes sense&quot;? Obviously, smaller projects less than a month definitely wouldn't justify the setup time it takes to get the automation working. Do you have any rule of thumb as to when you should look at making the investment in automation testing?
  • 5 Sean Corfield // Jan 29, 2010 at 1:45 PM

    Selenium itself really is very lightweight and easy to set up so I don't think there are many projects where it's too much investment.

    Automation via ant isn't a huge deal either and, for any multi-developer project, I'd prefer to have automated testing in place.