This week Gojko Adzic wrote about the anatomy of a good acceptance test. After having read his elaboration, I remembered how I came up with the preparation for the EWT19 session some weeks ago. We used RobotFramework to automate tests for the Parking Lot Calculator that we asked Weekend Testing participants a few weeks earlier with manual Exploratory Testing. To get testers started we provided them with three examples that I prepared before the session. We then asked testers to automate their tests for the ParkCalc website based on one of the examples we provided. Here is my write-up how I came up with the examples, and what I had in mind.

Introduction

First of all, we had picked the ParkCalc from the Gerald R. Ford International Airport as product to be tested. Adam Goucher was kind enough to provide a copy where we could run our test against without bringing down some production system of an airport. That said, the parking rates on the official airport page served as an oracle for the correct prices.

Since RobotFramework provides great abilities to test this application when used together with the Selenium Library for Robot Framework, we asked the participants to install the tools before joining our session. Additionally we were happy to have Pekka Klärck, the main developer of the Robot Framework project, with us.

I won’t get into details on how to install Robot Framework or the Selenium Library. There is a great introduction for Robot Framework on their wiki, as well as a demo for the selenium library which should get you started. In case you run into problems, please drop the authors a note. They will be happy to help you out on this one.

The examples we provided to the participants can be downloaded here. There are three examples included in the .zip file, called calc1, calc2, and calc3. The examples are based upon the selenium library demo mentioned above.

The first example

So, I needed to get a first test started. We explored the product a few weeks earlier, and focused mainly on problems related to date time calculation. The mission now was to get automation in place, since all of these bugs were going to be removed from the code. Therefore a good regression test suite should serve the purpose to ensure the underlying business rules will still be met with the new code. A perfect part for test automation.

After fiddling a bit with the id’s in the form and getting a first understanding how robot framework works, I ended up with a straight-forward test that had everything inlined. Here is the essence of the test:

Basic Test
    Open Browser  http://adam.goucher.ca/parkcalc/  firefox
    Set Selenium Speed  0
    Title Should Be  Parking Calculator
    Select From List  Lot  Economy Parking
    Input Text  EntryTime  01:23
    Select Radio Button  EntryTimeAMPM  AM
    Input Text  EntryDate  02/28/2000
    Input Text  ExitTime  12:34
    Select Radio Button  ExitTimeAMPM  AM
    Input Text  ExitDate  03/01/2000
    Click Button  Submit
    Page Should Contain   (1 Days, 23 Hours, 11 Minutes)
    [Teardown]  Close Browser

Looks like a good test, doesn’t it? It expresses that parking in the economy lot from 02/28/2000 01:23am to 03/01/2000 12:34am calculates to a duration of 1 day, 23 hours, 11 minutes. Wonderful.

Of course, there are several problems with this test. It uses no variables, it does not abstract from the implementation of the test script to the business rules, and it has many values hard-code which make the test messy to read. At work, I would consider this one to be a spike solution and throw everything away, not even submitting it to the source code repository. To be honest, the first test serves the purpose to learn about the tool and get enough knowledge about the tool domain to get the real tests started.

A data-driven test

Now consider the parkCalc3 test. Here is the test:

Basic Test
    Park Calc  Valet Parking  05/04/2010  12:00  AM  05/05/2010  12:00  AM  $ 42.00

This expresses that for Valet Parking from 05/04/2010 12:00am until 05/05/2010 12:00am a charge of $42.00 will be calculated. Regardless whether or not this is correct, the application prints this one out.

Now, does this test look any better compared to the former one? First of all there is a single keyword now driving the application from the tests. Taking a look into the resource.txt file in the same directory as the test itself, shows up the following implementation:

Park Calc  [Arguments]  ${arg1}  ${arg2}  ${arg3}  ${arg4}  ${arg5}  ${arg6}  ${arg7}  ${arg8}
    Open Browser  http://adam.goucher.ca/parkcalc/  firefox
    Set Selenium Speed  0
    Title Should Be  Parking Calculator
    Select From List  Lot  ${arg1}
    Input Text  EntryTime  ${arg3}
    Select Radio Button  EntryTimeAMPM  ${arg4}
    Input Text  EntryDate  ${arg2}
    Input Text  ExitTime  ${arg6}
    Select Radio Button  ExitTimeAMPM  ${arg7}
    Input Text  ExitDate  ${arg5}
    Click Button  Submit
    Page Should Contain  ${arg8}
    Close Browser

It seems that we simply have applied some extract method to the first test, and ended up with the same script here. The browser as well as the url are hard-coded which may make future adaptations tricky – in this case it’s just the extraction of a variable. The names of the parameters are awkward – and indeed I confused them when I wrote this example down. When a new parameter is introduced, we’ll be pretty much doomed with this test. At least we should then take action and rename those parameters to that we won’t be confused any longer.

Finally a good one?

So, let’s take a look on the second example.

Basic Test
    Open ParkCalc
    Input Parking Lot  Short-Term Parking
    Input Entry Date  12/31/2010  12:00  AM
    Input Leaving Date  01/01/2011  12:00  AM 
    Calculated Cost Should Be  $ 28.00
    [Teardown]  Close Browser

This test checks that for short-term parking from 12/31/2010 12:00am until 01/01/2011 12:00am a cost of $28.00 will be calculated.

Now, is this one any better? In terms of good tests, it isn’t. There are some keywords in use in this test, but the names still reflect the nature of the user interface. If these tests should serve as executable specifications we surely may want to get rid of the inputs and of the closing of the browser. It shouldn’t bother us at this time. Taking a look on the resource file include in the second examples shows us the following implementations of the keywords:

Open ParkCalc
    Open Browser  ${PARKCALC URL}  ${BROWSER}
    Set Selenium Speed  ${DELAY}
    Title Should Be  Parking Calculator

Input Parking Lot  [Arguments]  ${lot}
    Select From List  Lot  ${lot}

Input Entry Date  [Arguments]  ${date}  ${time}  ${ampm}
    Input Text  EntryDate  ${date}
    Input Text  EntryTime  ${time}
    Select Radio Button  EntryTimeAMPM  ${ampm}

Input Leaving Date  [Arguments]  ${date}  ${time}  ${ampm}
    Input Text  ExitDate  ${date}
    Input Text  ExitTime  ${time}
    Select Radio Button  ExitTimeAMPM  ${ampm}

Calculated Cost Should Be  [Arguments]  ${cost}
    Click Button  Submit
    ${actual} =  Get Text  xpath=//tr[td/div[@class='SubHead'] = 'COST']/td/span/font/b
    Log  Actual costs: ${actual}
    Page Should Contain  ${cost}

So, there seems to be some structure to it. This looks a bit better, but still not optimal to serve as executable specifications.

Which one is better?

Now, as all the tests indicate a simple basic example, and we want to dig deeper on the business rules, we now should decide which one to use in order to start off our next few tests. As mentioned before, the first example is a spike solution from my perspective. It simply serves the purpose to learn something about selenium keywords, and how to get started with the new tool domain. It’s surely the best option to forget completely about that.

The third example is a data-driven test. It’s current implementation of the keyword used is a bit sloppy, but it can be used to express the business rules in a simple table format. So, it might be worth taking a closer look into it.

The second example is a keyword-driven test. It’s current implementation is sloppy, too, but it can be improved. Additionally by combining new keywords this test may also be used for bad path tests in the future maybe. So, it might be worth taking a closer look in it, too.

From my perspective it’s a matter of personal taste whether to use the data-driven or the keyword-driven example in this case. I preferred the keyword-driven test, but I can also see the benefits of a data-driven test in this example. Therefore I will spend in a follow-up post some time to automate some tests in a data-driven format, and in another follow-up post I will show how I work through a keyword-driven approach. Meanwhile you may want to try yourself on improving the tests as shown – maybe with some of the insights you get from Gojko’s blog entry mentioned earlier. Stay tuned.