Best practice: focus on repeatability of your automated tests

This is the first installment in a series of posts on test automation best practices. Notwithstanding the rapid growth and evolution of the test automation field, a number of best practices can be identified that stand the test of time. Adhering to these best practices will improve the added value of your automated tests, no matter the scale or scope of the test, or the technology or tools that are used to design, implement or execute automated tests.

In order for your automated test suites to be truly efficient, they should be set up with repeatability in mind. This means that your tests should execute with the click of a button, or by entering a command in the command prompt, again and again, without the need for manual intervention during or in between test runs. They should also yield the same (or comparable) test results every single time. Except when the system under test changes or fails, of course.

To achieve or improve repeatability, you need to pay attention to a number of things during the implementation of your automated tests. I will address some of these in this article. There are probably lots of other aspects to be considered, but these stand out for me.

One disclaimer: the repeatability factor does not apply (or applies to a far lesser extent) to projects where there’s just a single test run to be executed, for instance after a conversion or a migration project. If you’re involved in such a project, it’s probably not worth it to put extra effort in achieving repeatability of your automated tests.

Start small
As with every software development project, it is best to start small when implementing automated tests. Automate one or two test cases, or even just one or two steps of the process to be automated, and execute them over and over again to make sure they are stable and repeatable. Once your small test cases are proven to be repeatable you can build on them to create larger test suites. Make sure you prove that every major change you make to your tests does not compromise the repeatability of your tests.

Watch your test data
An important issue when designing and implementing repeatable automated tests is the use of test data. Scripts that alter or consume test data need some extra attention with regards to repeatability. A test data object, such as a customer, an order, etc., used in a certain test run may be altered or removed during that run, rendering it unsuitable or unavailable for subsequent test runs.

Roughly speaking, there are three possible approaches for dealing with test data that is altered during a test run:

  • Create the test data during the test run. For example, if your test script covers the processing of an order, have your script create a new order before processing it to make sure there’s always an order to be processed
  • Reset the test data to its original state after the test run. For example, if your test script covers changing a customer’s address to a foreign location, reset it to its original value after the test script has been executed (through whichever interface available).
  • Select the test data to be used at the start of your test run. Rather than using previously defined sets of test data, have your script perform a query on the available test data set to select a test data object to use in a particular test run. Make sure that your script can handle occasions where there’s no suitable test data object available.

Be ready for continuous integration
With the current trend of test and development teams working closer together in increasingly shorter development and test cycles (think Agile / Scrum and DevOps), continuous integration (CI) is applied not only for development tasks, but for system and integration testing as well. In order to be able to keep up with the development team, automated tests should seamlessly integrate with the continuous integration platform in use. Most open source and COTS automated test tools provide a command line interface to execute test runs and export and distribute test result reports. This doesn’t make your test scripts automagically suited for use in a CI environment. Only test scripts or frameworks that can be run again and again without the need for manual intervention can be successfully integrated in the CI process, so make sure yours fit the bill!

A schematic representation of continuous integration

A schematic representation of continuous integration

Effects of repeatability on the acceptance and the ROI of test automation
Once you have managed to control the repeatability of your automated test scripts, you should see some pretty positive results with regards to the acceptance of automated testing and the ROI associated with the test automation project:

  • Repeatable tests can be run on demand, as often as required, leading to a dramatic reduction of the cost per test run and the time needed to complete a development/test cycle
  • Automated tests that can run unattended and that can be repeated on demand will appeal to everybody from developers to upper management, increasing its perceived value and ultimately also increasing the trust in the product delivered by your team.

Are your tests as repeatable as they can be? Let me know how you achieved your degree of repeatability and the issues you had to overcome!

Generating and deploying web service stubs using WSO2

In case you need a relatively simple stub that simulates a third-party web service or application during your testing process, you have several options at your disposal. In this article, I will introduce the WSO2 Enterprise Service Bus, which is one of these options, and show you how you can use it to quickly simulate an existing web service. This is particularly useful when the actual web service or application is not (always) available during testing, or when you want to simulate particular behavior in your testing process.

The WSO2 Enterprise Service Bus
The WSO2 Enterprise Service Bus is a lightweight, open source ESB implementation that has gained significant popularity in recent years. It is used by companies such as Ebay, British Airways and Volvo. Downloading and installation is simple: go to the product website and download a zip file from there. Unpack it and run wso2server.bat (Windows) or (Linux) from the /bin directory. Note that you need a Java JDK to be installed for WSO2 ESB to run on your system.

Developing WSO2 stubs in Eclipse
As we are going to generate our own web service stub and deploy it using WSO2, it is a good idea to do all the work from within our IDE. I prefer to use Eclipse for this. Stop WSO2 using Ctrl+C as we are going to restart it from within Eclipse again later on. Start Eclipse and install the WSO2 Developer Studio using the Eclipse Marketplace.


Generating a web service skeleton
For this example, I used a sample web service providing weather forecasts for cities in the US. Its WSDL is available here. In order to be able to generate a stub for this web service using WSO2, you should download it to your local machine.

In Eclipse, choose File > New > Project … > WSO2 > Service Hosting > Project Types > Axis2 Service Project. Next, select ‘Create New Axis2 Service Using WSDL File’ (this is why we need a copy of the WSDL on our system). Select the WSDL file and give your project a name:


Click finish to generate a web service skeleton for the service described in the WSDL.

Deploying the web service
To deploy our web service and interact with it, we need to carry out two steps. First, create a Carbon Application Project in Eclipse. This project bundles our web service project so it can be deployed. Select File > New … > Carbon Application Project, give the project a name, select our generated service as a dependency and click Finish.


Next, we need to create a server in Eclipse on which to deploy the web service. Add a new server and choose WSO2 > WSO2 Carbon 4.0 based server as the server type. Click Next and select the installation directory of WSO2 as your CARBON_HOME folder. Click Next twice and add the Carbon Application project for our web service to the web server in the ‘Add and Remove’ window (this can be done later as well by right-clicking on the server and selecting ‘Add and Remove’). Finally, start the server.


Testing the web service
Now that our web service skeleton has been deployed, let’s see whether we can communicate with it. The WSDL for the local web service implementation can be found using the WSO2 Management Console (user: admin, pass: admin) that is automatically opened when you start the server in Eclipse. You can then use a tool such as SmartBear SoapUI or Parasoft SOAtest to test the web service skeleton.

By default, none of the operations in the web service will be implemented yet, and sending a request message will result in a response containing a SOAP Fault:

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="">
   <faultstring>Please implement</faultstring>

The good news is that we have successfully created a web service skeleton from the WSDL web service description, that we deployed it on a local web server for testing purposes and that we are able to communicate with the web service, without having to write a single line of code.

Implementing the web service operations
In order for our web service simulation to return more intelligent responses, we will have to implement the methods that construct the actual responses. When you generate a web service skeleton using the method and example WSDL above, these methods will all be located in the file. After generating the skeleton, the only action performed by each method is to throw the exception message we have seen when we first communicated with the web service:

public getCityForecastByZIP ( getCityForecastByZIP) {
            //TODO : fill this with the necessary business logic
            throw new  java.lang.UnsupportedOperationException("Please implement " + this.getClass().getName() + "#getCityForecastByZIP");

A simple implementation of this method, generating the same response no matter what is in the request message,is shown below:

public GetCityForecastByZIPResponse getCityForecastByZIP(GetCityForecastByZIP getCityForecastByZIP) {
	 // generate new response for the web service
	 try {
		 GetCityForecastByZIPResponse cityForecastResponse = GetCityForecastByZIPResponse.class.newInstance();
		 // create a new response object and the required objects to fill it
		 ForecastReturn fr = new ForecastReturn();
		 ArrayOfForecast aof = new ArrayOfForecast();
		 Forecast forecast = new Forecast();
		 Calendar cal = Calendar.getInstance();
		 Temp temp = new Temp();
		 POP pop = new POP();
		 // set field values
		 fr.setResponseText("Forecast for Boulder, generated on " + new SimpleDateFormat("MM/dd/yyyy HH:mm:ss").format(cal.getTime()));
		 // create weather forecast array and forecast entry
		 forecast.setDesciption("Sample forecast");
		 // add entry to forecast array
		 // add forecast to response and return it
		 return cityForecastResponse;
	 } catch (Exception e) {
		 throw new UnsupportedOperationException("Unexpected error occurred during message processing");

When we redeploy our web service (by right-clicking on the server in Eclipse and selecting ‘Redeploy’) and call the GetCityForecastByZIP operation again, we now get a sensible response from our web service:

<?xml version="1.0" encoding="UTF-8"?>
  <ns1:GetCityForecastByZIPResponse xmlns:ns1="">
    <ns1:ResponseText>Forecast for Boulder, generated on 11/07/2013 09:02:54</ns1:ResponseText>
      <ns1:Desciption>Sample forecast</ns1:Desciption>

In future posts, I will introduce ways to improve the functionality and flexibility of our simulated web service, and make it even more useful when used in our testing processes.

A very basic web service test tool

For those of us involved in testing web services and SOA-based applications, there are lots of different test tools on the market that can help speed up the testing and make them repeatable and easy to maintain. Some examples of these tools are SoapUI from SmartBear and SOAtest from Parasoft. SoapUI is available both as a freeware product and as an enterprise edition offering additional functionality. SOAtest is only available in a paid version.

One of the biggest downsides of the freeware version of SoapUI is the lack of data driven test support. This feature is only available in the paid version. In a previous post I introduced a simple way of implementing data driven testing in Selenium Webdriver tests, using Microsoft Excel sheets as the test data container format. As I am very regularly involved in web service testing myself, and often want to use data driven testing to create maintainable and flexible test suites, I thought it would be a good idea to see whether it’s possible to quickly create a very basic web service testing tool that supports data driven testing.

The test data source
As in the previous example, let’s create a test data source first. I chose to base the solution presented here on predefined XML request message files as this saves us the trouble of creating XML objects in our code. There are obvious downsides to this, of course, but for now we will focus on the ability to call a web service and subsequently capture and validate the result.

Our test data source looks like this:


It contains columns to identify the current test case, the path to the XML file containing the SOAP message to be sent and the endpoint to which the message should be sent. The last two columns contain the name of an element in the web service response we would like to validate and its expected value, respectively.

Reading our test data source
In our very basic web service test tool, we will process this Excel sheet just as we did in the Selenium example posted earlier. For every test data row, we will then execute the following steps:

  • Create a SOAP request from the XML file
  • Send the SOAP request to the right web service endpoint and capture the response
  • Extract the value from a response message element and compare it to the expected value

Create a SOAP request from the XML file
This is an easy step, as all we need to do is open the file, read all of its contents and transform it into a SOAP message object:

private static SOAPMessage createSOAPRequest(String strPath) throws Exception {
    // Create a SOAP message from the XML file located at the given path
    FileInputStream fis = new FileInputStream(new File(strPath));
    MessageFactory factory = MessageFactory.newInstance();
    SOAPMessage message = factory.createMessage(new MimeHeaders(), fis);
    return message;

Send the SOAP request to the right web service and capture the response
This is fairly straightforward as well and can be done using standard Java methods:

private static SOAPMessage getSOAPResponse(SOAPMessage soapRequest, String strEndpoint) throws Exception, SOAPException {
    // Send the SOAP request to the given endpoint and return the corresponding response
    SOAPConnectionFactory soapConnectionFactory = SOAPConnectionFactory.newInstance();
    SOAPConnection soapConnection = soapConnectionFactory.createConnection();
    SOAPMessage soapResponse =, strEndpoint);
    return soapResponse;	

Validating elements from the response message
Finally, we will parse the response message and validate the value of one of its elements. The validation result is sent to the stdout:

private static void validateValue(SOAPMessage soapMsg, String strEl, String strExpected) throws Exception {
    // Get all elements with the requested element tag from the SOAP message
    SOAPBody soapBody = soapMsg.getSOAPBody();
    NodeList elements = soapBody.getElementsByTagName(strEl);
    // Check whether there is exactly one element with the given tag
    if (elements.getLength() != 1){
        System.out.println("Expected exactly one element " + strEl + "in message, but found " + Integer.toString(elements.getLength()));
    } else {
        // Validate the element value against the expected value
        String strActual = elements.item(0).getTextContent();
        if (strActual.equals(strExpected)) {
        	System.out.println("Actual value " + strActual + " for element " + strEl + " matches expected value");
        } else {
        	System.out.println("Expected value " + strExpected + " for element " + strEl + ", but found " + strActual + " instead");

Running the test
When we run the test, we see that the web service to be tested is called three times. Two tests succeed, the last test case fails (on purpose, to show that the validation is executed properly):


Extensions to our tool
As stated earlier in this post, there are a lot of improvements to be made to this very basic web service test tool. For instance, we could add:

  • Dynamic request message generation based on a template and data source values
  • XSD validation for the response messages
  • Support for non-SOAP-based (or plain XML) web services or even for transport protocols other than HTTP

Some of these improvements will probably be featured in later articles at For questions or suggestions on topics to be covered on this blog, please do not hesitate to contact me at bas AT or leave a reply through the comment form below.

An example Eclipse project using the pattern described above can be downloaded here.