Running Selenium Webdriver tests in Jenkins using Ant

In a previous post I introduced a very simple and straightforward way to run data-driven tests in Selenium Webdriver using test data stored in an Excel sheet. In this post, I want to show how to run these tests using a continuous integration (CI-) solution.

My preferred CI-tool is Jenkins, as it is open source, very flexible and easy to use.
First, make sure that Jenkins is set up properly and is running as a service. Installation is very easy, so I won’t go into details here.

I also recommend using Ant as a software build tool to further ease the process of compiling and running our tests. While it is not strictly necessary to use Ant, it will make life a lot easier for us. Again, install Ant and make sure it is running smoothly by typing ant on the command prompt. If it starts asking for a build.xml file, it’s running properly.

ant_running

Next, open the Selenium Webdriver project in your IDE. Again, I prefer using Eclipse, so the images shown here will be based on Eclipse. Other IDEs such as IntelliJ usually provide the same functionality, it’s just hidden behind different menu options.

Before we start configuring our project for using Ant and running in Jenkins, we are going to add some flexibility to it. Jenkins uses its own workspace and might be running on another server altogether. However, our Selenium project contains a reference to a locally stored Excel data source. Therefore, we are going to add the possibility to provide the path to the data source to be used as an argument to our main method in the Selenium test. In this way, we can simply specify the location of the data source when we run the test. Not only is this necessary to have our test run smoothly in Jenkins, it also adds the possibility to execute several test runs with separate test data sets.

public static void main (String args[]) {
		
		String strPath;
		
		if (args.length == 1) {
			strPath = args[0];
		} else {
			strPath = "Z:\\Documents\\Bas\\blog\\datasources\\testdata.xls";
		}
		
		try {
			// Open the Excel file
			FileInputStream fis = new FileInputStream(new File(strPath).getAbsolutePath());

If an argument is specified when running the main method of our test, we assume this is a relative path to our Excel data source. In order to be able to use it, all we have to do is to get the absolute path for it and off we go. If no argument is specified, we use the default Excel file.

Next, we create a build.xml file that provides Ant with the necessary instructions and details on how to build and run our test. In Eclipse, this can be done easily by right-clicking our project and selecting ‘Export > General > Ant Buildfiles’. After selecting the appropriate project, a build.xml file is generated and added to the root of our project. The example below is a part of the resulting file:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- WARNING: Eclipse auto-generated file.
              Any modifications will be overwritten.
              To include a user specific buildfile here, simply create one in the same
              directory with the processing instruction <?eclipse.ant.import?>
              as the first entry and export the buildfile again. --><project basedir="." default="build" name="seleniumTest">
<property environment="env"/>
<property name="ECLIPSE_HOME" value="C:/Tools/eclipse"/>
<property name="debuglevel" value="source,lines,vars"/>
<property name="target" value="1.7"/>
<property name="source" value="1.7"/>
<path id="seleniumTest.classpath">
    <pathelement location="bin"/>
    <pathelement location="../../../../../vmware-host/Shared Folders/Documents/Bas/blog/libs/selenium-server-standalone-2.37.0.jar"/>
    <pathelement location="../../../../../vmware-host/Shared Folders/Documents/Bas/blog/libs/poi-3.9-20121203.jar"/>
</path>
<target name="init">
    <mkdir dir="bin"/>
    <copy includeemptydirs="false" todir="bin">
        <fileset dir="src">
            <exclude name="**/*.java"/>
        </fileset>
    </copy>
</target>
<target name="clean">
    <delete dir="bin"/>
</target>

Before we can run our tests using Ant, we need to make two modifications.

First, we need to clean up the classpath as we want to use clear and relative paths. Make sure that the necessary libraries can be found on the designated locations. These are specified relative to the location of the build.xml file.

<path id="seleniumTest.classpath">
    <pathelement location="bin"/>
    <pathelement location="libs/selenium-server-standalone-2.37.0.jar"/>
    <pathelement location="libs/poi-3.9-20121203.jar"/>
</path>

Next, we need to make sure that our Excel data source is specified as an argument in the designated Ant target:

<target name="ExcelDataDriven">
    <java classname="com.ontestautomation.selenium.ExcelDataDriven" failonerror="true" fork="yes">
        <arg line="datasources/testdata.xls"/>
        <classpath refid="seleniumTest.classpath"/>
    </java>
</target>

Now, we can execute our test using Ant either in Eclipse or at the command line. When you choose the latter, go to the subdirectory for the Selenium project in your workspace (build.xml should be located there) and execute ant <> (ant ExcelDataDriven in this case). You’ll see that the test is run successfully using Ant.

ant_run_test

The final step is to have this step performed by Jenkins. This should be very straightforward now. Create a new job in Jenkins and add ‘Invoke Ant’ as a build step. Specify the correct target (again, ExcelDataDriven in our case).

jenkins_ant_target

Make sure that all referenced libraries and data sources can be found at the correct locations in the workspace for the Jenkins job (this is where using relative paths comes in handy!). Normally, you would do this using some sort of version control sytem such as Subversion. Next, schedule a build for the job, which should run smoothly now:

jenkins_test_output

That’s it, we’ve now successfully run our Selenium Webdriver tests in Jenkins using Ant. One step closer to a successful continuous integration approach!

Best practice: focus on repeatability of your automated tests

This is the first installment in a series of posts on test automation best practices. Notwithstanding the rapid growth and evolution of the test automation field, a number of best practices can be identified that stand the test of time. Adhering to these best practices will improve the added value of your automated tests, no matter the scale or scope of the test, or the technology or tools that are used to design, implement or execute automated tests.

In order for your automated test suites to be truly efficient, they should be set up with repeatability in mind. This means that your tests should execute with the click of a button, or by entering a command in the command prompt, again and again, without the need for manual intervention during or in between test runs. They should also yield the same (or comparable) test results every single time. Except when the system under test changes or fails, of course.

To achieve or improve repeatability, you need to pay attention to a number of things during the implementation of your automated tests. I will address some of these in this article. There are probably lots of other aspects to be considered, but these stand out for me.

One disclaimer: the repeatability factor does not apply (or applies to a far lesser extent) to projects where there’s just a single test run to be executed, for instance after a conversion or a migration project. If you’re involved in such a project, it’s probably not worth it to put extra effort in achieving repeatability of your automated tests.

Start small
As with every software development project, it is best to start small when implementing automated tests. Automate one or two test cases, or even just one or two steps of the process to be automated, and execute them over and over again to make sure they are stable and repeatable. Once your small test cases are proven to be repeatable you can build on them to create larger test suites. Make sure you prove that every major change you make to your tests does not compromise the repeatability of your tests.

Watch your test data
An important issue when designing and implementing repeatable automated tests is the use of test data. Scripts that alter or consume test data need some extra attention with regards to repeatability. A test data object, such as a customer, an order, etc., used in a certain test run may be altered or removed during that run, rendering it unsuitable or unavailable for subsequent test runs.

Roughly speaking, there are three possible approaches for dealing with test data that is altered during a test run:

  • Create the test data during the test run. For example, if your test script covers the processing of an order, have your script create a new order before processing it to make sure there’s always an order to be processed
  • Reset the test data to its original state after the test run. For example, if your test script covers changing a customer’s address to a foreign location, reset it to its original value after the test script has been executed (through whichever interface available).
  • Select the test data to be used at the start of your test run. Rather than using previously defined sets of test data, have your script perform a query on the available test data set to select a test data object to use in a particular test run. Make sure that your script can handle occasions where there’s no suitable test data object available.

Be ready for continuous integration
With the current trend of test and development teams working closer together in increasingly shorter development and test cycles (think Agile / Scrum and DevOps), continuous integration (CI) is applied not only for development tasks, but for system and integration testing as well. In order to be able to keep up with the development team, automated tests should seamlessly integrate with the continuous integration platform in use. Most open source and COTS automated test tools provide a command line interface to execute test runs and export and distribute test result reports. This doesn’t make your test scripts automagically suited for use in a CI environment. Only test scripts or frameworks that can be run again and again without the need for manual intervention can be successfully integrated in the CI process, so make sure yours fit the bill!

A schematic representation of continuous integration

A schematic representation of continuous integration

Effects of repeatability on the acceptance and the ROI of test automation
Once you have managed to control the repeatability of your automated test scripts, you should see some pretty positive results with regards to the acceptance of automated testing and the ROI associated with the test automation project:

  • Repeatable tests can be run on demand, as often as required, leading to a dramatic reduction of the cost per test run and the time needed to complete a development/test cycle
  • Automated tests that can run unattended and that can be repeated on demand will appeal to everybody from developers to upper management, increasing its perceived value and ultimately also increasing the trust in the product delivered by your team.

Are your tests as repeatable as they can be? Let me know how you achieved your degree of repeatability and the issues you had to overcome!

Generating and deploying web service stubs using WSO2

In case you need a relatively simple stub that simulates a third-party web service or application during your testing process, you have several options at your disposal. In this article, I will introduce the WSO2 Enterprise Service Bus, which is one of these options, and show you how you can use it to quickly simulate an existing web service. This is particularly useful when the actual web service or application is not (always) available during testing, or when you want to simulate particular behavior in your testing process.

The WSO2 Enterprise Service Bus
The WSO2 Enterprise Service Bus is a lightweight, open source ESB implementation that has gained significant popularity in recent years. It is used by companies such as Ebay, British Airways and Volvo. Downloading and installation is simple: go to the product website and download a zip file from there. Unpack it and run wso2server.bat (Windows) or wso2server.sh (Linux) from the /bin directory. Note that you need a Java JDK to be installed for WSO2 ESB to run on your system.

Developing WSO2 stubs in Eclipse
As we are going to generate our own web service stub and deploy it using WSO2, it is a good idea to do all the work from within our IDE. I prefer to use Eclipse for this. Stop WSO2 using Ctrl+C as we are going to restart it from within Eclipse again later on. Start Eclipse and install the WSO2 Developer Studio using the Eclipse Marketplace.

stub_install_wso2_developer_studio

Generating a web service skeleton
For this example, I used a sample web service providing weather forecasts for cities in the US. Its WSDL is available here. In order to be able to generate a stub for this web service using WSO2, you should download it to your local machine.

In Eclipse, choose File > New > Project … > WSO2 > Service Hosting > Project Types > Axis2 Service Project. Next, select ‘Create New Axis2 Service Using WSDL File’ (this is why we need a copy of the WSDL on our system). Select the WSDL file and give your project a name:

stub_generate_skeleton_from_wsdl

Click finish to generate a web service skeleton for the service described in the WSDL.

Deploying the web service
To deploy our web service and interact with it, we need to carry out two steps. First, create a Carbon Application Project in Eclipse. This project bundles our web service project so it can be deployed. Select File > New … > Carbon Application Project, give the project a name, select our generated service as a dependency and click Finish.

stub_create_carbon_project

Next, we need to create a server in Eclipse on which to deploy the web service. Add a new server and choose WSO2 > WSO2 Carbon 4.0 based server as the server type. Click Next and select the installation directory of WSO2 as your CARBON_HOME folder. Click Next twice and add the Carbon Application project for our web service to the web server in the ‘Add and Remove’ window (this can be done later as well by right-clicking on the server and selecting ‘Add and Remove’). Finally, start the server.

stub_service_deployed

Testing the web service
Now that our web service skeleton has been deployed, let’s see whether we can communicate with it. The WSDL for the local web service implementation can be found using the WSO2 Management Console (user: admin, pass: admin) that is automatically opened when you start the server in Eclipse. You can then use a tool such as SmartBear SoapUI or Parasoft SOAtest to test the web service skeleton.

By default, none of the operations in the web service will be implemented yet, and sending a request message will result in a response containing a SOAP Fault:

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
 <soapenv:Body>
  <soapenv:Fault>
   <faultcode>soapenv:Server</faultcode>
   <faultstring>Please implement com.cdyne.ws.weatherws.WeatherSkeleton#getCityForecastByZIP</faultstring>
   <detail/>
  </soapenv:Fault>
 </soapenv:Body>
</soapenv:Envelope>

The good news is that we have successfully created a web service skeleton from the WSDL web service description, that we deployed it on a local web server for testing purposes and that we are able to communicate with the web service, without having to write a single line of code.

Implementing the web service operations
In order for our web service simulation to return more intelligent responses, we will have to implement the methods that construct the actual responses. When you generate a web service skeleton using the method and example WSDL above, these methods will all be located in the WeatherSkeleton.java file. After generating the skeleton, the only action performed by each method is to throw the exception message we have seen when we first communicated with the web service:

public com.cdyne.ws.weatherws.GetCityForecastByZIPResponse getCityForecastByZIP (com.cdyne.ws.weatherws.GetCityForecastByZIP getCityForecastByZIP) {
            //TODO : fill this with the necessary business logic
            throw new  java.lang.UnsupportedOperationException("Please implement " + this.getClass().getName() + "#getCityForecastByZIP");
}

A simple implementation of this method, generating the same response no matter what is in the request message,is shown below:

public GetCityForecastByZIPResponse getCityForecastByZIP(GetCityForecastByZIP getCityForecastByZIP) {
	 
	 // generate new response for the web service
	 try {
		 GetCityForecastByZIPResponse cityForecastResponse = GetCityForecastByZIPResponse.class.newInstance();
		 
		 // create a new response object and the required objects to fill it
		 ForecastReturn fr = new ForecastReturn();
		 ArrayOfForecast aof = new ArrayOfForecast();
		 Forecast forecast = new Forecast();
		 Calendar cal = Calendar.getInstance();
		 Temp temp = new Temp();
		 POP pop = new POP();
		 
		 // set field values
		 fr.setCity("Boulder");
		 fr.setState("Colorado");
		 fr.setWeatherStationCity("Boulder");
		 fr.setResponseText("Forecast for Boulder, generated on " + new SimpleDateFormat("MM/dd/yyyy HH:mm:ss").format(cal.getTime()));
		 fr.setSuccess(true);
		 
		 // create weather forecast array and forecast entry
		 forecast.setDate(cal);
		 forecast.setDesciption("Sample forecast");
		 temp.setDaytimeHigh("90");
		 temp.setMorningLow("40");
		 forecast.setTemperatures(temp);
		 pop.setDaytime("12:00:00");
		 pop.setNighttime("00:00:00");
		 forecast.setProbabilityOfPrecipiation(pop);
		 
		 // add entry to forecast array
		 aof.addForecast(forecast);
		 
		 // add forecast to response and return it
		 fr.setForecastResult(aof);
		 cityForecastResponse.setGetCityForecastByZIPResult(fr);
		 return cityForecastResponse;
		 
	 } catch (Exception e) {
		 throw new UnsupportedOperationException("Unexpected error occurred during message processing");
	 }
 }

When we redeploy our web service (by right-clicking on the server in Eclipse and selecting ‘Redeploy’) and call the GetCityForecastByZIP operation again, we now get a sensible response from our web service:

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope
   xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
 <soapenv:Body>
  <ns1:GetCityForecastByZIPResponse xmlns:ns1="http://ws.cdyne.com/WeatherWS/">
   <ns1:GetCityForecastByZIPResult>
    <ns1:Success>true</ns1:Success>
    <ns1:ResponseText>Forecast for Boulder, generated on 11/07/2013 09:02:54</ns1:ResponseText>
    <ns1:State>Colorado</ns1:State>
    <ns1:City>Boulder</ns1:City>
    <ns1:WeatherStationCity>Boulder</ns1:WeatherStationCity>
    <ns1:ForecastResult>
     <ns1:Forecast>
      <ns1:Date>2013-11-07T09:02:54.274+01:00</ns1:Date>
      <ns1:WeatherID>0</ns1:WeatherID>
      <ns1:Desciption>Sample forecast</ns1:Desciption>
      <ns1:Temperatures>
       <ns1:MorningLow>40</ns1:MorningLow>
       <ns1:DaytimeHigh>90</ns1:DaytimeHigh>
      </ns1:Temperatures>
      <ns1:ProbabilityOfPrecipiation>
       <ns1:Nighttime>00:00:00</ns1:Nighttime>
       <ns1:Daytime>12:00:00</ns1:Daytime>
      </ns1:ProbabilityOfPrecipiation>
     </ns1:Forecast>
    </ns1:ForecastResult>
   </ns1:GetCityForecastByZIPResult>
  </ns1:GetCityForecastByZIPResponse>
 </soapenv:Body>
</soapenv:Envelope>

In future posts, I will introduce ways to improve the functionality and flexibility of our simulated web service, and make it even more useful when used in our testing processes.