Running Selenium JUnit tests from Jenkins

In this post I want to show you how to use Jenkins to automatically execute Selenium tests written in JUnit format, and how the results from these tests can be directly reported back to Jenkins. To achieve this, we need to complete the following steps:

  • Write some Selenium tests in JUnit format that we want to execute
  • Create a build file that runs these tests and writes the reports to disk
  • Set up a Jenkins job that runs these tests and interprets the results

Note: First of all a point of attention: I couldn’t get this to work while Jenkins was installed as a Windows service. This has something to do with Jenkins opening browser windows and subsequently not having suitable permissions to access sites and handle Selenium calls. I solved this by starting Jenkins ‘by hand’ by downloading the .war file from the Jenkins site and running it using java -jar jenkins.war

Creating Selenium tests to run
First, we need to have some tests that we would like to run. I’ve created three short tests in JUnit-format, where one has an intentional error for demonstration purposes – it’s good practice to see if any test defects actually show up in Jenkins! Using the JUnit-format implies that tests can be run independently, so there can’t be any dependencies between tests. My test class looks like this (I’ve removed two tests and all import statements for brevity):

package com.ontestautomation.selenium.ci;

public class SeleniumCITest {
	
	static WebDriver driver;
	
	@Before
	public void setup() {
		
		driver = new FirefoxDriver();
		driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);				
	}
	
	@Test
	public void successfulLoginLogout() {
		
		driver.get("http://parabank.parasoft.com");
		Assert.assertEquals(driver.getTitle(), "ParaBank | Welcome | Online Banking");
		driver.findElement(By.name("username")).sendKeys("john");
		driver.findElement(By.name("password")).sendKeys("demo");
		driver.findElement(By.cssSelector("input[value='Log In']")).click();
		Assert.assertEquals(driver.getTitle(), "ParaBank | Accounts Overview");
		driver.findElement(By.linkText("Log Out")).click();
		Assert.assertEquals(driver.getTitle(), "ParaBank | Welcome | Online Banking");
	}
	
	@After
	public void teardown() {
		driver.quit();
	}	
}

Pretty straightforward, but good enough.

Creating a build file to run tests automatically
Now to create a build file to run our tests automatically. I used Ant for this, but Maven should work as well. I had Eclipse generate an Ant build-file for me, then changed it to allow Jenkins to run the tests as well. In my case, I only needed to change the location of the imports (the Selenium and the JUnit .jar files) to a location where Jenkins could find them:

<path id="seleniumCI.classpath">
    <pathelement location="bin"/>
    <pathelement location="C:/libs/selenium-server-standalone-2.44.0.jar"/>
    <pathelement location="C:/libs/junit-4.11.jar"/>
</path>

Note that I ran my tests on my own system, so in this case it’s OK to use absolute paths to the .jar files, but it’s by no means good practice to do so! It’s better to use paths relative to your Jenkins workspace, so tests and projects are transferable and can be run on any system without having to change the build.xml.

Actual test execution is a matter of using the junit and junitreport tasks:

<target name="SeleniumCITest">
    <mkdir dir="${junit.output.dir}"/>
    <junit fork="yes" printsummary="withOutAndErr">
        <formatter type="xml"/>
        <test name="com.ontestautomation.selenium.ci.SeleniumCITest" todir="${junit.output.dir}"/>
        <classpath refid="seleniumCI.classpath"/>
        <bootclasspath>
            <path refid="run.SeleniumCITest (1).bootclasspath"/>
        </bootclasspath>
    </junit>
</target>
<target name="junitreport">
    <junitreport todir="${junit.output.dir}">
        <fileset dir="${junit.output.dir}">
            <include name="TEST-*.xml"/>
        </fileset>
        <report format="frames" todir="${junit.output.dir}"/>
    </junitreport>
</target>

This is generated automatically when you create your build.xml using Eclipse, by the way.

Running your tests through Jenkins
The final step is setting up a Jenkins job that simply calls the correct Ant target in a build step:
Ant build step
After tests have been run, Jenkins should pick up the JUnit test results from the folder specified in the junitreport task in the Ant build.xml:
JUnit report post build action
If everything is set up correctly, you should now be able to run your tests through Jenkins and have the results displayed:
Build result in Jenkins
You can also view details on the individual test results by clicking on the error message:
Error details in Jenkins

The Eclipse project I have used for this example can be downloaded here.

Running Selenium Webdriver tests in Jenkins using Ant

In a previous post I introduced a very simple and straightforward way to run data-driven tests in Selenium Webdriver using test data stored in an Excel sheet. In this post, I want to show how to run these tests using a continuous integration (CI-) solution.

My preferred CI-tool is Jenkins, as it is open source, very flexible and easy to use.
First, make sure that Jenkins is set up properly and is running as a service. Installation is very easy, so I won’t go into details here.

I also recommend using Ant as a software build tool to further ease the process of compiling and running our tests. While it is not strictly necessary to use Ant, it will make life a lot easier for us. Again, install Ant and make sure it is running smoothly by typing ant on the command prompt. If it starts asking for a build.xml file, it’s running properly.

ant_running

Next, open the Selenium Webdriver project in your IDE. Again, I prefer using Eclipse, so the images shown here will be based on Eclipse. Other IDEs such as IntelliJ usually provide the same functionality, it’s just hidden behind different menu options.

Before we start configuring our project for using Ant and running in Jenkins, we are going to add some flexibility to it. Jenkins uses its own workspace and might be running on another server altogether. However, our Selenium project contains a reference to a locally stored Excel data source. Therefore, we are going to add the possibility to provide the path to the data source to be used as an argument to our main method in the Selenium test. In this way, we can simply specify the location of the data source when we run the test. Not only is this necessary to have our test run smoothly in Jenkins, it also adds the possibility to execute several test runs with separate test data sets.

public static void main (String args[]) {
		
		String strPath;
		
		if (args.length == 1) {
			strPath = args[0];
		} else {
			strPath = "Z:\\Documents\\Bas\\blog\\datasources\\testdata.xls";
		}
		
		try {
			// Open the Excel file
			FileInputStream fis = new FileInputStream(new File(strPath).getAbsolutePath());

If an argument is specified when running the main method of our test, we assume this is a relative path to our Excel data source. In order to be able to use it, all we have to do is to get the absolute path for it and off we go. If no argument is specified, we use the default Excel file.

Next, we create a build.xml file that provides Ant with the necessary instructions and details on how to build and run our test. In Eclipse, this can be done easily by right-clicking our project and selecting ‘Export > General > Ant Buildfiles’. After selecting the appropriate project, a build.xml file is generated and added to the root of our project. The example below is a part of the resulting file:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- WARNING: Eclipse auto-generated file.
              Any modifications will be overwritten.
              To include a user specific buildfile here, simply create one in the same
              directory with the processing instruction <?eclipse.ant.import?>
              as the first entry and export the buildfile again. --><project basedir="." default="build" name="seleniumTest">
<property environment="env"/>
<property name="ECLIPSE_HOME" value="C:/Tools/eclipse"/>
<property name="debuglevel" value="source,lines,vars"/>
<property name="target" value="1.7"/>
<property name="source" value="1.7"/>
<path id="seleniumTest.classpath">
    <pathelement location="bin"/>
    <pathelement location="../../../../../vmware-host/Shared Folders/Documents/Bas/blog/libs/selenium-server-standalone-2.37.0.jar"/>
    <pathelement location="../../../../../vmware-host/Shared Folders/Documents/Bas/blog/libs/poi-3.9-20121203.jar"/>
</path>
<target name="init">
    <mkdir dir="bin"/>
    <copy includeemptydirs="false" todir="bin">
        <fileset dir="src">
            <exclude name="**/*.java"/>
        </fileset>
    </copy>
</target>
<target name="clean">
    <delete dir="bin"/>
</target>

Before we can run our tests using Ant, we need to make two modifications.

First, we need to clean up the classpath as we want to use clear and relative paths. Make sure that the necessary libraries can be found on the designated locations. These are specified relative to the location of the build.xml file.

<path id="seleniumTest.classpath">
    <pathelement location="bin"/>
    <pathelement location="libs/selenium-server-standalone-2.37.0.jar"/>
    <pathelement location="libs/poi-3.9-20121203.jar"/>
</path>

Next, we need to make sure that our Excel data source is specified as an argument in the designated Ant target:

<target name="ExcelDataDriven">
    <java classname="com.ontestautomation.selenium.ExcelDataDriven" failonerror="true" fork="yes">
        <arg line="datasources/testdata.xls"/>
        <classpath refid="seleniumTest.classpath"/>
    </java>
</target>

Now, we can execute our test using Ant either in Eclipse or at the command line. When you choose the latter, go to the subdirectory for the Selenium project in your workspace (build.xml should be located there) and execute ant <> (ant ExcelDataDriven in this case). You’ll see that the test is run successfully using Ant.

ant_run_test

The final step is to have this step performed by Jenkins. This should be very straightforward now. Create a new job in Jenkins and add ‘Invoke Ant’ as a build step. Specify the correct target (again, ExcelDataDriven in our case).

jenkins_ant_target

Make sure that all referenced libraries and data sources can be found at the correct locations in the workspace for the Jenkins job (this is where using relative paths comes in handy!). Normally, you would do this using some sort of version control sytem such as Subversion. Next, schedule a build for the job, which should run smoothly now:

jenkins_test_output

That’s it, we’ve now successfully run our Selenium Webdriver tests in Jenkins using Ant. One step closer to a successful continuous integration approach!