Creating HTML reports for your Selenium tests using ExtentReports

In a comment on a previous post I wrote on creating custom HTML reports for Selenium tests, Anshoo Arora of Relevant Codes pointed me to ExtentReports, his own solution for generating HTML reports. This week, I have been playing around with it for a while and I must say I’m pretty impressed by its ease of use and the way the reports it generates turn out. Let’s have a look.

Note: this post is based on ExtentReports version 1.4 and therefore might not reflect any changes made in more recent versions.

Downloading and installation
Installation of ExtentReports is a breeze. Just download the latest .jar from the website, add it as a dependency to your Java project and you’re all set.

An example test and report
To illustrate the workings of ExtentReports, we’ll use two simple login tests that are executed against the ParaBank demo application on the Parasoft website. Both tests perform a login action, where the first will be successful and the second one will be unsuccessful due to invalid credentials. We’ll have ExtentReports generate a HTML report that contain the appropriate test results.
Continue reading

Using the Page Object Model pattern in Selenium + TestNG tests

After having introduced the Selenium + TestNG combination in my previous post, I would like to show you how to apply the Page Object Model, an often used method for improving maintainability of Selenium tests, to this setup. To do so, we need to accomplish the following steps:

  • Create Page Objects representing pages of a web application that we want to test
  • Create methods for these Page Objects that represent actions we want to perform within the pages that they represent
  • Create tests that perform these actions in the required order and performs checks that make up the test scenario
  • Run the tests as TestNG tests and inspect the results

Creating Page Objects for our test application
For this purpose, again I use the ParaBank demo application that can be found here. I’ve narrowed the scope of my tests down to just three of the pages in this application: the login page, the home page (where you end up after a successful login) and an error page (where you land after a failed login attempt). As an example, this is the code for the login page:

package com.ontestautomation.seleniumtestngpom.pages;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;

public class LoginPage {
	
	private WebDriver driver;
	
	public LoginPage(WebDriver driver) {
		
		this.driver = driver;
		
		if(!driver.getTitle().equals("ParaBank | Welcome | Online Banking")) {
			driver.get("http://parabank.parasoft.com");
		}		
	}
	
	public ErrorPage incorrectLogin(String username, String password) {
		
		driver.findElement(By.name("username")).sendKeys(username);
		driver.findElement(By.name("password")).sendKeys(password);
		driver.findElement(By.xpath("//input[@value='Log In']")).click();
		return new ErrorPage(driver);
	}
	
	public HomePage correctLogin(String username, String password) {
		
		driver.findElement(By.name("username")).sendKeys(username);
		driver.findElement(By.name("password")).sendKeys(password);
		driver.findElement(By.xpath("//input[@value='Log In']")).click();
		return new HomePage(driver);
	}
}

It contains a constructor that returns a new instance of the LoginPage object as well as two methods that we can use in our tests: incorrectLogin, which sends us to the error page and correctLogin, which sends us to the home page. Likewise, I’ve constructed Page Objects for these two pages as well. A link to those implementations can be found at the end of this post.

Note that this code snippet isn’t optimized for maintainability – I used direct references to element properties rather than some sort of element-level abstraction, such as an Object Repository.

Creating methods that perform actions on the Page Objects
You’ve seen these for the login page in the code sample above. I’ve included similar methods for the other two pages. A good example can be seen in the implementation of the error page Page Object:

package com.ontestautomation.seleniumtestngpom.pages;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;

public class ErrorPage {
	
	private WebDriver driver;
	
	public ErrorPage(WebDriver driver) {
		
		this.driver = driver;
	}
	
	public String getErrorText() {
		
		return driver.findElement(By.className("error")).getText();
	}
}

By implementing a getErrorText method to retrieve the error message that is displayed on the error page, we can call this method in our actual test script. It is considered good practice to separate the implementation of your Page Objects from the actual assertions that are performed in your test script (separation of responsibilities). If you need to perform additional checks, just add a method that returns the actual value displayed on the screen to the associated page object and add assertions to the scripts where this check needs to be performed.

Create tests that perform the required actions and execute the required checks
Now that we have created both the page objects and the methods that we want to use for the checks in our test scripts, it’s time to create these test scripts. This is again pretty straightforward, as this example shows (imports removed for brevity):

package com.ontestautomation.seleniumtestngpom.tests;

public class TestNGPOM {
	
	WebDriver driver;
	
	@BeforeSuite
	public void setUp() {
		
		driver = new FirefoxDriver();
		driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
	}
	
	@Parameters({"username","incorrectpassword"})
	@Test(description="Performs an unsuccessful login and checks the resulting error message")
	public void testLoginNOK(String username, String incorrectpassword) {
		
		LoginPage lp = new LoginPage(driver);
		ErrorPage ep = lp.incorrectLogin(username, incorrectpassword);
		Assert.assertEquals(ep.getErrorText(), "The username and password could not be verified.");
	}
	
	@AfterSuite
	public void tearDown() {
		
		driver.quit();
	}
}

Note the use of the page objects and the check being performed using methods in these page object implementations – in this case the getErrorText method in the error page page object.

As we have designed our tests as Selenium + TestNG tests, we also need to define a testng.xml file that defines which tests we need to run and what parameter values the parameterized testLoginOK test takes. Again, see my previous post for more details.

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
 
<suite name="My first TestNG test suite" verbose="1" >
  <parameter name="username" value="john"/>
  <parameter name="password" value="demo"/>
  <test name="Login tests">
    <packages>
      <package name="com.ontestautomation.seleniumtestngpom.tests" />
   </packages>
 </test>
</suite>

Run the tests as TestNG tests and inspect the results
Finally, we can run our tests again by right-clicking on the testng.xml file in the Package Explorer and selecting ‘Run As > TestNG Suite’. After test execution has finished, the test results will appear in the ‘Results of running suite’ tab in Eclipse. Again, please note that using meaningful names for tests and test suites in the testng.xml file make these results much easier to read and interpret.

TestNG test results in Eclipse

An extended HTML report can be found in the test-output subdirectory of your project:

TestNG HTML test results

The Eclipse project I have used for the example described in this post, including a sample HTML report as generated by TestNG, can be downloaded here.

Running Selenium JUnit tests from Jenkins

In this post I want to show you how to use Jenkins to automatically execute Selenium tests written in JUnit format, and how the results from these tests can be directly reported back to Jenkins. To achieve this, we need to complete the following steps:

  • Write some Selenium tests in JUnit format that we want to execute
  • Create a build file that runs these tests and writes the reports to disk
  • Set up a Jenkins job that runs these tests and interprets the results

Note: First of all a point of attention: I couldn’t get this to work while Jenkins was installed as a Windows service. This has something to do with Jenkins opening browser windows and subsequently not having suitable permissions to access sites and handle Selenium calls. I solved this by starting Jenkins ‘by hand’ by downloading the .war file from the Jenkins site and running it using java -jar jenkins.war

Creating Selenium tests to run
First, we need to have some tests that we would like to run. I’ve created three short tests in JUnit-format, where one has an intentional error for demonstration purposes – it’s good practice to see if any test defects actually show up in Jenkins! Using the JUnit-format implies that tests can be run independently, so there can’t be any dependencies between tests. My test class looks like this (I’ve removed two tests and all import statements for brevity):

package com.ontestautomation.selenium.ci;

public class SeleniumCITest {
	
	static WebDriver driver;
	
	@Before
	public void setup() {
		
		driver = new FirefoxDriver();
		driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);				
	}
	
	@Test
	public void successfulLoginLogout() {
		
		driver.get("http://parabank.parasoft.com");
		Assert.assertEquals(driver.getTitle(), "ParaBank | Welcome | Online Banking");
		driver.findElement(By.name("username")).sendKeys("john");
		driver.findElement(By.name("password")).sendKeys("demo");
		driver.findElement(By.cssSelector("input[value='Log In']")).click();
		Assert.assertEquals(driver.getTitle(), "ParaBank | Accounts Overview");
		driver.findElement(By.linkText("Log Out")).click();
		Assert.assertEquals(driver.getTitle(), "ParaBank | Welcome | Online Banking");
	}
	
	@After
	public void teardown() {
		driver.quit();
	}	
}

Pretty straightforward, but good enough.

Creating a build file to run tests automatically
Now to create a build file to run our tests automatically. I used Ant for this, but Maven should work as well. I had Eclipse generate an Ant build-file for me, then changed it to allow Jenkins to run the tests as well. In my case, I only needed to change the location of the imports (the Selenium and the JUnit .jar files) to a location where Jenkins could find them:

<path id="seleniumCI.classpath">
    <pathelement location="bin"/>
    <pathelement location="C:/libs/selenium-server-standalone-2.44.0.jar"/>
    <pathelement location="C:/libs/junit-4.11.jar"/>
</path>

Note that I ran my tests on my own system, so in this case it’s OK to use absolute paths to the .jar files, but it’s by no means good practice to do so! It’s better to use paths relative to your Jenkins workspace, so tests and projects are transferable and can be run on any system without having to change the build.xml.

Actual test execution is a matter of using the junit and junitreport tasks:

<target name="SeleniumCITest">
    <mkdir dir="${junit.output.dir}"/>
    <junit fork="yes" printsummary="withOutAndErr">
        <formatter type="xml"/>
        <test name="com.ontestautomation.selenium.ci.SeleniumCITest" todir="${junit.output.dir}"/>
        <classpath refid="seleniumCI.classpath"/>
        <bootclasspath>
            <path refid="run.SeleniumCITest (1).bootclasspath"/>
        </bootclasspath>
    </junit>
</target>
<target name="junitreport">
    <junitreport todir="${junit.output.dir}">
        <fileset dir="${junit.output.dir}">
            <include name="TEST-*.xml"/>
        </fileset>
        <report format="frames" todir="${junit.output.dir}"/>
    </junitreport>
</target>

This is generated automatically when you create your build.xml using Eclipse, by the way.

Running your tests through Jenkins
The final step is setting up a Jenkins job that simply calls the correct Ant target in a build step:
Ant build step
After tests have been run, Jenkins should pick up the JUnit test results from the folder specified in the junitreport task in the Ant build.xml:
JUnit report post build action
If everything is set up correctly, you should now be able to run your tests through Jenkins and have the results displayed:
Build result in Jenkins
You can also view details on the individual test results by clicking on the error message:
Error details in Jenkins

The Eclipse project I have used for this example can be downloaded here.