Up and running with: TestNG

This is the fifth article in our series on new, popular or otherwise interesting tools used in test automation. You can read all posts within this series by clicking here.

What is TestNG?
From the TestNG.org website: TestNG is a testing framework inspired from JUnit and NUnit but introducing some new functionalities that make it more powerful and easier to use, such as annotations and support for data driven testing.

Where can I get TestNG?
TestNG can be downloaded from this site. For Eclipse users, it is highly recommended to install the TestNG plugin for Eclipse for maximum ease of use. IDEA IntelliJ supports TestNG natively as of version 7.

How do I install and configure TestNG?
Since TestNG is supported natively from IntelliJ 7 onwards, there’s no need for additional configuration for IntelliJ users. When you install the TestNG for Eclipse plugin as described, you’re set to create your first TestNG test as well, as can be read here. In other situations, you can download the .jar files from here as well.

Creating a first TestNG test
As I have done in the past, I’ll (ab)use the ParaBank demo application at the Parasoft website for our first TestNG test. I’ll use Selenium WebDriver to perform the test steps, and will use TestNG to perform the checks that we want to do and the reporting. Let’s say we want to verify that we can login successfully to the ParaBank application, given the right credentials. A Selenium + TestNG test that performs this test looks like this (import statements removed for brevity):

public class ParabankTestNG {
	
	WebDriver driver;
	
	@BeforeSuite
	public void setUp() {
		
		driver = new FirefoxDriver();
		driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
	}
	
	@Test(description="Tests a successful login")
	public void testLoginOK() {
		
		driver.get("http://parabank.parasoft.com");
		driver.findElement(By.name("username")).sendKeys("john");
		driver.findElement(By.name("password")).sendKeys("demo");
		driver.findElement(By.xpath("//input[@value='Log In']")).click();
		Assert.assertEquals("ParaBank | Accounts Overview",driver.getTitle());
		driver.findElement(By.partialLinkText("Log Out")).click();
	}
	
	@AfterSuite
	public void tearDown() {
		
		driver.quit();
	}
}

Note that this test looks almost identical to a Selenium + JUnit test. The only difference is the use of the @BeforeSuite and @AfterSuite annotations for test setup and teardown, where we would use @Before and @After in JUnit. TestNG uses a large variety of annotations that can be used to enhance your tests and test suites.

Running the test
Again, as I’m an Eclipse user, I’ll show you how to execute your tests in Eclipse only. Please refer to the TestNG website for instructions for other IDEs.

For those that installed the TestNG plugin for Eclipse, there are two simple ways to start a TestNG test. First, we can simply right-click our test .java file in the Package Explorer and select ‘Run As > TestNG Test’. This is perfectly suitable when you have a single class containing all of your tests.

However, for larger projects, this will typically not be the case. For those situations, we can create an XML file called testng.xml that contains instructions on which tests to run and how they should be run. You can find instructions on the use of testng.xml files here. As an example, we can run all TestNG tests in a specific package using the following instructions:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
 
<suite name="My first TestNG test suite" verbose="1" >
  <test name="Login tests"   >
    <packages>
      <package name="com.ontestautomation.selenium.testng" />
   </packages>
 </test>
</suite>

The verbose attribute specifies the verbosity of information logged to the console, where 1 is low and 10 is high.

Running a test using the testng.xml file can be done just as easily by right-clicking on it in the Package Explorer and selecting ‘Run As > TestNG Suite’. Test results can be reviewed in the ‘Results of running suite’ tab in Eclipse. Note that using meaningful names for tests and test suites in the testng.xml file make these results much easier to read and interpret:

TestNG test results in Eclipse

Using the testng.xml file also makes it easy to specify exactly which tests are run in a specific test suite, and also in which order they are executed. By default, the order in which the tests appear in the testng.xml file defines the order in which the tests are run.

Useful features
one very useful feature of TestNG is the ability to easily parameterize tests from testng.xml. For example, if we want to parameterize the input values from the login test above, we first redefine the method in the test class to take parameters:

@Parameters({"username","password"})
@Test(description="Tests a successful login")
public void testLoginOK(String username, String password) {
		
	driver.get("http://parabank.parasoft.com");
	driver.findElement(By.name("username")).sendKeys(username);
	driver.findElement(By.name("password")).sendKeys(password);
	driver.findElement(By.xpath("//input[@value='Log In']")).click();
	Assert.assertEquals("ParaBank | Accounts Overview",driver.getTitle());
	driver.findElement(By.partialLinkText("Log Out")).click();
}

Note the use of the @Parameter annotation to link the arguments of our test method to the parameters you define in the testng.xml file:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
 
<suite name="My first TestNG test suite" verbose="1" >
  <parameter name="username" value="john"/>
  <parameter name="password" value="demo"/>
  <test name="Login tests"   >
    <packages>
      <package name="com.ontestautomation.selenium.testng" />
   </packages>
 </test>
</suite>

Another useful option of TestNG is the fact that it automatically generates readable HTML reports containing the test results. By default, these are created in a test-output directory relative to your project location. The HTML report generated from the test above, for example, looks as follows:
TestNG HTML test report
You can further personalize and adjust the reports as described here.

Further reading
An Eclipse project including the TestNG test I’ve demonstrated above and the reports that have been generated can be downloaded here.

Happy TestNG testing! Since I’ve got to know TestNG in the past couple of weeks, I’ve discovered quite a few interesting possibilities that I want to share with you in future posts, so stay tuned.

Running Selenium JUnit tests from Jenkins

In this post I want to show you how to use Jenkins to automatically execute Selenium tests written in JUnit format, and how the results from these tests can be directly reported back to Jenkins. To achieve this, we need to complete the following steps:

  • Write some Selenium tests in JUnit format that we want to execute
  • Create a build file that runs these tests and writes the reports to disk
  • Set up a Jenkins job that runs these tests and interprets the results

Note: First of all a point of attention: I couldn’t get this to work while Jenkins was installed as a Windows service. This has something to do with Jenkins opening browser windows and subsequently not having suitable permissions to access sites and handle Selenium calls. I solved this by starting Jenkins ‘by hand’ by downloading the .war file from the Jenkins site and running it using java -jar jenkins.war

Creating Selenium tests to run
First, we need to have some tests that we would like to run. I’ve created three short tests in JUnit-format, where one has an intentional error for demonstration purposes – it’s good practice to see if any test defects actually show up in Jenkins! Using the JUnit-format implies that tests can be run independently, so there can’t be any dependencies between tests. My test class looks like this (I’ve removed two tests and all import statements for brevity):

package com.ontestautomation.selenium.ci;

public class SeleniumCITest {
	
	static WebDriver driver;
	
	@Before
	public void setup() {
		
		driver = new FirefoxDriver();
		driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);				
	}
	
	@Test
	public void successfulLoginLogout() {
		
		driver.get("http://parabank.parasoft.com");
		Assert.assertEquals(driver.getTitle(), "ParaBank | Welcome | Online Banking");
		driver.findElement(By.name("username")).sendKeys("john");
		driver.findElement(By.name("password")).sendKeys("demo");
		driver.findElement(By.cssSelector("input[value='Log In']")).click();
		Assert.assertEquals(driver.getTitle(), "ParaBank | Accounts Overview");
		driver.findElement(By.linkText("Log Out")).click();
		Assert.assertEquals(driver.getTitle(), "ParaBank | Welcome | Online Banking");
	}
	
	@After
	public void teardown() {
		driver.quit();
	}	
}

Pretty straightforward, but good enough.

Creating a build file to run tests automatically
Now to create a build file to run our tests automatically. I used Ant for this, but Maven should work as well. I had Eclipse generate an Ant build-file for me, then changed it to allow Jenkins to run the tests as well. In my case, I only needed to change the location of the imports (the Selenium and the JUnit .jar files) to a location where Jenkins could find them:

<path id="seleniumCI.classpath">
    <pathelement location="bin"/>
    <pathelement location="C:/libs/selenium-server-standalone-2.44.0.jar"/>
    <pathelement location="C:/libs/junit-4.11.jar"/>
</path>

Note that I ran my tests on my own system, so in this case it’s OK to use absolute paths to the .jar files, but it’s by no means good practice to do so! It’s better to use paths relative to your Jenkins workspace, so tests and projects are transferable and can be run on any system without having to change the build.xml.

Actual test execution is a matter of using the junit and junitreport tasks:

<target name="SeleniumCITest">
    <mkdir dir="${junit.output.dir}"/>
    <junit fork="yes" printsummary="withOutAndErr">
        <formatter type="xml"/>
        <test name="com.ontestautomation.selenium.ci.SeleniumCITest" todir="${junit.output.dir}"/>
        <classpath refid="seleniumCI.classpath"/>
        <bootclasspath>
            <path refid="run.SeleniumCITest (1).bootclasspath"/>
        </bootclasspath>
    </junit>
</target>
<target name="junitreport">
    <junitreport todir="${junit.output.dir}">
        <fileset dir="${junit.output.dir}">
            <include name="TEST-*.xml"/>
        </fileset>
        <report format="frames" todir="${junit.output.dir}"/>
    </junitreport>
</target>

This is generated automatically when you create your build.xml using Eclipse, by the way.

Running your tests through Jenkins
The final step is setting up a Jenkins job that simply calls the correct Ant target in a build step:
Ant build step
After tests have been run, Jenkins should pick up the JUnit test results from the folder specified in the junitreport task in the Ant build.xml:
JUnit report post build action
If everything is set up correctly, you should now be able to run your tests through Jenkins and have the results displayed:
Build result in Jenkins
You can also view details on the individual test results by clicking on the error message:
Error details in Jenkins

The Eclipse project I have used for this example can be downloaded here.

A couple of presentations on service virtualization

Hey all!

First of all, apologies for the lack of recent postings on this blog. I have been quite busy with my day job and haven’t had time to write new posts in a while. I definitely intend to get working on some new stuff soon enough, so please do keep posted.

It’s been great to see that even though I haven’t added any new posts in a while, questions on existing posts are still coming in quite regularly. This is a great motivator for me to keep working on this site, and yet another reason to get my lazy behind into gear and start working on new material.

One of the reasons I’ve been quite busy is the fact that I’ve been doing a couple of presentations at conferences here in the Netherlands, and I thought it would be nice to share these with you as well.

The first presentation I held in November at the Dutch Testing Conference. The presentation shows the case study for a project I’ve been working on for the last 8 months or so. In this project, we have successfully introduced service virtualization as a means to get rid of some major blockers in our test environment. Using virtualized services that emulate the behaviour of the dependencies that were causing trouble, we have been able to speed up the development and testing process significantly. Introducing SV has also been an enabler for test automation as well – as in: we couldn’t do test automation without these virtualized services.

I gave the second presentation at the first Continuous Delivery Conference, also here in the Netherlands. As you can guess from the name, this conference was more about Continuous Delivery, rather than just about testing. However, as our case study showed some pretty significant improvements in the CD area as well, we decided to present it there as well. I did this talk at the request of Parasoft.

Both conferences were great to attend, and especially the CD conference gave me a lot of inspiration for future work and areas to explore. It really showed that testing isn’t just an activity in itself anymore (if it ever has been), but it has become an integral part of a much larger story, that of continuously delivering high quality software at ever increasing pace. Very interesting to hear and see the perspectives of some of the inspirators in the CD field on this topic..