Up and running with: TestNG

This is the fifth article in our series on new, popular or otherwise interesting tools used in test automation. You can read all posts within this series by clicking here.

What is TestNG?
From the TestNG.org website: TestNG is a testing framework inspired from JUnit and NUnit but introducing some new functionalities that make it more powerful and easier to use, such as annotations and support for data driven testing.

Where can I get TestNG?
TestNG can be downloaded from this site. For Eclipse users, it is highly recommended to install the TestNG plugin for Eclipse for maximum ease of use. IDEA IntelliJ supports TestNG natively as of version 7.

How do I install and configure TestNG?
Since TestNG is supported natively from IntelliJ 7 onwards, there’s no need for additional configuration for IntelliJ users. When you install the TestNG for Eclipse plugin as described, you’re set to create your first TestNG test as well, as can be read here. In other situations, you can download the .jar files from here as well.

Creating a first TestNG test
As I have done in the past, I’ll (ab)use the ParaBank demo application at the Parasoft website for our first TestNG test. I’ll use Selenium WebDriver to perform the test steps, and will use TestNG to perform the checks that we want to do and the reporting. Let’s say we want to verify that we can login successfully to the ParaBank application, given the right credentials. A Selenium + TestNG test that performs this test looks like this (import statements removed for brevity):

public class ParabankTestNG {
	
	WebDriver driver;
	
	@BeforeSuite
	public void setUp() {
		
		driver = new FirefoxDriver();
		driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
	}
	
	@Test(description="Tests a successful login")
	public void testLoginOK() {
		
		driver.get("http://parabank.parasoft.com");
		driver.findElement(By.name("username")).sendKeys("john");
		driver.findElement(By.name("password")).sendKeys("demo");
		driver.findElement(By.xpath("//input[@value='Log In']")).click();
		Assert.assertEquals("ParaBank | Accounts Overview",driver.getTitle());
		driver.findElement(By.partialLinkText("Log Out")).click();
	}
	
	@AfterSuite
	public void tearDown() {
		
		driver.quit();
	}
}

Note that this test looks almost identical to a Selenium + JUnit test. The only difference is the use of the @BeforeSuite and @AfterSuite annotations for test setup and teardown, where we would use @Before and @After in JUnit. TestNG uses a large variety of annotations that can be used to enhance your tests and test suites.

Running the test
Again, as I’m an Eclipse user, I’ll show you how to execute your tests in Eclipse only. Please refer to the TestNG website for instructions for other IDEs.

For those that installed the TestNG plugin for Eclipse, there are two simple ways to start a TestNG test. First, we can simply right-click our test .java file in the Package Explorer and select ‘Run As > TestNG Test’. This is perfectly suitable when you have a single class containing all of your tests.

However, for larger projects, this will typically not be the case. For those situations, we can create an XML file called testng.xml that contains instructions on which tests to run and how they should be run. You can find instructions on the use of testng.xml files here. As an example, we can run all TestNG tests in a specific package using the following instructions:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
 
<suite name="My first TestNG test suite" verbose="1" >
  <test name="Login tests"   >
    <packages>
      <package name="com.ontestautomation.selenium.testng" />
   </packages>
 </test>
</suite>

The verbose attribute specifies the verbosity of information logged to the console, where 1 is low and 10 is high.

Running a test using the testng.xml file can be done just as easily by right-clicking on it in the Package Explorer and selecting ‘Run As > TestNG Suite’. Test results can be reviewed in the ‘Results of running suite’ tab in Eclipse. Note that using meaningful names for tests and test suites in the testng.xml file make these results much easier to read and interpret:

TestNG test results in Eclipse

Using the testng.xml file also makes it easy to specify exactly which tests are run in a specific test suite, and also in which order they are executed. By default, the order in which the tests appear in the testng.xml file defines the order in which the tests are run.

Useful features
one very useful feature of TestNG is the ability to easily parameterize tests from testng.xml. For example, if we want to parameterize the input values from the login test above, we first redefine the method in the test class to take parameters:

@Parameters({"username","password"})
@Test(description="Tests a successful login")
public void testLoginOK(String username, String password) {
		
	driver.get("http://parabank.parasoft.com");
	driver.findElement(By.name("username")).sendKeys(username);
	driver.findElement(By.name("password")).sendKeys(password);
	driver.findElement(By.xpath("//input[@value='Log In']")).click();
	Assert.assertEquals("ParaBank | Accounts Overview",driver.getTitle());
	driver.findElement(By.partialLinkText("Log Out")).click();
}

Note the use of the @Parameter annotation to link the arguments of our test method to the parameters you define in the testng.xml file:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
 
<suite name="My first TestNG test suite" verbose="1" >
  <parameter name="username" value="john"/>
  <parameter name="password" value="demo"/>
  <test name="Login tests"   >
    <packages>
      <package name="com.ontestautomation.selenium.testng" />
   </packages>
 </test>
</suite>

Another useful option of TestNG is the fact that it automatically generates readable HTML reports containing the test results. By default, these are created in a test-output directory relative to your project location. The HTML report generated from the test above, for example, looks as follows:
TestNG HTML test report
You can further personalize and adjust the reports as described here.

Further reading
An Eclipse project including the TestNG test I’ve demonstrated above and the reports that have been generated can be downloaded here.

Happy TestNG testing! Since I’ve got to know TestNG in the past couple of weeks, I’ve discovered quite a few interesting possibilities that I want to share with you in future posts, so stay tuned.

Up and running with: JBehave

This is the fourth article in our series on new, popular or otherwise interesting tools used in test automation. You can read all posts within this series by clicking here.

What is JBehave?
From the JBehave.org website: JBehave is a framework for Behaviour-Driven Development (BDD). BDD is an evolution of test-driven development (TDD) and acceptance-test driven design, and is intended to make these practices more accessible and intuitive to newcomers and experts alike. It shifts the vocabulary from being test-based to behaviour-based, and positions itself as a design philosophy.

Using JBehave follows these five simple steps, which we will follow in the examples provided in this article as well:

  1. Write your user story
  2. Map the steps in the user story to Java code
  3. Configure your user stories
  4. Run your JBehave tests
  5. Review the test results

Where can I get JBehave?
JBehave can be downloaded from this site.

How do I install and configure JBehave?
JBehave consists of a number of .jar files. All you need to do is to add these .jar files as a dependency to your Java project and you’re ready to start using JBehave. You can also set up and configure JBehave using Maven. See for more details the extensive JBehave reference guide here.

Creating a first JBehave user story and corresponding test
Before we start writing our first JBehave test, we need a Java class that we can test. For this purpose, I have written a very simple POJO class Flipper with two variables: a String state (which is either ‘even’ or ‘odd’) and an Integer value. The source code for this class is included in the download at the end of this post. Among the methods for this class is a method flipState, which flips the state from ‘even’ to ‘odd’ (or the other way round) and increases the value by 1. Useless, maybe, but it does the trick for this example!

Step 1: Write your user story
A user story is written in the ‘given-when-then’ format that is characteristic for the BDD approach. In our example, we want to make sure (i.e., test) that given a fresh instance of the Flipper class, when we call the flipState method, then the value variable of the Flipper class is assigned the value 2. See how I used the ‘given-when-then’ strcuture to describe my test case? Translated to a user story that is JBehave-readable, our test case looks like this:

Scenario: when a user flips a flipper, its value is increased by 1

Given a flipper
Given the flipper has value 1
When the user flips the flipper
Then the value of the flipper must become 2

Step 2: Map the steps in the user story to Java code
One of the nicer features of JBehave is that you can run your tests as JUnit tests, which makes for easy integration with the rest of your testing and / or continuous integration framework. To do this, your test class containing the implementation of your story steps should extend the JUnitStories class. JBehave uses the @Given, @When and @Then annotations to identify story steps and their nature. For example, our test class and the implementation of the first ‘given’ step might look like this:

public class JBehaveTest extends JUnitStories {

	private Flipper flipper;

	@Given("a flipper")
	public void aFlipper() {

		flipper = new Flipper();
}

Now, whenever JBehave encounters the line ‘Given a flipper’ in one of the stories that is executed, it will automatically execute the aFlipper() method.

We can do the same for the ‘when’ and ‘then’ clauses of our story, using the appropriate annotations:

@When("the user flips the flipper")
public void whenTheUserFlipsTheFlipper() {

	flipper.flipState();
}

@Then("the value of the flipper must become 2")
public void valueOfFlipperMustBecomeTwo() {

	Assert.assertEquals(2, flipper.getValue());
}

Step 3: Configure your user stories
In order for our stories to be run, we need to tell JBehave where our stories are located, and which configuration should be used to execute them. For now, we are going to use the default configuration.

@Override
public Configuration configuration() {
	return new MostUsefulConfiguration().useStoryLoader(new LoadFromClasspath(getClass().getClassLoader())).useStoryReporterBuilder(new StoryReporterBuilder().withFormats(Format.CONSOLE));
}

@Override
public List<CandidateSteps> candidateSteps() {
	return new InstanceStepsFactory(configuration(), this).createCandidateSteps();
}
	
@Override
protected List<String> storyPaths() {
	return Arrays.asList("com/ontestautomation/jbehave/demo/test_value.story");
}

@Override
@Test
public void run() throws Throwable {
	super.run();
}

The configuration method loads the default configuration for running our JBehave tests, known as the MostUsefulConfiguration. The candidateSteps method loads the story steps defined in the current class (‘this’). The storyPaths method is used to define which stories are going to be executed. I included my sample story in the Eclipse project I used, but you can load story definitions from everywhere, including from an URL. Finally, the run method allows us to run our class as a test.

Step 4: Run your JBehave tests
Since we defined our JBehave tests as JUnit tests, running them is as easy as running regular JUnit tests. For example, in Eclipse, you can just right-click on your JBehave test class and select Run As > JUnit Test.

Step 5: Review the test results
Again, since our JBehave tests are just another type of JUnit tests, we can review the results for our tests like we do for any other JUnit test. Again, in Eclipse, a new tab displaying our test results opens automatically after test execution:
JBehave test results
In the Eclipse console, we can also see that our story has been processed successfully:

Processing system properties {}
Using controls EmbedderControls[batch=false,skip=false,generateViewAfterStories=true,ignoreFailureInStories=false,ignoreFailureInView=false,verboseFailures=false,verboseFiltering=false,storyTimeoutInSecs=300,threads=1]

(BeforeStories)

Running story com/ontestautomation/jbehave/demo/test_value.story

(com/ontestautomation/jbehave/demo/test_value.story)
Scenario: when a user flips a flipper, its value is increased by 1
Given a flipper
Given the flipper has value 1
When the user flips the flipper
Then the value of the flipper must become 2

Useful features
JBehave offers a lot of useful features, all of which are described in the online reference manual which can be found here. Some of the most notable features are:

Further reading
An Eclipse project including my Flipper class implementation and the JBehave test story I’ve demonstrated above can be downloaded here.

Happy JBehaving!

Up and running with: Mockito

This is the third article in our series on new, popular or otherwise interesting tools used in test automation. You can read all posts within this series by clicking here.

What is Mockito?
Mockito is an open source testing framework for Java that allows developers to create mock objects for use in automated unit tests. It is often used in the Test Driven Development and Behavior Driven Development approaches. Using Mockito mock objects, developers can mock external dependencies and ensure their code interacts with it in the expected manner.

Where can I get Mockito?
Mockito can be downloaded from this site.

How do I install and configure Mockito?
As the Mockito framework consists of just a single .jar file, all you need to do is to add this .jar as a dependency to your Java project and you’re ready to start mocking!

Creating a first Mockito mock object
Before we can start creating mock objects using Mockito, we first need a Java class for which we can mock the behavior. To illustrate the workings of Mockito I created a simple class Login, consisting of a single constructor, two String variables username and password and a single method login(username,password). The class implementation is included in the download link at the end of this post.

Now, assume that this Login class is either not readily available when you want to perform unit tests that have this class as a dependency, or that you want to emulate a certain type of behavior. To do so, let’s create a Mockito mock for the Login class:

public static Login createLoginMock() {

	// create and return a mock instance of the Login class	
	return mock(Login.class);
}

Easy, right? Now, let’s see what we can do with this mock.

Using your Mockito mock
First, let’s check whether we can interact with the methods of our mock correctly. Mockito provides a verify function for this:

public static void runVerifyTestWithMockito() {
		
	// create mock instance of the login class
	Login loginMock = createLoginMock();
		
	// call methods on the mock instance
	loginMock.setUsername("bas");
	loginMock.setPassword("ok");
		
	// verify that methods are called properly
	verify(loginMock).setUsername("bas");
	verify(loginMock).setPassword("ok");
		
	// attempt to verify a method that hasn't been called (this generates an error)
	verify(loginMock).getUsername();	
}

The last call generated an error, as we try to verify an interaction that hasn’t taken place.

Now, let’s use Mockito to stub the behavior of a method of the Login class, so that we can use our mock to simulate specific behavior in our unit tests. In Mockito, this is done using a when … thenReturn … construction:

public static void runStubTestWithMockito() {
		
	// create mock instance of the login class
	Login loginMock = createLoginMock();
		
	// stub mock behavior
	when(loginMock.login()).thenReturn("login_ok");
		
	// call mock method and verify result
	Assert.assertEquals("login_ok",loginMock.login());
}

Finally, we can also have a class method of our mock return an exception, even when the actual class implementation might not always do that. This is useful when testing exception handling in your code and is done using a when … thenThrow … construction:

public static void runExceptionTestWithMockito() {

	// create mock instance of the login class
	Login loginMock = createLoginMock();

	// have the mock return an Exception
	when(loginMock.getPassword()).thenThrow(new RuntimeException());

	// call the mock method that throws the exception
	loginMock.getPassword();
}

Useful features
I’ve showed you some of the basic features of Mockito. However, the framework offers loads more, such as:

  • Verification that a method is invoked exactly once / at least x times / never / …
  • Verification that methods are invoked in a particular order
  • Explicitly verify that certain mock classes have not been interacted with

An extensive list of Mockito features, along with examples, can be found here.

Further reading
There is a lot of documentation to be found on Mockito on the web, apart from the reading material which I’ve linked to above. For example, a nice tutorial, including using Mockito for Android testing, can be found on the site of Lars Vogel. For those that prefer actual books, Tomek Kaczanowski has written a book on Mockito and TestNG, which can be purchased here.

An Eclipse project including my Login class implementation and the tests I’ve demonstrated above can be downloaded here.

Happy mocking!