Up and running with: JUnit

This is the first article in a new series on tools used in test automation. Each of the articles in it will introduce a specific test tool and will show you how to get up and running with it. The focus will be on free and / or open source test tools as this allows everyone with an interest in the tool presented to get started using it right away.

What is JUnit?
JUnit is a unit testing framework for Java. Is it part of a family of unit testing frameworks for a variety of programming languages, known collectively as xUnit. With JUnit, you can quickly develop and run unit tests for Java classes to verify their correctness.

Where can I get JUnit?
JUnit can be downloaded from here. However, when you use an IDE such as Eclipse or IntelliJ, JUnit comes with the installation, making getting started with JUnit test development even easier.

How do I install and configure JUnit?
As I prefer using Eclipse, the instructions below show you how to start developing and running JUnit tests in Eclipse. Those using IntelliJ are referred to the IntelliJ homepage.

To start using JUnit to run tests on your code, all you need to do is to add the JUnit library to your existing Java project. To do this, right-click on your project, select Properties, go to the Libraries tab and press the ‘Add Library’ button. Select JUnit from the list and click ‘Next’. Then, select ‘JUnit 4’ as the library version and click ‘Finish’. JUnit has now been added to your project libraries and you’re ready to go.

junit_add_library

Creating a first test script
Now, let’s create a first JUnit test script. First of all, as JUnit tests are Java classes in themselves, we create a new source folder in our project to prevent test code getting mixed up with application code. Right-click on your project, select Properties and in the Source tab, select ‘Add Folder’. Name this folder ‘test’ for instance and add it to your project.

It’s a good idea to structure the test code in your test folder similar to the application code to be tested. So, if the Java class you are writing tests for is in package x.y.z in the ‘src’ folder, your test code for this class is placed in package x.y.z in the ‘test’ folder. Create this package in the ‘test’ folder by right-clicking it and selecting ‘New > Package’.

One of the features of Eclipse is that it can automatically generate JUnit test skeletons for you based on the definition of the class you’re writing tests for. To do so, right-click on the package you’ll place the test code in and select ‘New > JUnit Test Case’. Name your test class – I prefer adding ‘Test’ to the name of the class to be tested, so tests for class Apple.java are placed in AppleTest.java – and select the class under test using the button ‘Select’ next to ‘Class under test:’. Click ‘Next’ and select the methods for which you want to generate JUnit tests.

junit_select_methods

Click ‘Finish’ to generate JUnit test skeletons for the selected methods. The code generated should be similar to this:

package com.ontestautomation.selenium.objectmap;

import static org.junit.Assert.*;

import org.junit.Test;

public class ObjectMapTest {

	@Test
	public void testObjectMap() {
		fail("Not yet implemented");
	}

	@Test
	public void testGetLocator() {
		fail("Not yet implemented");
	}

}

The annnotation @Test is used by JUnit to indicate the start of a test case. The generated test code is neatly placed in the right package in your Java project:

junit_package_explorer

Running your test
To run your JUnit tests, simply right-click the test code file and select ‘Run As > JUnit Test’. Your test methods will be executed and a new tab opens displaying the test results:

junit_test_results_1

Both tests fail for now as they have not yet been implemented. That is, no actual tests are being executed so far as we haven’t written the tests itself yet. Below you see a simple test for the getLocator method that tests whether the object retrieved from the object map using this method is equal to the object that is expected.

@Test
public void testGetLocator() {
	ObjectMap objMap = new ObjectMap("objectmap.properties");
	try {
		By testLocator = objMap.getLocator("bing.homepage.textbox");
		assertEquals("Check testLocator object",By.id("sb_form_q"),testLocator);
	} catch (Exception e) {
		System.out.println("Error during JUnit test execution:\n" + e.toString());
	}
		
}

If we rerun the test code, we now see that the test for the getLocator method passes, meaning we have successfully implemented and run our first JUnit test!

junit_test_results_2

Useful features
To write more advanced and better maintainable tests, JUnit provides some nice features. The most important of these features are:

  • The ability to test for expected exceptions. If you want to validate that the method you developed throws the right exception under the right circumstances, you can easily verify this with JUnit, using the expected=NameOfException.class notation:
@Test(expected=Exception.class)
public void testGetLocatorException() throws Exception {
	ObjectMap objMap = new ObjectMap("objectmap.properties");
	By testLocator = objMap.getLocator("unknownobject");
}
  • The ability to create test suites using the @Suite annotation. Using this you can run a set of test classes by calling a single test suite class.
  • The ability to parameterize tests with test data, using the @Parameterized annotation. Using this you can make your unit tests data driven and run the same tests using multiple sets of test data without the need for duplicate test code.
  • The ability to easily export test results to continuous integration frameworks, such as Jenkins. JUnit generates reports in an XML format that can easily be interpreted by Jenkins and similar CI frameworks. This results in a very readable graphical representation of your JUnit test results:

build_result_test_detail

Further reading
For more information on JUnit, you can visit the sites below:

An Eclipse project including the tests I’ve demonstrated above and the reports that have been generated can be downloaded here.

Happy unit testing!

Building and using an Object Repository in Selenium Webdriver

One of the main burdens of automated GUI test script maintainability is the amount of maintenance needed when object properties change within the application under test. A very common way of minimizing the time it takes to update your automated test scripts is the use of a central object repository (or object map as it’s also referred to sometimes). A basic object repository can be implemented as a collection of key-value pairs, with the key being a logical name identifying the object and the value containing unique objects properties used to identify the object on a screen.

Selenium Webdriver offers no object repository implementation by default. However, implementing and using a basic object repository is pretty straightforward. In this article, I will show you how to do it and how to lighten the burden of test script maintenance in this way.

Note that all code samples below are written in Java. However, the object repository concept as explained here can be used with your language of choice just as easily.

Creating the object repository
First, we are going to create a basic object repository and fill it with some objects that we will use in our test script. In this article, I am going to model a very basic scenario: go to the Bing search engine, search for a particular search query and determine the number of search results returned by Bing. To execute this scenario, our script needs to manipulate three screen objects:

  • The textbox where the search string is typed
  • The search button to be clicked in order to submit the search query
  • The text field that displays the number of search results

Our object map will a simple .properties text file that we add to our Selenium project:
Our object mapThe key for each object, for example bing.homepage.textbox, is a logical name for the object that we will use in our script. The corresponding value consists of two parts: the attribute type used for uniquely identifying the object on screen and the corresponding attribute value. For example, the aforementioned text box is uniquely identified by its id attribute, which has the value sb_form_q.

Retrieving objects from the object repository
To retrieve objects from our newly created object map, we will define an ObjectMap with a constructor taking a single argument, which is the path to the .properties file:

public class ObjectMap {
	
	Properties prop;
	
	public ObjectMap (String strPath) {
		
		prop = new Properties();
		
		try {
			FileInputStream fis = new FileInputStream(strPath);
			prop.load(fis);
			fis.close();
		}catch (IOException e) {
			System.out.println(e.getMessage());
		}
	}

The class contains a single method getLocator, which returns a By object that is used by the Selenium browser driver object (such as a HtmlUnitDriver or a FirefoxDriver):

public By getLocator(String strElement) throws Exception {
		
		// retrieve the specified object from the object list
		String locator = prop.getProperty(strElement);
		
		// extract the locator type and value from the object
		String locatorType = locator.split(":")[0];
		String locatorValue = locator.split(":")[1];
		
		// for testing and debugging purposes
		System.out.println("Retrieving object of type '" + locatorType + "' and value '" + locatorValue + "' from the object map");
		
		// return a instance of the By class based on the type of the locator
		// this By can be used by the browser object in the actual test
		if(locatorType.toLowerCase().equals("id"))
			return By.id(locatorValue);
		else if(locatorType.toLowerCase().equals("name"))
			return By.name(locatorValue);
		else if((locatorType.toLowerCase().equals("classname")) || (locatorType.toLowerCase().equals("class")))
			return By.className(locatorValue);
		else if((locatorType.toLowerCase().equals("tagname")) || (locatorType.toLowerCase().equals("tag")))
			return By.className(locatorValue);
		else if((locatorType.toLowerCase().equals("linktext")) || (locatorType.toLowerCase().equals("link")))
			return By.linkText(locatorValue);
		else if(locatorType.toLowerCase().equals("partiallinktext"))
			return By.partialLinkText(locatorValue);
		else if((locatorType.toLowerCase().equals("cssselector")) || (locatorType.toLowerCase().equals("css")))
			return By.cssSelector(locatorValue);
		else if(locatorType.toLowerCase().equals("xpath"))
			return By.xpath(locatorValue);
		else
			throw new Exception("Unknown locator type '" + locatorType + "'");
	}

As you can see, objects can be identified using a number of different properties, including object IDs, CSS selectors and XPath expressions.

Using objects in your test script
Now that we can retrieve objects from our object map, we can use these in our scripts to execute the desired scenario:

public static void main (String args[]) {

		// Create a new instance of the object map
		ObjectMap objMap = new ObjectMap("objectmap.properties");

		// Start a browser driver and navigate to Google
		WebDriver driver = new HtmlUnitDriver();
        driver.get("http://www.bing.com");

        // Execute our test
        try {
        	
        	// Retrieve search text box from object map and type search query
        	WebElement element = driver.findElement(objMap.getLocator("bing.homepage.textbox"));
			element.sendKeys("Alfa Romeo");
			
			// Retrieve search button from object map and click it
			element = driver.findElement(objMap.getLocator("bing.homepage.searchbutton"));
			element.click();
			
			// Retrieve number of search results using results object from object map
			element = driver.findElement(objMap.getLocator("bing.resultspage.results"));
			System.out.println("Search result string: " + element.getText());
			
			// Verify page title
			Assert.assertEquals(driver.getTitle(), "Alfa Romeo - Bing");
			
		} catch (Exception e) {
			System.out.println("Error during test execution:\n" + e.toString());
		}
        
	}

You can see from this code sample that using an object from the object map in your test is as easy as referring to its logical name (i.e., the key in our object map).

Object repository maintenance
With this straightforward mechanism we have been able to vastly reduce the amount of time needed for script maintenance in case object properties change. All it takes is an update of the appropriate entries in the object map and we’re good to go and run our tests again.

Thanks to Selenium Master for explaining this concept clearly for me to apply.

An example Eclipse project using the pattern described above can be downloaded here.

Generating automated tests from specifications

In most cases, the implementation of automated testing focuses on the automated execution of previously defined test cases. These test cases come in the form of test scripts or use cases that are prepared by test analists or test engineers. Often, before the introduction of automated testing in a project and / or team, these test cases were executed by hand in the past.

However, what if we could also automate the creation of test cases? When applied correctly, this would yield an even more efficient testing process. Furthermore, the number of mistakes in the derivation of test cases would be reduced to 0 (given that the test case generator is functioning correctly). These two factors indicate that an interesting case might be made, given the right circumstances.

Of course, this does not apply everywhere. To be able to automatically generate test cases, the following points should be observed:

Are my specifications recorded in a format that is suitable for automated test case generation?
Of course, the specifications should be recorded in such a way that they can be interpreted by your test case generator. It is highly unlikely that specifications written in natural language in a Word document can be read and understood by a piece of code to the extent that correct test case specifications can be derived from it (however, if you have ever achieved this, please let me know :). On the other hand, when (part of) the specifications for your SUT are captured in a formal model, certain types of test cases might be derived from this by an intelligent piece of software. This is sometimes achieved in Model Based Testing approaches.

Also, in order to validate whether business rules are implemented correctly in your SUT, it might be feasible to automatically derive test cases from these rules, given that they are specified in some sort of (semi-)formal language.

I’m sure there are more examples where generating test cases with a single keystroke or mouse click is feasible.

Is there a profit to be gained from the ability to generate test cases automatically?
Of course, the fact that something can be automated doesn’t mean it necessarily should be automated. However, when there are a lot of test cases to be generated, or when specs change frequently throughout the software development process, which occurs more and more often with the rise in popularity of Agile and Scrum methodologies, developing and testing a test case generator might be beneficial. As with the previous factor, I’m sure there are more possibilities for profit here, but these two stand out for me at the moment.

A case study
In a recent project I have been able to successfully implement not only the automated execution of test cases, but also the generation of these test cases directly from the specifications as provided by our client.

The project concerned the testing of a web service that validates fixed-length messages for syntactical correctness, semantical integrity and adherence to a number of business rules. There were around 20 types of messages, each consisting of a couple of hundreds of fields. Each field and each business rule has a corresponding error code that is returned by the server in case either the field specifications or the business rules are violated. Needless to say, that’s a lot of test cases.

Specifying these test cases by hand would take a very long time, would be tiresome as the test cases are highly similar and would therefore be pretty error-prone as well. Even testers get bored every now and then.. However, the similarity and the number of the set of test cases also makes a pretty good case for automating test case generation.

First, we need to determine what types of test cases we can identify and which of these can be generated automatically. In this article, I will focus on the negative test cases, i.e., the test cases for which the web service should return one or more error codes. For this project, I identified the following types of negative test cases:

  • Leave mandatory fields empty
  • Assign alphanumeric values to numeric fields
  • For fields that may only contain values from a predefined enumeration: enter values that are not in the enumeration
  • Simple business rules such as ‘IF field A contains value X THEN field B should contain value Y’
  • More complex business rules such as ‘The message may contain up to three address records whenever the client is of type Z’

I chose to skip the test case where the mandatory field was omitted altogether. In fixed length messages, this will cause all subsequent fields to be out of place, effectively rendering the message useless.

Next, I wrote some lines of code (in Jython, in my case) to read the message specifications (these were provided in Excel format, not ideal but manageable using the Apache POI library) and derive test cases from them. For example, a field with the following characteristics:

  • Mandatory
  • Length 1
  • Numerical
  • Values allowed: 1,2,9

yields the following negative test cases:

  • Leave the field empty
  • Enter value ‘A’ in the field
  • Enter value 3 in the field

Writing the code to generate these test cases took me a day, far shorter than it would have taken to generate all test cases by hand. Let alone what would happen when a new version of the message specs would have arrived somewhere halfway through the testing process..

Next, I had to generate the test messages for each and every test case. I did this by using a test message template, which was effectively a valid message (i.e., it does not generate error messages upon validation), containing all possible fields. Then, for every negative test case, I replaced the correct value for the field under test in that test case with an incorrect value depending on the type of test case. For example, when the test message template looks like this:

10002398TEST141401

and the field under test is the last position in the test message, the generated test messages for the test cases mentioned above look like this:

10002398TEST14140
10002398TEST14140A
10002398TEST141403

Then, I’d send the test message to the validation web service and checked whether the response from the web service contained a message stating that the field under test had an error in it, and whether the correct error code was returned by the service.

As a result, I was able to quickly generate literally thousands of test cases, many of which had never been executed before (as there was no time and no-one had though of generating test cases instead of manually specifying them). This resulted in very high test coverage and quick turnaround time, and a very happy customer overall.

Hopefully the case study above shows an example of the power of generating test cases. Again, it might not be achievable or economical everywhere, but when it does, the results may be very beneficial. Whenever system specifications are available in a format that can be processed automatically (such as Excel, text or XML), there might just be an opportunity to save lots of time using test case generation.