Three practices for creating readable test code

You’ve probably read before that writing code that is readable and easy to understand is considered good practice for any software developer. Readable code takes less time to understand, to hand over to your colleagues or clients and to update if needs be. Since writing automated checks is essentially just another form of software development (this was true in 2005 and it sure hasn’t changed since then), it makes sense to apply the ‘write readable code’ practice there as well. But how do you go about doing that? In this post, I would like to introduce three practices that are guaranteed to have a positive impact on the readability of your test code.

Practice: Use fluent assertions libraries
Some other fans of readable code with far better development skills than yours truly have created some useful libraries that allow you to write assertions in a style that is almost equal to natural languauge, so-called fluent assertions. Two examples of these fluent assertion libraries for Java are AssertJ and Truth.

Let’s take AssertJ as an example. Consider the following TestNG assertions:

assertEquals(4, calculator.add(2,2));

assertTrue(muppets.hasMember("Kermit"));

assertEquals(0, mammals.getAnimalsByType("fish").size());

When converted to AssertJ assertions, these look like:

assertThat(calculator.add(2,2)).isEqualTo(4);

assertThat(muppets.hasMember("Kermit")).isTrue();

assertThat(mammals.getAnimalsByType("fish")).isEmpty();

The exact same checks are being executed, but the AssertJ assertions are the better readable ones in my opinion, since you can read them from the left to the right without having to jump back in order to understand what the assertion does. AssertJ has many more features for improving the readability of your test code, so be sure to check it out. The library also comes with a script that automatically converts your TestNG assertions to AssertJ assertions for you.

Practice: Have Page Object methods return Page Objects
This practice applies mostly to Selenium WebDriver code, although I can imagine there are other uses for it as well. When you use the Page Object model in your code, you can easily improve the readability of your tests by assigning Page Object return types to all methods in your Page Object. Each method represents a specific user action that is to be performed on the page, and the type of Page Object that is returned depends on the page where the user will end up after that specific action is performed. For example, a simple LoginPage object could look like this:

public class LoginPage {
	
	private WebDriver driver;
	
	@FindBy(id="txtFieldUsername")
	private WebElement txtFieldUsername;
	
	@FindBy(id="txtFieldPassword")
	private WebElement txtFieldPassword;
	
	@FindBy(id="btnLogin")
	private WebElement btnLogin;
	
	public LoginPage(WebDriver driver) {
		
		this.driver = driver;
		
		if(!driver.getTitle().equals("Login Page")) {
			driver.get("http://example.com/login");
		}
		
		PageFactory.initElements(driver, this);
	}
	
	public LoginPage setUsername(String username) {
		
		txtFieldUsername.sendKeys(username);
		return this;
	}
	
	public LoginPage setPassword(String password) {
		
		txtFieldPassword.sendKeys(password);
		return this;
	}
	
	public HomePage clickLoginButton() {
		
		btnLogin.click();
		return new HomePage(driver);
	}
}

Note that this Page Object also uses the PageFactory pattern, which improves Page Object code readability as well. This pattern is beyond the scope of this blog post, though.

Assuming we also have a HomePage object with a method that returns a boolean indicating we are indeed on the home page, we can write a simple login test that looks like this:

@Test
public void loginTest() {
		
	new LoginPage(driver)
		.setUsername("myUser")
		.setPassword("myPassword")
		.clickLoginButton();
		
	Assert.assertTrue(new HomePage(driver).isAt(), "Home page has been loaded successfully after login action");
}

You can instantly see what this test does, due to the fact that all of our Page Object methods return a specific type of Page Object. This enables us to chain methods from either the same or different Page Objects (as long as their return types match up) and create this type of readable tests.

Practice: Employ a Given/When/Then structure for your tests wherever possible
Another way to improve the readability of your test code is to structure your tests following the Given/When/Then format that is often used in Behaviour Driven Development (BDD). This applies even when you’re not doing BDD, by the way. Just think of it like this:

  • Given: set up test data and other preconditions
  • When: perform a specific action
  • Then: check the results

Some test tools support this format explicitly, for example REST Assured:

@Test
public void testMD5() {
		
	given().
		parameters("text", "test").
	when().
		get("http://md5.jsontest.com").
	then().
		body("md5",equalTo("098f6bcd4621d373cade4e832627b4f6")).
	and().
		body("original", equalTo("test"));
}

For other tools, you may need to apply this implicitly:

@Test
public void testCalculatorAddition() {
		
	//Given
	Calculator calculator = new Calculator();
	
	//When
	calculator.add(2);
		
	//Then
	assertEquals(calculator.getResult(), 2);		
}

An additional benefit of writing your tests in this way is that it makes it much easier to discuss them with people without any substantial programming background. Most people will recognize what a test does if you break them down in this way, even when they lack programming knowledge.

I hope there’s at least something that you can take away from this post and apply it to your existing test suite in order to make it more readable. You’ll be doing yourself, your clients and your fellow team members a huge favour.

Using the TestNG ITestContext to create smarter REST Assured tests

In this post, I would like to demonstrate two different concepts that I think work very well together:

  • How to store and retrieve data objects using the TestNG ITestContext for better code maintainability
  • How to communicate with RESTful web services that use basic or OAuth2 authorization using REST Assured

Using the PayPal sandbox API as an example, I will show you how you can create readable and maintainable tests for secured APIs using TestNG and REST Assured.

The TestNG ITestContext
If you have a suffficiently large test suite, chances are high that you want to be able to share objects between individual tests to make your tests shorter and easier to maintain. For example, if you are calling a web service multiple times throughout your test suite and that web service requires an authentication token in order to be able to consume it, you might want to request and store that authentication token in the setup phase of your test suite, then retrieve and use it in all subsequent tests where this web service is invoked. This is exactly the scenario we’ll see in this blog post.

TestNG offers a means of storing and retrieving objects between tests through the ITestContext interface. This interface allows you to store (using the inherited setAttribute() method) and retrieve (using getAttribute()) objects. Since the ITestContext is created once and remains active for the duration of your test run, this is the perfect way to implement object sharing in your test suite. Making the ITestContext available in your test methods is easy: just pass it as a parameter to your test method (we’ll see an example further down).

REST Assured authentication options
As you might have read in one of my previous blog posts, REST Assured is a Java library that allows you to write and execute readable tests for RESTful web services. Since we’re talking about secured APIs here, it’s good to know that REST Assured supports the following authentication mechanisms:

  • Basic
  • Digest
  • OAuth (version 1 and 2)
  • Form

In the examples in this post, we’ll take a closer look at both Basic authentication (for requesting an OAuth token) and OAuth2 authentication (for invoking secured web service operations) in REST Assured.

The PayPal sandbox API
To illustrate the concepts introduced above I chose to use the PayPal sandbox API. This is a sandbox version of the ‘live’ PayPal API that can be used to test applications that integrate with PayPal, as well as to goof around. It’s free to use for anybody that has an active PayPal account. You can find all documentation on the API here.

Retrieving an Oauth2 access token
The first step – after creating the necessary test accounts in the sandbox environment – is to construct a call in REST Assured that retrieves an OAuth2 authentication token from the PayPal web service. This request uses basic authentication and looks like this:

@BeforeSuite
public void requestToken(ITestContext context) {

	String response =
			given().
				parameters("grant_type","client_credentials").
				auth().
				preemptive().
				basic("client_id","secret").
			when().
				post("https://api.sandbox.paypal.com/v1/oauth2/token").
				asString();
}

The actual values for client_id and secret are specific to the PayPal sandbox account. Note that we have stored the JSON response as a string. This makes it easier to parse it, as we will see in a moment. The response to this request contains our OAuth2 authentication token:

Our OAuth2 access token

In order to store this token for use in our actual tests, we need to extract it from the response and store it in the TestNG ITestContext:

JsonPath jsonPath = new JsonPath(response);

String accessToken = jsonPath.getString("access_token");
		
context.setAttribute("accessToken", accessToken);

System.out.println("Access token: " + context.getAttribute("accessToken"));

The System.out.println output shows us we have successfully stored the OAuth2 access token in the ITestContext:

Access token has been stored in the ITestContext

Using the OAuth2 access token in your tests
Next, we want to use the previously stored token in subsequent API calls that require OAuth2 authentication. This is fairly straightforward: see for example this test that verifies that no payments have been made for the current test account:

@Test
public void checkNumberOfAssociatedPaymentsIsEqualToZero(ITestContext context) {

	given().
		contentType("application/json").
		auth().
		oauth2(context.getAttribute("accessToken").toString()).
	when().
		get("https://api.sandbox.paypal.com/v1/payments/payment/").
	then().
		assertThat().
		body("count", equalTo(0));
}

Note the use of context.getAttribute() to retrieve the token from the ITestContext. This test passes, which not only tells us that no payments have yet been made by this account, but also that our authentication worked as expected (otherwise, we would have received an authentication error).

Download an example project
The Maven project containing all code from this post can be downloaded here.

An introduction to property-based testing with JUnit-Quickcheck

Even though I consider having an extensive set of unit and unit integration tests a good thing in almost any case, there is still a fundamental shortcoming with automated checks that I feel should be addressed more often. Almost all checks I come across are very much example-based, meaning that only a single combination of input values is being checked. This goes for unit-, API-level as well as UI-driven tests, by the way. Of course, this is already far better than not checking anything at all, but how do you make sure that your check passes (and thus your application works for all, or at least for a considerable subset of all possible input parameter combinations?

In most cases, it is computationally impossible to perform a check for every combination of all possible input values (have you ever tried to work out what all possible values for a single string input parameter are?), so we need an approach that is able to generate a lot of (preferably random) input values that satisfy a set of predicates, and subsequently verify whether the check holds for all these input values and value combinations. Enter property-based testing (see this post on Jessica Kerr’s blog for an introduction to the concept).

What is property based testing?
In short, property-based testing pretty much exactly addresses the problem described above: based on an existing check, randomly generate a lot of input parameter combinations that satisfy predefined properties and see if the check passes for each of these combinations. This includes using negative and empty parameter values and other edge cases, all to try and break the system (or prove its robustness, if you want to look at it from the positive side…). You can do this either manually – although this does constitute a lot of effort – or, and given the nature of this blog this is of course the preferred method, use a tool to do the menial work for you.

Why should you use property based testing?
Again, the answer is in the first paragraph: because property-based testing will give you far more information about the functional correctness and the robustness of your software than executing mere example-based checks. Especially when combined with mutation testing, property-based testing you will likely have you end up with a far more powerful and effective unit- and unit integration test suite.

Tool: JUnit-Quickcheck
So, what tools are available for property-based testing? The archetypical tool is QuickCheck for the Haskell language. Most other property-based testing tools are somehow derived from QuickCheck. This is also the case for JUnit-Quickcheck, the tool that will be used in the rest of this blog post. As the name suggests, it is a tool for the Java language, based on JUnit.

Our system under test: the Calculator class
For the examples in this blog post, we will use the same Calculator class that was used when we talked about mutation testing. In particular, we are going to perform property-based testing on the add() method of this simple calculator:

public class Calculator {

	int valueDisplayed;

	public Calculator() {
		this.valueDisplayed = 0;
	}

	public void add(int x) {
		this.valueDisplayed += x;
	}

	public int getResult() {
		return this.valueDisplayed;
	}
}

An example-based unit test for the add() method of this admittedly very basic calculator could look like this:

@Test
public void testAdditionExampleBased() {
		
	Calculator calculator = new Calculator();
	calculator.add(2);
	assertEquals(calculator.getResult(), 2);		
}

Using JUnit-Quickcheck, we can replace this example-based unit test to a property-based test as follows:

@Property(trials = 5)
public void testAddition(int number) {
		
	System.out.println("Generated number for testAddition: " + number);
		
	Calculator calculator = new Calculator();
	calculator.add(number);
	assertEquals(calculator.getResult(), number);
}

The @Property annotation defines this test as a property-based test, while the @RunWith class-level annotation defines that these tests are to be run using JUnit-Quickcheck. Note that you can mix example-based and property-based tests in the same test class. As long as you run your test class using JUnit, all public void no-parameter methods annotated with @Test will be run just as plain JUnit would do, while all tests annotated with @Property will be run as property-based tests.

The trials = 5 attribute tells JUnit-Quickcheck to generate 5 random parameter values (also known as samples). Default is 100. The System.out.println call is used to write the generated parameter value to the console, purely for demonstrational purposes:

Output of basic JUnit-Quickcheck property

As you can see, JUnit-Quickcheck randomly generated integer values and performed the check using each of these values. The property passes, telling us that our add() method seems to work quite well, even with quite large or negative integers. This is information you wouldn’t get when using purely example-based tests. Sweet!

Constraining property values
Purely random integers seem to work well for our simple add() method, but obviously there are cases where you want to define some sort of constraints on the values generated by your property-based testing tool. JUnit-Quickcheck provides a number of options to do so:

1. Using the JUnit Assume class
Using Assume, you can define assumptions on the values generated by JUnit-Quickcheck. For example, if you only want to test your add() method using positive integers, you could do this:

@Property(trials = 5)
public void testAdditionUsingAssume(int number) {
		
	assumeThat(number, greaterThan(0));
		
	System.out.println("Generated number for testAdditionUsingAssume: " + number);
		
	Calculator calculator = new Calculator();
	calculator.add(number);
	assertEquals(calculator.getResult(), number);
}

When you run these tests, you can see that values generated by JUnit-Quickcheck that do not satisfy the assumption (in this case, three of them) are simply discarded:

Output of JUnit-Quickcheck using Assume

2. Using the @InRange annotation With this annotation, you can actually constrain the values that are generated by JUnit-Quickcheck:

@Property(trials = 5)
public void testAdditionUsingInRange(@InRange(minInt = 0) int number) {
		
	System.out.println("Generated number for testAdditionUsingInRange: " + number);
		
	Calculator calculator = new Calculator();
	calculator.add(number);
	assertEquals(calculator.getResult(), number);
}

Contrary to the approach that uses the Assume class, where values generated by JUnit-Quickcheck are filtered post-generation, when you’re using the @InRange approach you will always end up with the required number of samples:

Output of JUnit-Quickcheck using InRange

3. Using constraint expressions
Here, just like when you’re using the Assume approach, values generated by JUnit-Quickcheck are filtered after generation by using a satisfies predicate:

@Property(trials = 5)
public void testAdditionUsingSatisfies(@When(satisfies = "#_ >= 0") int number) {
				
	System.out.println("Generated number for testAdditionUsingSatisfies: " + number);
		
	Calculator calculator = new Calculator();
	calculator.add(number);
	assertEquals(calculator.getResult(), number);
}

The difference is that when the discard ratio (the percentage of generated values that do not satisfy the constraints defined) exceeds a certain threshold (0.5 by default), the property fails:

Output of JUnit-Quickcheck using the satisfies predicate

Error generated by JUnit-Quickcheck when discard threshold is not met

Depending on your preferences and project requirements, you can select any of these strategies for constraining your property values.

Additional JUnit-Quickcheck features
JUnit-Quickcheck comes with some other useful features as well:

  • Fixing the seed used to generate the random property values. You can use this to have JUnit-Quickcheck generate the same values for each and every test run. You may want to use this feature when a property fails, so that you can test the property over and over again with the same set of generated values that caused the failure in the first place.
  • When a property fails for a given set of values, JUnit-Quickcheck will attempt to find smaller sets of values that also fail the property, a technique called shrinking. See the JUnit-Quickcheck documentation for an example.

Download an example project
You can download a Maven project containing the Calculator class and all JUnit-Quickcheck tests that have been demonstrated in this post here. Happy property-based testing!