An introduction to mutation testing and PIT

This blog post will cover the basics of the concept of mutation testing. I have been made aware of mutation testing only recently, but I have discovered it’s a very powerful and interesting technique for:

  • analysis and improvement of unit tests
  • detection of dead code in your application

Two things that are always worth taking a look at if you ask me. I will illustrate the mutation testing concept using a tool called PIT and a simple piece of code and accompanying set of unit tests.

What is mutation testing?
From Wikipedia:

Mutation testing is used to design new software tests and evaluate the quality of existing software tests. Mutation testing involves modifying a program in small ways. Each mutated version is called a mutant and tests detect and reject mutants by causing the behavior of the original version to differ from the mutant. This is called killing the mutant. Test suites are measured by the percentage of mutants that they kill.

In other words, mutation testing is a technique that allows you to evaluate not only the percentage of code that is executed when running your tests (i.e., code coverage), but also the ability of your tests to detect any defects in the executed code. This makes mutation testing a very powerful and very useful technique I think anyone involved in software development and testing should at least be aware of.

Introducing PIT
I will try and illustrate the power of mutation testing using PIT, a Java mutation test tool which can be downloaded here. I chose PIT over other available mutation test tools mainly because of its ease of installation and use.

Assuming you’re also using Maven, you can configure your Java project for mutation testing using PIT by adding the following to your pom.xml:

<build>
	<plugins>
		<plugin>
			<groupId>org.pitest</groupId>
			<artifactId>pitest-maven</artifactId>
			<version>PIT-VERSION</version>
			<configuration>
				<targetClasses>
					<param>package.root.containing.classes.to.mutate*</param>
				</targetClasses>
				<targetTests>
					<param>package.root.containing.test.classes*</param>
				</targetTests>
			</configuration>
		</plugin>
	</plugins>
</build>

Simply replace the package locators with those appropriate for your project and be sure not to forget the asterisk at the end. Also replace PIT-VERSION with the PIT version you want to use (the latest is 1.1.4 at the moment of writing this blog post) and you’re good to go.

The code class and test class to be subjected to mutation testing
I created a very simple Calculator class that, you guessed it, performs simple arithmetic on integers. My calculator only does addition, subtraction and power calculations:

public class Calculator {

	int valueDisplayed;

	public Calculator() {
		this.valueDisplayed = 0;
	}
	
	public Calculator(int initialValue) {
		this.valueDisplayed = initialValue;
	}

	public void add(int x) {
		this.valueDisplayed += x;
	}
	
	public void subtract(int x) {
		this.valueDisplayed -= x;
	}
	
	public void power(int x) {
		this.valueDisplayed = (int) Math.pow(this.valueDisplayed, x);
	}

	public int getResult() {
		return this.valueDisplayed;
	}
	
	public void set(int x) {
		this.valueDisplayed = x;
	}

	public boolean setConditional(int x, boolean yesOrNo) {
		if(yesOrNo) {
			set(x);
			return true;
		} else {
			return false;
		}
	}
}

To test the calculator, I have created a couple of TestNG unit tests that call the various methods my calculator supports. Note that PIT supports both JUnit and TestNG.

public class CalculatorTest {
	
	@Test
	public void testAddition() {
		
		Calculator calculator = new Calculator();
		calculator.add(2);
		Assert.assertEquals(calculator.getResult(), 2);
	}
	
	@Test
	public void testPower() {
		
		Calculator calculator = new Calculator(2);
		calculator.power(3);
		Assert.assertEquals(calculator.getResult(), 8);
	}
	
	@Test
	public void testConditionalSetTrue() {
		
		Calculator calculator = new Calculator();
		Assert.assertEquals(calculator.setConditional(2, true), true);
	}
	
	@Test
	public void testConditionalSetFalse() {
		
		Calculator calculator = new Calculator();
		Assert.assertEquals(calculator.setConditional(3, false), false);
	}
}

To illustrate the capabilities of PIT and mutation testing in general, I ‘forgot’ to include a test for the subtract() method. Also, I created what is known as a ‘weak test’: a test that passes but doesn’t check whether all code is actually called (in this case, no check is done to see whether set() is called when calling setConditional()). Now, when we run PIT on our code and test classes using:

mvn org.pitest:pitest-maven:mutationCoverage

an HTML report is generated displaying our mutation test results:

The report as generated by PIT

When we drill down to our Calculator class we can see the modifications that have been made by PIT and the effect it had on our tests:

Class-level details of the PIT mutation test results

This clearly shows that our unit test suite has room for improvement:

  • The fact that subtract() is never called in our test suite (i.e., our code coverage can be improved) is detected
  • The fact that the call to set() can be removed from the code without our test results being affected (i.e., our tests are lacking defect detection power) is detected

These holes in our test coverage and test effectiveness might go unnoticed for a long time, especially since all tests pass when run using TestNG. This goes especially for the second flaw as a regular code coverage tool would not pick this up: the call to set() is made after all, but it does not have any effect on the outcome of our tests!

Additional PIT features
The PIT documentation discusses a lot of features that make your mutation testing efforts even more powerful. You can configure the set of mutators used to tailor the result set to your needs, you can use mutation filters to filter out any unwanted results, and much more. However, even in the default configuration, using PIT (or potentially any other mutation testing tool as listed here will tell you a lot about the quality of your unit testing efforts.

Removing dead code from your codebase based on mutation test results
Apart from evaluating the quality of your unit tests, mutation test results can also give you insight into which parts of your application code are never executed (dead code). Consider the call to the set() method in the example above. The mutation test results indicated that this call could be removed without the results of the unit test being altered. Now, in our case it is pretty obvious that this indicates a lack of coverage in our unit tests (if you want to set the Calculator value, you’d better call the set() method), but it isn’t hard to imagine a situation where such a method call can be removed without any further consequences. In this case, the results of the mutation tests will point you to potentially dead code that might be a candidate for refactoring or removal. Thanks go to Markus Schirp for pointing out this huge advantage of mutation testing to me on Twitter.

Example project
The Maven project that was used to generate the results demonstrated in this post can be downloaded here. You can simply import this project and run

mvn org.pitest:pitest-maven:mutationCoverage

to recreate my test results and review the generated report. This will serve as a good starting point for your further exploration of the power and possibilities of mutation testing.

Using the LoadableComponent pattern for better Page Object handling in Selenium

I’m sure that by now, everybody who is at least remotely involved in the creation of automated user interface tests using Selenium WebDriver is aware of the Page Object pattern (read more about it here and here) for introducing modularity and reusability in their automated tests. In this post I will introduce the LoadableComponent pattern, an extension to this pattern that standardizes handling the loading and verifying loading status of these page objects.

Why use the LoadableComponent pattern?
Out of the box, Selenium WebDriver does a good job of determining whether an HTML page has been loaded by using the document.readyState property, which is part of the W3C WebDriver specification. However, this property alone is not always enough to assert that all dynamic content on your page has been fully loaded and that all elements required in your test script are present on the page. This is especially the case when your web page uses a JavaScript-heavy framework such as the popular AngularJS or KnockoutJS frameworks.

So, even when Selenium tells you the page is loaded and continues to execute the next step in your test, this step might fail because the actual element needed for that step is not yet visible, clickable or otherwise ready. You can solve this problem to an extent by using suitable wrapper methods for the standard Selenium API methods, but wouldn’t it be nice to enhance our Page Objects with a generic approach to evaluate page load status? This is where the LoadableComponent pattern comes in.

Introducing the LoadableComponent pattern
To explain the concept of the LoadableComponent pattern, we will again turn to the ParaBank application. LoadableComponent is a base class in Selenium, which means that you can simply define your Page Objects as an extension of the LoadableComponent class. So, for example, we can simply define a LoginPage object as follows:

public class LoginPage extends LoadableComponent<LoginPage> {
  // class implementation will be explained later 
}

This does nothing more than defining this class as a LoadableComponent that loads the LoginPage page.

Now, by having our Page Objects extend the LoadableComponent base class, we need to implement two new methods, load() and isLoaded() (note that in C#, these are called ExecuteLoad() and EvaluateLoadedStatus() for some reason). These methods provide the added value of using the LoadableComponent pattern. The load() method contains the code that is executed to navigate to the page, while the isLoaded() method is used to evaluate whether we are on the correct page and whether page loading has finished successfully. Using LoadableComponent, our LoginPage class now looks like this:

public class LoginPage extends LoadableComponent<LoginPage> {
	
	private WebDriver driver;
	
	public LoginPage(WebDriver driver) {
		
		this.driver = driver;
		driver.get("http://parabank.parasoft.com");
	}
	
	@Override
	protected void isLoaded() throws Error {
		
		if(!PageLoad.myElementIsClickable(this.driver, By.name("username"))) {
			throw new Error("LoginPage was not successfully loaded");
		}
	}

	@Override
	protected void load() {		
	}
}

Note the use of a custom wrapper boolean method myElementIsClickable() to determine whether the page has loaded successfully. It can be used as a generic way to specify which element must be present for any given Page Object in order to consider it fully loaded:

public static boolean myElementIsClickable (WebDriver driver, By by) {
		
	try
	{
		new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(by));
	}
	catch (WebDriverException ex)
	{
		return false;
	}
	return true;		
}

Also, for most pages I don’t actually put code inside the body of the load() method. This is because in a typical application, most Page Objects are only accessed by means of navigation via other Page Objects instead of being accessed directly. Even for the LoginPage I have included the navigate() call in the constructor, because in this way we do not need to wait for the timeout in isLoaded() to be exceeded before the navigate() method is called from within load(). The example project I will link to at the end of this post shows you how I implemented the LoadableComponent pattern in my latest project.

So, now that we’ve implemented our Page Objects as LoadableComponents, we can use it in our tests simply by doing this:

new LoginPage(driver).get();

The get() method is implemented as follows:

public T get() {
	try {
		isLoaded();
		return (T) this;
	} catch (Error e) {
		load();
	}
 
	isLoaded();
 
	return (T) this;
}

In plain English: it calls isLoaded() first to see whether the page is loaded. If this is not the case, it calls load() to load the page. Afterwards, it calls isLoaded() again to see if the page is now successfully loaded.

Leveraging LoadableComponents in your tests
To ensure in your tests that your Page Objects are loaded before moving on, you can simply call get() each time a Page Object is instantiated:

@Test
public void validLoginTest() {

	// Load login page
	LoginPage loginPage = new LoginPage(driver).get();

	// Log in using valid credentials
	HomePage homePage = loginPage.correctLogin("john", "demo").get();

	// Load home page and check welcome text
	Assert.assertEquals("Welcome text is correct","Welcome John Smith", homePage.getWelcomeString());
}

The assertion in isLoaded() will fail if MyElementIsClickable() returns false, leading to the test as a whole to fail:

LoadableComponent error

Enhancing the LoadableComponent class
Finally, another interesting use of the LoadableComponent pattern is writing your own implementation of the class. In that way you can, for example, generate specific logging to the console or to an ExtentReports report in case you use it.

The source code for the original LoadableComponent implementation can be found here. A simple addition is writing the error message that is thrown by isLoaded() (which in turn is the error message thrown by the performed assertion) to the console:

public abstract class CustomLoadableComponent<T extends CustomLoadableComponent<T>> {

	@SuppressWarnings("unchecked")
	public T get() {
		try {
			isLoaded();
			return (T) this;
		} catch (Error e) {
			// This is the extra line of code 
			System.out.println("Error encountered during page load: " + e.getMessage());
			load();
		}

		isLoaded();

		return (T) this;
	}

	protected abstract void load();

	protected abstract void isLoaded() throws Error;
}

Now you can simply define your Page Objects as an extension of the CustomLoadableComponent class:

public class LoginPage extends CustomLoadableComponent<LoginPage> {
  // class implementation remains the same 
}

When we run our tests and encounter an error, we see that the error message is written to the console:
Custom LoadableComponent output

A working Eclipse project (created using Maven) containing all of the code samples included in this post can be downloaded here. Just run the tests in the tests package to see for yourself how it works.

Using wrapper methods for better error handling in Selenium

Frequent users of Selenium WebDriver might have come across the problem that when they’re using Selenium for testing responsive, dynamic web applications, timing and synchronization can be a major hurdle in creating useful and reliable automated tests. In my current project, I’m creating WebDriver-based automated tests for a web application that heavily relies on Javascript for page rendering and content retrieval and display, which makes waiting for the right element to be visible, clickable or whatever state it needs to be in significantly less than trivial.

Using the standard Selenium methods such as click() and sendKeys() directly in my script often results in failures as a result of the weg page element being invisible, disabled or reloaded (this last example resulting in a StaleElementReferenceException). Even using implicit waits when creating the WebDriver object didn’t always lead to stable tests, and I refuse to use Thread.sleep() (and so should you!). Also, I didn’t want to use individual WebDriverWait calls for every single object that needed to be waited on, since that introduces a lot of extra code that needs to be maintained. So I knew I had to do something more intelligent for my tests to be reliable – and therefore valuable as opposed to a waste of time and money.

Wrapper methods
The solution to this problem lies in using wrapper methods for the standard Selenium methods. So instead of doing this every time I need to perform a click:

(new WebDriverWait(driver, 10)).until(ExpectedConditions.elementToBeClickable(By.id("loginButton")));
driver.findElement(By.id("loginButton")).click();

I have created a wrapper method click() in a MyElements class that looks like this:

public static void click(WebDriver driver, By by) {
	(new WebDriverWait(driver, 10)).until(ExpectedConditions.elementToBeClickable(by));
	driver.findElement(by).click();
}

Of course, the 10 second timeout is arbitrary and it’s better to replace this with some constant value. Now, every time I want to perform a click in my test I can simply call:

MyElements.click(driver, By.id("loginButton");

which automatically performs a WebDriverWait, resulting in much stabler, better readable and maintainable scripts.

Extending your wrapper methods: error handling
Using wrapper methods for Selenium calls has the additional benefit of making error handling much more generic as well. For example, if you often encounter a StaleElementReferenceException (which those of you writing tests for responsive and dynamic web applications might be all too familiar with), you can simply handle this in your wrapper method and be done with it once and for all:

public static void click(WebDriver driver, By by) {
	try {
		(new WebDriverWait(driver, 10)).until(ExpectedConditions.elementToBeClickable(by));
		driver.findElement(by).click();
	catch (StaleElementReferenceException sere) {
		// simply retry finding the element in the refreshed DOM
		driver.findElement(by).click();
	}
}

This is essentially my own version of this excellent solution for handling a StaleElementReferenceException. By using wrapper methods, you can easily handle any type of exception in its own specific way, which will improve your tests significantly.

Extending your wrapper methods: logging
Your wrapper methods also allow for better logging capabilities. For example, if you’re using the ExtentReports library for creating HTML reports for Selenium tests (as I do in my current project), you can create a log entry every time an object is not clickable after the WebDriverWait times out:

public static void click(WebDriver driver, By by) {
	try {
		(new WebDriverWait(driver, 10)).until(ExpectedConditions.elementToBeClickable(by));
		driver.findElement(by).click();
	catch (StaleElementReferenceException sere) {
		// simply retry finding the element in the refreshed DOM
		driver.findElement(by).click();
	}
	catch (TimeoutException toe) {
		test.log(logStatus.Error, "Element identified by " + by.toString() + " was not clickable after 10 seconds");
	}
}

Here, test is the ExtentTest object representing the log for the current test. See how easy this is?

Other possible benefits
There’s a couple of other benefits to using these wrapper methods, some of which I use in my current project as well, such as:

  • Automatically failing the JUnit / TestNG (or in my case NUnit) test whenever a specific error occurs by including an Assert.Fail();
  • Restoring or recreating specific conditions or a specific application state after an error occurs to prepare for the next test (e.g. ending the current user session)
  • Automatically restarting a test in case of a specific error

And I’m sure there are many more.

Wrapping this up (pun intended!), using these wrapper methods for standard Selenium calls can be very beneficial to your Selenium experience and those of your clients or employers. They certainly saved me a lot of unnecessary code and frustration over brittle tests..