Creating executable specifications with Spectrum

One of the most important features of a good set of automated tests is that they require a minimal amount of explanation and/or documentation. When I see someone else’s test, I’d like to be able to instantly see what its purpose is and how it’s constructed in terms of setting up > execution > verification (for example using given/when/then or arrange/act/assert). That’s one of the reasons that I’ve been using Cucumber (or SpecFlow, depending on the client) in several of my automated test solutions, even though the team wasn’t doing Behaviour Driven Development.

It’s always a good idea to look for alternatives, though. Last week I was made aware of Spectrum, which is, as creator Greg Haskins calls it “A BDD-style test runner for Java 8. Inspired by Jasmine, RSpec, and Cucumber.”. I’ve been working with Jasmine a little before, and even though it took me a while to get used to the syntax and lambda notation style, I liked the way it provided a way to document your tests directly in the code and produce human readable results. Since Spectrum is created as a Jasmine-like test runner for Java, plus it provides support for the (at least in the Java world) much more common Gherkin given/when/then syntax, I thought it’d be a good idea to check it out.

Spectrum is ‘just’ yet another Java library, so adding it to your project is a breeze when using Maven or Gradle. Note that since Spectrum uses lambda functions, it won’t work unless you’re using Java 8. Spectrum runs on top of JUnit, so you’ll need that too. It works with all kinds of assertion libraries, so if you’re partial to Hamcrest, for example, that’s no problem at all.

As I said, Spectrum basically supports two types of specs: the Jasmine-style describe/it specification and the Gherkin-style given/when/then specification using features and scenarios. Let’s take a quick look at the Jasmine-style specifications first. For this example, I’m resorting once again to REST Assured tests. I’d like to verify whether Max Verstappen is in the list of 2017 drivers in one test, and whether both Fernando Alonso and Lewis Hamilton are in that list too in another test. This is how that looks like with Spectrum:

@RunWith(Spectrum.class)
public class SpectrumTests {{

	describe("The list of drivers for the 2017 season", () -> {

		ValidatableResponse response = get("http://ergast.com/api/f1/2017/drivers.json").then();

		String listOfDriverIds = "MRData.DriverTable.Drivers.driverId";

		it("includes Max Verstappen", () -> {

			response.assertThat().body(listOfDriverIds, hasItem("max_verstappen"));
		});

		it("also includes Fernando Alonso and Lewis Hamilton", () -> {

			response.assertThat().body(listOfDriverIds, hasItems("alonso","hamilton"));
		});
	});
}}

Since Spectrum runs ‘on top of’ JUnit, executing this specification is a matter of running it as a JUnit test. This results in the following output:

Spectrum output for Jasmine-style specs

Besides this (admittedly quite straightforward) example, Spectrum also comes with support for setup (using beforeEach and beforeAll) and teardown (using afterEach and afterAll), focusing on or ignoring specific specs, tagging specs, and more. You can find the documentation here.

The other type of specification supported by Spectrum is the Gherkin syntax. Let’s say I want to recreate the same specifications as above in the given/when/then format. With Spectrum, that looks like this:

@RunWith(Spectrum.class)
public class SpectrumTestsGherkin {{

	feature("2017 driver list verification", () -> {

		scenario("Verify that max_verstappen is in the list of 2017 drivers", () -> {

			final Variable<String> endpoint = new Variable<>();
			final Variable<Response> response = new Variable<>();

			given("We have an endpoint that gives us the list of 2017 drivers", () -> {
				
				endpoint.set("http://ergast.com/api/f1/2017/drivers.json");
			});

			when("we retrieve the list from that endpoint", () -> {
				
				response.set(get(endpoint.get()));
			});
			then("max_verstappen is in the driver list", () -> {
				
				response.get().then().assertThat().body("MRData.DriverTable.Drivers.driverId", hasItem("max_verstappen"));
			});
		});

		scenarioOutline("Verify that there are also some other people in the list of 2017 drivers", (driver) -> {

			final Variable<String> endpoint = new Variable<>();
			final Variable<Response> response = new Variable<>();

			given("We have an endpoint that gives us the list of 2017 drivers", () -> {
				
				endpoint.set("http://ergast.com/api/f1/2017/drivers.json");
			});

			when("we retrieve the list from that endpoint", () -> {
				
				response.set(get(endpoint.get()));
			});
			then(driver + " is in the driver list", () -> {
				
				response.get().then().assertThat().body("MRData.DriverTable.Drivers.driverId", hasItem(driver));
			});
		},

		withExamples(
				example("hamilton"),
				example("alonso"),
				example("vettel")
			)
		);
	});
}}

Running this spec shows that it does work indeed:

Spectrum output for Gherkin-style specs

There are two things that are fundamentally different from using the Jasmine-style syntax (the rest is ‘just’ syntactical):

  • Support for scenario outlines enables you to create data driven tests easily. Maybe this can be done too using the Jasmine syntax, but I haven’t figured it out so far.
  • If you want to pass variables between the given, the when and the then steps you’ll need to do so by using the Variable construct. This works with the Jasmine-style syntax too, but you’ll likely need to use it more in the Gherkin case (since ‘given/when/then’ are three steps, where ‘it’ is just one). When your tests get larger and more complex, having to use get() and set() every time you want to access or assign a variable might get cumbersome.

Having said that, I think Spectrum is a good addition to the test runner / BDD-supporting tool set available for Java, and something that you might want to consider using for your current (or next) test automation project. After all, any library or tool that makes your tests and/or test results more readable is worth taking note of. Right?

You can find a small Maven project containing the examples featured in this blog post here.

Getting started with JavaScript testing using Jasmine

New project, new environment, new tools to explore! I just started working as a freelance test engineer on a new project involving some of Holland’s most visited job sites. An environment where a lot of PHP and JavaScript is used, as opposed to the familiar Java and C#. This also means I get to learn a lot of new stuff (I’ve not worked much in web development projects before) and a lot of new and interesting test automation tools and concepts to explore and apply. Which I can then use to bore you.

The first tool I get to explore is Jasmine, a BDD-style tool for testing JavaScript code. As with most other BDD tools, it’s only truly a BDD tool if you use it as such and I’m not yet convinced that we’re doing that right in this project. However, Jasmine offers enough interesting features for a blog post, so here we go..

What is Jasmine and where can I get it?
Simply put, Jasmine is a stand-alone JavaScript testing tool that supports writing your tests in a BDD-like manner. It does not require a DOM and therefore a browser to run, which makes tests run incredibly fast. Its syntax is also pretty clear and simple, as we will see in this post.

You can download the latest Jasmine release from here, after which you can unzip it. The package contains the Jasmine source (also written in JavaScript) and some sample code and tests.

The code to be tested
My example code that will be tested is a simple Car object that supports a two methods, one to add a driver to a car and one to start driving the car:

function Car() {
}

Car.prototype.addDriver = function() {
  if (this.hasDriver) {
	  throw new Error("Someone's already driving this car");
  }
  if (this.isDriving) {
	  throw new Error("Can't add a driver when the car is driving");
  }
  this.hasDriver = true;
};

Car.prototype.startDriving = function() {
	this.isDriving = true;
}

A first Jasmine test
Let’s say we want to write a simple test to see whether we can actually add a driver to a car. In Jasmine, this is done as follows:

describe("A Car", function() {

  var car;
  var person;

  beforeEach(function() {
    car = new Car();
    person = new Person();
  });

  it("should be able to be assigned a driver", function() {
	car.addDriver(person);
	expect(car.hasDriver).toBeTruthy();
  });
});

The describe keyword starts a new test suite and takes two parameters: the first is a string providing a description for the test suite, the second is a method that implements the test suite. A test suite consists of a number of tests (or specs in Jasmine terminology), each defined using the it keyword. Just as with Java testing frameworks such as JUnit and TestNG, you have the option to perform setup and teardown actions, in this case using beforeEach.

Assertions are performed using the expect keyword and a matcher, in this case toBeTruthy(). This matcher checks whether the value of a variable evaluates to true (note that values such as 1 and ‘test’ also evaluate to true in JavaScript). Jasmine comes with a rich set of matchers, which are described here.

Running your tests
Running Jasmine tests can be done by creating an HTML file that includes both the source code and the specs files in the and opening it in a browser. The HTML runner file for my demo project looks like this:

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>Jasmine Car Demo Spec Runner v2.4.1</title>

  <link rel="shortcut icon" type="image/png" href="lib/jasmine-2.4.1/jasmine_favicon.png">
  <link rel="stylesheet" href="lib/jasmine-2.4.1/jasmine.css">

  <!-- include Jasmine -->
  <script src="lib/jasmine-2.4.1/jasmine.js"></script>
  <script src="lib/jasmine-2.4.1/jasmine-html.js"></script>
  <script src="lib/jasmine-2.4.1/boot.js"></script>

  <!-- include source files -->
  <script src="src/Car.js"></script>
  <script src="src/Person.js"></script>

  <!-- include spec files -->
  <script src="spec/SpecHelper.js"></script>
  <script src="spec/CarSpec.js"></script>
</head>
<body/>
</html>

When we open this HTML file in our browser, the specs are run and we can see the following result:

Results of first Jasmine test run

Hooray!

By the way, if the assertion would fail, the Jasmine output would look like this:

Example of a failing Jasmine check

Checking for expected errors
Returning to the code for the Car object, we can see that some methods throw errors. If you want to check that these errors are indeed thrown, you can do that in Jasmine like this:

it("should throw an error when assigned a second driver", function() {
  car.addDriver(person);
  
  expect(function() {
	  car.addDriver(person);
  }).toThrowError("Someone's already driving this car");
});

Adding and using custom matchers
Even though Jasmine includes a variety of matchers to perform all kinds of checks, sometimes you might have to perform very specific checks not covered by one of the standard matchers. In this case, it’s also possible to write a custom matcher (which is called toBeDriving in this case):

beforeEach(function () {
  jasmine.addMatchers({
    toBeDriving: function () {
      return {
        compare: function (actual, expected) {
          var car = actual;

          return {
            pass: car.isDriving
          };
        }
      };
    }
  });
});

This custom matcher can then be used in Jasmine specs just like any other matcher:

it("should be able to start driving", function() {
  car.startDriving();
  expect(car).toBeDriving();		
});

Jasmine offers lots more features, such as the ability to write custom reporters. Check it out!

The example project I have used in this blog post can be downloaded here. It includes Jasmine 2.4.1 and a working HTML runner, so you can start experimenting with Jasmine right away.

An introduction to property-based testing with JUnit-Quickcheck

Even though I consider having an extensive set of unit and unit integration tests a good thing in almost any case, there is still a fundamental shortcoming with automated checks that I feel should be addressed more often. Almost all checks I come across are very much example-based, meaning that only a single combination of input values is being checked. This goes for unit-, API-level as well as UI-driven tests, by the way. Of course, this is already far better than not checking anything at all, but how do you make sure that your check passes (and thus your application works for all, or at least for a considerable subset of all possible input parameter combinations?

In most cases, it is computationally impossible to perform a check for every combination of all possible input values (have you ever tried to work out what all possible values for a single string input parameter are?), so we need an approach that is able to generate a lot of (preferably random) input values that satisfy a set of predicates, and subsequently verify whether the check holds for all these input values and value combinations. Enter property-based testing (see this post on Jessica Kerr’s blog for an introduction to the concept).

What is property based testing?
In short, property-based testing pretty much exactly addresses the problem described above: based on an existing check, randomly generate a lot of input parameter combinations that satisfy predefined properties and see if the check passes for each of these combinations. This includes using negative and empty parameter values and other edge cases, all to try and break the system (or prove its robustness, if you want to look at it from the positive side…). You can do this either manually – although this does constitute a lot of effort – or, and given the nature of this blog this is of course the preferred method, use a tool to do the menial work for you.

Why should you use property based testing?
Again, the answer is in the first paragraph: because property-based testing will give you far more information about the functional correctness and the robustness of your software than executing mere example-based checks. Especially when combined with mutation testing, property-based testing you will likely have you end up with a far more powerful and effective unit- and unit integration test suite.

Tool: JUnit-Quickcheck
So, what tools are available for property-based testing? The archetypical tool is QuickCheck for the Haskell language. Most other property-based testing tools are somehow derived from QuickCheck. This is also the case for JUnit-Quickcheck, the tool that will be used in the rest of this blog post. As the name suggests, it is a tool for the Java language, based on JUnit.

Our system under test: the Calculator class
For the examples in this blog post, we will use the same Calculator class that was used when we talked about mutation testing. In particular, we are going to perform property-based testing on the add() method of this simple calculator:

public class Calculator {

	int valueDisplayed;

	public Calculator() {
		this.valueDisplayed = 0;
	}

	public void add(int x) {
		this.valueDisplayed += x;
	}

	public int getResult() {
		return this.valueDisplayed;
	}
}

An example-based unit test for the add() method of this admittedly very basic calculator could look like this:

@Test
public void testAdditionExampleBased() {
		
	Calculator calculator = new Calculator();
	calculator.add(2);
	assertEquals(calculator.getResult(), 2);		
}

Using JUnit-Quickcheck, we can replace this example-based unit test to a property-based test as follows:

@Property(trials = 5)
public void testAddition(int number) {
		
	System.out.println("Generated number for testAddition: " + number);
		
	Calculator calculator = new Calculator();
	calculator.add(number);
	assertEquals(calculator.getResult(), number);
}

The @Property annotation defines this test as a property-based test, while the @RunWith class-level annotation defines that these tests are to be run using JUnit-Quickcheck. Note that you can mix example-based and property-based tests in the same test class. As long as you run your test class using JUnit, all public void no-parameter methods annotated with @Test will be run just as plain JUnit would do, while all tests annotated with @Property will be run as property-based tests.

The trials = 5 attribute tells JUnit-Quickcheck to generate 5 random parameter values (also known as samples). Default is 100. The System.out.println call is used to write the generated parameter value to the console, purely for demonstrational purposes:

Output of basic JUnit-Quickcheck property

As you can see, JUnit-Quickcheck randomly generated integer values and performed the check using each of these values. The property passes, telling us that our add() method seems to work quite well, even with quite large or negative integers. This is information you wouldn’t get when using purely example-based tests. Sweet!

Constraining property values
Purely random integers seem to work well for our simple add() method, but obviously there are cases where you want to define some sort of constraints on the values generated by your property-based testing tool. JUnit-Quickcheck provides a number of options to do so:

1. Using the JUnit Assume class
Using Assume, you can define assumptions on the values generated by JUnit-Quickcheck. For example, if you only want to test your add() method using positive integers, you could do this:

@Property(trials = 5)
public void testAdditionUsingAssume(int number) {
		
	assumeThat(number, greaterThan(0));
		
	System.out.println("Generated number for testAdditionUsingAssume: " + number);
		
	Calculator calculator = new Calculator();
	calculator.add(number);
	assertEquals(calculator.getResult(), number);
}

When you run these tests, you can see that values generated by JUnit-Quickcheck that do not satisfy the assumption (in this case, three of them) are simply discarded:

Output of JUnit-Quickcheck using Assume

2. Using the @InRange annotation With this annotation, you can actually constrain the values that are generated by JUnit-Quickcheck:

@Property(trials = 5)
public void testAdditionUsingInRange(@InRange(minInt = 0) int number) {
		
	System.out.println("Generated number for testAdditionUsingInRange: " + number);
		
	Calculator calculator = new Calculator();
	calculator.add(number);
	assertEquals(calculator.getResult(), number);
}

Contrary to the approach that uses the Assume class, where values generated by JUnit-Quickcheck are filtered post-generation, when you’re using the @InRange approach you will always end up with the required number of samples:

Output of JUnit-Quickcheck using InRange

3. Using constraint expressions
Here, just like when you’re using the Assume approach, values generated by JUnit-Quickcheck are filtered after generation by using a satisfies predicate:

@Property(trials = 5)
public void testAdditionUsingSatisfies(@When(satisfies = "#_ >= 0") int number) {
				
	System.out.println("Generated number for testAdditionUsingSatisfies: " + number);
		
	Calculator calculator = new Calculator();
	calculator.add(number);
	assertEquals(calculator.getResult(), number);
}

The difference is that when the discard ratio (the percentage of generated values that do not satisfy the constraints defined) exceeds a certain threshold (0.5 by default), the property fails:

Output of JUnit-Quickcheck using the satisfies predicate

Error generated by JUnit-Quickcheck when discard threshold is not met

Depending on your preferences and project requirements, you can select any of these strategies for constraining your property values.

Additional JUnit-Quickcheck features
JUnit-Quickcheck comes with some other useful features as well:

  • Fixing the seed used to generate the random property values. You can use this to have JUnit-Quickcheck generate the same values for each and every test run. You may want to use this feature when a property fails, so that you can test the property over and over again with the same set of generated values that caused the failure in the first place.
  • When a property fails for a given set of values, JUnit-Quickcheck will attempt to find smaller sets of values that also fail the property, a technique called shrinking. See the JUnit-Quickcheck documentation for an example.

Download an example project
You can download a Maven project containing the Calculator class and all JUnit-Quickcheck tests that have been demonstrated in this post here. Happy property-based testing!