Creating executable specifications with Spectrum

One of the most important features of a good set of automated tests is that they require a minimal amount of explanation and/or documentation. When I see someone else’s test, I’d like to be able to instantly see what its purpose is and how it’s constructed in terms of setting up > execution > verification (for example using given/when/then or arrange/act/assert). That’s one of the reasons that I’ve been using Cucumber (or SpecFlow, depending on the client) in several of my automated test solutions, even though the team wasn’t doing Behaviour Driven Development.

It’s always a good idea to look for alternatives, though. Last week I was made aware of Spectrum, which is, as creator Greg Haskins calls it “A BDD-style test runner for Java 8. Inspired by Jasmine, RSpec, and Cucumber.”. I’ve been working with Jasmine a little before, and even though it took me a while to get used to the syntax and lambda notation style, I liked the way it provided a way to document your tests directly in the code and produce human readable results. Since Spectrum is created as a Jasmine-like test runner for Java, plus it provides support for the (at least in the Java world) much more common Gherkin given/when/then syntax, I thought it’d be a good idea to check it out.

Spectrum is ‘just’ yet another Java library, so adding it to your project is a breeze when using Maven or Gradle. Note that since Spectrum uses lambda functions, it won’t work unless you’re using Java 8. Spectrum runs on top of JUnit, so you’ll need that too. It works with all kinds of assertion libraries, so if you’re partial to Hamcrest, for example, that’s no problem at all.

As I said, Spectrum basically supports two types of specs: the Jasmine-style describe/it specification and the Gherkin-style given/when/then specification using features and scenarios. Let’s take a quick look at the Jasmine-style specifications first. For this example, I’m resorting once again to REST Assured tests. I’d like to verify whether Max Verstappen is in the list of 2017 drivers in one test, and whether both Fernando Alonso and Lewis Hamilton are in that list too in another test. This is how that looks like with Spectrum:

@RunWith(Spectrum.class)
public class SpectrumTests {{

	describe("The list of drivers for the 2017 season", () -> {

		ValidatableResponse response = get("http://ergast.com/api/f1/2017/drivers.json").then();

		String listOfDriverIds = "MRData.DriverTable.Drivers.driverId";

		it("includes Max Verstappen", () -> {

			response.assertThat().body(listOfDriverIds, hasItem("max_verstappen"));
		});

		it("also includes Fernando Alonso and Lewis Hamilton", () -> {

			response.assertThat().body(listOfDriverIds, hasItems("alonso","hamilton"));
		});
	});
}}

Since Spectrum runs ‘on top of’ JUnit, executing this specification is a matter of running it as a JUnit test. This results in the following output:

Spectrum output for Jasmine-style specs

Besides this (admittedly quite straightforward) example, Spectrum also comes with support for setup (using beforeEach and beforeAll) and teardown (using afterEach and afterAll), focusing on or ignoring specific specs, tagging specs, and more. You can find the documentation here.

The other type of specification supported by Spectrum is the Gherkin syntax. Let’s say I want to recreate the same specifications as above in the given/when/then format. With Spectrum, that looks like this:

@RunWith(Spectrum.class)
public class SpectrumTestsGherkin {{

	feature("2017 driver list verification", () -> {

		scenario("Verify that max_verstappen is in the list of 2017 drivers", () -> {

			final Variable<String> endpoint = new Variable<>();
			final Variable<Response> response = new Variable<>();

			given("We have an endpoint that gives us the list of 2017 drivers", () -> {
				
				endpoint.set("http://ergast.com/api/f1/2017/drivers.json");
			});

			when("we retrieve the list from that endpoint", () -> {
				
				response.set(get(endpoint.get()));
			});
			then("max_verstappen is in the driver list", () -> {
				
				response.get().then().assertThat().body("MRData.DriverTable.Drivers.driverId", hasItem("max_verstappen"));
			});
		});

		scenarioOutline("Verify that there are also some other people in the list of 2017 drivers", (driver) -> {

			final Variable<String> endpoint = new Variable<>();
			final Variable<Response> response = new Variable<>();

			given("We have an endpoint that gives us the list of 2017 drivers", () -> {
				
				endpoint.set("http://ergast.com/api/f1/2017/drivers.json");
			});

			when("we retrieve the list from that endpoint", () -> {
				
				response.set(get(endpoint.get()));
			});
			then(driver + " is in the driver list", () -> {
				
				response.get().then().assertThat().body("MRData.DriverTable.Drivers.driverId", hasItem(driver));
			});
		},

		withExamples(
				example("hamilton"),
				example("alonso"),
				example("vettel")
			)
		);
	});
}}

Running this spec shows that it does work indeed:

Spectrum output for Gherkin-style specs

There are two things that are fundamentally different from using the Jasmine-style syntax (the rest is ‘just’ syntactical):

  • Support for scenario outlines enables you to create data driven tests easily. Maybe this can be done too using the Jasmine syntax, but I haven’t figured it out so far.
  • If you want to pass variables between the given, the when and the then steps you’ll need to do so by using the Variable construct. This works with the Jasmine-style syntax too, but you’ll likely need to use it more in the Gherkin case (since ‘given/when/then’ are three steps, where ‘it’ is just one). When your tests get larger and more complex, having to use get() and set() every time you want to access or assign a variable might get cumbersome.

Having said that, I think Spectrum is a good addition to the test runner / BDD-supporting tool set available for Java, and something that you might want to consider using for your current (or next) test automation project. After all, any library or tool that makes your tests and/or test results more readable is worth taking note of. Right?

You can find a small Maven project containing the examples featured in this blog post here.

Data driven JavaScript tests using Jasmine

In my previous post I introduced Jasmine as a tool for testing JavaScript code. Now that I’ve had some more time to play around with it, combining it with Protractor for user interface-driven tests for one of my client’s websites, I’ve noticed that one feature was sorely missing: built-in support for data driven testing. All BDD tools (or tools that say they support BDD) that I have worked with so far, such as Cucumber, SpecFlow and JGiven, support the creation of data driven scenarios. However, Jasmine doesn’t, at least not as far as I have seen in the limited time I have been using it so far.

As I think data driven testing is key when creating maintainable and reusable scenarios, I fired up Google to check whether someone had already found a solution. From what I’ve read, there are (at least) two different ways to do data-driven testing with Jasmine. Let’s have a look at both.

Specifying your test data in an array variable
I found the first approach in the comments of this StackOverflow post. The comment submitted by peterhendrick suggests the use of a JSON file containing the test data. I slightly simplified this to use a simple array of key-value pairs and have the Jasmine spec loop through the array. You can simply populate this array any way you like, for example from an external .json file as suggested, but in any other way as well. In this example, I hard-coded it in the spec itself:

describe("Calculator (data driven tests with test data specified in array)", function() {
  var calc;
  var testdata = [{"x":1,"y":1,"sum":2},{"x":2,"y":3,"sum":5},{"x":4,"y":2,"sum":6}];
  
  beforeEach(function() {
    calc = new Calculator();
  });

	for(var i in testdata) {
		it("should be able to add two numbers, using (" + testdata[i].x + "," + testdata[i].y + "," + testdata[i].sum + ")", function() {
			
			expect(calc.add(testdata[i].x, testdata[i].y)).toEqual(testdata[i].sum);
		});	
	};
});

I particularly like this example because it enables you to add the actual values used to the test output. You can see this when you run your spec:

Test results from driving test data through an array

Data driven testing with a custom using function
This blog post from JP Castro presents an alternative way to do data driven testing in Jasmine. His solution seemed easy enough for me to apply directly to my own specs. Let’s see what it looks like.

The basic idea behind his solution is to wrap each it block in a spec with a using block that passes the parameter values to the it block:

describe("Calculator (data driven tests with using() function)", function() {
  var calc;
  
  beforeEach(function() {
    calc = new Calculator();
  });

  using('parameters',[[1,1,2],[2,3,5],[4,2,6]], function(parameters){
	it("should be able to add two numbers, using (" + parameters[0] + "," + parameters[1] + "," + parameters[2] + ")", function() {
		expect(calc.add(parameters[0],parameters[1])).toEqual(parameters[2]);
	});
  });
});

The using function he presents is implemented as follows:

function using(name, values, func) {
	for(var i = 0, count = values.length; i < count; i++) {
		if(Object.prototype.toString.call(values[i]) !== '[Object Array]') {
			values[i] = [values[i]];
		}
		func.apply(this,values[i]);
	}
}

When we run the spec above, the output looks like this:

Test results from driving test data through a using function

So, now we have two simple (and admittedly similar) workarounds for what I think is still a fundamental shortcoming in Jasmine. The code demonstrated in this blog post can be downloaded here.

Getting started with JavaScript testing using Jasmine

New project, new environment, new tools to explore! I just started working as a freelance test engineer on a new project involving some of Holland’s most visited job sites. An environment where a lot of PHP and JavaScript is used, as opposed to the familiar Java and C#. This also means I get to learn a lot of new stuff (I’ve not worked much in web development projects before) and a lot of new and interesting test automation tools and concepts to explore and apply. Which I can then use to bore you.

The first tool I get to explore is Jasmine, a BDD-style tool for testing JavaScript code. As with most other BDD tools, it’s only truly a BDD tool if you use it as such and I’m not yet convinced that we’re doing that right in this project. However, Jasmine offers enough interesting features for a blog post, so here we go..

What is Jasmine and where can I get it?
Simply put, Jasmine is a stand-alone JavaScript testing tool that supports writing your tests in a BDD-like manner. It does not require a DOM and therefore a browser to run, which makes tests run incredibly fast. Its syntax is also pretty clear and simple, as we will see in this post.

You can download the latest Jasmine release from here, after which you can unzip it. The package contains the Jasmine source (also written in JavaScript) and some sample code and tests.

The code to be tested
My example code that will be tested is a simple Car object that supports a two methods, one to add a driver to a car and one to start driving the car:

function Car() {
}

Car.prototype.addDriver = function() {
  if (this.hasDriver) {
	  throw new Error("Someone's already driving this car");
  }
  if (this.isDriving) {
	  throw new Error("Can't add a driver when the car is driving");
  }
  this.hasDriver = true;
};

Car.prototype.startDriving = function() {
	this.isDriving = true;
}

A first Jasmine test
Let’s say we want to write a simple test to see whether we can actually add a driver to a car. In Jasmine, this is done as follows:

describe("A Car", function() {

  var car;
  var person;

  beforeEach(function() {
    car = new Car();
    person = new Person();
  });

  it("should be able to be assigned a driver", function() {
	car.addDriver(person);
	expect(car.hasDriver).toBeTruthy();
  });
});

The describe keyword starts a new test suite and takes two parameters: the first is a string providing a description for the test suite, the second is a method that implements the test suite. A test suite consists of a number of tests (or specs in Jasmine terminology), each defined using the it keyword. Just as with Java testing frameworks such as JUnit and TestNG, you have the option to perform setup and teardown actions, in this case using beforeEach.

Assertions are performed using the expect keyword and a matcher, in this case toBeTruthy(). This matcher checks whether the value of a variable evaluates to true (note that values such as 1 and ‘test’ also evaluate to true in JavaScript). Jasmine comes with a rich set of matchers, which are described here.

Running your tests
Running Jasmine tests can be done by creating an HTML file that includes both the source code and the specs files in the and opening it in a browser. The HTML runner file for my demo project looks like this:

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>Jasmine Car Demo Spec Runner v2.4.1</title>

  <link rel="shortcut icon" type="image/png" href="lib/jasmine-2.4.1/jasmine_favicon.png">
  <link rel="stylesheet" href="lib/jasmine-2.4.1/jasmine.css">

  <!-- include Jasmine -->
  <script src="lib/jasmine-2.4.1/jasmine.js"></script>
  <script src="lib/jasmine-2.4.1/jasmine-html.js"></script>
  <script src="lib/jasmine-2.4.1/boot.js"></script>

  <!-- include source files -->
  <script src="src/Car.js"></script>
  <script src="src/Person.js"></script>

  <!-- include spec files -->
  <script src="spec/SpecHelper.js"></script>
  <script src="spec/CarSpec.js"></script>
</head>
<body/>
</html>

When we open this HTML file in our browser, the specs are run and we can see the following result:

Results of first Jasmine test run

Hooray!

By the way, if the assertion would fail, the Jasmine output would look like this:

Example of a failing Jasmine check

Checking for expected errors
Returning to the code for the Car object, we can see that some methods throw errors. If you want to check that these errors are indeed thrown, you can do that in Jasmine like this:

it("should throw an error when assigned a second driver", function() {
  car.addDriver(person);
  
  expect(function() {
	  car.addDriver(person);
  }).toThrowError("Someone's already driving this car");
});

Adding and using custom matchers
Even though Jasmine includes a variety of matchers to perform all kinds of checks, sometimes you might have to perform very specific checks not covered by one of the standard matchers. In this case, it’s also possible to write a custom matcher (which is called toBeDriving in this case):

beforeEach(function () {
  jasmine.addMatchers({
    toBeDriving: function () {
      return {
        compare: function (actual, expected) {
          var car = actual;

          return {
            pass: car.isDriving
          };
        }
      };
    }
  });
});

This custom matcher can then be used in Jasmine specs just like any other matcher:

it("should be able to start driving", function() {
  car.startDriving();
  expect(car).toBeDriving();		
});

Jasmine offers lots more features, such as the ability to write custom reporters. Check it out!

The example project I have used in this blog post can be downloaded here. It includes Jasmine 2.4.1 and a working HTML runner, so you can start experimenting with Jasmine right away.