Creating executable specifications with Spectrum

One of the most important features of a good set of automated tests is that they require a minimal amount of explanation and/or documentation. When I see someone else’s test, I’d like to be able to instantly see what its purpose is and how it’s constructed in terms of setting up > execution > verification (for example using given/when/then or arrange/act/assert). That’s one of the reasons that I’ve been using Cucumber (or SpecFlow, depending on the client) in several of my automated test solutions, even though the team wasn’t doing Behaviour Driven Development.

It’s always a good idea to look for alternatives, though. Last week I was made aware of Spectrum, which is, as creator Greg Haskins calls it “A BDD-style test runner for Java 8. Inspired by Jasmine, RSpec, and Cucumber.”. I’ve been working with Jasmine a little before, and even though it took me a while to get used to the syntax and lambda notation style, I liked the way it provided a way to document your tests directly in the code and produce human readable results. Since Spectrum is created as a Jasmine-like test runner for Java, plus it provides support for the (at least in the Java world) much more common Gherkin given/when/then syntax, I thought it’d be a good idea to check it out.

Spectrum is ‘just’ yet another Java library, so adding it to your project is a breeze when using Maven or Gradle. Note that since Spectrum uses lambda functions, it won’t work unless you’re using Java 8. Spectrum runs on top of JUnit, so you’ll need that too. It works with all kinds of assertion libraries, so if you’re partial to Hamcrest, for example, that’s no problem at all.

As I said, Spectrum basically supports two types of specs: the Jasmine-style describe/it specification and the Gherkin-style given/when/then specification using features and scenarios. Let’s take a quick look at the Jasmine-style specifications first. For this example, I’m resorting once again to REST Assured tests. I’d like to verify whether Max Verstappen is in the list of 2017 drivers in one test, and whether both Fernando Alonso and Lewis Hamilton are in that list too in another test. This is how that looks like with Spectrum:

@RunWith(Spectrum.class)
public class SpectrumTests {{

	describe("The list of drivers for the 2017 season", () -> {

		ValidatableResponse response = get("http://ergast.com/api/f1/2017/drivers.json").then();

		String listOfDriverIds = "MRData.DriverTable.Drivers.driverId";

		it("includes Max Verstappen", () -> {

			response.assertThat().body(listOfDriverIds, hasItem("max_verstappen"));
		});

		it("also includes Fernando Alonso and Lewis Hamilton", () -> {

			response.assertThat().body(listOfDriverIds, hasItems("alonso","hamilton"));
		});
	});
}}

Since Spectrum runs ‘on top of’ JUnit, executing this specification is a matter of running it as a JUnit test. This results in the following output:

Spectrum output for Jasmine-style specs

Besides this (admittedly quite straightforward) example, Spectrum also comes with support for setup (using beforeEach and beforeAll) and teardown (using afterEach and afterAll), focusing on or ignoring specific specs, tagging specs, and more. You can find the documentation here.

The other type of specification supported by Spectrum is the Gherkin syntax. Let’s say I want to recreate the same specifications as above in the given/when/then format. With Spectrum, that looks like this:

@RunWith(Spectrum.class)
public class SpectrumTestsGherkin {{

	feature("2017 driver list verification", () -> {

		scenario("Verify that max_verstappen is in the list of 2017 drivers", () -> {

			final Variable<String> endpoint = new Variable<>();
			final Variable<Response> response = new Variable<>();

			given("We have an endpoint that gives us the list of 2017 drivers", () -> {
				
				endpoint.set("http://ergast.com/api/f1/2017/drivers.json");
			});

			when("we retrieve the list from that endpoint", () -> {
				
				response.set(get(endpoint.get()));
			});
			then("max_verstappen is in the driver list", () -> {
				
				response.get().then().assertThat().body("MRData.DriverTable.Drivers.driverId", hasItem("max_verstappen"));
			});
		});

		scenarioOutline("Verify that there are also some other people in the list of 2017 drivers", (driver) -> {

			final Variable<String> endpoint = new Variable<>();
			final Variable<Response> response = new Variable<>();

			given("We have an endpoint that gives us the list of 2017 drivers", () -> {
				
				endpoint.set("http://ergast.com/api/f1/2017/drivers.json");
			});

			when("we retrieve the list from that endpoint", () -> {
				
				response.set(get(endpoint.get()));
			});
			then(driver + " is in the driver list", () -> {
				
				response.get().then().assertThat().body("MRData.DriverTable.Drivers.driverId", hasItem(driver));
			});
		},

		withExamples(
				example("hamilton"),
				example("alonso"),
				example("vettel")
			)
		);
	});
}}

Running this spec shows that it does work indeed:

Spectrum output for Gherkin-style specs

There are two things that are fundamentally different from using the Jasmine-style syntax (the rest is ‘just’ syntactical):

  • Support for scenario outlines enables you to create data driven tests easily. Maybe this can be done too using the Jasmine syntax, but I haven’t figured it out so far.
  • If you want to pass variables between the given, the when and the then steps you’ll need to do so by using the Variable construct. This works with the Jasmine-style syntax too, but you’ll likely need to use it more in the Gherkin case (since ‘given/when/then’ are three steps, where ‘it’ is just one). When your tests get larger and more complex, having to use get() and set() every time you want to access or assign a variable might get cumbersome.

Having said that, I think Spectrum is a good addition to the test runner / BDD-supporting tool set available for Java, and something that you might want to consider using for your current (or next) test automation project. After all, any library or tool that makes your tests and/or test results more readable is worth taking note of. Right?

You can find a small Maven project containing the examples featured in this blog post here.

2 thoughts on “Creating executable specifications with Spectrum

  1. Pingback: Java Weekly, Issue 186 | Baeldung

  2. Pingback: Testing Bits – 7/16/17 – 7/22/17 | Testing Curator Blog

Leave a Reply

Your email address will not be published. Required fields are marked *