Creating executable specifications with Spectrum

One of the most important features of a good set of automated tests is that they require a minimal amount of explanation and/or documentation. When I see someone else’s test, I’d like to be able to instantly see what its purpose is and how it’s constructed in terms of setting up > execution > verification (for example using given/when/then or arrange/act/assert). That’s one of the reasons that I’ve been using Cucumber (or SpecFlow, depending on the client) in several of my automated test solutions, even though the team wasn’t doing Behaviour Driven Development.

It’s always a good idea to look for alternatives, though. Last week I was made aware of Spectrum, which is, as creator Greg Haskins calls it “A BDD-style test runner for Java 8. Inspired by Jasmine, RSpec, and Cucumber.”. I’ve been working with Jasmine a little before, and even though it took me a while to get used to the syntax and lambda notation style, I liked the way it provided a way to document your tests directly in the code and produce human readable results. Since Spectrum is created as a Jasmine-like test runner for Java, plus it provides support for the (at least in the Java world) much more common Gherkin given/when/then syntax, I thought it’d be a good idea to check it out.

Spectrum is ‘just’ yet another Java library, so adding it to your project is a breeze when using Maven or Gradle. Note that since Spectrum uses lambda functions, it won’t work unless you’re using Java 8. Spectrum runs on top of JUnit, so you’ll need that too. It works with all kinds of assertion libraries, so if you’re partial to Hamcrest, for example, that’s no problem at all.

As I said, Spectrum basically supports two types of specs: the Jasmine-style describe/it specification and the Gherkin-style given/when/then specification using features and scenarios. Let’s take a quick look at the Jasmine-style specifications first. For this example, I’m resorting once again to REST Assured tests. I’d like to verify whether Max Verstappen is in the list of 2017 drivers in one test, and whether both Fernando Alonso and Lewis Hamilton are in that list too in another test. This is how that looks like with Spectrum:

public class SpectrumTests {{

	describe("The list of drivers for the 2017 season", () -> {

		ValidatableResponse response = get("").then();

		String listOfDriverIds = "MRData.DriverTable.Drivers.driverId";

		it("includes Max Verstappen", () -> {

			response.assertThat().body(listOfDriverIds, hasItem("max_verstappen"));

		it("also includes Fernando Alonso and Lewis Hamilton", () -> {

			response.assertThat().body(listOfDriverIds, hasItems("alonso","hamilton"));

Since Spectrum runs ‘on top of’ JUnit, executing this specification is a matter of running it as a JUnit test. This results in the following output:

Spectrum output for Jasmine-style specs

Besides this (admittedly quite straightforward) example, Spectrum also comes with support for setup (using beforeEach and beforeAll) and teardown (using afterEach and afterAll), focusing on or ignoring specific specs, tagging specs, and more. You can find the documentation here.

The other type of specification supported by Spectrum is the Gherkin syntax. Let’s say I want to recreate the same specifications as above in the given/when/then format. With Spectrum, that looks like this:

public class SpectrumTestsGherkin {{

	feature("2017 driver list verification", () -> {

		scenario("Verify that max_verstappen is in the list of 2017 drivers", () -> {

			final Variable<String> endpoint = new Variable<>();
			final Variable<Response> response = new Variable<>();

			given("We have an endpoint that gives us the list of 2017 drivers", () -> {

			when("we retrieve the list from that endpoint", () -> {
			then("max_verstappen is in the driver list", () -> {
				response.get().then().assertThat().body("MRData.DriverTable.Drivers.driverId", hasItem("max_verstappen"));

		scenarioOutline("Verify that there are also some other people in the list of 2017 drivers", (driver) -> {

			final Variable<String> endpoint = new Variable<>();
			final Variable<Response> response = new Variable<>();

			given("We have an endpoint that gives us the list of 2017 drivers", () -> {

			when("we retrieve the list from that endpoint", () -> {
			then(driver + " is in the driver list", () -> {
				response.get().then().assertThat().body("MRData.DriverTable.Drivers.driverId", hasItem(driver));


Running this spec shows that it does work indeed:

Spectrum output for Gherkin-style specs

There are two things that are fundamentally different from using the Jasmine-style syntax (the rest is ‘just’ syntactical):

  • Support for scenario outlines enables you to create data driven tests easily. Maybe this can be done too using the Jasmine syntax, but I haven’t figured it out so far.
  • If you want to pass variables between the given, the when and the then steps you’ll need to do so by using the Variable construct. This works with the Jasmine-style syntax too, but you’ll likely need to use it more in the Gherkin case (since ‘given/when/then’ are three steps, where ‘it’ is just one). When your tests get larger and more complex, having to use get() and set() every time you want to access or assign a variable might get cumbersome.

Having said that, I think Spectrum is a good addition to the test runner / BDD-supporting tool set available for Java, and something that you might want to consider using for your current (or next) test automation project. After all, any library or tool that makes your tests and/or test results more readable is worth taking note of. Right?

You can find a small Maven project containing the examples featured in this blog post here.

On crossing the bridge into unit testing land

Maybe it’s just the people and organizations I meet and work with, but no matter how active they’re trying to implement automation and involve testers therein, there’s one bridge that’s often too far for those that are tasked with test automation, and that’s the bridge to unit testing land. When asking them for the reasons that testers aren’t involved in unit testing, I typically get one (or more, or all) of the following answers:

  • ‘That’s the responsibility of our developers’
  • ‘I don’t know how to write unit tests’
  • ‘I’m already busy with other types of automation and I don’t have time for that’

While these answers might sound perfectly reasonable to some, I think there’s something inherently wrong with all of them. Let’s take a look:

  • With more and more teams becoming multidisciplinary, we can’t simply shift responsibility for any task to a specific subgroup. If ‘we’ (i.e., the testers) keep saying that unit testing is a developer’s responsibility, we’ll never get rid of the silos we’re trying to break down.
  • While you might not know how to actually write unit tests yourself, there’s a lot you CAN do to contribute to their value and effectiveness. Try reviewing them, for example: has the developer of the unit test missed some obvious cases?
  • Not having time to concern yourself with unit testing reminds me of the picture below. Really, if something can be covered with a decent set of unit tests, there really is no need to write integration or even (shudder) end-to-end tests for it.

Are you too busy to pay attention to unit testing?

I’m not a devotee of the test automation pyramid per se, but there IS a lot of truth to the concept that a decent set of unit tests should be the foundation of every solid test automation effort. Unit tests are relatively easy to write (even though it might not look that way to some), they run fast (no need for waiting until web pages are loaded and complete their client-side processing, for example..) and therefore they’re the best way to provide that fast feedback that development teams are looking for those in this age of Continuous Integration / Delivery / Deployment / Testing / Everything / … .

To put it in even bolder terms, as a tester, I think you have the responsibility of familiarizing yourself with the unit testing activities that your development team is undertaking. Offer to review them. Try to understand what they do, what they check and where coverage could be improved. Yes, this might require you to actually talk to your developers! But it’ll be worth it, not just to you, but to the whole team and, in the end, also to your product and your customers. Over time, you might even write some unit tests yourself, though, again, that’s not a necessity for you to provide value in the land of unit testing. Plus, you’ll likely learn some new tricks and skills by doing so, and that’s always a good thing, right?

P.S.: Remember Tesults, the SaaS solution for storing and displaying test results I wrote about a couple of months ago? The people behind Tesults recently let me know they now offer a free forever plan as well. So if you were interested in using their services but could not afford or justify the investment, it might be a good idea to check their new plan out here. And again, I am in no way, shape or form associated with, nor do I have a commercial interest in Tesults as an organization or a product. I still think it’s a great platform, though.

How to move up from being good to being great?

Summer holidays.. A perfect time for taking a step or two back from work and reflect on what I’ve been doing this year, work-wise. Having just returned from my vacation, I can confirm that this year has been no different. One theme (or rather, one question) kept coming back over and over again this time:

How do I go from being ‘good’ to being ‘great’?

I consider myself to be a decent, sometimes maybe even a good consultant / writer / teacher / whatever it is I’m doing that day. While this is enough to be pretty fully booked at the moment, and even have the luxury of being able to say ‘no’ to tasks or projects I don’t really want to take on, I feel I can do better in many ways. It doesn’t feel like I’m failing, but there’s a lot of things that I feel I can improve.

Here’s a(n incomplete) list of things I’d like to accomplish on my journey from good to great:

  • Improving my consulting skills
  • Improving my public speaking skills
  • Improving my teaching skills
  • Improving my writing skills

These improvements would, in turn, hopefully lead to the following results (and who knows what else):

  • I’d like to grow a (small) back log of clients that I keep in close touch with and that I can help with their automation on a periodic basis, moving away from the hourly billing model towards a daily fee or a retainer
  • I’d like to speak at more conferences, and not just at testing conferences (no need for siloing knowledge sharing when the whole world seems to consist of multidisciplinary teams nowadays)
  • I’d love to get more bookings for my training and workshop offerings
  • I’d like to write and publish a short (e)book containing my views on the test automation craft

Achieving these goals is only part of the story, though. What would happen if I accomplished the above tasks? Would I somehow automatically start to see myself as a great consultant, rather than ‘just’ a good one? Being the cynical and self-criticizing b*st*rd that I can be, will I ever consider myself being more than half decent? I’m not exactly sure, but I’d like to think that hitting the above targets moves me in the right direction, at least.

One thing that struck me when reviewing the list, though, is that they’re all totally unrelated to getting more knowledgeable with regards to specific test automation tools. I just don’t think that that’s going to get me much closer to what I want to be. Of course, tools are important (no test automation without tools), but I don’t think that’s key to the step I want to take.

According to you, readers of this blog, how do you think one moves up from ‘good’ to ‘great’? I’d love to see your input on this matter! In the meantime, I will start to reach out to people that I consider to be on the ‘great’ level, see how they got to where they are now and what I can learn from them. Call it looking for a mentor (or two). In time, hopefully I can be a mentor myself. I’d love to be able to do that and teach others what I know.

If there’s one thing that my time off work has made me realize this year, though, it’s that there’s so, so many things that I don’t know or don’t master! I’m already looking forward to what the future will bring.