Do you really need that Cucumber with your Selenium?

Note: this blog post is NOT meant to discredit the value of Cucumber, SpecFlow and similar tools. Quite the contrary. I think these are fantastic tools, created and maintained by great people.

Somewhere last week I watched the recording of ‘Is Cucumber Automation Killing Your Project?‘, a SauceLabs webinar presented by Nikolay Advolodkin. In this webinar, Nikolay showed some interesting figures: 68% of the participants indicated that they don’t collaborate with others to create business specs in three amigos sessions. However, 54% of the participants said they used Cucumber.

That means that there’s a significant amount of participants that do use Cucumber without actively collaborating on the creation of specifications through practices like three amigos sessions, Specification by Example and Example Mapping. That’s not the strong point of a tool like Cucumber, though. These tools really shine when they’re used to support collaboration, as discussed in this blog post from Aslak Hellesøy, creator of and core contributor to the Cucumber project.

I must say that the above statistics don’t surprise me. Many clients that I work with use Cucumber (or SpecFlow) in the same way, including my current one. Their reasoning?

“We want everybody in our team to understand what we’re testing with our tests”

And for a long time, I supported this. I, too, thought that using Cucumber on top of your test automation code could be a good idea, even if you’re not practicing Behaviour Driven Development. I’ve even written an article on the Cucumber.io blog that says something to that extent. Yes, I’ve put in some pitfalls to avoid and things to consider, but I don’t think that blog post covers my current point of view well enough.

That’s where this blog post comes in. I’ve come to think that in a lot of projects where Cucumber is used solely as another layer in the automation stack, it does more harm than good. The only people that really read the Given-When-Then specifications are the people who create them (the automation engineers, most of the time), without regard for the additional time and effort it requires to implement and maintain this abstraction layer. There’s no discussion, no validation, no Example Mapping, just an automation engineer writing scenarios and implementing them, because readability.

That, though, is not the point of this blog post. What I do want to show here are a couple of techniques you can employ to make your test methods read (almost) like prose, without resorting to adding another abstraction layer like Cucumber.

Our application under test, once again, is ParaBank, the world’s least safe online bank (or rather, a demo web application from Parasoft. In this demo application, you can perform a variety of different scenarios related to online banking, such as opening a new checking or savings account.

With Cucumber, an example scenario that describes part of the behaviour of ParaBank around opening new accounts might look something like this:

Given John is an existing ParaBank customer
And he has an existing checking account with a balance of 5000 dollars
When he opens a new savings account
Then a confirmation message containing the new account number is shown

Not too bad, right? It’s readable, plain English, and (when you know that the initial balance is required for the deposit into the new savings account) describes the intended behaviour in a clear and unambiguous manner.

But here’s the thing: unless this specification has been conjured up before the software was written, by the three amigos, using techniques like Specification by Example and Example Mapping, you don’t need it. It’s perfectly possible to write test code that is nearly just as readable without the additional abstraction layer and dependency that a tool like Cucumber is.

I mean, if the automation engineer is the only person to read the specifications, why even bother creating them? This only presents a maintenance burden that a lot of projects could do without.

As an example, this is what the same test could look like without the Cucumber layer, but with some design decisions that are included for readability (an important aspect of test code, if you’d ask me) and which I’ll describe in more detail below:

private WebDriver driver;

@Before
public void initializeDatabaseAndLogin() {

    ApiHelpers.initializeDatabaseBeforeTest();

    driver = DriverHelpers.createADriverOfType(DriverType.CHROME);

    Credentials johnsCredentials = Credentials.builder().username("john").password("demo").build();

    new LoginPage(driver).
        load().
        loginUsing(johnsCredentials);
}

@Test
public void openAccount_withSufficientFunds_shouldSucceed() {

    Account aNewCheckingAccount =
        Account.builder().type(AccountType.CHECKING).build();

    Account depositingFromAccount =
        Account.builder().id(12345).build();

    new OpenAccountPage(driver).
        load().
        open(aNewCheckingAccount, depositingFromAccount);

    boolean newAccountIdIsDisplayed = new OpenAccountResultPage(driver).newAccountIdIsDisplayed();

    assertThat(newAccountIdIsDisplayed).isTrue();
}

Now, I don’t know about you, but to me, that’s almost as readable as the Cucumber scenario we’ve seen earlier. And remember: if we opted to use Cucumber instead, we would have had to write the same code anyway. So if there’s no upfront communication happening around these scenarios (or in this case, I’d rather just call them tests) anyway, why bother including the Cucumber layer in the first place?

Let’s look at some of the things I’ve implemented to make this code as readable as possible:

Short tests
This is probably the most important one of them all, and that’s why I mention it first. Your tests should be short, sweet and to the point. Ideally, they should check one thing only. Need specific data to be set up prior to the actual test? Try and do that using an API or directly in a database.

In this example, I’m calling a method initializeDatabaseBeforeTest() to reset the database to a known state via an API. There’s plenty of reading material out there on why your tests should be short, so I’m not going to dive into this too deeply here.

Model business concepts as types in your code
If you want to write tests that are human readable, it really helps to model business concepts that mean something to humans as object types in your code. For example, in the test above, we’re creating a new account. An account, in the context of an online banking system, is an entity that has specific properties. In this case, an account has a type, a unique id and a balance:

@Data
@Builder
@AllArgsConstructor
public class Account {

    private AccountType type;
    private int id;
    private double balance;

    public Account(){}
}

I’m using Lombok here to generate getters and setters as well as a builder to allow for fluid object creation in my test method.

It’s important that everybody understands and agrees on the definition of these POJOs (Plain Old Java Objects), such as the Account object here. This massively helps people that are not as familiar with the code as the person who wrote it to understand what’s happening. Not using Cucumber doesn’t absolve you from communicating with your amigos!

Another tip: if a property of a business object can only have specific values, use an enum, like we did here using AccountType:

public enum AccountType {
    CHECKING,
    SAVINGS
}

This prevents objects and properties to accidentally being assigned a wrong value and it increases readability. Winner!

Think hard about the methods exposed by your Page Objects
To further improve test readability, your Page Objects should (only) expose methods that have business meaning. Looking at the example above, the meat of the test happens on the OpenAccount page, where the new account is created. Next to a load() method used to navigate to the page directly (only use these for pages that you can load directly), it has an open() method that takes two arguments, both of type Account, the POJO we’ve seen before. The first one represents the new account, the second represents the account from which the initial deposit into the new account is made.

If you look at the page where you can open an account in the ParaBank application, you’ll see that there’s not much else to do than opening an account, so it makes sense to expose this action to the test methods that use the OpenAccount Page Object.

Choose good names, then choose better ones
You’ve hopefully seen by now that I tried to choose the names I use in my code very carefully, so as to maximize readability. This is hard. I changed the names of my variables and methods many times when I created this example, and I feel that there’s still more room for improvement.

Long variable and method names aren’t inherently bad, as long as they stick to the point. That’s why, for example, I chose to name the method that opens a new account on the OpenAccount page as open() instead of openAccount().

From the context, it’s clear that we’re opening an account here. It’s a method of the OpenAccount page, and its arguments are of type Account. No need to mention it again in the method name, as I did in an earlier iteration. By the way, I learned this from the Clean Code book, which I think is a very valuable read for automation engineers. Lots of good stuff in there.

Use libraries that help you with readability
Apart from Lombok, I also used the AssertJ library to help me write more readable assertions. So, instead of using the default JUnit assertTrue() method, I can now write

assertThat(newAccountIdIsDisplayed).isTrue();

which I think is easier to read. AssertJ has a lot of methods that can help you write more readable assertions, and I think it’s worth checking out for everybody writing Java test code.

So, all in all, I hope that the example above has shown you that it is possible to write (automation) code that is human readable without adding another layer of abstraction in the form of a tool like Cucumber or SpecFlow. This GitHub repository contains the examples I’ve shown here, plus a couple more tests to show some more example of readable (Selenium) test code.

I’m sure there’s still more room for improvement, and I’d love to hear your suggestions on how to further improve the readability of the test code shown here. My main point, though, is to show you that you don’t need Cucumber to make your tests readable to humans.

Writing BDD tests using Selenium and Cucumber

In this article, I want to introduce another Behaviour Driven Development (BDD) tool to you. It’s similar to JBehave, about which I’ve been talking in a previous post, but with a couple of advantages:

  • It is being updated regularly, in contrast to JBehave, which has not seen any updates for quite some time now
  • It works completely separately from your Selenium WebDriver implementation, which means you can simply add it as another dependency to your existing and new Selenium projects. Again, this is unlike JBehave, which uses its own Selenium browser driver implementation, and because these aren’t updated regularly, it’s likely you’ll run into compatibility issues when trying to run Selenium + JBehave tests on newer browser versions.

Installing and configuring Cucumber
The Cucumber Java implementation (Cucumber JVM) can be downloaded from the Cucumber homepage. Once you’ve downloaded the binaries, just add the required .jar files as dependencies to your Selenium project and you’re good to go.

Defining the user story to be tested
Like with JBehave, Cucumber BDD tests are written as user stories in the Given-When-Then format. I am going to use the following login scenario as an example user story in this post:

Feature: Login action
	As a user of ParaBank
	I want to be able to log in
	So I can use the features provided
	
	Scenario: An authorized user logs in successfully
	Given the ParaBank home page is displayed
	When user "john" logs in using password "demo"
	Then the login is successful

ParaBank is a demo application from the guys at Parasoft that I use (or abuse) a lot in my demos.

In order for Cucumber to automatically detect the stories (or features, as they’re known in Cucumber), you need to make sure that they carry the .feature file extension. For example, in this case, I’ve named my user story login.feature.

Translating the test steps to code
As with JBehave, the next step is to translate the steps from the user story into runnable code. First, we are going to define the actual class to be run, which looks like this:

import org.junit.runner.RunWith;
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;

@RunWith(Cucumber.class)
@CucumberOptions(format = "pretty")
public class SeleniumCucumber {
	
}

I’ve configured my tests to run as JUnit tests, because this will make it easy to incorporate them into a CI configuration, and it auto-generates reporting in Eclipse. The @RunWith annotation defines the tests to run as Cucumber tests, whereas the @CucumberOptions annotation is used to set options for your tests (more info here). Cucumber automatically looks for step definitions and user stories in the same package, so there’s no need for any additional configuration here. Note that in Cucumber, the class to be run cannot contain any actual implementation of the test steps. These are defined in a different class:

public class SeleniumCucumberSteps {

	WebDriver driver;
	
	@Before
	public void beforeTest() {
		driver = new FirefoxDriver();
	}

	@Given("^the ParaBank home page is displayed$")
	public void theParaBankHomepageIsDisplayed() {

		driver.get("http://parabank.parasoft.com");
	}

	@When("^user john logs in using password demo$")
	public void userJohnLogsInUsingPasswordDemo() {

		driver.findElement(By.name("username")).sendKeys("john");
		driver.findElement(By.name("password")).sendKeys("demo");
		driver.findElement(By.cssSelector("input[value='Log In']")).click();
	}

	@Then("^the login is successful$")
	public void theLoginIsSuccessful() {
		Assert.assertEquals("ParaBank | Accounts Overview",driver.getTitle());
	}
	
	@After
	public void afterTest() {
		driver.quit();
	}
}

Similar to what we’ve seen with JBehave, each step in our user story is translated to runnable code. As the user story is quite straightforward, so is its implementation, as you can see.

I’ve used the JUnit @Before and @After annotations for set-up and tear-down test steps. These are steps to set up your test environment before tests are run and to restore the test environment to its original setup after test execution is finished. In our case, the only set-up step we take is the creation of a new browser instance, and the only tear-down action is closing that same browser instance. All other steps are part of the actual user story to be executed.

I’ve used the JUnit Assert feature to execute the verification from the Then part of the user story, so it shows up nicely in the reports.

That’s all there is to creating a first Cucumber test.

Running the test
Now, if we run the tests we defined above by running the SeleniumCucumber class, Cucumber automatically detects the user story to be executed (as it is in the same directory) and the associated step implementations (as they’re in the same package too). The result is that the user story is executed successfully and a nicely readable JUnit-format report is generated and displayed:
Cucumber test resultsAs you can see, the test result even displays the name of the feature and the scenario that has been executed, which makes it all the more readable and usable.

Using parameters in test steps
In Cucumber, we can use parameters to make our step definitions reusable. This is done by replacing the values in the step definitions with regular expressions and adding parameters to the methods that implement the steps:

@When("^user \"(.*)\" logs in using password \"(.*)\"$")
public void userJohnLogsInUsingPasswordDemo(String username, String password) {

	driver.findElement(By.name("username")).sendKeys(username);
	driver.findElement(By.name("password")).sendKeys(password);
	driver.findElement(By.cssSelector("input[value='Log In']")).click();
}

When running the tests, Cucumber will replace the regular expressions with the values defined in the user story and pass these as parameters to the method that is executed. Easy, right? You can do the same for integer, double and other parameter types just as easily if you know your way around regular expressions.

Overall, I’ve found Cucumber to be really easy to set up and use so far, even for someone who is not a true developer (I’m definitely not!). In a future post, I’ll go into more detail on adding these tests to a Continuous Integration solution.

An Eclipse project containing the example code and the user story / feature I’ve used in this post can be downloaded here.