Do you really need that Cucumber with your Selenium?

Note: this blog post is NOT meant to discredit the value of Cucumber, SpecFlow and similar tools. Quite the contrary. I think these are fantastic tools, created and maintained by great people.

Somewhere last week I watched the recording of ‘Is Cucumber Automation Killing Your Project?‘, a SauceLabs webinar presented by Nikolay Advolodkin. In this webinar, Nikolay showed some interesting figures: 68% of the participants indicated that they don’t collaborate with others to create business specs in three amigos sessions. However, 54% of the participants said they used Cucumber.

That means that there’s a significant amount of participants that do use Cucumber without actively collaborating on the creation of specifications through practices like three amigos sessions, Specification by Example and Example Mapping. That’s not the strong point of a tool like Cucumber, though. These tools really shine when they’re used to support collaboration, as discussed in this blog post from Aslak Hellesøy, creator of and core contributor to the Cucumber project.

I must say that the above statistics don’t surprise me. Many clients that I work with use Cucumber (or SpecFlow) in the same way, including my current one. Their reasoning?

“We want everybody in our team to understand what we’re testing with our tests”

And for a long time, I supported this. I, too, thought that using Cucumber on top of your test automation code could be a good idea, even if you’re not practicing Behaviour Driven Development. I’ve even written an article on the Cucumber.io blog that says something to that extent. Yes, I’ve put in some pitfalls to avoid and things to consider, but I don’t think that blog post covers my current point of view well enough.

That’s where this blog post comes in. I’ve come to think that in a lot of projects where Cucumber is used solely as another layer in the automation stack, it does more harm than good. The only people that really read the Given-When-Then specifications are the people who create them (the automation engineers, most of the time), without regard for the additional time and effort it requires to implement and maintain this abstraction layer. There’s no discussion, no validation, no Example Mapping, just an automation engineer writing scenarios and implementing them, because readability.

That, though, is not the point of this blog post. What I do want to show here are a couple of techniques you can employ to make your test methods read (almost) like prose, without resorting to adding another abstraction layer like Cucumber.

Our application under test, once again, is ParaBank, the world’s least safe online bank (or rather, a demo web application from Parasoft. In this demo application, you can perform a variety of different scenarios related to online banking, such as opening a new checking or savings account.

With Cucumber, an example scenario that describes part of the behaviour of ParaBank around opening new accounts might look something like this:

Given John is an existing ParaBank customer
And he has an existing checking account with a balance of 5000 dollars
When he opens a new savings account
Then a confirmation message containing the new account number is shown

Not too bad, right? It’s readable, plain English, and (when you know that the initial balance is required for the deposit into the new savings account) describes the intended behaviour in a clear and unambiguous manner.

But here’s the thing: unless this specification has been conjured up before the software was written, by the three amigos, using techniques like Specification by Example and Example Mapping, you don’t need it. It’s perfectly possible to write test code that is nearly just as readable without the additional abstraction layer and dependency that a tool like Cucumber is.

I mean, if the automation engineer is the only person to read the specifications, why even bother creating them? This only presents a maintenance burden that a lot of projects could do without.

As an example, this is what the same test could look like without the Cucumber layer, but with some design decisions that are included for readability (an important aspect of test code, if you’d ask me) and which I’ll describe in more detail below:

private WebDriver driver;

@Before
public void initializeDatabaseAndLogin() {

    ApiHelpers.initializeDatabaseBeforeTest();

    driver = DriverHelpers.createADriverOfType(DriverType.CHROME);

    Credentials johnsCredentials = Credentials.builder().username("john").password("demo").build();

    new LoginPage(driver).
        load().
        loginUsing(johnsCredentials);
}

@Test
public void openAccount_withSufficientFunds_shouldSucceed() {

    Account aNewCheckingAccount =
        Account.builder().type(AccountType.CHECKING).build();

    Account depositingFromAccount =
        Account.builder().id(12345).build();

    new OpenAccountPage(driver).
        load().
        open(aNewCheckingAccount, depositingFromAccount);

    boolean newAccountIdIsDisplayed = new OpenAccountResultPage(driver).newAccountIdIsDisplayed();

    assertThat(newAccountIdIsDisplayed).isTrue();
}

Now, I don’t know about you, but to me, that’s almost as readable as the Cucumber scenario we’ve seen earlier. And remember: if we opted to use Cucumber instead, we would have had to write the same code anyway. So if there’s no upfront communication happening around these scenarios (or in this case, I’d rather just call them tests) anyway, why bother including the Cucumber layer in the first place?

Let’s look at some of the things I’ve implemented to make this code as readable as possible:

Short tests
This is probably the most important one of them all, and that’s why I mention it first. Your tests should be short, sweet and to the point. Ideally, they should check one thing only. Need specific data to be set up prior to the actual test? Try and do that using an API or directly in a database.

In this example, I’m calling a method initializeDatabaseBeforeTest() to reset the database to a known state via an API. There’s plenty of reading material out there on why your tests should be short, so I’m not going to dive into this too deeply here.

Model business concepts as types in your code
If you want to write tests that are human readable, it really helps to model business concepts that mean something to humans as object types in your code. For example, in the test above, we’re creating a new account. An account, in the context of an online banking system, is an entity that has specific properties. In this case, an account has a type, a unique id and a balance:

@Data
@Builder
@AllArgsConstructor
public class Account {

    private AccountType type;
    private int id;
    private double balance;

    public Account(){}
}

I’m using Lombok here to generate getters and setters as well as a builder to allow for fluid object creation in my test method.

It’s important that everybody understands and agrees on the definition of these POJOs (Plain Old Java Objects), such as the Account object here. This massively helps people that are not as familiar with the code as the person who wrote it to understand what’s happening. Not using Cucumber doesn’t absolve you from communicating with your amigos!

Another tip: if a property of a business object can only have specific values, use an enum, like we did here using AccountType:

public enum AccountType {
    CHECKING,
    SAVINGS
}

This prevents objects and properties to accidentally being assigned a wrong value and it increases readability. Winner!

Think hard about the methods exposed by your Page Objects
To further improve test readability, your Page Objects should (only) expose methods that have business meaning. Looking at the example above, the meat of the test happens on the OpenAccount page, where the new account is created. Next to a load() method used to navigate to the page directly (only use these for pages that you can load directly), it has an open() method that takes two arguments, both of type Account, the POJO we’ve seen before. The first one represents the new account, the second represents the account from which the initial deposit into the new account is made.

If you look at the page where you can open an account in the ParaBank application, you’ll see that there’s not much else to do than opening an account, so it makes sense to expose this action to the test methods that use the OpenAccount Page Object.

Choose good names, then choose better ones
You’ve hopefully seen by now that I tried to choose the names I use in my code very carefully, so as to maximize readability. This is hard. I changed the names of my variables and methods many times when I created this example, and I feel that there’s still more room for improvement.

Long variable and method names aren’t inherently bad, as long as they stick to the point. That’s why, for example, I chose to name the method that opens a new account on the OpenAccount page as open() instead of openAccount().

From the context, it’s clear that we’re opening an account here. It’s a method of the OpenAccount page, and its arguments are of type Account. No need to mention it again in the method name, as I did in an earlier iteration. By the way, I learned this from the Clean Code book, which I think is a very valuable read for automation engineers. Lots of good stuff in there.

Use libraries that help you with readability
Apart from Lombok, I also used the AssertJ library to help me write more readable assertions. So, instead of using the default JUnit assertTrue() method, I can now write

assertThat(newAccountIdIsDisplayed).isTrue();

which I think is easier to read. AssertJ has a lot of methods that can help you write more readable assertions, and I think it’s worth checking out for everybody writing Java test code.

So, all in all, I hope that the example above has shown you that it is possible to write (automation) code that is human readable without adding another layer of abstraction in the form of a tool like Cucumber or SpecFlow. This GitHub repository contains the examples I’ve shown here, plus a couple more tests to show some more example of readable (Selenium) test code.

I’m sure there’s still more room for improvement, and I’d love to hear your suggestions on how to further improve the readability of the test code shown here. My main point, though, is to show you that you don’t need Cucumber to make your tests readable to humans.

11 thoughts on “Do you really need that Cucumber with your Selenium?

  1. Most of the pain points mentioned by Nikolay in his video could be mitigated with good understanding of the BDD tool.

    Whilst I agree that there is no point in using BDD if no-one cares besides yourself, you do end up with something that is readable by pretty much everyone. Your example is readable from a coding perspective, but not from a general perspective. What’s ‘boolean’? What’s ‘builder’? What’s ‘new’ etc

    Once you know the tool, a step effectively is just an abstraction over a piece of code. It takes no time at all to implement and you get the same benefits as if it wasn’t there, e.g. you could use your ‘Given John is an existing ParaBank customer’ step in hundreds/thousands of scenarios/tests with no code duplication.

    The tool can also encourage good framework design. You start to think in terms of features and scenarios and user journeys that you may struggle to know how to best organise if you are using the fairly blank slate of a typical test runner like NUnit, MSTest etc.

    IMO it’s fine to suggest BDD automation is not needed if your company as a whole has not adopted BDD, but to suggest it can’t do certain things or it takes significantly longer to implement are misguided.

    • Hey Matt,

      I politely disagree. I’ve seen just too many implementations that suffered from engineers adding the Cucumber abstraction layer without success. And I do know the tool, I’ve been using and teaching it for years. Yes, it’s quick to implement at first, but once it grows, maintenance becomes harder and more time-consuming. I could also pretty easily reuse parts of my test code (like the user creation examples you mentioned) without adding Cucumber to my setup (by using base tests, for example).

      Also, I don’t think I gave examples of what it can’t do anywhere? I just wanted to give a different perspective and show how to write readable code without using a tool like Cucumber. If you still use it with good results, even when you’re not practicing BDD, that’s great. I’ve seen it go downhill more times than not, unfortunately.

      • The can’t do reference was regarding Nikolays sauce labs video. One example was parallelisation limited to feature files but this is not the case.

        What maintenance issues are you running into and what bits get more time consuming as it grows? I’m finding the opposite. The more the suite grows the more reusable code I have and the quicker I can craft scenarios. If there is a complicated test setup, chances are it’s already been done or only requires a minor tweak to fit the new functionality.

        Maybe I’ve just been lucky. I have 500 tests at the moment and I don’t know how that compares to others in the wild.

        • I wouldn’t call it lucky, I’d say you’re doing things right 🙂 Out of interest, how many people are working on your automation project? If you’re the only one I can see how that makes things easier. Most of the projects I’m working on, there’s at least a couple and sometimes a whole group of people working on the same automation code base, and that requires more communication about what steps are available, what makes a good step, etc. And that’s often where things go wrong..

          • I have run into those same problems. In fact any code base is likely to experience something similar to a certain degree.

            I’m my experience unfortunately it’s generally been due to poor programming skills. Automation testers were often previously manual testers, and they start with the selenium library rather than learning traditional programming concepts.

            The issue can be somewhat mitigated with clearly defined coding standards and wrappers that provide abstractions that make it easier to navigate through a test without running into synchronisation issues.

  2. I appreciate this post greatly. I’m currently struggling with an inherited Cucumber framework, have always created my frameworks with Cucumber, and have never had any of the promised “three amigos” level of collaboration.

    Faced with building out a new framework at a new gig in the very near future, you’ve convinced me it’s okay to run away from the seemingly useless added layer of abstraction that’s currently giving me fits.

    • If it doesn’t work for you, it’s OK to leave it. If it works for you, it’s OK to leave it in. I was only trying to give some suggestions on how you could improve things without Cucumber. Not trying to say ‘thou shalt not use the Cucumber!’. I’ve long moved on from those kinds of views. There’s always situations where one thing works, and others where the same thing doesn’t work.

      • Oh of course it works well for some. My work is all mobile and since the native frameworks have come around, Cucumber is proving harder and harder to use (Xcode only supports it through a very limited and no longer supported third party hack).

  3. Pingback: Java Weekly, Issue 326 | Baeldung

  4. Thank you for the post. It’s nice to see that people start to realize that approach (BDD, in the current case) is not a silver bullet, and it needs to be adopted based on the project context (the level of team collaboration and the issues it is about to solve), not vice versa. Otherwise it turns into a cargo cult.

    Unfortunately, I also saw such cases in practice, with only reasoning “We need to show to the client, that we are familiar with BDD”.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.