Do you really need that Cucumber with your Selenium?

Note: this blog post is NOT meant to discredit the value of Cucumber, SpecFlow and similar tools. Quite the contrary. I think these are fantastic tools, created and maintained by great people.

Somewhere last week I watched the recording of ‘Is Cucumber Automation Killing Your Project?‘, a SauceLabs webinar presented by Nikolay Advolodkin. In this webinar, Nikolay showed some interesting figures: 68% of the participants indicated that they don’t collaborate with others to create business specs in three amigos sessions. However, 54% of the participants said they used Cucumber.

That means that there’s a significant amount of participants that do use Cucumber without actively collaborating on the creation of specifications through practices like three amigos sessions, Specification by Example and Example Mapping. That’s not the strong point of a tool like Cucumber, though. These tools really shine when they’re used to support collaboration, as discussed in this blog post from Aslak Hellesøy, creator of and core contributor to the Cucumber project.

I must say that the above statistics don’t surprise me. Many clients that I work with use Cucumber (or SpecFlow) in the same way, including my current one. Their reasoning?

“We want everybody in our team to understand what we’re testing with our tests”

And for a long time, I supported this. I, too, thought that using Cucumber on top of your test automation code could be a good idea, even if you’re not practicing Behaviour Driven Development. I’ve even written an article on the Cucumber.io blog that says something to that extent. Yes, I’ve put in some pitfalls to avoid and things to consider, but I don’t think that blog post covers my current point of view well enough.

That’s where this blog post comes in. I’ve come to think that in a lot of projects where Cucumber is used solely as another layer in the automation stack, it does more harm than good. The only people that really read the Given-When-Then specifications are the people who create them (the automation engineers, most of the time), without regard for the additional time and effort it requires to implement and maintain this abstraction layer. There’s no discussion, no validation, no Example Mapping, just an automation engineer writing scenarios and implementing them, because readability.

That, though, is not the point of this blog post. What I do want to show here are a couple of techniques you can employ to make your test methods read (almost) like prose, without resorting to adding another abstraction layer like Cucumber.

Our application under test, once again, is ParaBank, the world’s least safe online bank (or rather, a demo web application from Parasoft. In this demo application, you can perform a variety of different scenarios related to online banking, such as opening a new checking or savings account.

With Cucumber, an example scenario that describes part of the behaviour of ParaBank around opening new accounts might look something like this:

Given John is an existing ParaBank customer
And he has an existing checking account with a balance of 5000 dollars
When he opens a new savings account
Then a confirmation message containing the new account number is shown

Not too bad, right? It’s readable, plain English, and (when you know that the initial balance is required for the deposit into the new savings account) describes the intended behaviour in a clear and unambiguous manner.

But here’s the thing: unless this specification has been conjured up before the software was written, by the three amigos, using techniques like Specification by Example and Example Mapping, you don’t need it. It’s perfectly possible to write test code that is nearly just as readable without the additional abstraction layer and dependency that a tool like Cucumber is.

I mean, if the automation engineer is the only person to read the specifications, why even bother creating them? This only presents a maintenance burden that a lot of projects could do without.

As an example, this is what the same test could look like without the Cucumber layer, but with some design decisions that are included for readability (an important aspect of test code, if you’d ask me) and which I’ll describe in more detail below:

private WebDriver driver;

@Before
public void initializeDatabaseAndLogin() {

    ApiHelpers.initializeDatabaseBeforeTest();

    driver = DriverHelpers.createADriverOfType(DriverType.CHROME);

    Credentials johnsCredentials = Credentials.builder().username("john").password("demo").build();

    new LoginPage(driver).
        load().
        loginUsing(johnsCredentials);
}

@Test
public void openAccount_withSufficientFunds_shouldSucceed() {

    Account aNewCheckingAccount =
        Account.builder().type(AccountType.CHECKING).build();

    Account depositingFromAccount =
        Account.builder().id(12345).build();

    new OpenAccountPage(driver).
        load().
        open(aNewCheckingAccount, depositingFromAccount);

    boolean newAccountIdIsDisplayed = new OpenAccountResultPage(driver).newAccountIdIsDisplayed();

    assertThat(newAccountIdIsDisplayed).isTrue();
}

Now, I don’t know about you, but to me, that’s almost as readable as the Cucumber scenario we’ve seen earlier. And remember: if we opted to use Cucumber instead, we would have had to write the same code anyway. So if there’s no upfront communication happening around these scenarios (or in this case, I’d rather just call them tests) anyway, why bother including the Cucumber layer in the first place?

Let’s look at some of the things I’ve implemented to make this code as readable as possible:

Short tests
This is probably the most important one of them all, and that’s why I mention it first. Your tests should be short, sweet and to the point. Ideally, they should check one thing only. Need specific data to be set up prior to the actual test? Try and do that using an API or directly in a database.

In this example, I’m calling a method initializeDatabaseBeforeTest() to reset the database to a known state via an API. There’s plenty of reading material out there on why your tests should be short, so I’m not going to dive into this too deeply here.

Model business concepts as types in your code
If you want to write tests that are human readable, it really helps to model business concepts that mean something to humans as object types in your code. For example, in the test above, we’re creating a new account. An account, in the context of an online banking system, is an entity that has specific properties. In this case, an account has a type, a unique id and a balance:

@Data
@Builder
@AllArgsConstructor
public class Account {

    private AccountType type;
    private int id;
    private double balance;

    public Account(){}
}

I’m using Lombok here to generate getters and setters as well as a builder to allow for fluid object creation in my test method.

It’s important that everybody understands and agrees on the definition of these POJOs (Plain Old Java Objects), such as the Account object here. This massively helps people that are not as familiar with the code as the person who wrote it to understand what’s happening. Not using Cucumber doesn’t absolve you from communicating with your amigos!

Another tip: if a property of a business object can only have specific values, use an enum, like we did here using AccountType:

public enum AccountType {
    CHECKING,
    SAVINGS
}

This prevents objects and properties to accidentally being assigned a wrong value and it increases readability. Winner!

Think hard about the methods exposed by your Page Objects
To further improve test readability, your Page Objects should (only) expose methods that have business meaning. Looking at the example above, the meat of the test happens on the OpenAccount page, where the new account is created. Next to a load() method used to navigate to the page directly (only use these for pages that you can load directly), it has an open() method that takes two arguments, both of type Account, the POJO we’ve seen before. The first one represents the new account, the second represents the account from which the initial deposit into the new account is made.

If you look at the page where you can open an account in the ParaBank application, you’ll see that there’s not much else to do than opening an account, so it makes sense to expose this action to the test methods that use the OpenAccount Page Object.

Choose good names, then choose better ones
You’ve hopefully seen by now that I tried to choose the names I use in my code very carefully, so as to maximize readability. This is hard. I changed the names of my variables and methods many times when I created this example, and I feel that there’s still more room for improvement.

Long variable and method names aren’t inherently bad, as long as they stick to the point. That’s why, for example, I chose to name the method that opens a new account on the OpenAccount page as open() instead of openAccount().

From the context, it’s clear that we’re opening an account here. It’s a method of the OpenAccount page, and its arguments are of type Account. No need to mention it again in the method name, as I did in an earlier iteration. By the way, I learned this from the Clean Code book, which I think is a very valuable read for automation engineers. Lots of good stuff in there.

Use libraries that help you with readability
Apart from Lombok, I also used the AssertJ library to help me write more readable assertions. So, instead of using the default JUnit assertTrue() method, I can now write

assertThat(newAccountIdIsDisplayed).isTrue();

which I think is easier to read. AssertJ has a lot of methods that can help you write more readable assertions, and I think it’s worth checking out for everybody writing Java test code.

So, all in all, I hope that the example above has shown you that it is possible to write (automation) code that is human readable without adding another layer of abstraction in the form of a tool like Cucumber or SpecFlow. This GitHub repository contains the examples I’ve shown here, plus a couple more tests to show some more example of readable (Selenium) test code.

I’m sure there’s still more room for improvement, and I’d love to hear your suggestions on how to further improve the readability of the test code shown here. My main point, though, is to show you that you don’t need Cucumber to make your tests readable to humans.

On no longer wanting to depend on ExpectedConditions in my C# Selenium WebDriver tests

Those of you that have been writing tests with Selenium in C# for a while might have noticed that as of version 3.11, some often-used things have been marked as deprecated, most notably the PageFactory implementation and the ExpectedConditions class. For those of you that have not read the motivation behind this, here is the rationale behind it, as written by Jim Evans, core contributor to the Selenium C# bindings.

These implementations have been moved to a new location and have been made available as separate NuGet packages (DotNetSeleniumExtras.PageObjects and DotNetSeleniumExtras.WaitHelpers, respectively). As of the time of writing this blog post, however, there’s still no new maintainer for this GitHub repository, meaning that there’s no support available and no certainty as to its future.

Notice of ExpectedConditions class being deprecated, as visible in Visual Studio

And while that is no problem yet, since you can still use the deprecated classes from the Selenium bindings or use the DotNetSeleniumExtras packages, I think it’s a risk to keep on relying on code that is effectively unsupported.

In my case, for PageFactory, that’s not a problem, since I don’t typically use it anyway. For ExpectedConditions, however, it’s a different story. When I’m writing and teaching Selenium automation, I’m pretty much still following the general approach I described a couple of years ago in this post. It still works, it’s readable, easy to maintain and explain and has proven to be effective in many different projects I’ve worked on over the years.

However, it also relies on the use of ExpectedConditions, and that’s something that is bothering me more and more often.

So, while preparing for a training session I delivered recently, I started looking for an alternative way of writing my wrapper methods in C#. And it turns out that C# offers an elegant way of doing so, out of the box, in the form of lambda expressions.

Lambda expressions (also known as anonymous functions) is a function that’s not bound to an identifier, but instead can be passed as an argument to a higher-order function. The concept stems from functional programming (it has been a long while since I’ve done any of that, by the way) but has been part of C# for a while. By the way, lambda expressions are available in Java too, from Java 8 onwards, for those of you not working with C#.

Let’s look at an example. Here’s my implementation for a wrapper function around the default Selenium SendKeys() method that takes care of synchronization and exception handling using ExpectedConditions:

public void SendKeys(By by, string valueToType)
{
    try
    {
      new WebDriverWait(_driver, TimeSpan.FromSeconds(Constants.DEFAULT_TIMEOUT)).Until(ExpectedConditions.ElementToBeClickable(by));
        _driver.FindElement(by).Clear();
        _driver.FindElement(by).SendKeys(valueToType);
    }
    catch (Exception ex) when (ex is NoSuchElementException || ex is WebDriverTimeoutException)
    {
         Assert.Fail($"Exception occurred in SeleniumHelper.SendKeys(): element located by {by.ToString()} could not be located within {Constants.DEFAULT_TIMEOUT} seconds.");
    }
}

Here’s the same wrapper, but now passing a lambda expression of our own as an argument to the Until() method instead of a method in ExpectedConditions:

public void SendKeys(By by, string valueToType)
{
    WebDriverWait wait = new WebDriverWait(_driver, TimeSpan.FromSeconds(Constants.DEFAULT_TIMEOUT));
            
    try
    {
        IWebElement myElement = wait.Until<IWebElement>(driver =>
        {
            IWebElement tempElement = _driver.FindElement(by);
            return (tempElement.Displayed && tempElement.Enabled) ? tempElement : null;
        }
        );
        myElement.Clear();
        myElement.SendKeys(valueToType);
    }
    catch (WebDriverTimeoutException)
    {
        Assert.Fail($"Exception in SendKeys(): element located by {by.ToString()} not visible and enabled within {Constants.DEFAULT_TIMEOUT} seconds.");
    }
}

From the Selenium docs, we learn that the Until() method repeatedly (with a default but configurable polling interval of 500 milliseconds) evaluates the lambda expression, until one of these conditions applies:

  • The expression returns neither null nor false;
  • The function throws an exception that is not in the list of ignored expressions;
  • The specified timeout expires

This explains why I created the lambda expression like I did. Before clicking on an element, I want to wait until it is clickable, which I define as it being both visible (expressed through the Displayed property) and enabled (the Enabled property). If both are true, I return the element, otherwise null. If this happens within the timeout set, I’ll clear the text field and use the standard SendKeys() method to type the specified value. If a WebDriverTimeOutException occurs, I’ll handle it accordingly, in this case by failing the test with a readable message.

Yes, implementing my helper methods like this makes my code a little more verbose, but you could argue that this is a small price to pay for not having to rely on (part of) a library that has been deprecated and of which the future is uncertain. Plus, because I define all these helpers in a single class, there’s really only one place in my code that is affected. The Page Objects and my test code remain wholly unaffected.

This GitHub repository contains a couple of runnable example tests that are written against the ParaBank demo application. This is an updated version of the solution described here. An updated version of this last blog post is coming soon, by the way, so that it reflects some opinions and preferences of mine that have changed since I wrote that one. No longer having to rely on ExpectedConditions, as I discussed here, is but one of those changes.

Remember what your tests are trying to verify

Lately, I’ve been working a lot on Selenium-based test automation solutions. And even though I’m still not overly enthusiastic about creating lots of user interface-driven automated tests, now that I’m getting more skilled at creating robust and stable Selenium tests, I am starting to appreciate the tool and what you can do with it more and more. As everybody working with Selenium can tell you, there are situations where things can get, well, interesting. And by interesting, I mean tricky. Dealing with dynamic front end frameworks and unpredictable modals and overlays can ask a lot of you in terms of debugging and exception handling skills.

Not yet being fluent in Selenese, I find myself on Google and (subsequently) StackOverflow a lot, trying to find suitable solutions for problems I encounter. While doing so, there’s one thing that I see a lot of people do, yet strikes me as really odd, given what I think is the purpose of these user interface-driven tests:

Forcing your tests to do something your users can’t.

From what I’ve understood, studying and working in the test automation field for a while now, user interface-driven tests, such as these Selenium tests, should be used to verify that the user of your user interface is able to complete a sequence of predefined actions. If you’re working in a hip and happening environment, these are often called ‘customer journeys’ or ‘user journeys’. But that’s not my point. What my point is, is that quite often I see workarounds suggested that go beyond what a regular user could do with his or her keyboard and / or mouse.

For example, take an element that is invisible until you hover over it with your mouse. If you just try to do a click(), your test will probably throw an exception stating that the element was not visible. Now, there are (at least) two ways to deal with this situation:

  1. Use the Actions class to simulate a mouseover, then click the element.
  2. Use a JavaScriptExecutor to perform the click.

While both approaches might result in a passing test, I am of the opinion that one is useful and the other is a horrible idea. Based on what I’ve written so far, can you guess which of the two options I’d suggest?

Indeed, option #1 is the way to go, for two reasons:

  • User interface-driven tests should mimic actual user interaction as closely as possible. I’ve never seen a user execute some JavaScript on a page to make an element visible. Either that, or I’m hanging around the wrong group of users..
  • What happens if a front-end developer makes a mistake (I know, they never do, but let’s assume so anyway) which causes the element not to become visible, even on a mouseover? With option #1, your test will fail, for the right reason. With #2, hello false negative!

There are some exceptions to the rule, though. The prime example I can think of is handling file uploads by directly sending the absolute path of the file to be uploaded to the input element responsible for the file upload using sendKeys(), instead of clicking on it and handling the file dialog. I’ve tried the latter before, and it’s a pain, first because you can’t do it with the standard Selenium API (because the file dialog is native to the operating system), second because different browsers use different file dialog layouts, resulting in a lot of pesky code that easily breaks down. In this case, I prefer to bypass the file dialog altogether (it’s probably not the subject of my test anyway).

In (almost) all other cases though, I’d advise you to stick to simulating your end user behavior as closely as possible. The job of your user interface-driven tests, and therefore of you as its creator, is not to force a pass, but to simulate end user interaction and see if that leads to a successfully executed scenario. Don’t fool yourself and your stakeholders by underwater tricks that obscure potential user interface issues.

Remember what your tests are trying to verify.