Do you really need that Cucumber with your Selenium?

Note: this blog post is NOT meant to discredit the value of Cucumber, SpecFlow and similar tools. Quite the contrary. I think these are fantastic tools, created and maintained by great people.

Somewhere last week I watched the recording of ‘Is Cucumber Automation Killing Your Project?‘, a SauceLabs webinar presented by Nikolay Advolodkin. In this webinar, Nikolay showed some interesting figures: 68% of the participants indicated that they don’t collaborate with others to create business specs in three amigos sessions. However, 54% of the participants said they used Cucumber.

That means that there’s a significant amount of participants that do use Cucumber without actively collaborating on the creation of specifications through practices like three amigos sessions, Specification by Example and Example Mapping. That’s not the strong point of a tool like Cucumber, though. These tools really shine when they’re used to support collaboration, as discussed in this blog post from Aslak Hellesøy, creator of and core contributor to the Cucumber project.

I must say that the above statistics don’t surprise me. Many clients that I work with use Cucumber (or SpecFlow) in the same way, including my current one. Their reasoning?

“We want everybody in our team to understand what we’re testing with our tests”

And for a long time, I supported this. I, too, thought that using Cucumber on top of your test automation code could be a good idea, even if you’re not practicing Behaviour Driven Development. I’ve even written an article on the Cucumber.io blog that says something to that extent. Yes, I’ve put in some pitfalls to avoid and things to consider, but I don’t think that blog post covers my current point of view well enough.

That’s where this blog post comes in. I’ve come to think that in a lot of projects where Cucumber is used solely as another layer in the automation stack, it does more harm than good. The only people that really read the Given-When-Then specifications are the people who create them (the automation engineers, most of the time), without regard for the additional time and effort it requires to implement and maintain this abstraction layer. There’s no discussion, no validation, no Example Mapping, just an automation engineer writing scenarios and implementing them, because readability.

That, though, is not the point of this blog post. What I do want to show here are a couple of techniques you can employ to make your test methods read (almost) like prose, without resorting to adding another abstraction layer like Cucumber.

Our application under test, once again, is ParaBank, the world’s least safe online bank (or rather, a demo web application from Parasoft. In this demo application, you can perform a variety of different scenarios related to online banking, such as opening a new checking or savings account.

With Cucumber, an example scenario that describes part of the behaviour of ParaBank around opening new accounts might look something like this:

Given John is an existing ParaBank customer
And he has an existing checking account with a balance of 5000 dollars
When he opens a new savings account
Then a confirmation message containing the new account number is shown

Not too bad, right? It’s readable, plain English, and (when you know that the initial balance is required for the deposit into the new savings account) describes the intended behaviour in a clear and unambiguous manner.

But here’s the thing: unless this specification has been conjured up before the software was written, by the three amigos, using techniques like Specification by Example and Example Mapping, you don’t need it. It’s perfectly possible to write test code that is nearly just as readable without the additional abstraction layer and dependency that a tool like Cucumber is.

I mean, if the automation engineer is the only person to read the specifications, why even bother creating them? This only presents a maintenance burden that a lot of projects could do without.

As an example, this is what the same test could look like without the Cucumber layer, but with some design decisions that are included for readability (an important aspect of test code, if you’d ask me) and which I’ll describe in more detail below:

private WebDriver driver;

@Before
public void initializeDatabaseAndLogin() {

    ApiHelpers.initializeDatabaseBeforeTest();

    driver = DriverHelpers.createADriverOfType(DriverType.CHROME);

    Credentials johnsCredentials = Credentials.builder().username("john").password("demo").build();

    new LoginPage(driver).
        load().
        loginUsing(johnsCredentials);
}

@Test
public void openAccount_withSufficientFunds_shouldSucceed() {

    Account aNewCheckingAccount =
        Account.builder().type(AccountType.CHECKING).build();

    Account depositingFromAccount =
        Account.builder().id(12345).build();

    new OpenAccountPage(driver).
        load().
        open(aNewCheckingAccount, depositingFromAccount);

    boolean newAccountIdIsDisplayed = new OpenAccountResultPage(driver).newAccountIdIsDisplayed();

    assertThat(newAccountIdIsDisplayed).isTrue();
}

Now, I don’t know about you, but to me, that’s almost as readable as the Cucumber scenario we’ve seen earlier. And remember: if we opted to use Cucumber instead, we would have had to write the same code anyway. So if there’s no upfront communication happening around these scenarios (or in this case, I’d rather just call them tests) anyway, why bother including the Cucumber layer in the first place?

Let’s look at some of the things I’ve implemented to make this code as readable as possible:

Short tests
This is probably the most important one of them all, and that’s why I mention it first. Your tests should be short, sweet and to the point. Ideally, they should check one thing only. Need specific data to be set up prior to the actual test? Try and do that using an API or directly in a database.

In this example, I’m calling a method initializeDatabaseBeforeTest() to reset the database to a known state via an API. There’s plenty of reading material out there on why your tests should be short, so I’m not going to dive into this too deeply here.

Model business concepts as types in your code
If you want to write tests that are human readable, it really helps to model business concepts that mean something to humans as object types in your code. For example, in the test above, we’re creating a new account. An account, in the context of an online banking system, is an entity that has specific properties. In this case, an account has a type, a unique id and a balance:

@Data
@Builder
@AllArgsConstructor
public class Account {

    private AccountType type;
    private int id;
    private double balance;

    public Account(){}
}

I’m using Lombok here to generate getters and setters as well as a builder to allow for fluid object creation in my test method.

It’s important that everybody understands and agrees on the definition of these POJOs (Plain Old Java Objects), such as the Account object here. This massively helps people that are not as familiar with the code as the person who wrote it to understand what’s happening. Not using Cucumber doesn’t absolve you from communicating with your amigos!

Another tip: if a property of a business object can only have specific values, use an enum, like we did here using AccountType:

public enum AccountType {
    CHECKING,
    SAVINGS
}

This prevents objects and properties to accidentally being assigned a wrong value and it increases readability. Winner!

Think hard about the methods exposed by your Page Objects
To further improve test readability, your Page Objects should (only) expose methods that have business meaning. Looking at the example above, the meat of the test happens on the OpenAccount page, where the new account is created. Next to a load() method used to navigate to the page directly (only use these for pages that you can load directly), it has an open() method that takes two arguments, both of type Account, the POJO we’ve seen before. The first one represents the new account, the second represents the account from which the initial deposit into the new account is made.

If you look at the page where you can open an account in the ParaBank application, you’ll see that there’s not much else to do than opening an account, so it makes sense to expose this action to the test methods that use the OpenAccount Page Object.

Choose good names, then choose better ones
You’ve hopefully seen by now that I tried to choose the names I use in my code very carefully, so as to maximize readability. This is hard. I changed the names of my variables and methods many times when I created this example, and I feel that there’s still more room for improvement.

Long variable and method names aren’t inherently bad, as long as they stick to the point. That’s why, for example, I chose to name the method that opens a new account on the OpenAccount page as open() instead of openAccount().

From the context, it’s clear that we’re opening an account here. It’s a method of the OpenAccount page, and its arguments are of type Account. No need to mention it again in the method name, as I did in an earlier iteration. By the way, I learned this from the Clean Code book, which I think is a very valuable read for automation engineers. Lots of good stuff in there.

Use libraries that help you with readability
Apart from Lombok, I also used the AssertJ library to help me write more readable assertions. So, instead of using the default JUnit assertTrue() method, I can now write

assertThat(newAccountIdIsDisplayed).isTrue();

which I think is easier to read. AssertJ has a lot of methods that can help you write more readable assertions, and I think it’s worth checking out for everybody writing Java test code.

So, all in all, I hope that the example above has shown you that it is possible to write (automation) code that is human readable without adding another layer of abstraction in the form of a tool like Cucumber or SpecFlow. This GitHub repository contains the examples I’ve shown here, plus a couple more tests to show some more example of readable (Selenium) test code.

I’m sure there’s still more room for improvement, and I’d love to hear your suggestions on how to further improve the readability of the test code shown here. My main point, though, is to show you that you don’t need Cucumber to make your tests readable to humans.

On no longer wanting to depend on ExpectedConditions in my C# Selenium WebDriver tests

Those of you that have been writing tests with Selenium in C# for a while might have noticed that as of version 3.11, some often-used things have been marked as deprecated, most notably the PageFactory implementation and the ExpectedConditions class. For those of you that have not read the motivation behind this, here is the rationale behind it, as written by Jim Evans, core contributor to the Selenium C# bindings.

These implementations have been moved to a new location and have been made available as separate NuGet packages (DotNetSeleniumExtras.PageObjects and DotNetSeleniumExtras.WaitHelpers, respectively). As of the time of writing this blog post, however, there’s still no new maintainer for this GitHub repository, meaning that there’s no support available and no certainty as to its future.

Notice of ExpectedConditions class being deprecated, as visible in Visual Studio

And while that is no problem yet, since you can still use the deprecated classes from the Selenium bindings or use the DotNetSeleniumExtras packages, I think it’s a risk to keep on relying on code that is effectively unsupported.

In my case, for PageFactory, that’s not a problem, since I don’t typically use it anyway. For ExpectedConditions, however, it’s a different story. When I’m writing and teaching Selenium automation, I’m pretty much still following the general approach I described a couple of years ago in this post. It still works, it’s readable, easy to maintain and explain and has proven to be effective in many different projects I’ve worked on over the years.

However, it also relies on the use of ExpectedConditions, and that’s something that is bothering me more and more often.

So, while preparing for a training session I delivered recently, I started looking for an alternative way of writing my wrapper methods in C#. And it turns out that C# offers an elegant way of doing so, out of the box, in the form of lambda expressions.

Lambda expressions (also known as anonymous functions) is a function that’s not bound to an identifier, but instead can be passed as an argument to a higher-order function. The concept stems from functional programming (it has been a long while since I’ve done any of that, by the way) but has been part of C# for a while. By the way, lambda expressions are available in Java too, from Java 8 onwards, for those of you not working with C#.

Let’s look at an example. Here’s my implementation for a wrapper function around the default Selenium SendKeys() method that takes care of synchronization and exception handling using ExpectedConditions:

public void SendKeys(By by, string valueToType)
{
    try
    {
      new WebDriverWait(_driver, TimeSpan.FromSeconds(Constants.DEFAULT_TIMEOUT)).Until(ExpectedConditions.ElementToBeClickable(by));
        _driver.FindElement(by).Clear();
        _driver.FindElement(by).SendKeys(valueToType);
    }
    catch (Exception ex) when (ex is NoSuchElementException || ex is WebDriverTimeoutException)
    {
         Assert.Fail($"Exception occurred in SeleniumHelper.SendKeys(): element located by {by.ToString()} could not be located within {Constants.DEFAULT_TIMEOUT} seconds.");
    }
}

Here’s the same wrapper, but now passing a lambda expression of our own as an argument to the Until() method instead of a method in ExpectedConditions:

public void SendKeys(By by, string valueToType)
{
    WebDriverWait wait = new WebDriverWait(_driver, TimeSpan.FromSeconds(Constants.DEFAULT_TIMEOUT));
            
    try
    {
        IWebElement myElement = wait.Until<IWebElement>(driver =>
        {
            IWebElement tempElement = _driver.FindElement(by);
            return (tempElement.Displayed && tempElement.Enabled) ? tempElement : null;
        }
        );
        myElement.Clear();
        myElement.SendKeys(valueToType);
    }
    catch (WebDriverTimeoutException)
    {
        Assert.Fail($"Exception in SendKeys(): element located by {by.ToString()} not visible and enabled within {Constants.DEFAULT_TIMEOUT} seconds.");
    }
}

From the Selenium docs, we learn that the Until() method repeatedly (with a default but configurable polling interval of 500 milliseconds) evaluates the lambda expression, until one of these conditions applies:

  • The expression returns neither null nor false;
  • The function throws an exception that is not in the list of ignored expressions;
  • The specified timeout expires

This explains why I created the lambda expression like I did. Before clicking on an element, I want to wait until it is clickable, which I define as it being both visible (expressed through the Displayed property) and enabled (the Enabled property). If both are true, I return the element, otherwise null. If this happens within the timeout set, I’ll clear the text field and use the standard SendKeys() method to type the specified value. If a WebDriverTimeOutException occurs, I’ll handle it accordingly, in this case by failing the test with a readable message.

Yes, implementing my helper methods like this makes my code a little more verbose, but you could argue that this is a small price to pay for not having to rely on (part of) a library that has been deprecated and of which the future is uncertain. Plus, because I define all these helpers in a single class, there’s really only one place in my code that is affected. The Page Objects and my test code remain wholly unaffected.

This GitHub repository contains a couple of runnable example tests that are written against the ParaBank demo application. This is an updated version of the solution described here. An updated version of this last blog post is coming soon, by the way, so that it reflects some opinions and preferences of mine that have changed since I wrote that one. No longer having to rely on ExpectedConditions, as I discussed here, is but one of those changes.

An introduction to REST API testing in Go with Resty

Just over a year ago, I was working in a team that, among other things, was responsible for developing and running some microservices that delivered data to a front end. It turned out those services were written in Go, a programming language that I had vaguely heard of before but never worked with.

However, as the tester and automation engineer in that team, I was tasked with writing some automated checks that could be made part of the build pipeline, so I decided that it would be a good opportunity to pick up some new skills and tools and write those checks in the same language. Now, Go is a relatively young language, first appearing in 2009. This also meant that the tool set around Go isn’t as mature as in other languages like Java or C#. Despite that, I was able to get these tests up and running fairly quickly.

Recently, I stumbled upon Go again (can’t really remember when and where, not that it’s important) and thought it would be a good idea to revisit the language. In this blog post, I’d like to show you some examples of how to write and run tests against a REST API in Go. The API I’m using for this is once again the Zippopotam.us API.

Now, Go has support for testing built-in, so unlike languages like Java or C#, you don’t need to add a unit testing framework to your project. All you need to do is import the testing library in your code, have your Go file name end with _test.go, write a method with a name starting with Test, run

go test

and you’re good to go. For setting up requests and capturing, parsing and checking the response, I found the Resty library to be very useful. It’s pretty similar to the requests library in Python, or RestSharp for C#. Here’s what a first test, checking the response status code for a correctly formatted GET requests to our API, looks like in Go with Resty:

func Test_GetUs90210_StatusCodeShouldEqual200(t *testing.T) {

	client := resty.New()

	resp, _ := client.R().Get("http://api.zippopotam.us/us/90210")

	if resp.StatusCode() != 200 {
		t.Errorf("Unexpected status code, expected %d, got %d instead", 200, resp.StatusCode())
	}
}

Note that I’m not going to explain all the nuts and bolts of Go and its syntax here. If you’re interested to learn more, I’d recommend this book or taking the Tour of Go.

If we run this test using the go test command, you’ll see that it passes:

Each test method takes an argument of type T (from the testing library), which is used to manage test state. What might strike you as odd (it did for me) is that there’s no assertion in this test like you would expect if you’re familiar with unit testing frameworks like JUnit, NUnit or pytest. The reason is simple: they don’t exist in the Go testing library. The people behind Go have their reasons for it, but this is something I don’t really like. I prefer using assertions, because they make my test code easier to read, plus it saves me from writing all these if-then-else statements myself.

Fortunately, other people thought the same, so there are third party libraries available that let you write assertions. I chose to use Testify, because it not only provides assertions, it also allows you to create test suites and use setup and teardown methods in a way similar to other languages. Here’s what the same test looks like, but now using an assertion provided by Testify:

func Test_GetUs90210_StatusCodeShouldEqual200(t *testing.T) {

	client := resty.New()

	resp, _ := client.R().Get("http://api.zippopotam.us/us/90210")

	assert.Equal(t, 200, resp.StatusCode())
}

Much better, in my opinion. Now, let’s see if we can also check a response header value:

func Test_GetUs90210_ContentTypeShouldEqualApplicationJson(t *testing.T) {

	client := resty.New()

	resp, _ := client.R().Get("http://api.zippopotam.us/us/90210")

	assert.Equal(t, "application/json", resp.Header().Get("Content-Type"))
}

That is pretty straightforward. Next, how about extracting and checking a response body value? The easiest way to do that in Go is to unmarshal (deserialize) the response body into a struct (a data structure in Go). If we want to do that, we first have to define the struct like this:

type LocationResponse struct {
	Country string `json:"country"`
}

This defines a struct LocationResponse with a single element Country. The json:”country” tag tells Go that it should populate this element with the value of the country element in the JSON response body. We don’t have to worry about the other elements in the response, those simply will not be mapped (unless you need them in a check, then you’ll need to add them to the struct, too).

Now, we can write a test that maps the response body to a struct of type LocationResponse and then check the value that has been assigned to the Country element:

func Test_GetUs90210_CountryShouldEqualUnitedStates(t *testing.T) {

	client := resty.New()

	resp, _ := client.R().Get("http://api.zippopotam.us/us/90210")

	myResponse := LocationResponse{}

	err := json.Unmarshal(resp.Body(), &myResponse)

	if err != nil {
		fmt.Println(err)
		return
	}

	assert.Equal(t, "United States", myResponse.Country)
}

As a final example, I’d like to show you how to create a setup method for creating an initial state for all the tests in a suite. Here’s how to do that using Testify and its suite module:

type ZippopotamUsTestSuite struct {
	suite.Suite
	ApiClient *resty.Client
}

func (suite *ZippopotamUsTestSuite) SetupTest() {
	suite.ApiClient = resty.New()
}

func (suite *ZippopotamUsTestSuite) Test_GetUs90210_StatusCodeShouldEqual200() {
	resp, _ := suite.ApiClient.R().Get("http://api.zippopotam.us/us/90210")

	assert.Equal(suite.T(), 200, resp.StatusCode())
}

First, we create a struct ZippopotamUsTestSuite that contains all the objects shared by all tests. In this case, all I need is a Client (a class in Resty) called ApiClient (in Go the variable name comes before the data type when you declare a new variable). We can then write a method called SetupTest() that does the setup for all of our tests, and we use (suite *ZippopotamUsTestSuite) to make our existing test methods part of the suite we defined.

To actually run our test suite, we need to create a ‘regular’ test method and pass the suite we have created to the suite.Run() method of Testify:

func TestZippopotamUsSuite(t *testing.T) {
	suite.Run(t, new(ZippopotamUsTestSuite))
}

If you omit this step, go test will not run the test suite!

All in all, given my experience is mostly with Java, C# and Python, I’ve found writing tests in Go to be a little more cumbersome than in those languages. However, with the right tool set, it is perfectly possible to write readable and well-structured tests in Go, as the examples in this blog post hopefully demonstrated.

I’m keen to further explore writing tests (and other software) with Go, so I’ve recently started learning the language through this Coursera course series. It’s mainly targeted towards those wanting to become a Go developer, and there isn’t much talk of testing, but I’m looking forward to it anyway. I’ll try to share some more tips and tricks on writing tests in Go in the near future.

If you want to read more about writing tests in Go, I recommend you reading this article by Alex Ellis or going through this tutorial.

All code that I’ve used in this blog post can be found here.