Remember what your tests are trying to verify

Lately, I’ve been working a lot on Selenium-based test automation solutions. And even though I’m still not overly enthusiastic about creating lots of user interface-driven automated tests, now that I’m getting more skilled at creating robust and stable Selenium tests, I am starting to appreciate the tool and what you can do with it more and more. As everybody working with Selenium can tell you, there are situations where things can get, well, interesting. And by interesting, I mean tricky. Dealing with dynamic front end frameworks and unpredictable modals and overlays can ask a lot of you in terms of debugging and exception handling skills.

Not yet being fluent in Selenese, I find myself on Google and (subsequently) StackOverflow a lot, trying to find suitable solutions for problems I encounter. While doing so, there’s one thing that I see a lot of people do, yet strikes me as really odd, given what I think is the purpose of these user interface-driven tests:

Forcing your tests to do something your users can’t.

From what I’ve understood, studying and working in the test automation field for a while now, user interface-driven tests, such as these Selenium tests, should be used to verify that the user of your user interface is able to complete a sequence of predefined actions. If you’re working in a hip and happening environment, these are often called ‘customer journeys’ or ‘user journeys’. But that’s not my point. What my point is, is that quite often I see workarounds suggested that go beyond what a regular user could do with his or her keyboard and / or mouse.

For example, take an element that is invisible until you hover over it with your mouse. If you just try to do a click(), your test will probably throw an exception stating that the element was not visible. Now, there are (at least) two ways to deal with this situation:

  1. Use the Actions class to simulate a mouseover, then click the element.
  2. Use a JavaScriptExecutor to perform the click.

While both approaches might result in a passing test, I am of the opinion that one is useful and the other is a horrible idea. Based on what I’ve written so far, can you guess which of the two options I’d suggest?

Indeed, option #1 is the way to go, for two reasons:

  • User interface-driven tests should mimic actual user interaction as closely as possible. I’ve never seen a user execute some JavaScript on a page to make an element visible. Either that, or I’m hanging around the wrong group of users..
  • What happens if a front-end developer makes a mistake (I know, they never do, but let’s assume so anyway) which causes the element not to become visible, even on a mouseover? With option #1, your test will fail, for the right reason. With #2, hello false negative!

There are some exceptions to the rule, though. The prime example I can think of is handling file uploads by directly sending the absolute path of the file to be uploaded to the input element responsible for the file upload using sendKeys(), instead of clicking on it and handling the file dialog. I’ve tried the latter before, and it’s a pain, first because you can’t do it with the standard Selenium API (because the file dialog is native to the operating system), second because different browsers use different file dialog layouts, resulting in a lot of pesky code that easily breaks down. In this case, I prefer to bypass the file dialog altogether (it’s probably not the subject of my test anyway).

In (almost) all other cases though, I’d advise you to stick to simulating your end user behavior as closely as possible. The job of your user interface-driven tests, and therefore of you as its creator, is not to force a pass, but to simulate end user interaction and see if that leads to a successfully executed scenario. Don’t fool yourself and your stakeholders by underwater tricks that obscure potential user interface issues.

Remember what your tests are trying to verify.

Let your test automation talk to you

Most of my current projects involve me paving the automation way by discussing requirements and needs with stakeholders, deciding on an approach and the tools used, setting up a solution and some initial tests and then instructing others and handing over the project. I love doing this type of projects, as it allows me (who’s bored quite easily and quickly) to move from project to project and from client to client regularly (sometimes once every couple of weeks). One of the most important factors in facilitating an easy handover of ‘my’ (it isn’t really mine, of course) automation solution to those who are going to continue and expand on it is by making the automation setup as self-explanatory as possible.

This is especially so when those that are going to take over don’t have as much experience with the tools and architecture that has been decided upon (or, in some cases, that I’ve decided to use). Let’s take a look at two ways to make it easy to hand over automation (results) to your successor (stakeholder), or, as I’d like to call it, to have your automation talk to you about what it’s doing and what the results of executing the automated tests are. Not literally talk to you (though that would be nice!), but you know what I mean..

Through the code
One of the most effective ways of making your automation solution easily explainable to others is through the code. Not only should your code be well structured and maintainable, good code requires little to no comments (who’s got time or motivation to write comments anyway?) because it is self-explanatory. Some prime examples of self describing code that I’ve used or seen are:

Having your Page Object methods in Selenium return Page Objects.
When you write your Page Object Methods like this:

public LoginPage SetEmailAddressTo(string emailAddress)
{
    PElements.SendKeys(_driver, textfieldEmailAddress, emailAddress);
    return this;
}

public LoginPage SetPasswordTo(string password)
{
    PElements.SendKeys(_driver, textfieldPassword, password);
    return this;
}

public void ClickLoginButton()
{
    PElements.Click(_driver, buttonLogin);
}

you can write your tests (or your step definition implementations, if you’re using Cucumber or SpecFlow) like this:

[When(@"he logs in using his credentials")]
public void WhenHeLogsInUsingHisCredentials()
{
    new LoginPage().
        SetEmailAddressTo("user@example.com").
        SetPassword("mypassword").
        ClickLoginButton();
}

It’s instantly clear what your tests is doing here, all by means of allowing method chaining through well-chosen method return types and expressive method names.

Choosing to work with libraries that offer fluent APIs
Two prime examples I often use are REST Assured and WireMock. For example, REST Assured allows you to create powerful, yet highly readable tests for RESTful APIs, while abstracting away boilerplate code like this:

@Test
public void verifyCountryForZipCode() {
						
	given().
	when().
		get("http://localhost:9876/us/90210").
	then().
		assertThat().
		body("country", equalTo("United States"));
}

WireMock does something similar, but for creating over-the-wire mocks:

public void setupExampleStub() {

	stubFor(post(urlEqualTo("/pingpong"))
		.withRequestBody(matching("<input>PING</input>"))
		.willReturn(aResponse()
			.withStatus(200)
			.withHeader("Content-Type", "application/xml")
			.withBody("<output>PONG</output>")));
}

One of the main reasons I like to work with both is that it’s so easy to read the code and explain to others what it does, without loss of power and features.

Create fluent assertions
Arguably the most important part of your test code is where the assertions are being made. Of course you need readable arrange (given) and act (when) sections too, but when you’re explaining or demonstrating your code to others, having readable assert (then) sections will be of great help, simply because that’s where the magic happens (so to say). You can make your assertions pretty much self-explanatory by using libraries such as Hamcrest when you’re using Java, or FluentAssertions when you’re working with C#. The latter even allows you to create better readable error messages when an assertion fails:

IEnumerable collection = new[] { 1, 2, 3 };
collection.Should().HaveCount(4, "because we thought we put four items in the collection"));

results in the following error message:

Expected <4> items because we thought we put four items in the collection, but found <3>.

Through the reporting
Now that you’ve got your code all cleaned up and readable, it’s time to shift our attention towards the output of the test execution: your reporting. We’ve seen the first step towards clearly readable and therefore useful reporting above: readable error messages. But good reporting is more than that. First, you need to think of the audience for your report. Who’s going to be informed by your report?

  • Is it a developer? In that case you might want to include details about the test data used, steps to reproduce the erroneous situation and detailed information about the failure (such as stack traces and screenshots) in the report. Pretty much everything that makes it easy for your devs to analyze the failure, so to say. In my experience, more is better here.
  • Is it a manager or a product owner? In that case he or she is probably only interested in the overall outcome (did the tests pass?). Maybe also in test coverage (what is it that we covered with these automated tests?). They’ll probably be less interested in stack traces, though.
  • Is it a build tool, such as Jenkins or Bamboo? In that case the highest priority is that the results are presented in a way that can be interpreted automatically by that tool. A prime example is the xUnit test output format that frameworks such as JUnit and TestNG produce. These can be picked up, parsed and presented by Jenkins, Bamboo and any other CI tool that’s worth its salt without any additional configuration or programming.

Again, think about the audience for your test automation reports, and include the information that’s valuable and useful to them. Nothing less, but nothing more either. This might require you to talk with these stakeholders. This might even require you to create more than a single report for every test run. Doesn’t matter. Have your automation talk to you and to your stakeholders by means of proper reporting.

Using JsonPath and XmlPath in REST Assured

While preparing my REST Assured workshop for the Romanian Testing Conference next month, I ran into a subject I feel I didn’t cover enough in the previous times I hosted the workshop: how to effectively use JsonPath and XmlPath to extract specific elements and element groups in RESTful API responses before verifying them. Here are a couple of tricks I learned since and worked into the exercises that’ll be part of the workshop from now on.

The examples in this post are all based on the following XML response:

<?xml version="1.0" encoding="UTF-8" ?>
<cars>
	<car make="Alfa Romeo" model="Giulia">
		<country>Italy</country>
		<year>2016</year>
	</car>
	<car make="Aston Martin" model="DB11">
		<country>UK</country>
		<year>1949</year>
	</car>
	<car make="Toyota" model="Auris">
		<country>Japan</country>
		<year>2012</year>
	</car>
</cars>

The syntax for JsonPath is very, very similar, except for the obvious lack of support for attributes in JsonPath (JSON does not have attributes).

Extracting a single element based on its index
Let’s get started with an easy example. Say I want to check that the first car in the list is made in Italy. To do this, we can simply traverse the XML tree until we get to the right element, using the index [0] to select the first car in the list:

@Test
public void checkCountryForFirstCar() {
						
	given().
	when().
		get("http://path.to/cars").
	then().
		assertThat().
		body("cars.car[0].country", equalTo("Italy"));
}

Similarly, we can check that the last car came on the market in 2012, using the [-1] index (this points us to the last item in a list):

@Test
public void checkYearForLastCar() {
						
	given().
	when().
		get("http://path.to/cars").
	then().
		assertThat().
		body("cars.car[-1].year", equalTo("2012"));
}

Extracting an attribute value
Just as easily, you can extract and check the value of an attribute in an XML document. If we want to check that the model of the second car in the list is ‘DB11’, we can do so using the ‘@’ notation:

@Test
public void checkModelForSecondCar() {
						
	given().
	when().
		get("http://path.to/cars").
	then().
		assertThat().
		body("cars.car[1].@model", equalTo("DB11"));
}

Counting the number of occurrences of a specific value
Now for something a little more complex: let’s assume we want to check that there’s only one car in the list that is made in Japan. To do this, we’ll need to apply a findAll filter to the country element, and subsequently count the number of items in the list using size():

@Test
public void checkThereIsOneJapaneseCar() {
		
	given().
	when().
		get("http://path.to/cars").
	then().
		assertThat().
		body("cars.car.findAll{it.country=='Japan'}.size()", equalTo(1));
}

Likewise, we can also check that there are two cars that are made either in Italy or in the UK, using the in operator:

@Test
public void checkThereAreTwoCarsThatAreMadeEitherInItalyOrInTheUK() {
		
	given().
	when().
		get("http://path.to/cars").
	then().
		assertThat().
		body("cars.car.findAll{it.country in ['Italy','UK']}.size()", equalTo(2));
}

Performing a search for a specific string of characters
Finally, instead of looking for exact attribute or element value matches, we can also filter on substrings. This is done using the grep() method (very similar to the Unix command). If we want to check the number of cars in the list whose make starts with an ‘A’, we can do so like this:

@Test
public void checkThereAreTwoCarsWhoseMakeStartsWithAnA() {
		
	given().
	when().
		get("http://localhost:9876/xml/cars").
	then().
		assertThat().
		body("cars.car.@make.grep(~/A.*/).size()", equalTo(2));
}

If you know of more examples, or if I missed another example of how to use JsonPath / XmlPath, do let me know!