On handling processing time in your integration tests with Awaitility, and on using the Gmail API

For someone that writes and talks about REST Assured a lot, it has taken me a lot of time to find an opportunity to use it in an actual client project. Thankfully, my current project finally gives me the opportunity to use it on a scale broader than the examples and exercises from my workshop. Being able to do so has made me realize that some of the concepts and features I teach in the workshop deserve a more prominent spot, while others can be taught later on, or more briefly.

But that’s not what I wanted to talk about today. No, now that I actually use REST Assured for integration testing, as in: the integration between a system and its adjacent components, there’s something I have to deal with that I haven’t yet had to tackle before: processing time, or the time it takes for a message triggered using the REST API of system A to reach and be processed by system B, and being able to verify its result through consuming the REST API of system B.

Unlike writing tests for a single API, which is how I have been using and demonstrating REST Assured until now, I need a mechanism that helps me wait exactly long enough for the processing to be finished. Similar to user interface-driven automation, I could theoretically solve this by using calls to Thread.sleep(), but that’s ugly and vile and… just no.

Instead, I needed a mechanism that allowed me to poll an API until its response indicated a certain state change had occurred before performing the checks I needed to perform. In this case, and this is the example I’ll use in the remainder of this blog post, I invoked the REST API of system A to trigger a password reset for a given user account, and wanted to check if that resulted in a ‘password reset’ email message arriving in system B, system B being a specific Gmail inbox here.

Triggering the password reset
Triggering the password reset is done by means of a simple API call, which I perform (as expected) using REST Assured:

    body(new PasswordForgottenRequestBody()).

Waiting until the password reset email arrives
As stated above, I could just include a fixed waiting period of, say, 10 seconds before checking Gmail and seeing whether the email message arrived as expected. But again, Thread.sleep() is evil and dirty and… and should be avoided at all times. No, I wanted a better approach. Preferably one that didn’t result in unreadable code, both because I use my code in demos and I’d like to spend as little time as possible explaining my tests to others, and therefore want to keep it as readable as possible. Looking for a suitable library (why reinvent the wheel.. ) I was pointed to a solution that was created by Johan Haleby (not coincidentally also the creator of REST Assured), called Awaitility. From the website:

Awaitility is a DSL that allows you to express expectations of an asynchronous system in a concise and easy to read manner.

I’m not going to write about all of the features provided by Awaitility here (the usage guide does that way better than I ever could), but to demonstrate its expression power, here’s how I used it in my test:

    atMost(10, TimeUnit.SECONDS).
    pollInterval(1, TimeUnit.SECONDS).
    until(() -> this.getNumberOfEmails() == 1);

This does exactly what it says on the tin: it executes a method called getNumberOfEmails() once per second for a duration of 10 seconds, until the result returned by that method equals 1 (in which case my test execution continues) or until the 10 second timeout period has been exceeded, resulting in an exception being thrown. All with a single line of readable code. That’s how powerful it is.

In this example, the getNumberOfEmails() is a method that retrieves the contents for a specific Gmail mailbox and returns the number of messages in it. Before the test starts, I empty the mailbox completely to make sure that no old messages remain there and cause false positives in my test. Here’s how it looks:

private int getNumberOfEmails() {

    return given().

This method retrieves the number of emails in a Gmail inbox (the required OAuth2 authentication details, base URL and base path are specified in the gmailApiRequestSpec RequestSpecification) by means of a GET call to /messages and extracting and returning the value of the resultSizeEstimate field of the JSON response returned by the API. If you want to know more about the Gmail API, its documentation can be found here, by the way.

Checking the content of the password reset message
So, now that we know that an email message has arrived in the Gmail inbox, all that’s left for us to do is check whether it is a password reset message and not any other type of email message that might have arrived during the execution of our test. All we need to do is to once more retrieve the contents of the mailbox, extract the message ID of the one email message in it, use that to retrieve the details for that message and check whatever we want to check (in this case, whether it has landed in the inbox and whether the subject line has the correct value):

String messageID =
        body("resultSizeEstimate", equalTo(1)).

    // Retrieve email and check its contents
        pathParam("messageID", messageID).
        body("payload.headers.find{it.name=='Subject'}.value",equalTo("Password reset"));

Gmail authentication
It took me a little while to figure out how to consume the Gmail API. In the end, this proved to be quite simple but I spent a couple of hours fiddling with OAuth2, authentication codes, OAuth2 access and refresh tokens and the way Google has implemented this. Describing how all this goes is beyond the scope of this blog post, but you can find instructions here. Once you’ve obtained a refresh token, store it, because that’s the code you can use to generate a new access token through the API, without having to deal with the pesky authentication user interface. For those of you more experienced with OAuth2, this might sound obvious, but it took me a while to figure it out. Still, it’s far better than writing automation against the Gmail user interface, though (seriously, DON’T DO THAT).

So, to wrap things up, there are two lessons here.

One, if you’re looking for a library that helps you deal with processing times in integration tests in a flexible and readable manner, and you’re writing your tests in Java, I highly recommend taking a look at Awaitility. I’ve only recently discovered it but I’m sure this one won’t leave my tool belt anytime soon.

Two, if you want to include checking email into your integration or possibly even your end-to-end tests, skip the user interface and go the API route instead. Alternatively, you could try an approach like Angie Jones presents in a recent blog post, leveraging the JavaMail API, instead. Did I say don’t use the Gmail (or Outlook, or Yahoo, or whatever) user interface?

On automation implementation frustration

Recently (as in, in the last couple of months) I’ve wrapped up a few automation projects that have left me less than satisfied. Not because there were no technical challenges – there were plenty. Not because I felt bored – if I’m getting bored, I simply move on. Not because I didn’t learn anything new – I’ve become a much better engineer in the last couple of months and learned lots about creating useful automation efforts.

No, my dissatisfaction was caused by something different. Something I should have seen coming. Something I, in retrospect, should have addressed earlier.

But before I explain what caused this dissatisfaction, first, let’s take a quick look at what a typical project I’m working on looks like. Usually, it starts with a client with a test automation-related challenge. Sometimes they’re just getting started. Sometimes they’ve tried some things already, only to see it fail. In any case, at some point in time, they decide it’s a good idea to hire me (maybe that’s where things go wrong, no comment.).

I then get to work, usually starting out by asking a lot of questions. What does the application under test do? What does the software development process look like? What do the testers do? Where’s the (perceived) risk? What do the stakeholders think to gain from introducing test automation? What have they tried already and why did it fail? All questions that are important for me when deciding on the best possible next step(s).

Then, I usually start getting involved in the actual automation. Sometimes that means building a brand new solution. Sometimes it’s training others to do so, or to maintain and extend what I built. In other projects, it’s running awareness workshops to remind people why they’re implementing test automation in the first place, and to help them get realistic about the expectations around it. Often, it’s a mixture of all of these activities.

I’m not one to boast, but most of the time, things tend to go well during the project. I’ve seen and written enough horrible automation in the past to recognize and know what works and what doesn’t, and as a result, most of the time, I’m able to figure out an approach that brings value to the development process instead of being a time and money drain. So, that’s not the problem. There IS a problem though, and that’s when I start wrapping up.

Too often, by the time I am preparing for my exit from the project, I get the feeling that a lot of the work I’ve done has been in vain. There are no tangible clues that support this feeling, but still, I sometimes just know that once I walk out of the office for the last time, the automation I’ve created will become shelfware. The most important reasons for that? Teams that do not see test automation as software development, and teams that continuously give priority to feature delivery, often pushed by management setting deadlines. The latter is especially cruel when those same organizations claim they ‘do Agile’ and by that are able to ‘deliver software faster’. Sure, that might work in the short term, but it’s not a strategy that will result in a sustainable pace of delivery in the longer term. But I digress.

Now, I’ll be the first one to (at least partly) blame myself for the test automation starting to gather dust once I’m gone. In retrospect, in these past projects I did things right, like deciding on what to automate before starting out, and deciding what approach would be the most effective in terms of coverage, speed of execution and maintainability. However, in some cases, I seem to have forgotten something even more important: creating the right level of awareness and setting the right expectations for the automation. Looking back, I should have made a bigger effort showing the teams and organizations I work with that test automation isn’t something you do once. Or when you have the time. Instead, it’s a software development project within a software development project, and therefore it should be treated as such.

Writing about this here and now means I’ve learned a valuable lesson. But more importantly, I hope to remind you to not make the same mistakes I made too often in the past, by just getting started without keeping the end in mind. I know I will do better in the future. I hope you do too. Start asking questions like ‘who will be responsible for maintaining and extending the automation once I’m gone?’, ‘how are we going to make and keep automation an integral part of our development process?’, and so on. Don’t repeat my mistakes. Start with the end in mind.

As I said, I learned my lesson here. In the project I’m working on at the moment, I am currently working hard at creating the right amount of awareness, and helping the organization decide who is going to take ownership of the automation solutions I create once I’m gone. I set my last day of working at this project on April 26th, so there’s plenty of time. But as with a lot of things, time flies, and making your exit as smooth and as fulfilling as possible isn’t something you can start doing two weeks before you’ll bring the goodbye cake. And this project has an additional complicating factor in that it is the first time that they are executing a software development project on this scale. It’ll make for an interesting three months, I’m sure. But I’m also sure that once I do say goodbye to this team, I know that the automation I delivered will be in good hands. I’m looking forward to walking away much more satisfied this time.

On looking back on 2017 and looking forward to 2018

As I like to do every year, now that 2017 has almost come to an end, I’m carving out some time to reflect on all that I’ve been working on this year. What has been successful? What needs working on? And most importantly, what are the things I’ll put my energy towards in 2018?

The start of On Test Automation – the sole proprietorship
Perhaps the most significant change I made this year was to quit working with The Future Group to venture out on my own under the On Test Automation name. On the other hand, nothing much has changed at all since then. I’m still working as a freelancer, I’m still doing a bit of consulting, some teaching, some writing, pretty much what I’ve been doing before I made this move. The only thing that has changed is the financial side of things, and the fact that I’m now really working for myself instead of mostly for myself. It’s just that little extra bit of freedom, albeit only psychologically. I would be very surprised if, at the end of 2018, I won’t still be freelancing like this. It has proven to be the optimal way of working for me, with total freedom over what I’m working on at what time and with whom. I’d like to further reduce the time I spend commuting a little more in 2018, though.

On the consulting side of things, it has been a pretty strong year. I’ve been working on projects for 5 or 6 different clients, mostly as the person responsible for developing automation solutions, but also sometimes acting more like an automation coach of sorts. I’ve been lucky enough never to have to look for a new project for long. The job market for experienced automation engineers and consultants is so good over here I’ve had to turn down more projects than I’ve been able to accept. I’m considering myself a very lucky person in this respect, although I do like to think that the time I invest in learning, spreading my knowledge and networking has at least partly brought me to where I am now.

Where 80-90% of my working hours in 2017 was spent on consulting work, however, in 2018, I would like to slowly bring that down to around 50% to free up time for other activities.

This year, I’ve written a couple of blog posts, most notably this one, about what I’d like to see changed in education around test automation and what I think good test automation training should look like. As you might have seen elsewhere on this site, I currently offer a couple of courses, and I am looking to expand my offerings in 2018. More importantly, I’d like to deliver significantly more training next year. Counting quickly, I’ve delivered about 10 days of training in 2017, mostly with clients, but I also did a very enjoyable workshop at the Romanian Testing Conference.

For 2018, I’d like to work towards teaching 5 or more days each month. This will require significant effort from my side, not only in actual teaching, but more importanty in marketing and promotion to make sure that I can deliver them in the first place. I’ve front-loaded some of that work by closing partnerships with a couple of other players in the field, and I’ve landed a couple of teaching gigs already (more on that in a future blog post, undoubtedly), but there’s much more work to be done if I want to achieve the ‘5 days of teaching per month’ goal.

2017 was a relatively quiet year for me with regards to conferences. In the Netherlands, I only attended 2 (both organized by TestNet). Abroad, there was the previously mentioned Romanian Testing Conference in May as well as the splendid TestBash Manchester in October, making for a grand total of 4 conferences.

I expect 2018 to be busier on the conference front. In fact, I’ve got my agenda for the spring conference season pretty much filled up already with TestBash Netherlands (where I’ll be doing a workshop together with Ard Kramer), the TestNet spring conference (where I probably won’t be speaking or hosting a workshop for the first time in a while), my second Romanian Testing Conference (where I’ll be doing both a workshop and a talk this time) and the Test Automation Day (which I missed this year due to being on holiday and where I hope to be accepted as a speaker for the first time this coming year). So that’s four conferences before the summer. And that’s not counting the Agile Tour Vienna meetup in March, to which I’m invited to do a talk / live coding session as well. And I haven’t even started to think about the fall conference season yet (that’s a lie, some negotiations are underway).

Including this one, I’ve published 46 blog posts on this site in 2017. I started out with the promise of publishing a blog post every week, and I’ve kept true to my word for most of the year, but last month I came to the conclusion that I’ve been spreading myself a little thin on the writing front. Apart from these blog posts, I also wrote and published 15 articles on other websites, including TechBeacon, StickyMinds and LinkedIn. That’s a lot of writing, I can tell you.

Next year, I’ll probably be blogging less, in an effort to create higher quality output. I’ll also still be doing articles for other websites (I’m working on two of those as we speak). I’m aiming to publish at least one quality blog post on here each month, plus some reviews of conferences, books and other resources whenever I feel like it. That should free up some time to invest in other interesting things that I encounter.

All in all, 2017 has been a great year for me, I’ve met many interesting people and worked on a lot of interesting stuff. 2018 will hopefully be a year of spreading myself a little less thin, instead focusing more on the good stuff. As always, I’ll keep you posted.