Is your user interface-driven test automation worth the effort?

I don’t know what’s happening on your side of things, dear reader, but in the projects I’m currently working on, and those that I have been working on in the recent past, there’s been a big focus on implementing user interface-driven test automation, almost always using some implementation of Selenium WebDriver (be it Java, C# or JavaScript). While that isn’t a bad thing in itself (I think Selenium is a great and powerful tool), I’m sometimes wondering whether all the effort that is being put into creating, stabilizing and maintaining these scripts is worth the effort in the end.

Recently, I’ve been thinking and talking about this question especially often, either when teaching different forms of my test automation awareness workshop, giving a talk on trusting your automation or just evaluating and thinking about my own work and projects. Yes, I too am sometimes guilty of getting caught up in the buzz of creating those perfectly stable, repeatable and maintainable Selenium tests, spending hours or sometimes even days on getting it right, thereby losing sight of the far more important questions of ‘why am I creating this test in the first place?’ and ‘will this test pay me back for the effort that I’m putting into creating it?’.

Sure, there are plenty of valid applications for user interface-driven tests. Here’s a little checklist that might be of use to you. Feel free to critique or add to it in the comments or via email. In my opinion, it is likely you’re creating a valuable test if all of these conditions apply:

  • The test simulates how an end user or customer interacts with the application and receives feedback from it (example: user searches for an item in a web shop, adds it to the cart, goes through checkout and receives feedback on the successful purchase)
  • There’s significant risk associated with the end user not being able to complete the interaction (example: not being able to complete a purchase and checkout leads to loss of revenue)
  • There’s no viable alternative available through which to perform the interaction (example: the web shop might provide an API that’s being called by the UI throughout the process, but this does not allow you to check that the end user is able to perform said interaction via the user interface. Web shop customer typically do not use APIs for their shopping needs..)
  • The test is repeatable (without an engineer having to intervene with regards to test environments or test data)

Checking all of the above boxes is no guarantee for a valuable user interface-driven test, but I tend to think it is far more likely you’re creating one if it does.

At the other end of the spectrum, a lot of useless (or ‘less valuable’ if you want the PC version) user interface-driven tests are created. And there’s more than one type of ‘useless’ here:

  • Tests that use the UI to test business logic that’s exposed through an API (use an API-level test instead!) or implemented in code (how about those unit tests?). Not testing at the right level supports shallow feedback and increased execution time. Goodbye fast feedback.
  • Tests that are unreliable with regards to execution and result consistency. ‘Flaky’ is the word I see used a lot, but I prefer ‘unreliable’ or ‘untrustworthy’. ‘Flaky’ sounds like snow to me. And I like snow. I don’t like unreliable tests, though. And user interface-driven tests are the tests that are most likely to be unreliable, in my experience.

What it boils down to is that these user interface-driven tests are by far the hardest to implement correctly. There’s so much to be taken care of: waiting for page loads and element state, proper exception handling, test data and test environment management. Granted, those last two are not limited to just this type of tests, but I find that people that know how to work on the unit or API level are also far more likely to be able to work with mocks, stubs and other simulations to deal with issues related to test data or test environments.

Here’s a tweet by Alan Page that recently appeared in my timeline and that sums it all up pretty well:

So, having read this post, are you still sure that all these hours you’re putting into creating, stabilizing and maintaining your Selenium tests are worth it in the end? If so, I tip my hat to you. But for the majority of people working on user interface-driven tests (again, including myself), it wouldn’t hurt to take a step back every now and then, lose the ‘have to get it working’ tunnel vision and think for a while whether your tests are actually delivering enough value to justify the efforts put into creating them.

So, are your UI automation efforts worth it?

On supporting Continuous Testing with FITR test automation

Last week I had the chance to participate as a contributor to my very first webinar. The people at SeaLights, an Israeli company that offers a management platform for Continuous Testing, asked me to come on the webinar and share my views on test automation and continuous testing. In this post, I’ll share some of the thoughts and opinions I talked about there.

Test automation is everywhere, however…
Test automation is everywhere. That’s probably nothing new.

A lot of organizations are adopting Continuous Integration and Continuous Delivery. Also nothing new.

To be able to ‘do’ CI/CD, a lot of organizations are relying on their automated tests to help safeguard quality thresholds while increasing release speed. Again, no breaking news here.

However, to safeguard quality in CI and CD you’ll need to be able to do Continuous Testing (CT). Here’s my definition of CT, which I used in the webinar:

Continuous Testing is a process that allows you to gauge the quality of your software on demand. No matter if you’re building and deploying once a month or once a minute, CT allows you to get insight into application quality at all times.

It won’t come as a surprise to you that automated tests often form a big part of an organization’s CT strategy. However, just having automated tests is not enough to be able to support CT. Your automated test approach should not just cover application functionality and coverage, but also:

  • A solid test data management strategy
  • Effective test environment management
  • Informative reporting, targeted towards all relevant audiences

And probably many more things that I’m forgetting here..

In order to be able to leverage your automated tests successfully for supporting CT, I’ve come up with a model based on four pillars that need to be in place:

From AT to CT with FITR tests

Let’s take a quick look at each of these FITR pillars and how they are necessary in CT.

Focused
Automated tests need to be focused to effectively support CT. ‘Focused’ has two dimensions here.

First of all, your tests should be targeted at the right application component and/or layer. It does not make sense to use a user interface-driven test to test application logic that’s exposed through an API (and subsequently presented through the user interface), for example. Similarly, it does not make sense to write API-level tests that validate the inner workings of a calculation algorithm if unit tests can provide the same level of coverage. By now, most of you will be familiar with the test automation pyramid. While I think it’s not to be used as a guideline, I find that the pyramid does provide a good starting point for discussion when it comes to focusing your tests at the right level. Use the model to your advantage.

The second aspect of focused automated tests is that your tests should test what they can do effectively. This boils down to sticking to what your test solution and tools in it do best, and leaving the rest either to other tools or to testers, depending on what’s there to be tested. Don’t try and force your tool to do things it isn’t supposed to (here’s an example).

If your tests are unfocused, they are far more likely to be slow to run, to have high maintenance costs and to provide inaccurate or shallow feedback on application quality.

Informative
Touching upon shallow or inaccurate feedback, automated tests also need to be informative to effectively support CT. ‘Informative’ also has two separate dimensions.

Most importantly, the results produced and the feedback provided by your automated tests should allow you, or the system that’s doing the interpretation for you (such as an automated build tool), make important decisions based on that feedback. Make sure that the test results and reporting provided contain clear results, information and error messages, targeted towards the intended audience. Keep in mind that every audience has its own requirements when it comes to this information. Developers likely want to see stack trace, whereas managers don’t. Find out what the target audience for your reporting and test results is, what their requirements are, and then cater to them as best as you can. This might mean creating more that one report (or source of information in general) for a single test run. That’s OK.

Another important aspect of informative automated tests is that it should be clear what they do (and what they don’t do). You can make your tests themselves be more informative through various means, including (but not limited to) using naming conventions, using a BDD tool such as Cucumber or SpecFlow to create living documentation for your tests (blasphemy? maybe.. but if it works, it works), and following good programming practices to make your code better readable and maintainable.

When automated test solutions and the results they produce are not informative, valuable time is wasted analyzing shallow feedback, or gathering missing information, which evidently breaks the ‘continuous’ part of CT.

Trustworthy
When you’re relying on your automated tests to make important decisions in your CT activities, you’d better make sure they’re trustworthy. As I described in more detail in previous posts, automated tests that cannot be trusted are essentially worthless. Make sure to eliminate false positives (tests that fail when they shouldn’t), but also false negatives (tests that pass when they shouldn’t).

Repeatable
The essential idea behind CT (referring to the definition I gave at the beginning of this blog post) is that you’re able to determine application quality on demand. Which means you should be able to run your automated tests on demand. Especially when you’re including API-level and end-to-end tests, this is often not as easy as it sounds. There are two main factors that can hinder the repeatability of your tests:

  • Test data. This is in my opinion one of the hardest ones to get right, especially when talking end-to-end tests. Lots of applications I see and work with have (overly) complex data models or share test data with other systems. And if you’re especially lucky, you’ll get both. A solid test data strategy should be put in place to do CT, meaning that you’ll either have to create fresh test data at the start of every test run or have the ability to restore test data before every test run. Unfortunately, both options can be quite time consuming (if at all attainable and manageable), drawing you further away from the ‘C’ in CT instead of bringing you closer.
  • Test environments. If your application communicates with other components, applications or systems (and pretty much all of them do nowadays), you’ll need suitable test environments for each of these dependencies. This is also easier said than done. One possible way to deal with this is by using a form of simulation, such as mocking or service virtualization. Mocks or virtual assets are under your full control, allowing you to speed up your testing efforts, or even enable them in the first place. Use simulation carefully, though, since it’s yet another thing to be managed and maintained, and make sure to test against the real thing periodically for optimal results.

Having the above four pillars in place does not guarantee that you’ll be able to perform your testing as continuously as your CI/CD process requires, but it will likely give it a solid push in the right direction.

Also, the FITR model I described here is far from finished. If there’s anything I forgot or got wrong, feel free to comment or contact me through email. I’d love to get feedback.

Finally, if you’re interested in the webinar I talked about earlier, but haven’t seen it, it’s freely available on YouTube:

Romanian Testing Conference 2017 was a blast!

Last week I had the pleasure of taking part in the 2017 edition of the Romanian Testing Conference. I was contacted by Andrei from the organizing committee in August of last year, initially to host a workshop at what would be the first edition of a spin-off conference of the main RTC event. That conference unfortunately had to be cancelled, but Andrei from the organizing committee was kind enough to extend the invitation to this year’s edition of the original event. And what an excellent couple of days they’ve been!

Wednesday: Cluj
Wednesday saw a very early start to the day, with my alarm set at 3.45. My plane to Munich set off at 7.00, and after a quick and easy transfer I suddenly found myself in Romania! After getting into the country through customs I was faced with the first sign of how excellently organized this whole event would be: there was a car with a driver waiting for me at the arrivals hall to drive me from the airport to the hotel. I felt spoiled already!

The official RTC 2017 car

After checking in to the luxurious Grand Hotel Italia I decided to go and see the city for a bit, as this day would be the only day where I’d have a little time to do so. I’m not really a city person (I spent an afternoon in NYC and thought that was enough..) but I’m making a habit of seeing more of the area I’m visiting than just an airport, a hotel and a conference venue. Luckily, the weather was gorgeous and there’s some really good coffee stalls to be found on the streets of Cluj, so it was time well spent.

Upon returning to the hotel, I met some of the other speakers, as well as Rob, the conference chairman. The rest of the day was fairly uneventful, with dinner in my room, watching Office Space for the umpteenth time and an early night. The day had been long enough, plus I thought it might be a good idea to be fresh and well rested in the morning for my workshop.

Thursday: workshop day
Thursday was show time for me, the day of my workshop on REST Assured (mostly) and WireMock (a bit). I heard in advance that my workshop was fully booked, which meant that there were 30 people that registered for it. Normally, when I do training, I’ll try and get no more than 12-15 people, but since this was the fourth or fifth time I’d be giving this workshop and I received exactly 0 emails from attendees that had trouble completing the preparation instructions I’d sent them a couple of weeks in advance, I wasn’t too uncomfortable with that.

Attendees hard at work during my workshop

I was pleasantly surprised that all participants were fully prepared, which doesn’t happen regularly. A great start to the day, because that means no time lost setting up people’s laptops. Instead, we were able to dive into REST Assured directly. I felt the workshop went rather well, the only thing I had a bit of trouble with is getting the interaction going. People asked me enough questions one-on-one when I was walking around when they were working on the exercises I provided, but I wasn’t able to get a lot of plenary discussion going. As a result, it was a bit hard to gauge whether or not people were engaged and interested, or bored and distracted. They seemed to be happy enough with the way I delivered the workshop, though. This was reflected in the ratings I received afterwards:

Ratings for my workshop

For those of you who are interested in what I covered in the workshop, you can find all of the slides, the exercises and the answers on my GitHub page here. Feel free to review, steal and otherwise use them for your own fun and profit. Or book me to deliver it to a place near you 😉

After the workshops were over, it was time for the official speakers dinner. We took taxis to a nice restaurant (the name of it escapes me for now) and I spent a great couple meeting new people (Keith, Beren, Nicola, Elizabeth, Kamila, Viktor and so many others) and catching up with others I met before (Ard, Huib, Rick and others as well). One of the highlights of the whole event for me, even though I felt somewhat knackered after a full day of teaching. After dinner, it was time for a last couple of drinks in the hotel lobby (not a bad place to spend some time either, as you can see below) and off to bed.

The Grand Hotel Italia lobby

Friday: the conference
Because the hard part was over for me after delivering my workshop, I got to enjoy the conference day without the stress that comes with having to do a talk or anything else. This meant I could pick and attend the talks I liked, spend some time wandering around and talking to people, or just zoning out whenever I felt like it. The programme that was put together by the organizing committee was of very high quality, so most of the time there was at least one talk that was worth attending.

During the day, I enjoyed talks about finding and holding on to your passion (Santhosh), AI and Machine Learning (Cristina), introversion (Elizabeth), not talking about testing (Keith), bitter truths in test automation (Viktor, who seems to be able to read my mind), wrapping up projects and moving on (Nicola) and a closing keynote about youngsters and game testing by the awesome Harry (and yes, he’s really only 12).

All in all, another great day, but an exhausting one too. I wasn’t planning on attending the conference after party, but in the end I spent a couple of hours there anyway, talking some more to other speakers and attendees and reflecting on what was simply a wonderful event and an experience I’ll be remembering for a long time.

Saturday: back home
Unfortunately, the plane was scheduled to take off quite early on Saturday morning (my own fault!), but the flights home were uneventful and in the end I was happy to see my wife and kids again. When I’m writing this, I’m still feeling somewhat tired, but it was all more than worth it in the end.

If you’re ever considering attending (or better: speaking at) the Romanian Testing Conference yourself, I can only really recommend it. The organizing committee have put together a wonderful, high quality event and both speakers and attendees are taken care of in the best possible manner. And even though I’m trying to visit events in as many different countries as possible, I’m already considering going again next year!