Is your user interface-driven test automation worth the effort?

I don’t know what’s happening on your side of things, dear reader, but in the projects I’m currently working on, and those that I have been working on in the recent past, there’s been a big focus on implementing user interface-driven test automation, almost always using some implementation of Selenium WebDriver (be it Java, C# or JavaScript). While that isn’t a bad thing in itself (I think Selenium is a great and powerful tool), I’m sometimes wondering whether all the effort that is being put into creating, stabilizing and maintaining these scripts is worth the effort in the end.

Recently, I’ve been thinking and talking about this question especially often, either when teaching different forms of my test automation awareness workshop, giving a talk on trusting your automation or just evaluating and thinking about my own work and projects. Yes, I too am sometimes guilty of getting caught up in the buzz of creating those perfectly stable, repeatable and maintainable Selenium tests, spending hours or sometimes even days on getting it right, thereby losing sight of the far more important questions of ‘why am I creating this test in the first place?’ and ‘will this test pay me back for the effort that I’m putting into creating it?’.

Sure, there are plenty of valid applications for user interface-driven tests. Here’s a little checklist that might be of use to you. Feel free to critique or add to it in the comments or via email. In my opinion, it is likely you’re creating a valuable test if all of these conditions apply:

  • The test simulates how an end user or customer interacts with the application and receives feedback from it (example: user searches for an item in a web shop, adds it to the cart, goes through checkout and receives feedback on the successful purchase)
  • There’s significant risk associated with the end user not being able to complete the interaction (example: not being able to complete a purchase and checkout leads to loss of revenue)
  • There’s no viable alternative available through which to perform the interaction (example: the web shop might provide an API that’s being called by the UI throughout the process, but this does not allow you to check that the end user is able to perform said interaction via the user interface. Web shop customer typically do not use APIs for their shopping needs..)
  • The test is repeatable (without an engineer having to intervene with regards to test environments or test data)

Checking all of the above boxes is no guarantee for a valuable user interface-driven test, but I tend to think it is far more likely you’re creating one if it does.

At the other end of the spectrum, a lot of useless (or ‘less valuable’ if you want the PC version) user interface-driven tests are created. And there’s more than one type of ‘useless’ here:

  • Tests that use the UI to test business logic that’s exposed through an API (use an API-level test instead!) or implemented in code (how about those unit tests?). Not testing at the right level supports shallow feedback and increased execution time. Goodbye fast feedback.
  • Tests that are unreliable with regards to execution and result consistency. ‘Flaky’ is the word I see used a lot, but I prefer ‘unreliable’ or ‘untrustworthy’. ‘Flaky’ sounds like snow to me. And I like snow. I don’t like unreliable tests, though. And user interface-driven tests are the tests that are most likely to be unreliable, in my experience.

What it boils down to is that these user interface-driven tests are by far the hardest to implement correctly. There’s so much to be taken care of: waiting for page loads and element state, proper exception handling, test data and test environment management. Granted, those last two are not limited to just this type of tests, but I find that people that know how to work on the unit or API level are also far more likely to be able to work with mocks, stubs and other simulations to deal with issues related to test data or test environments.

Here’s a tweet by Alan Page that recently appeared in my timeline and that sums it all up pretty well:

So, having read this post, are you still sure that all these hours you’re putting into creating, stabilizing and maintaining your Selenium tests are worth it in the end? If so, I tip my hat to you. But for the majority of people working on user interface-driven tests (again, including myself), it wouldn’t hurt to take a step back every now and then, lose the ‘have to get it working’ tunnel vision and think for a while whether your tests are actually delivering enough value to justify the efforts put into creating them.

So, are your UI automation efforts worth it?

7 thoughts on “Is your user interface-driven test automation worth the effort?

  1. “The test simulates how an end user or customer interacts with the application and receives feedback from it (example: user searches for an item in a web shop, adds it to the cart, goes through checkout and receives feedback on the successful purchase)”

    Where does, in your opinion, the exploratory phase of testing your product UI come into play?

    For instance, you mention significant risk associated with consumers not being able to purchase but can automated testing pick this up reliably 100%?

    • Hey Nick,

      automated tests only do what you tell them to do (and that’s only if they’re built sturdily enough). It will never replace the exploratory testing you’re referring to. However, in my experience, that exploratory testing is usually done for features and applications that are under development, whereas automated tests, especially those acting on the UI level, are much better suited for regression testing, i.e., checking that a change in another part of the application has not led to a breaking change in existing processes. Such as, in this case, the process of ordering and checking out a product in a web shop.

      Both types of tests have their place.

      And no, automated testing does not tend to be able to pick up everything. And you shouldn’t try making it do so. What it CAN reliably do, though, is ensure that after any given change or update the customer can still complete predefined flows through the application.

  2. Bas,
    Not to be snarky, but I think you are missing a fundamental reason why GUI automation is typically done as a first attempt. And I know this will sound lame, but here goes… Because that is what has always been done in the past. Meaning that up until recent times there wasn’t an easy way to get at the Code for Unit Tests or at an API/Service by the Testers. Unit Test and some type of Interface Test (API/Service) was done primarily by the Dev people, and if at all. This is also due to the Dev people not exposing the API/Services in the past for the Test people to try to get at it.

    This is a bad habit that is just now starting to be fixed. Over the last 25+ years I’ve been working with automation I’ve had to primarily deal with the GUI level because the API’s to the application were not exposed for me to get at. For a typical Win32 Desktop application you had to use COM (Common Object Model) to “hook” the API you wanted to test against. That’s if it was compiled as an “exposed” API Entry Point for the DLL. Ninety nine percent of the time the Developers didn’t do that, you had to ask them to do it. And if you were lucky you they begrudgingly did it. With early Web / Browser based applications we didn’t have a “service” to call directly to or the tools in place to do so. Only over the last 8-10 years have we seen the use of SOA come to rise on the majority of systems. SOA allowed for API’s like SOAP to be accessed and leveraged. Now REST is the next thing to work with. Naturally as part of the work of exposing the Service/API call do we have access to them for testing purposes.

    Also, realize the skill/knowledge level of most Test people. I would say a large percentage are not technically knowledgeable and skilled enough to do this type of work. Me, I’m an oddball because I came from a development background and I have lived in this niche for years. But I know a lot of Test people who did not. So they don’t know better. And Management definitely don’t know any better, and they are a key factor to supporting and funding this type of work. Bit of a Catch-22 situation that I’ve seen.

    So I think the first question that should be asked is not should you do GUI automation, it should be is your application under test setup so you can access it at the different layers. If not then how easy is it to get Dev to open it up. How do you get them to build in “Testability”. Once you do that then you have more options to work with.

    But in the end I agree with your statements and say it is time for Testers to become more skilled in technical ways, it can only help them be better at their job.

    • Hey Jim,

      no problem, I value your input highly, as always. What you say about the GUI having been the only way to effectively automated against an application for a long time is something I remember as well (I haven’t been in the field nearly as long as you though 😉 ). And yes, for a lot of testers moving towards or being given (either with their consent or not) the task of writing automated tests the GUI is their first and only entry point into the application, whether or not there are other options. That’s understandable too, yet something I feel needs to change. I think we agree on this.

      The main point I wanted to make, though, was that it doesn’t hurt to periodically take a step back from your vigorous GUI automation attempts to check whether all the effort you’re putting into creating these tests is worth it all. GUI automation has its place, but it should be used with care. I’m pretty sure you’ll agree with me on this as well 🙂

      Again, thanks again for all of your valuable input.

      • Bas,
        Yes, agreed on all points. And that is why I come here to read your posts. You’re teaching an “old dog some new tricks”.

        As your description says “…a test automation and service virtualization consultant who’s always looking for more intelligent ways to use tools to improve test processes and software quality.” And I’m not saying this be a kissass, you are doing some really good work here on your blog. Keep at it. You are keeping me on my toes, and that is a good thing.

        Best to you.

  3. Pingback: Java Web Weekly, Issue 179 | Baeldung

  4. Pingback: Java Testing Weekly 23 / 2017

Leave a Reply

Your email address will not be published. Required fields are marked *