Continuous Delivery and user interface-driven test automation: does that compute?

In this post, I’d like to take a closer look at the combination of Continuous Delivery on the one hand and automated tests at the user interface level on the other hand. Is this a match made in heaven? In hell? Or is the truth somewhere out there in between? (Hint: as with so many things in life, it is..).

Continuous Delivery (CD) is an approach in which software development teams produce software and release it into the production environment in very short cycles. Automation of building, testing and deploying the software often is a vital part in achieving CD. Since this is a blog on testing and test automation, I’ll focus on that, leaving the topics of build and deployment automation to those more experienced in that field.

Automated tests on the user interface level (such as those built using Selenium) traverse your complete application from the user interface to the database and back and can therefore be considered as end-to-end tests. These tests are often:

  • relatively slow to execute, since they require firing up an actual browser instance, rendering pages, dealing with page loading times, etc.,
  • demanding in terms of maintenance, since the user interface is among the components of an application that are most prone to change during the software life cycle, and
  • brittle, because object on a web page or in an application are often dynamically generated and rendered and because wait times are not always predictable, making synchronization a tough issue to tackle correctly.

So, we have CD striving for fast and reliable test feedback (CD is hard to do properly when you’ve got flaky tests stalling your builds) one the one hand, and user interface tests and their issues on the other hand. So, is there a place for these tests in the CD pipeline? I’d like to argue there is, if they satisfy a number of criteria.

The tests are actually verifying the user interface
There’s a difference between verifying the user interface itself, and using the user interface to verify (business) logic implemented in lower layers of your application. If you’re merely using the user interface as a means to verify, for example, API or database logic, you should reconsider moving the test to that specific level. The test automation pyramid isn’t popular without reason.. On the other hand, if it IS the user interface you’re testing, then you’re on the right track. But maybe there is a better option…

The user interface cannot be tested as a unit
Instead of verifying your user interface by running end-to-end tests, it might be worthwhile to see whether you can isolate the user interface in some way and test its logic as a unit instead. If this is possible, it will likely result in significant gains in terms of time needed to execute tests. I’ve recently written a blog post about this, so you might want to check that one out too.

The tests are verifying vital application logic
This point is more about the ‘what’ of the tests than the ‘how’. If you want to include end-to-end user interface-driven tests in your CD pipeline, they should verify business critical application features or logic. In other words, ask yourself ‘if this test fails, do we need to stop deploying into production?’ If the answer is yes, then the test has earned its place in the pipeline. If not, then maybe you should consider taking it out and running it periodically outside of the pipeline (no-one says that all tests need to be in the pipeline or that no testing can take place outside of the pipeline!). or maybe removing the test from your test set altogether, if it doesn’t provide enough value.

The tests are optimized in terms of speed and reliability
Once it’s clear that your user interface-driven end-to-end tests are worthy of being part of the CD pipeline, you should make sure that they’re as fast and stable as possible to prevent unnecessarily long delivery times and false negatives (and therefore blocked pipelines) due to flaky tests. For speed, you can for example make sure that there are no superfluous waits in your tests (Thread.sleep(), anyone?), and in case you have a lot of tests to execute – and all these tests should be run in the pipeline – you can see if it’s possible to parallelize test execution and have them run on different machines. For reliability, you should make sure that your error handling is top notch. For example, you should avoid any StaleElementReferenceException occurrence in Selenium, something you can achieve by implementing proper wrapper methods.

In short, I’d say you should free up a place for user-interface driven end-to-end tests in your CD pipeline, but it should be a very well earned place indeed.

An approach to test your user interface more efficiently

As returning readers of this blog might have read before, I am pretty skeptical about writing automated tests that interact with the application under test at the user interface level. These UI tests tend to be:

  • slow in execution, since they are typically end-to-end tests from an application-layer perspective, and
  • demanding when it comes to maintenance, because the user interface is typically a part of an application subject to frequent changes.

However, the user interface often is an important component of an application and therefore it requires testing effort. Since this blog is all about test automation, I’d like to talk about a different approach to user interface test automation in this blog post.

But first, let me explain a subtle yet important difference. On the one hand, there’s testing the user interface. Here, the actual logic incorporated in the user interface is isolated and tested as a unit. Everything that’s happening ‘behind’ the user interface is out of scope of the test and therefore usually mocked. On the other hand, there’s testing application logic through the user interface. This is generally done using tools such as Selenium. This type of automated tests uses the user interface as its point of entry and validation, even though the actual logic that processes the input and generates the output to be checked is not implemented at the user interface layer at all.

Testing the user interface versus testing through the user interface

Now, only when you specifically want to perform end-to-end application tests, or when there really isn’t any other option than to use the user interface drive tests that validate underlying application logic should you resort to tools such as Selenium. In all other cases, it might be a better idea to see if there’s a better option available.

User interface architectures and testability
In the remainder of this post I want to zoom in on a commonly used user interface pattern, namely Model-View-ViewModel, or MVVM in short, and what I think is a suitable approach for writing automated tests for user interfaces adhering to this pattern.

MVVM explained (briefly)
The MVVM pattern lets you separate presentation logic (that defines what information is shown on the screen) from the actual presentation (that defines how the information is displayed). Schematically, the pattern looks like this:

A schematic representation of the Model-View-ViewModel pattern

The View is the user interface, which is made up of textboxes, labels, buttons, etc. It is responsible for defining the structure, layout and appearance of what you seen on the screen. The Model is an implementation of the domain model. It consists of a data model along with business rules and validation logic. The View Model is the intermediary between the View and the Model, handling user interface actions (such as the click of a button) and interacting with models usually by invoking methods in the model classes (for example to update a record in a database). It is also responsible for presenting data from the model (for example the results of a query) to the view in a representation that can be easily displayed on screen.

One of the key ideas behind MVVM and other user interface architectures, such as Model-View-Controller (MVC) and Model-View-Presenter (MVP), is that the separation of concerns ingrained in these architectures makes them testable using unit tests, instead of having to rely on expensive and slow end-to-end tests.

Testing user interfaces built on MVVM
Since all business logic is contained in the view model, it makes sense to make it the center of attention for our testing efforts. Since view models are implemented as classes in object orientedn languages these tests are usually defined as unit tests. This immediately shows the value of asking yourself ‘am I testing the user interface or merely testing my application through the user interface?’. In case of the former, writing complicated, slow and brittle end-to-end tests with tools such as Selenium is pure overkill.

Instead, we write a simple unit test that covers the business logic in the view model. We mock the model part, since business and implementation logic further down the application stack can be tested using (again) unit tests, or when the view model for example invokes web services for storing, processing and retrieving data, we can write API-level tests to cover that part of our application. The view part can – when MVVM is applied correctly – be left out of scope, since buttons, labels and text boxes are usually standardized objects that ‘only’ need to be positioned correctly on screen (in reality, designing good user interfaces is a respectable and skillful job of it’s own, no disrespect intended). If you really want to write checks to verify that the view part is implemented correctly, there’s always tools such as Galen Framework that allow you to check your user interface at the design level (i.e., visual checks).

Links to example implementations and tests
Rather than writing my own example here, I’d like to link to some well-written and understandable examples of the application of MVVM and the way you can write unit tests for your MVVM-based user interface here. I’m currently in the early stages of implementing a suitable testing approach for the user interface of the key application at my current project, so you can expect some real-life examples from my own hand in a future post. For now, I encourage you to take a look at the following blog posts and examples:

Best practice: use UI-driven automated testing only when necessary

Question: what do you think of when I ask ‘what does test automation look like’? Chances are high that you think of a tool that replays user interaction with an application using the graphical user interface. This user interaction is either captured through recording functionality in the tool that is used and subsequently replayed, or it is programmed by a test automation engineer, or a mixture of both approaches is used.

Traditionally, these tools use HTML object attributes to uniquely identify and manipulate objects on the screen. Recently, a number of tools have emerged that use image recognition to look for and manipulate screen objects. Object recognition approach notwithstanding, all of these tools use the user interface to interact with the application under test.

This approach to automated testing is one of the most popular ones out there. For starters, it looks good in demos and sales pitches. More importantly though, it most closely represents how a manual test engineer or an end user would interact with the application under test. However, there’s a fundamental problem attached to UI-based automated testing. Too often, the investment just isn’t outweighed by the profits. Test automation engineers spend hours upon hours on crafting and maintaining wonderful frameworks and intricate scripts, with no one evaluating the ROI for these efforts.

Now, I don’t say that UI test automation shouldn’t be done. It definitely has a place within the overall spectrum of test automation, and I have seen it used with great results in various places. However, I do feel it is overused in a lot of projects. Most of the times, the test automation team or their managers fell into one of the pitfalls of UI test automation:

  • All test scripts are automated using a UI-based test automation approach, even those test cases that aren’t really about the UI.
  • Test automation engineers try to automate every test script available, and then some more. It might be due to their own poor judgment or to the wishes / demands from management to automate all test scripts, but bottom line is that not all test scripts should be automated just because it can be done.
  • The test automation approach is suboptimal with regards to maintainability. Several best practices exist to ensure maximum maintainability for automated UI test scripts, including use of object maps and keyword-driven test frameworks. If these are not applied or if they are applied incorrectly, more time might be required for automated test script maintenance.

Therefore, I’d recommend the use of UI-based test automation only when either (or both) of the following is true:

  1. The test script actually contains validations and/or verifications on the user interface level, or
  2. There is no alternative interface available for interacting with the application.

With regards to the second point, alternative interfaces could include interfaces on web service or on database level. Using these to drive your automated tests remove some of the major drawbacks of UI-based test automation, such as the maintenance burden due to changes in UI object properties and UI synchronization issues.

Case study
I have used the principles outlined above with good results in a project a couple of years ago In this project, I was responsible for setting up an automated test suite for an application built on the Cordys (now OpenText) Business Process Management suite. This application could be decomposed into four tiers: the user interface, a BPM tier, a web service tier and a database tier.

At first, I started out building automated tests on the user interface level, as this BPMS was still new to me and this was the obvious starting point. I soon realized that automating tests on the UI level was going to be very hard work as the user interface was very dynamic with a lot of active content (lots of Javascript and Xforms). If I was to deliver a complete set of automated test scripts, I would either have to invest a lot of time on maintenance or find some other way to achieve the desired results.

Luckily, by digging deeper into the Cordys BPMS and reading lots of material, I found out that it has a very powerful web service API. This API can be used not only to drive your application, but to query and even configure the BPMS itself as well. For instance, using this web service API, you can:

  • Create new instances of the BPM models used in the application under test,
  • Send triggers and messages to it, thus making the BPM instance change state and go to a subsequent state,
  • Verify whether the new state of the BPM instance matches the expected state, and so on..

Most of this could be done using predefined web service operations, so the risk of these interfaces changing during the course of the project was small to none. Using this API, I was able to set up an automated test for 80-90% of the application logic, as the user interface was nothing more than a user-friendly way to send messages to process instances display information about the current state and the next action(s) to take. Result!

Use UI testing only when the user interface is actually tested

Even better, in later projects where Cordys was used, I have been able to reuse most of the automated testing approach and the framework I used to set up and execute automated tests. Maximum reusability achieved and minimum maintenance required, all through a change of perspective on the automated testing approach.

Have you experienced similar results, simply by a shift of test automation perspective? Let me know.