Best practice: use UI-driven automated testing only when necessary

Question: what do you think of when I ask ‘what does test automation look like’? Chances are high that you think of a tool that replays user interaction with an application using the graphical user interface. This user interaction is either captured through recording functionality in the tool that is used and subsequently replayed, or it is programmed by a test automation engineer, or a mixture of both approaches is used.

Traditionally, these tools use HTML object attributes to uniquely identify and manipulate objects on the screen. Recently, a number of tools have emerged that use image recognition to look for and manipulate screen objects. Object recognition approach notwithstanding, all of these tools use the user interface to interact with the application under test.

This approach to automated testing is one of the most popular ones out there. For starters, it looks good in demos and sales pitches. More importantly though, it most closely represents how a manual test engineer or an end user would interact with the application under test. However, there’s a fundamental problem attached to UI-based automated testing. Too often, the investment just isn’t outweighed by the profits. Test automation engineers spend hours upon hours on crafting and maintaining wonderful frameworks and intricate scripts, with no one evaluating the ROI for these efforts.

Now, I don’t say that UI test automation shouldn’t be done. It definitely has a place within the overall spectrum of test automation, and I have seen it used with great results in various places. However, I do feel it is overused in a lot of projects. Most of the times, the test automation team or their managers fell into one of the pitfalls of UI test automation:

  • All test scripts are automated using a UI-based test automation approach, even those test cases that aren’t really about the UI.
  • Test automation engineers try to automate every test script available, and then some more. It might be due to their own poor judgment or to the wishes / demands from management to automate all test scripts, but bottom line is that not all test scripts should be automated just because it can be done.
  • The test automation approach is suboptimal with regards to maintainability. Several best practices exist to ensure maximum maintainability for automated UI test scripts, including use of object maps and keyword-driven test frameworks. If these are not applied or if they are applied incorrectly, more time might be required for automated test script maintenance.

Therefore, I’d recommend the use of UI-based test automation only when either (or both) of the following is true:

  1. The test script actually contains validations and/or verifications on the user interface level, or
  2. There is no alternative interface available for interacting with the application.

With regards to the second point, alternative interfaces could include interfaces on web service or on database level. Using these to drive your automated tests remove some of the major drawbacks of UI-based test automation, such as the maintenance burden due to changes in UI object properties and UI synchronization issues.

Case study
I have used the principles outlined above with good results in a project a couple of years ago In this project, I was responsible for setting up an automated test suite for an application built on the Cordys (now OpenText) Business Process Management suite. This application could be decomposed into four tiers: the user interface, a BPM tier, a web service tier and a database tier.

At first, I started out building automated tests on the user interface level, as this BPMS was still new to me and this was the obvious starting point. I soon realized that automating tests on the UI level was going to be very hard work as the user interface was very dynamic with a lot of active content (lots of Javascript and Xforms). If I was to deliver a complete set of automated test scripts, I would either have to invest a lot of time on maintenance or find some other way to achieve the desired results.

Luckily, by digging deeper into the Cordys BPMS and reading lots of material, I found out that it has a very powerful web service API. This API can be used not only to drive your application, but to query and even configure the BPMS itself as well. For instance, using this web service API, you can:

  • Create new instances of the BPM models used in the application under test,
  • Send triggers and messages to it, thus making the BPM instance change state and go to a subsequent state,
  • Verify whether the new state of the BPM instance matches the expected state, and so on..

Most of this could be done using predefined web service operations, so the risk of these interfaces changing during the course of the project was small to none. Using this API, I was able to set up an automated test for 80-90% of the application logic, as the user interface was nothing more than a user-friendly way to send messages to process instances display information about the current state and the next action(s) to take. Result!

Use UI testing only when the user interface is actually tested

Even better, in later projects where Cordys was used, I have been able to reuse most of the automated testing approach and the framework I used to set up and execute automated tests. Maximum reusability achieved and minimum maintenance required, all through a change of perspective on the automated testing approach.

Have you experienced similar results, simply by a shift of test automation perspective? Let me know.

Generating automated tests from specifications

In most cases, the implementation of automated testing focuses on the automated execution of previously defined test cases. These test cases come in the form of test scripts or use cases that are prepared by test analists or test engineers. Often, before the introduction of automated testing in a project and / or team, these test cases were executed by hand in the past.

However, what if we could also automate the creation of test cases? When applied correctly, this would yield an even more efficient testing process. Furthermore, the number of mistakes in the derivation of test cases would be reduced to 0 (given that the test case generator is functioning correctly). These two factors indicate that an interesting case might be made, given the right circumstances.

Of course, this does not apply everywhere. To be able to automatically generate test cases, the following points should be observed:

Are my specifications recorded in a format that is suitable for automated test case generation?
Of course, the specifications should be recorded in such a way that they can be interpreted by your test case generator. It is highly unlikely that specifications written in natural language in a Word document can be read and understood by a piece of code to the extent that correct test case specifications can be derived from it (however, if you have ever achieved this, please let me know :). On the other hand, when (part of) the specifications for your SUT are captured in a formal model, certain types of test cases might be derived from this by an intelligent piece of software. This is sometimes achieved in Model Based Testing approaches.

Also, in order to validate whether business rules are implemented correctly in your SUT, it might be feasible to automatically derive test cases from these rules, given that they are specified in some sort of (semi-)formal language.

I’m sure there are more examples where generating test cases with a single keystroke or mouse click is feasible.

Is there a profit to be gained from the ability to generate test cases automatically?
Of course, the fact that something can be automated doesn’t mean it necessarily should be automated. However, when there are a lot of test cases to be generated, or when specs change frequently throughout the software development process, which occurs more and more often with the rise in popularity of Agile and Scrum methodologies, developing and testing a test case generator might be beneficial. As with the previous factor, I’m sure there are more possibilities for profit here, but these two stand out for me at the moment.

A case study
In a recent project I have been able to successfully implement not only the automated execution of test cases, but also the generation of these test cases directly from the specifications as provided by our client.

The project concerned the testing of a web service that validates fixed-length messages for syntactical correctness, semantical integrity and adherence to a number of business rules. There were around 20 types of messages, each consisting of a couple of hundreds of fields. Each field and each business rule has a corresponding error code that is returned by the server in case either the field specifications or the business rules are violated. Needless to say, that’s a lot of test cases.

Specifying these test cases by hand would take a very long time, would be tiresome as the test cases are highly similar and would therefore be pretty error-prone as well. Even testers get bored every now and then.. However, the similarity and the number of the set of test cases also makes a pretty good case for automating test case generation.

First, we need to determine what types of test cases we can identify and which of these can be generated automatically. In this article, I will focus on the negative test cases, i.e., the test cases for which the web service should return one or more error codes. For this project, I identified the following types of negative test cases:

  • Leave mandatory fields empty
  • Assign alphanumeric values to numeric fields
  • For fields that may only contain values from a predefined enumeration: enter values that are not in the enumeration
  • Simple business rules such as ‘IF field A contains value X THEN field B should contain value Y’
  • More complex business rules such as ‘The message may contain up to three address records whenever the client is of type Z’

I chose to skip the test case where the mandatory field was omitted altogether. In fixed length messages, this will cause all subsequent fields to be out of place, effectively rendering the message useless.

Next, I wrote some lines of code (in Jython, in my case) to read the message specifications (these were provided in Excel format, not ideal but manageable using the Apache POI library) and derive test cases from them. For example, a field with the following characteristics:

  • Mandatory
  • Length 1
  • Numerical
  • Values allowed: 1,2,9

yields the following negative test cases:

  • Leave the field empty
  • Enter value ‘A’ in the field
  • Enter value 3 in the field

Writing the code to generate these test cases took me a day, far shorter than it would have taken to generate all test cases by hand. Let alone what would happen when a new version of the message specs would have arrived somewhere halfway through the testing process..

Next, I had to generate the test messages for each and every test case. I did this by using a test message template, which was effectively a valid message (i.e., it does not generate error messages upon validation), containing all possible fields. Then, for every negative test case, I replaced the correct value for the field under test in that test case with an incorrect value depending on the type of test case. For example, when the test message template looks like this:


and the field under test is the last position in the test message, the generated test messages for the test cases mentioned above look like this:


Then, I’d send the test message to the validation web service and checked whether the response from the web service contained a message stating that the field under test had an error in it, and whether the correct error code was returned by the service.

As a result, I was able to quickly generate literally thousands of test cases, many of which had never been executed before (as there was no time and no-one had though of generating test cases instead of manually specifying them). This resulted in very high test coverage and quick turnaround time, and a very happy customer overall.

Hopefully the case study above shows an example of the power of generating test cases. Again, it might not be achievable or economical everywhere, but when it does, the results may be very beneficial. Whenever system specifications are available in a format that can be processed automatically (such as Excel, text or XML), there might just be an opportunity to save lots of time using test case generation.

Best practice: focus on repeatability of your automated tests

This is the first installment in a series of posts on test automation best practices. Notwithstanding the rapid growth and evolution of the test automation field, a number of best practices can be identified that stand the test of time. Adhering to these best practices will improve the added value of your automated tests, no matter the scale or scope of the test, or the technology or tools that are used to design, implement or execute automated tests.

In order for your automated test suites to be truly efficient, they should be set up with repeatability in mind. This means that your tests should execute with the click of a button, or by entering a command in the command prompt, again and again, without the need for manual intervention during or in between test runs. They should also yield the same (or comparable) test results every single time. Except when the system under test changes or fails, of course.

To achieve or improve repeatability, you need to pay attention to a number of things during the implementation of your automated tests. I will address some of these in this article. There are probably lots of other aspects to be considered, but these stand out for me.

One disclaimer: the repeatability factor does not apply (or applies to a far lesser extent) to projects where there’s just a single test run to be executed, for instance after a conversion or a migration project. If you’re involved in such a project, it’s probably not worth it to put extra effort in achieving repeatability of your automated tests.

Start small
As with every software development project, it is best to start small when implementing automated tests. Automate one or two test cases, or even just one or two steps of the process to be automated, and execute them over and over again to make sure they are stable and repeatable. Once your small test cases are proven to be repeatable you can build on them to create larger test suites. Make sure you prove that every major change you make to your tests does not compromise the repeatability of your tests.

Watch your test data
An important issue when designing and implementing repeatable automated tests is the use of test data. Scripts that alter or consume test data need some extra attention with regards to repeatability. A test data object, such as a customer, an order, etc., used in a certain test run may be altered or removed during that run, rendering it unsuitable or unavailable for subsequent test runs.

Roughly speaking, there are three possible approaches for dealing with test data that is altered during a test run:

  • Create the test data during the test run. For example, if your test script covers the processing of an order, have your script create a new order before processing it to make sure there’s always an order to be processed
  • Reset the test data to its original state after the test run. For example, if your test script covers changing a customer’s address to a foreign location, reset it to its original value after the test script has been executed (through whichever interface available).
  • Select the test data to be used at the start of your test run. Rather than using previously defined sets of test data, have your script perform a query on the available test data set to select a test data object to use in a particular test run. Make sure that your script can handle occasions where there’s no suitable test data object available.

Be ready for continuous integration
With the current trend of test and development teams working closer together in increasingly shorter development and test cycles (think Agile / Scrum and DevOps), continuous integration (CI) is applied not only for development tasks, but for system and integration testing as well. In order to be able to keep up with the development team, automated tests should seamlessly integrate with the continuous integration platform in use. Most open source and COTS automated test tools provide a command line interface to execute test runs and export and distribute test result reports. This doesn’t make your test scripts automagically suited for use in a CI environment. Only test scripts or frameworks that can be run again and again without the need for manual intervention can be successfully integrated in the CI process, so make sure yours fit the bill!

A schematic representation of continuous integration

A schematic representation of continuous integration

Effects of repeatability on the acceptance and the ROI of test automation
Once you have managed to control the repeatability of your automated test scripts, you should see some pretty positive results with regards to the acceptance of automated testing and the ROI associated with the test automation project:

  • Repeatable tests can be run on demand, as often as required, leading to a dramatic reduction of the cost per test run and the time needed to complete a development/test cycle
  • Automated tests that can run unattended and that can be repeated on demand will appeal to everybody from developers to upper management, increasing its perceived value and ultimately also increasing the trust in the product delivered by your team.

Are your tests as repeatable as they can be? Let me know how you achieved your degree of repeatability and the issues you had to overcome!