Some thoughts on the future of test automation

As an independent consultant, keeping up with current trends in the field (in this case test automation, but this goes for all consultants regardless of the field they’re working in) is key to remaining relevant, hireable and able to land interesting projects and other gigs (writing, speaking, etc.). For me, the annual summer holiday is a great time to take a step back and reflect on what I’ve been working on the past year, and the trends I’ve seen emerge. In this post, I’d like to share some of the thoughts that have come up.

Test automation is getting closer to software development
The first observation I’ve made from the last couple of projects I’ve been working on is that test automation is moving closer to software development in two ways:

  • Test automation is regarded more and more as software development, including the use of development patterns and practices, and
  • test automation is adopted and executed more and more by developers

The first way is definitely a good thing. I’ve been saying this for some time now – and brighter persons have done so long before me – but test automation is finally starting to be seen as a proper software development task.

I believe that the second way is beneficial too (if only for the fact that developers are best at developing software, of which test automation is a form as we’ve just stated), although there are some side notes that need to be made. First of all, completely leaving test automation to developers probably isn’t a good thing – at least not yet. There’s the effects caused by developers marking their own paper, as well as the risk of losing the testers mindset. Angie Jones wrote a great blog post about this recently, which I think you should all read. I believe this means that the demand for proper test automation specialists (not being developers) will remain or even increase for the foreseeable future. Developers can work out the technical details for the test automation solution, something which they tend to like doing best in my experience, while the test automation specialists provides input, oversees that test automation remains efficient and meets business needs and helps with designing, development and implementation where necessary.

Can we please stop testing through the user interface?
I’ve elaborated on this in a recent blog post, but the amount of questions I see related to user interface-driven test automation (especially related to Selenium) keeps surprising me. Yes, I know a lot of current applications are web applications, but I refuse to believe that there aren’t more efficient ways to apply test automation to at least part of these applications other than automating all kinds of scenarios through the user interface. Again: please only use user interface-driven test automation when:

  • you’re actually testing the user interface itself, or
  • you’re performing end-to-end application tests (and these tests should make up only a small part of your overall test set)

In all other situations, please try and find more effective means of driving automated tests. This will result in faster, more stable and probably better maintainable test suites, which in turn will lead to a more positive attitude towards your efforts and test automation in general. The community thanks you.

The death of codeless test automation ‘solutions’ (finally!)
Also, my crystal ball told me that – finally – everybody starts to see that codeless test automation solutions just aren’t worth it. The points I made above only reinforce this statement. I’m seeing less and less advertising for these types of tools, and I think that’s a good thing. In my experience, these tools often do not live up to the hype past the first sales demo or two. Simply put: test automation equals software development equals writing code.

Increase of test automation scope
Finally, I think that the scope of test automation efforts is expanding beyond replicating the interaction between a user or a system and the application under test in an automated manner. Eloquently argumented by Richard Bradshaw in his presentation at the 2016 Test Automation Day, test automation is so much more than that. Automatically parsing log files generated through an exploratory testing session? That’s test automation. A script to set up or clear test data before or after a test session? That’s test automation. A tool that helps recording and sharing observations made when testing? You get the point. There’s so much to gain through smart applicatin of test tools other than simply automating predefined test scripts..

Please note that the above is not meant as a complete representation of my vision on the future of test automation, but rather as a list of a couple of thoughts I’ve had over a decent glass of wine (or two) during the holiday. Got any predictions about the recent or distant future of the test automation field yourself? Thought of anything brilliant during your own holiday? Please share in the comments!

Do you want a framework or a solution?

Reading through recent comments on this blog, as well as questions that are being asked in some of the LinkedIn groups I am a member of, something has started to strike me as odd. It seems like that in every other comment, someone is asking ‘how do I add so and so to my automation framework?’ or ‘how do I design a framework that does X, Y and Z?’. It isn’t the question as such that is the problem (if you can call me thinking of something as odd a problem in the first place), questions are good. You learn a lot by asking questions, and even more by asking the right questions. No, what gets to me is that all of these questions refer to automation frameworks. Why does it seem that so many people are so focused on ‘designing’ a ‘framework’? In this post, I’d like to explain why I think it’s better not to focus on frameworks in test automation, but shift your attention to something much more important.

First of all, what is a framework anyway? If you’d ask me, this is a framework:

Now this is what I'd call a framework

But within the test automation realm, what does a framework constitute? Wikipedia defines a test automation framework as:

A test automation framework is an integrated system that sets the rules of automation of a specific product. This system integrates the function libraries, test data sources, object details and various reusable modules. These components act as small building blocks which need to be assembled to represent a business process. The framework provides the basis of test automation and simplifies the automation effort.

So, it’s an integrated system of libraries, modules and other stuff that sets the rules of automation. And there’s exactly the main gripe I have with talking and thinking about automation in terms of frameworks. Your framework shouldn’t be the one setting the rules for automation. Instead, your application under test, the quality of that application you want to have achieved when deploying it into production and the tests and checks that form the evidence of this quality should be the ones setting the rules. Test automation is merely one of the possible means to adhere to these rules.

When you start your test automation efforts by building a framework, either from scratch or by glueing together two or more existing tools and only then try and fit in all tests you want to perform, you’re approaching the problem from the wrong end. Trying to fit everything into an existing framework limits flexibility, as your prime concern becomes ‘how can I perform this test using my framework?’ instead of ‘what is the best way to execute this test?’. It’s a bit like buying a two-bedroom apartment and only then realizing you’ve got six children. And a horse. Sure, it might fit, but it’s not necessarily going to be comfortable. It’s the same with test automation. Building a framework (the two-bedroom apartment) and then trying to fit in all sorts of tests (your six children, plus yourself and your significant other) will probably not lead to the most efficient (comfortable) situation. And don’t get me started on that horse..

Consider the following scenario: you have created a number of automated tests for your web application using Selenium (for browser automation), Cucumber (for BDD support) and a tool such as TestNG or JUnit (for assertions and possibly for reporting). At some point in time, you’re asked to also start writing automated tests for the newly built API that exposes part of the application logic. What do you do?

  1. Do you write some more tests that exercise the user interface, which in turn invoke the API ‘under the hood’?
  2. Do you search for an API testing tool that fits best inside your framework, glue it all together, start specifying features and scenarios for the API tests and automate away?
  3. Or do you look for the tool that best supports your actual API testing needs, use that and make it part of your overall testing activities, for example by making API tests a stage in your Continuous Integration or Continuous Delivery process?

If you chose #1, then I have three words for you (well, technically, four): don’t do that. If you opted for #2, you might have thought that API testing capabilities would be a nice addition to your framework, making it more versatile than it was before. But are you sure you can specify the behavior of your API in the same manner (i.e., describing the business value with the same amount of clarity) in Given/When/Then format? What if you discarded an API testing tool that fits the task at hand much better, just because it didn’t fit in your framework?

Option #1 and especially #2 are examples of what I’d like to call ‘framework think‘. While you think you’re building something flexible and reusable, you´re really limiting yourself by discarding solutions that do not fit the framework. Option #3, on the other hand, describes what I’d call ‘solution think‘, in my opinion a far better way of thinking and talking about test automation.

Test automation efforts should serve a higher purpose: solving the problem of giving fast and reliable insight into specific aspects of application quality. Every decision that’s made when implementing test automation, from the decision whether or not to automate at all down to selecting a specific tool to perform a certain test, should be made with this problem in mind. You should always ask: does this decision move me closer to providing a solution? And to what problem? The second question is equally, if not more important, since the perfect solution to a problem that’s irrelevant is just as wasteful as a solution that’s not right in the first place.

Stop framework think, start solution think

So, in short, here’s my advice: stop ‘framework think’. Instead, start practicing ‘solution think’. It might be only a change in mindset, but sometimes that’s all it takes to get you in the right direction.

Thanks to Alan Richardson for putting me on track to write this post with a quote from his 2015 Test Automation Day keynote. He also wrote a somewhat related post recently, which I think you should all read.

Continuous Delivery and user interface-driven test automation: does that compute?

In this post, I’d like to take a closer look at the combination of Continuous Delivery on the one hand and automated tests at the user interface level on the other hand. Is this a match made in heaven? In hell? Or is the truth somewhere out there in between? (Hint: as with so many things in life, it is..).

Continuous Delivery (CD) is an approach in which software development teams produce software and release it into the production environment in very short cycles. Automation of building, testing and deploying the software often is a vital part in achieving CD. Since this is a blog on testing and test automation, I’ll focus on that, leaving the topics of build and deployment automation to those more experienced in that field.

Automated tests on the user interface level (such as those built using Selenium) traverse your complete application from the user interface to the database and back and can therefore be considered as end-to-end tests. These tests are often:

  • relatively slow to execute, since they require firing up an actual browser instance, rendering pages, dealing with page loading times, etc.,
  • demanding in terms of maintenance, since the user interface is among the components of an application that are most prone to change during the software life cycle, and
  • brittle, because object on a web page or in an application are often dynamically generated and rendered and because wait times are not always predictable, making synchronization a tough issue to tackle correctly.

So, we have CD striving for fast and reliable test feedback (CD is hard to do properly when you’ve got flaky tests stalling your builds) one the one hand, and user interface tests and their issues on the other hand. So, is there a place for these tests in the CD pipeline? I’d like to argue there is, if they satisfy a number of criteria.

The tests are actually verifying the user interface
There’s a difference between verifying the user interface itself, and using the user interface to verify (business) logic implemented in lower layers of your application. If you’re merely using the user interface as a means to verify, for example, API or database logic, you should reconsider moving the test to that specific level. The test automation pyramid isn’t popular without reason.. On the other hand, if it IS the user interface you’re testing, then you’re on the right track. But maybe there is a better option…

The user interface cannot be tested as a unit
Instead of verifying your user interface by running end-to-end tests, it might be worthwhile to see whether you can isolate the user interface in some way and test its logic as a unit instead. If this is possible, it will likely result in significant gains in terms of time needed to execute tests. I’ve recently written a blog post about this, so you might want to check that one out too.

The tests are verifying vital application logic
This point is more about the ‘what’ of the tests than the ‘how’. If you want to include end-to-end user interface-driven tests in your CD pipeline, they should verify business critical application features or logic. In other words, ask yourself ‘if this test fails, do we need to stop deploying into production?’ If the answer is yes, then the test has earned its place in the pipeline. If not, then maybe you should consider taking it out and running it periodically outside of the pipeline (no-one says that all tests need to be in the pipeline or that no testing can take place outside of the pipeline!). or maybe removing the test from your test set altogether, if it doesn’t provide enough value.

The tests are optimized in terms of speed and reliability
Once it’s clear that your user interface-driven end-to-end tests are worthy of being part of the CD pipeline, you should make sure that they’re as fast and stable as possible to prevent unnecessarily long delivery times and false negatives (and therefore blocked pipelines) due to flaky tests. For speed, you can for example make sure that there are no superfluous waits in your tests (Thread.sleep(), anyone?), and in case you have a lot of tests to execute – and all these tests should be run in the pipeline – you can see if it’s possible to parallelize test execution and have them run on different machines. For reliability, you should make sure that your error handling is top notch. For example, you should avoid any StaleElementReferenceException occurrence in Selenium, something you can achieve by implementing proper wrapper methods.

In short, I’d say you should free up a place for user-interface driven end-to-end tests in your CD pipeline, but it should be a very well earned place indeed.