The tool is not important

One of the not-so-obvious reasons I spend a lot (too much?) time on LinkedIn is because it regularly provides me with inspiration for thinly veiled rants (also know as blog posts). This one is no exception. Here’s the question that caused this specific blog post to happen:

If a functional tester needs to learn automation, which tool (or programming language) would you recommend and why?

I’ll just forget about this ‘functional tester’ and why he or she ‘needs’ to learn automation (there is a gazillion other ways to contribute to quality software). No, my main gripe with this question is one that I’ve been talking about a lot on this blog, on other media as well as in person: it directly zooms in on the ‘how?’, skipping over the far more important questions of ‘why?’ and ‘what?’ to automate.

Note that I’m not blaming the person who asked the question, he probably meant well and really wanted an answer to his question. And yes, he got a lot of answers, most of them pointing him to Selenium and/or Java. No idea why, since there’s absolutely no context provided with regards to the application for which automated tests need to be written, the skill set of the people who are going to be made responsible for the automation, the overall software development process and lastly but probably most importantly, whether or not there is a need for automation at all.

Granted, I’m taking the question out of context here probably, or at least I’m way overthinking it, but the relentless focus on tools is a real problem in the test automation space, and one that needs fixing. I see it in questions like these, but (and that’s a far bigger problem) in job openings and project offers too. People contact me and ask me to help them introduce and set up test automation, but they’ve already made the decision to go with tool X and Y. The reasoning? ‘Well, we’ve got a couple of spare licenses for those tools’, or ‘Benny from accounting saw a demo and thought it was cool’, The stupid is strong in that one.

So, let me say this once more:

The. Tool. Is. Not. Important.

What you want to do with it is, though. And that’s the question that’s still too often forgotten. Funny thing is, other crafts mastered this a long time ago. For instance, would you trust a handyman that does not ask what needs to be done and why you want something done in the first place, but instead says ‘See you tomorrow at eight. I’ll bring my hammer.’? It could be that he’s psychic and knows you need some woodwork done. In that case, let’s hope he brings a ruler, a saw and some nails too. Or you probably won’t hire him because you’ve got a hunch that a hammer might not be all too useful for that paint job you need done. I think. I’m not really into DIY. But I hope you get my point anyway.

So why is it that we’re still directly jumping to conclusions on which tool to use, bring, buy or build when it comes to test automation? Wouldn’t it be far better if we asked why we need automation in the first place, and what exactly it is that needs to be automated, and in what way? I think it would lead to far better and more effective automation, and save the craft and the software development world of a lot of useless automation efforts.

Only after the ‘why?’ and the ‘what?’ has been covered and answered, it’s time to look at the ‘how?’. And that’s when the tool becomes important. Until then, it’s not.

Is your user interface-driven test automation worth the effort?

I don’t know what’s happening on your side of things, dear reader, but in the projects I’m currently working on, and those that I have been working on in the recent past, there’s been a big focus on implementing user interface-driven test automation, almost always using some implementation of Selenium WebDriver (be it Java, C# or JavaScript). While that isn’t a bad thing in itself (I think Selenium is a great and powerful tool), I’m sometimes wondering whether all the effort that is being put into creating, stabilizing and maintaining these scripts is worth the effort in the end.

Recently, I’ve been thinking and talking about this question especially often, either when teaching different forms of my test automation awareness workshop, giving a talk on trusting your automation or just evaluating and thinking about my own work and projects. Yes, I too am sometimes guilty of getting caught up in the buzz of creating those perfectly stable, repeatable and maintainable Selenium tests, spending hours or sometimes even days on getting it right, thereby losing sight of the far more important questions of ‘why am I creating this test in the first place?’ and ‘will this test pay me back for the effort that I’m putting into creating it?’.

Sure, there are plenty of valid applications for user interface-driven tests. Here’s a little checklist that might be of use to you. Feel free to critique or add to it in the comments or via email. In my opinion, it is likely you’re creating a valuable test if all of these conditions apply:

  • The test simulates how an end user or customer interacts with the application and receives feedback from it (example: user searches for an item in a web shop, adds it to the cart, goes through checkout and receives feedback on the successful purchase)
  • There’s significant risk associated with the end user not being able to complete the interaction (example: not being able to complete a purchase and checkout leads to loss of revenue)
  • There’s no viable alternative available through which to perform the interaction (example: the web shop might provide an API that’s being called by the UI throughout the process, but this does not allow you to check that the end user is able to perform said interaction via the user interface. Web shop customer typically do not use APIs for their shopping needs..)
  • The test is repeatable (without an engineer having to intervene with regards to test environments or test data)

Checking all of the above boxes is no guarantee for a valuable user interface-driven test, but I tend to think it is far more likely you’re creating one if it does.

At the other end of the spectrum, a lot of useless (or ‘less valuable’ if you want the PC version) user interface-driven tests are created. And there’s more than one type of ‘useless’ here:

  • Tests that use the UI to test business logic that’s exposed through an API (use an API-level test instead!) or implemented in code (how about those unit tests?). Not testing at the right level supports shallow feedback and increased execution time. Goodbye fast feedback.
  • Tests that are unreliable with regards to execution and result consistency. ‘Flaky’ is the word I see used a lot, but I prefer ‘unreliable’ or ‘untrustworthy’. ‘Flaky’ sounds like snow to me. And I like snow. I don’t like unreliable tests, though. And user interface-driven tests are the tests that are most likely to be unreliable, in my experience.

What it boils down to is that these user interface-driven tests are by far the hardest to implement correctly. There’s so much to be taken care of: waiting for page loads and element state, proper exception handling, test data and test environment management. Granted, those last two are not limited to just this type of tests, but I find that people that know how to work on the unit or API level are also far more likely to be able to work with mocks, stubs and other simulations to deal with issues related to test data or test environments.

Here’s a tweet by Alan Page that recently appeared in my timeline and that sums it all up pretty well:

So, having read this post, are you still sure that all these hours you’re putting into creating, stabilizing and maintaining your Selenium tests are worth it in the end? If so, I tip my hat to you. But for the majority of people working on user interface-driven tests (again, including myself), it wouldn’t hurt to take a step back every now and then, lose the ‘have to get it working’ tunnel vision and think for a while whether your tests are actually delivering enough value to justify the efforts put into creating them.

So, are your UI automation efforts worth it?

On supporting Continuous Testing with FITR test automation

Last week I had the chance to participate as a contributor to my very first webinar. The people at SeaLights, an Israeli company that offers a management platform for Continuous Testing, asked me to come on the webinar and share my views on test automation and continuous testing. In this post, I’ll share some of the thoughts and opinions I talked about there.

Test automation is everywhere, however…
Test automation is everywhere. That’s probably nothing new.

A lot of organizations are adopting Continuous Integration and Continuous Delivery. Also nothing new.

To be able to ‘do’ CI/CD, a lot of organizations are relying on their automated tests to help safeguard quality thresholds while increasing release speed. Again, no breaking news here.

However, to safeguard quality in CI and CD you’ll need to be able to do Continuous Testing (CT). Here’s my definition of CT, which I used in the webinar:

Continuous Testing is a process that allows you to gauge the quality of your software on demand. No matter if you’re building and deploying once a month or once a minute, CT allows you to get insight into application quality at all times.

It won’t come as a surprise to you that automated tests often form a big part of an organization’s CT strategy. However, just having automated tests is not enough to be able to support CT. Your automated test approach should not just cover application functionality and coverage, but also:

  • A solid test data management strategy
  • Effective test environment management
  • Informative reporting, targeted towards all relevant audiences

And probably many more things that I’m forgetting here..

In order to be able to leverage your automated tests successfully for supporting CT, I’ve come up with a model based on four pillars that need to be in place:

From AT to CT with FITR tests

Let’s take a quick look at each of these FITR pillars and how they are necessary in CT.

Focused
Automated tests need to be focused to effectively support CT. ‘Focused’ has two dimensions here.

First of all, your tests should be targeted at the right application component and/or layer. It does not make sense to use a user interface-driven test to test application logic that’s exposed through an API (and subsequently presented through the user interface), for example. Similarly, it does not make sense to write API-level tests that validate the inner workings of a calculation algorithm if unit tests can provide the same level of coverage. By now, most of you will be familiar with the test automation pyramid. While I think it’s not to be used as a guideline, I find that the pyramid does provide a good starting point for discussion when it comes to focusing your tests at the right level. Use the model to your advantage.

The second aspect of focused automated tests is that your tests should test what they can do effectively. This boils down to sticking to what your test solution and tools in it do best, and leaving the rest either to other tools or to testers, depending on what’s there to be tested. Don’t try and force your tool to do things it isn’t supposed to (here’s an example).

If your tests are unfocused, they are far more likely to be slow to run, to have high maintenance costs and to provide inaccurate or shallow feedback on application quality.

Informative
Touching upon shallow or inaccurate feedback, automated tests also need to be informative to effectively support CT. ‘Informative’ also has two separate dimensions.

Most importantly, the results produced and the feedback provided by your automated tests should allow you, or the system that’s doing the interpretation for you (such as an automated build tool), make important decisions based on that feedback. Make sure that the test results and reporting provided contain clear results, information and error messages, targeted towards the intended audience. Keep in mind that every audience has its own requirements when it comes to this information. Developers likely want to see stack trace, whereas managers don’t. Find out what the target audience for your reporting and test results is, what their requirements are, and then cater to them as best as you can. This might mean creating more that one report (or source of information in general) for a single test run. That’s OK.

Another important aspect of informative automated tests is that it should be clear what they do (and what they don’t do). You can make your tests themselves be more informative through various means, including (but not limited to) using naming conventions, using a BDD tool such as Cucumber or SpecFlow to create living documentation for your tests (blasphemy? maybe.. but if it works, it works), and following good programming practices to make your code better readable and maintainable.

When automated test solutions and the results they produce are not informative, valuable time is wasted analyzing shallow feedback, or gathering missing information, which evidently breaks the ‘continuous’ part of CT.

Trustworthy
When you’re relying on your automated tests to make important decisions in your CT activities, you’d better make sure they’re trustworthy. As I described in more detail in previous posts, automated tests that cannot be trusted are essentially worthless. Make sure to eliminate false positives (tests that fail when they shouldn’t), but also false negatives (tests that pass when they shouldn’t).

Repeatable
The essential idea behind CT (referring to the definition I gave at the beginning of this blog post) is that you’re able to determine application quality on demand. Which means you should be able to run your automated tests on demand. Especially when you’re including API-level and end-to-end tests, this is often not as easy as it sounds. There are two main factors that can hinder the repeatability of your tests:

  • Test data. This is in my opinion one of the hardest ones to get right, especially when talking end-to-end tests. Lots of applications I see and work with have (overly) complex data models or share test data with other systems. And if you’re especially lucky, you’ll get both. A solid test data strategy should be put in place to do CT, meaning that you’ll either have to create fresh test data at the start of every test run or have the ability to restore test data before every test run. Unfortunately, both options can be quite time consuming (if at all attainable and manageable), drawing you further away from the ‘C’ in CT instead of bringing you closer.
  • Test environments. If your application communicates with other components, applications or systems (and pretty much all of them do nowadays), you’ll need suitable test environments for each of these dependencies. This is also easier said than done. One possible way to deal with this is by using a form of simulation, such as mocking or service virtualization. Mocks or virtual assets are under your full control, allowing you to speed up your testing efforts, or even enable them in the first place. Use simulation carefully, though, since it’s yet another thing to be managed and maintained, and make sure to test against the real thing periodically for optimal results.

Having the above four pillars in place does not guarantee that you’ll be able to perform your testing as continuously as your CI/CD process requires, but it will likely give it a solid push in the right direction.

Also, the FITR model I described here is far from finished. If there’s anything I forgot or got wrong, feel free to comment or contact me through email. I’d love to get feedback.

Finally, if you’re interested in the webinar I talked about earlier, but haven’t seen it, it’s freely available on YouTube: