Tackle the hard problems first

When you’re given the task of creating a new test automation solution (or expanding upon an existing one, for that matter), it might be tempting to start working on creating tests right away. The more tests that are automated, the more time is freed to do other things, right? For any test automation solution to be successful and, more importantly, scalable, you’ll need to take care of a number of things, though. These may not seem to directly contribute to the coverage you’re getting with your automated tests, but failing to address them from the beginning will likely cause you some pretty decent headaches later on. So why not deal with them while things are still manageable?

Run tests from within a CI pipeline
If you’re like me, you’ll want to able to run tests on demand, which could be either on a scheduled or on a per-commit basis. You likely do not accept the ‘works on my machine’ attitude from a developer, do you? Also, I’ve been ranting on about how test automation is software development and should be treated as such, so start doing so! Have your tests uploaded to a build server and run them from there. This will ensure that there’s no funny stuff left in your test setup, such as references to absolute (or even relative) file paths that only exist on your machine, references to dependencies that are not automatically resolved, and so on. Also, from my experience, especially user interface-driven tests always seem to behave a little differently with regards to timing and syncing when run from a build server, so running them from there is the ultimate proof that they’re stable and can be trusted.

Take care of error handling and stability
Strongly related to the previous one is taking care of stabilizing your test execution and handling errors, both foreseen and unforeseen. This applies mainly to user interface-driven tests, but other types of automated tests should not be exempt of this. My preferred way of implementing error handling is by means of wrapper methods around API calls (here’s an example for Selenium). Don’t be tempted to skip this part of implementing tests and ‘make do’ with less than optimal error handling. The risk of spending a lot of time getting it right later on, having to implementing error handling in more than one place and spending a lot of time finding out why in the name of Pete your test run failed exactly are just too high. Especially when you stop running tests on your machine only, which, again, you should do as soon as humanly possible.

Have a solid test data strategy
In my experience, one of the hardest problems to get right when it comes to creating automated tests is managing the test data. The more you move towards end-to-end tests, and the more (external) dependencies and systems involved, the harder this is to get right. But even a system that’s operating in a relatively independent manner (and these are few and far between) can cause some headaches, simply because their data model can be so complex. No matter what the situation is you’re dealing with, having the right test data available at the right time (read: all the time) can be very hard to accomplish. And therefore this problem should be tackled as soon as possible. The earlier you think of a solid strategy, the more you’ll benefit from it in the long run. And don’t get complacent when everything is a-ok right now. There’s always the possibility that you’re simply working on those tests that do not yet need a better thought out test data approach, but that doesn’t mean you’re safe forever!

Get your test environment in order
With all these new technologies like containers and virtual machines readily available, I’m still quite surprised to see how hard it is for some organizations to get a test environment set up. I’m still seeing this taking weeks, sometimes. And then the environment is unavailable for hours or even days for every new deployment. Needless to say that’s very counter-effective when you want to be able to run your automated tests on demand. So my advice would be to try and get this sorted out as soon as possible. Make sure that everybody knows that having a reliable and available test environment is paramount to the success of test automation. Because it is. And since all modern systems are increasingly interconnected and ever more depending on components both inside the organization and beyond those walls, don’t stop at getting a test environment for your primary application. Make sure that there’s a suitable test environment available and ready on demand for all components that your integration and end-to-end tests touch upon. Resort to service virtualization when there’s no ‘real’ alternative. Make sure that you can test whenever you want.

In the end, writing more automated tests is never the hard problem. Making sure that all things are in place that enable you to grow a successful test automation suite is. So, tackle that first, while your suite is still small, and increasing the coverage of your automated tests will become so much easier.

Good test automation…

Good test automation…

  • Enhances testing instead of trying to replace it
  • Helps testers do their job, instead of trying to replace them
  • Is treated as a first class citizen software development project (including, but not limited to, proper planning, design and testing, as well as accounting for maintenance)
  • Goes beyond the programming of automated checks, if that helps the team
  • Is focused, informative, trustworthy and repeatable

How good is your automation?

Thanks to (among many, many others) Richard Bradshaw, Michael Bolton, Jim Hazen and Angie Jones for continuously driving forward discussions on and the development of the craft. I learn something new every day, thanks to all of you.

The tool is not important

One of the not-so-obvious reasons I spend a lot (too much?) time on LinkedIn is because it regularly provides me with inspiration for thinly veiled rants (also know as blog posts). This one is no exception. Here’s the question that caused this specific blog post to happen:

If a functional tester needs to learn automation, which tool (or programming language) would you recommend and why?

I’ll just forget about this ‘functional tester’ and why he or she ‘needs’ to learn automation (there is a gazillion other ways to contribute to quality software). No, my main gripe with this question is one that I’ve been talking about a lot on this blog, on other media as well as in person: it directly zooms in on the ‘how?’, skipping over the far more important questions of ‘why?’ and ‘what?’ to automate.

Note that I’m not blaming the person who asked the question, he probably meant well and really wanted an answer to his question. And yes, he got a lot of answers, most of them pointing him to Selenium and/or Java. No idea why, since there’s absolutely no context provided with regards to the application for which automated tests need to be written, the skill set of the people who are going to be made responsible for the automation, the overall software development process and lastly but probably most importantly, whether or not there is a need for automation at all.

Granted, I’m taking the question out of context here probably, or at least I’m way overthinking it, but the relentless focus on tools is a real problem in the test automation space, and one that needs fixing. I see it in questions like these, but (and that’s a far bigger problem) in job openings and project offers too. People contact me and ask me to help them introduce and set up test automation, but they’ve already made the decision to go with tool X and Y. The reasoning? ‘Well, we’ve got a couple of spare licenses for those tools’, or ‘Benny from accounting saw a demo and thought it was cool’, The stupid is strong in that one.

So, let me say this once more:

The. Tool. Is. Not. Important.

What you want to do with it is, though. And that’s the question that’s still too often forgotten. Funny thing is, other crafts mastered this a long time ago. For instance, would you trust a handyman that does not ask what needs to be done and why you want something done in the first place, but instead says ‘See you tomorrow at eight. I’ll bring my hammer.’? It could be that he’s psychic and knows you need some woodwork done. In that case, let’s hope he brings a ruler, a saw and some nails too. Or you probably won’t hire him because you’ve got a hunch that a hammer might not be all too useful for that paint job you need done. I think. I’m not really into DIY. But I hope you get my point anyway.

So why is it that we’re still directly jumping to conclusions on which tool to use, bring, buy or build when it comes to test automation? Wouldn’t it be far better if we asked why we need automation in the first place, and what exactly it is that needs to be automated, and in what way? I think it would lead to far better and more effective automation, and save the craft and the software development world of a lot of useless automation efforts.

Only after the ‘why?’ and the ‘what?’ has been covered and answered, it’s time to look at the ‘how?’. And that’s when the tool becomes important. Until then, it’s not.