Tackle the hard problems first

When you’re given the task of creating a new test automation solution (or expanding upon an existing one, for that matter), it might be tempting to start working on creating tests right away. The more tests that are automated, the more time is freed to do other things, right? For any test automation solution to be successful and, more importantly, scalable, you’ll need to take care of a number of things, though. These may not seem to directly contribute to the coverage you’re getting with your automated tests, but failing to address them from the beginning will likely cause you some pretty decent headaches later on. So why not deal with them while things are still manageable?

Run tests from within a CI pipeline
If you’re like me, you’ll want to able to run tests on demand, which could be either on a scheduled or on a per-commit basis. You likely do not accept the ‘works on my machine’ attitude from a developer, do you? Also, I’ve been ranting on about how test automation is software development and should be treated as such, so start doing so! Have your tests uploaded to a build server and run them from there. This will ensure that there’s no funny stuff left in your test setup, such as references to absolute (or even relative) file paths that only exist on your machine, references to dependencies that are not automatically resolved, and so on. Also, from my experience, especially user interface-driven tests always seem to behave a little differently with regards to timing and syncing when run from a build server, so running them from there is the ultimate proof that they’re stable and can be trusted.

Take care of error handling and stability
Strongly related to the previous one is taking care of stabilizing your test execution and handling errors, both foreseen and unforeseen. This applies mainly to user interface-driven tests, but other types of automated tests should not be exempt of this. My preferred way of implementing error handling is by means of wrapper methods around API calls (here’s an example for Selenium). Don’t be tempted to skip this part of implementing tests and ‘make do’ with less than optimal error handling. The risk of spending a lot of time getting it right later on, having to implementing error handling in more than one place and spending a lot of time finding out why in the name of Pete your test run failed exactly are just too high. Especially when you stop running tests on your machine only, which, again, you should do as soon as humanly possible.

Have a solid test data strategy
In my experience, one of the hardest problems to get right when it comes to creating automated tests is managing the test data. The more you move towards end-to-end tests, and the more (external) dependencies and systems involved, the harder this is to get right. But even a system that’s operating in a relatively independent manner (and these are few and far between) can cause some headaches, simply because their data model can be so complex. No matter what the situation is you’re dealing with, having the right test data available at the right time (read: all the time) can be very hard to accomplish. And therefore this problem should be tackled as soon as possible. The earlier you think of a solid strategy, the more you’ll benefit from it in the long run. And don’t get complacent when everything is a-ok right now. There’s always the possibility that you’re simply working on those tests that do not yet need a better thought out test data approach, but that doesn’t mean you’re safe forever!

Get your test environment in order
With all these new technologies like containers and virtual machines readily available, I’m still quite surprised to see how hard it is for some organizations to get a test environment set up. I’m still seeing this taking weeks, sometimes. And then the environment is unavailable for hours or even days for every new deployment. Needless to say that’s very counter-effective when you want to be able to run your automated tests on demand. So my advice would be to try and get this sorted out as soon as possible. Make sure that everybody knows that having a reliable and available test environment is paramount to the success of test automation. Because it is. And since all modern systems are increasingly interconnected and ever more depending on components both inside the organization and beyond those walls, don’t stop at getting a test environment for your primary application. Make sure that there’s a suitable test environment available and ready on demand for all components that your integration and end-to-end tests touch upon. Resort to service virtualization when there’s no ‘real’ alternative. Make sure that you can test whenever you want.

In the end, writing more automated tests is never the hard problem. Making sure that all things are in place that enable you to grow a successful test automation suite is. So, tackle that first, while your suite is still small, and increasing the coverage of your automated tests will become so much easier.