Should test automation be left to developers?

I am not a developer. I have a background in Computer Science, I know a thing or two about writing code, but I am definitely not a developer. This is similar to me knowing how to prepare food, but not being a cook. People that have met me in person probably remember and will possible be bored to death by me using this analogy (sorry!).

On the other hand, I also try to repeat as often as possible that test automation is software development, and that it should be treated as such. So, you might ask, how come I am working in test automation, yet I don’t call myself a developer? And more importantly, shouldn’t test automation be left to developers, if it really IS software development? Recent blog posts I’ve read and presentations I’ve heard have made me think a lot about this.

So, to start with the second and most important question: should test automation be left to developers? My answer would be yes.

And no.

Test automation should be left to developers, because
writing automated tests involves writing code. Personally, I don’t believe in codeless solutions that advertise that ‘anybody’ can create automated tests. So, since code writing is involved, and since nobody knows writing code better than a developer, I think that writing automated tests can best be done by a developer.

I’m going to use myself as an example here. Since writing code isn’t in my DNA, I find it takes me three times as long – at the minimum – to write automated tests compared to your average developer. And with high-pressure projects, short release cycles and the trend of having less and less testers for every developer in development teams, I just couldn’t keep up.

To make matters worse, new automation tools are being released what seems like every day. Especially within the Java and JavaScript worlds, although it might be just as good (or bad?) in other ecosystems, just less visible to me personally. I don’t know. For a while, as a non-developer, I’ve tried to keep up with all of these tools, but I just couldn’t manage. So, instead of frantically trying to keep up, I’ve made a career shift. This came naturally after I realized that

Test automation should not be left to developers, because
nobody knows testing better than testers. As I said, I (and possibly many people involved in test automation along with me) am not skilled enough to compete with the best on how to create automated tests. However, I think I (and a lot of others) can teach a lot of people a thing or two about the equally, if not more important questions of why you should (not) do test automation, and what tests should and should not be automated.

A lot of developers I have met don’t have a testing mindset (yet!) and tend to think solely along the happy path. ‘Let’s see if this works…’, rather than ‘let’s see what happens if I do this…’, so to say. When writing automated tests, just as with creating specs for the software that needs to be built, it requires a tester’s mindset to think beyond the obvious happy flow. This is why I think it isn’t wise to see test automation as a task purely for developers.

It also helps if people other than developers feel some sort of responsibility for creating automated tests. Given that most developers are busy people, they need to choose between writing tests and writing production code on a regular basis. Almost always, that decision will be made in favor of production code. This does make sense from a deadline perspective, but not always as much when you look at it from a quality and process point of view. Having people in other roles (testing, for example) stressing the need for a mature and stable automated testing suite will definitely improve the likelyhood of such a suite actually being created and maintained. Which might just benefit the people continuously fretting about those deadlines in the long end..

So, to answer the first question I posed at the beginning of this post: Why am I still working in test automation despite not being a developer? Because nowadays I mainly focus on helping people answer the why and the what of test automation. I do some stuff related to the how question as well, especially for smaller clients that are at the beginning of their test automation journey or that don’t have a lot of experienced developers in house, but not as much as before. I love to tinker with tools every now and then, if only just to keep up and remain relevant and hireable. I’m not as much involved in the day to day work of writing automated tests anymore, though. Which is fine with me, because it plays to my strengths. And I think that’ll benefit everybody, in the long end.

I think the test automation community could use a lot more highly skilled people that are (not) developers.

(Not so) useful metrics in test automation

In order to remain in control of test automation efforts, it’s a good idea to define metrics and track them, so that you can take action in case the metrics tell you your efforts aren’t yielding the right results. And even more important, it allows you to bask in glory if they do! But what exactly are good metrics when it comes to test automation? In this blog post, I’ll take a look at some metrics that I think are useful, and some that I think the automation world can easily do without. Note that I’m not even going to try and present a complete list of metrics that can be used to track your automation efforts, but hopefully the ones mentioned here can move you a little closer to the right path.

So, what do I think could be a useful metric when tracking the effectiveness and/or the results of your test automation efforts?

Feedback loop duration reduction
The first metric I’ll suggest here is not related to the quality of an application, but rather to the quality of the development process. At its heart, test automation is – or at least should be – meant to increase the effectiveness of testing efforts. One way to measure this is to track the time that elapses between the moment a developer commits a change and the moment (s)he is informed about the effects these changes have had on application quality. This time, also known as the feedback loop time, should ideally be as short as possible. If it takes two weeks before a developer hears about the negative (side) effects of a change, he or she will have long moved on to other things. Or projects. Or jobs. If, instead, feedback is given within minutest (or even seconds), it’s far easier to correct course directly. One way to shorten the feedback loop duration is by effective use of test automation, so use it wisely and track the effects that automation has on your feedback loop.

Customer journey completion rate
This might come as a surprise to some, but test automation, testing and software development in general is still an activity that serves a greater good: your customer base. In this light, it would make sense to have some metrics that relate directly to the extent to which a customer is able to use your application, right? A prime example of this would be the amount of predefined critical customer journeys that can (still) be completed by means of an automated test after a certain change to the software has been developed and deployed. By critical, I mean journeys that relate directly to revenue generation, customer acquisition and other such trivialities. The easier and more often you can verify (using automation) that these journeys can still be completed, the more trust you’ll have deploying that shiny new application version into your production environment.

False positives and negatives
Automated tests are only truly valuable if you can trust them fully to give you the feedback you need. That is: whenever an automated test case fails, it should be due to a defect (or an unnoticed change) in your application, not because your automation is failing (you don’t want false negatives). The other way around, whenever an automated test case passes, you should have complete trust that the component or application under test indeed works as intended (you want false positives even less). False negatives are annoying, but at least they don’t go by unnoticed. Fix their root cause and move on. If it can’t be fix, don’t be afraid to throw away the test, because if you can’t trust it, it’s worthless. False positives are the biggest pain in the backside, because they go by unnoticed. If all is green in the world of automation, it’s easy (and quite human) to trust the results, even when all you’re checking is that 1 equals 1 (see also below). One approach to detecting and fixing false positives, at least in the unit testing area, is the use of mutation testing. If this is not an option, be sure to regularly check your automated checks to see that they still have their desired defect detection dint (or 4D, coining my first ever useless automation acronym here!).

Where there are useful metrics, there are also ones that aren’t as valuable (or downright worthless)..

Code coverage
A metric that is often used to express the coverage of a suite of unit tests. The main problem I have with this metric is that in theory it all sounds perfectly sensible (‘every line of our code is executed at least once when we run our tests!’), but in practice, it doesn’t say anything about the quality and the effectiveness of the tests, nor does it say anything about actual application quality. For example, it’s perfectly possible to write unit tests that touch all lines of your code and then assert that 1 equals 1. Or that apples equal apples. These tests will run smoothly. They’ll pass every time, or at least until 1 does not equal 1 anymore (but I think we’re safe for the foreseeable future). Code coverage tools will show a nice and shiny ‘100%’. Yet it means nothing in terms of application quality.

Percentage of overall number of test cases that are automated
An exponent of the ‘automate all the things!!’ phenomenon. In theory, it looks cool: ‘We automated 83,56% of our tests!’. First of all, especially with exploratory testing (not my forte, so I might be wrong here), there is no such thing as a fixed and predetermined number of test cases anymore. As such, expressing the amount of automated tests as a percentage of a variable or even nonexistent number is worthless. Or downright lying (you pick one). There’s only one metric that counts in this respect, quoting Alan Page:

You should automate 100% of the tests that should be automated

Reduction in numbers of testers
Yes, it’s 2016. And yes, some organizations still think this way. I’m not even going to spend time explaining why ‘if we automate our tests, we can do with X fewer testers’ is horrible thinking anymore. However, in a blog post mentioning good, bad and downright ugly metrics related to test automation, it cannot be NOT mentioned.

Anyway, wrapping all this up, I think my view on automation metrics can be summarized like this: automation metrics should tell you something useful about the quality your application or your development process. Metrics related to automation itself or to something that automation cannot be used for (actual testing as a prime example) might not be worth tracking or even thinking about.

On asking ‘why?’ in test automation

This blog post is about something that’s been bothering me for a while, and that keeps coming up for some reason.. Whenever I talk to clients, see a discussion on LinkedIn or StackOverflow or read a blog post on something related to test automation, all too often it’s about ‘how can I solve problem X with tool Y’ (with tool Y suspiciously often being equal to Selenium). The word that buggers me most in this question is ‘how?‘. My knee-jerk reaction to a lot of these ‘how?’ questions I see is ‘why?’. Or more specifically: ‘why the &%$^* would you want to do that in the first place?’.

About half a year ago, I wrote a blog post related to this frustration of mine on LinkedIn. So far, it hasn’t changed the world, since I still see a lot of ‘how?’ where I think ‘why?’ would be a far better question. But since, as it is so eloquently said in Latin (although to me, all Latin sounds pretty eloquent):

“Repetitio mater studiorum est” (“Repetition is the mother of all learning”),

I think it’s worth repeating here as well: With all questions related to test automation, first ask yourself ‘why?’ before even thinking about the ‘how?’.

‘Why?’ prevents you from automation for automation’s sake
Before asking ‘how can I implement test automation most effectively?’, ask ‘why do I want to implement test automation in the first place?’. Implementation of test automation should be a conscious decision, motivated by tangible and significant benefits to the overall software development process (and in the end, to the business objectives of the organization), not an activity that is adopted just because it sounds cool, or because *shudder* everybody else is doing it.

‘Why?’ steers your efforts in the right direction
Before asking ‘how can I automate this test?’, ask ‘why do I want to automate this test in the first place?’. Don’t become the world’s best automator of useless tests. Instead, become the world’s best selector of useful tests to automate. Selecting those tests that give you the most valuable information about the quality and the risk associated with the application you’re developing and delivering can only be done by asking ‘why?’ first. Only after you’ve decided on the best possible set of tests, start exploring how you can automate those tests in the most effective way.

‘Why?’ makes sure you use your tools in the best possible way
Before asking ‘how can I use tool X to automate this test?’, ask ‘why should I use tool X to automate this test?’. In a recent blog post, I talked (or ranted) about abusing Selenium for API tests. Even though the blog post was meant to be – at least partly – satirical, I see similar things happening on a regular basis. Another big example is the use of Cucumber or SpecFlow as an automation tool.

In the surprisingly recent past, I’ve been guilty of the above mistakes myself too, by the way. No need to be elitist and pretend I know it all. I just hope that more people will start to think and ask the right questions before they automate.

On a final note, in test automation training too, a lot of attention is being paid to the ‘how?’, without the proper amount of focus on the ‘why?’, and subsequently on the ‘what?’. Since I’m a firm believer in ‘practice what you preach’, I’ve started to develop training material that I think will contribute to asking the right questions in test automation. I believe that doing so will always lead to better test automation in the end. I hope to be able to present this training material to you, and to the general test automation and testing community, somewhere early next year (creating course material takes time!).

May next year be the year of the ‘why?’ in test automation, not just of the ‘how?’.