Should test automation be left to developers?

I am not a developer. I have a background in Computer Science, I know a thing or two about writing code, but I am definitely not a developer. This is similar to me knowing how to prepare food, but not being a cook. People that have met me in person probably remember and will possible be bored to death by me using this analogy (sorry!).

On the other hand, I also try to repeat as often as possible that test automation is software development, and that it should be treated as such. So, you might ask, how come I am working in test automation, yet I don’t call myself a developer? And more importantly, shouldn’t test automation be left to developers, if it really IS software development? Recent blog posts I’ve read and presentations I’ve heard have made me think a lot about this.

So, to start with the second and most important question: should test automation be left to developers? My answer would be yes.

And no.

Test automation should be left to developers, because
writing automated tests involves writing code. Personally, I don’t believe in codeless solutions that advertise that ‘anybody’ can create automated tests. So, since code writing is involved, and since nobody knows writing code better than a developer, I think that writing automated tests can best be done by a developer.

I’m going to use myself as an example here. Since writing code isn’t in my DNA, I find it takes me three times as long – at the minimum – to write automated tests compared to your average developer. And with high-pressure projects, short release cycles and the trend of having less and less testers for every developer in development teams, I just couldn’t keep up.

To make matters worse, new automation tools are being released what seems like every day. Especially within the Java and JavaScript worlds, although it might be just as good (or bad?) in other ecosystems, just less visible to me personally. I don’t know. For a while, as a non-developer, I’ve tried to keep up with all of these tools, but I just couldn’t manage. So, instead of frantically trying to keep up, I’ve made a career shift. This came naturally after I realized that

Test automation should not be left to developers, because
nobody knows testing better than testers. As I said, I (and possibly many people involved in test automation along with me) am not skilled enough to compete with the best on how to create automated tests. However, I think I (and a lot of others) can teach a lot of people a thing or two about the equally, if not more important questions of why you should (not) do test automation, and what tests should and should not be automated.

A lot of developers I have met don’t have a testing mindset (yet!) and tend to think solely along the happy path. ‘Let’s see if this works…’, rather than ‘let’s see what happens if I do this…’, so to say. When writing automated tests, just as with creating specs for the software that needs to be built, it requires a tester’s mindset to think beyond the obvious happy flow. This is why I think it isn’t wise to see test automation as a task purely for developers.

It also helps if people other than developers feel some sort of responsibility for creating automated tests. Given that most developers are busy people, they need to choose between writing tests and writing production code on a regular basis. Almost always, that decision will be made in favor of production code. This does make sense from a deadline perspective, but not always as much when you look at it from a quality and process point of view. Having people in other roles (testing, for example) stressing the need for a mature and stable automated testing suite will definitely improve the likelyhood of such a suite actually being created and maintained. Which might just benefit the people continuously fretting about those deadlines in the long end..

So, to answer the first question I posed at the beginning of this post: Why am I still working in test automation despite not being a developer? Because nowadays I mainly focus on helping people answer the why and the what of test automation. I do some stuff related to the how question as well, especially for smaller clients that are at the beginning of their test automation journey or that don’t have a lot of experienced developers in house, but not as much as before. I love to tinker with tools every now and then, if only just to keep up and remain relevant and hireable. I’m not as much involved in the day to day work of writing automated tests anymore, though. Which is fine with me, because it plays to my strengths. And I think that’ll benefit everybody, in the long end.

I think the test automation community could use a lot more highly skilled people that are (not) developers.

Three reasons to start improving your API test automation skills

Modern applications and software development methods have changed the requirements for testers and the skills they need to possess to add real value to their clients and projects. One of these emerging and sought after skills is the ability to design and execute automated tests for APIs. In this post, I will give you three reasons why it might be useful for you to start improving your API test automation skills.

API word cloud

APIs are everywhere
The first reason why you should invest in your API test automation skills is a simple question of demand and supply: APIs are becoming ever more present in current IT solutions. From mobile applications to the Internet of Things, many modern systems and applications expose their data, their business logic or both through APIs. Whether you’re building an application that uses APIs to expose data or logic to the outside world, or you’re on the other side as an API consumer, you need to be able to perform tests on APIs. Otherwise, how are you going to ensure that an API and its integration with the outside world function as expected?

Oh, and if you’re testing an application that consumes a third-party API, please don’t fall into the trap of assuming that this API you’re using works perfectly as designed, or that integration with your own application will be seamless. Can anyone really assure you that that business-critical third party API you’re relying on has been tested for your specific situation and requirements? Thought not. Make sure it does and do some proper testing on it!

API test automation hits the sweet spot between speed and coverage
The second reason why API test automation can be very useful is that automated checks on the API level hit the sweet spot between speed of execution and coverage of application features. Compared to the two other types of automated tests in the test automation pyramid, API-level tests tend to:

  • execute faster than user interface-driven tests. User interface-driven automated tests, such as those written in Selenium WebDriver, need to fire up a browser and render several web pages every time a test is executed. When your tests go through a lot of different pages, execution time skyrockets. API-level tests, on the other hand, have to wait for a server responding to HTTP calls only. The only client-side processing that needs to be done is parsing the response and performing validation checks, for example on specific elements in the response. This is a lot faster than sending (possibly many) HTTP requests to a web server to fetch all objects required for a web page and then waiting until your browser has finished rendering the page.
  • cover more business logic than unit tests. Yes, you should have unit tests. And they should cover as much of the internal workings of your application that make up the business logic. However, there’s only so much unit tests can do. For example, unit tests can check whether the salary of a given employee is calculated correctly in the back end. They cannot guarantee that the same salary is correctly sent out to the front end layer of your application (or to the IRS) upon request, though. For this, you will need to perform tests at a higher level in your application, and API tests are usually perfect for that.

Automated API tests tend to be more reliable
Apart from having the right mixture of speed of execution and coverage of functional aspects, API-level automated checks have another big advantage over user interface-driven automated tests: they’re usually far more reliable. User interface-driven tests constantly have to walk the wobbly rope of synchronization, ever changing (or ‘improving’) user interface designs, dynamic element identification methods, etc. API definitions and interfaces on the other hand are amongst the most stable parts of an application: they follow standardized specification formats (such as WSDL for SOAP or WADL, RAML or Swagger for REST) and once agreed upon, an API does not usually change all that much. This applies especially to outward-facing APIs. For example, Google can easily change its Gmail user interface without this impacting end users (apart from annoyance, maybe, but that’s a different story altogether), but sudden radical changes to its Gmail API would render a lot of third-party applications that have Gmail integration useless. Therefore, changes to an API will usually be a lot fewer and further between, resulting in less maintenance required for your API-level automated tests.

A sample API test in REST Assured

Further reading
I hope this blog post has given you some insight into why I think API test automation skills are a valuable asset for any tester with an interest in test automation. If you want to read more on API testing and test automation, I highly recommend the API Testing Dojo on the SoapUI website. Additionally, you can also check out my other posts on API testing here.

The most important skill in test automation

On LinkedIn, as well as on various test-oriented discussion boards there’s a lot of talk about the necessary skills one needs to have or develop in order to be a good test automation engineer / consultant / << insert job title here>>. Example questions asked can be ‘What language is useful to learn?‘ or ‘How do I automate fancy technology XYZ using tool this-or-that?’ (too many hits on LinkedIn to count, often related to Selenium).

Another perspective on the discussion on skills needed for successful test automation is whether you’re best off being – or employing – a developer with a testing mindset or a tester with development skills (I’d go with the latter).

However, in all those discussions, one vital, even essential skill everybody in test automation should have seems to be overlooked again and again:

Knowing what not to automate.

Honestly, some of the questions that pop up.. ‘How do I check the values on this pie chart with this tool?’ ‘How can I have my test script read an email using Outlook?’ Seriously? Why not just do a quick check on the underlying values (in a database, for example) instead of spending days developing a buggy test script that performs the same check, only slower?

In the wise words of Alan Page: “You should automate 100% of the tests that should be automated“. No more, and if possible no less either. But, when in doubt, it’s probably better not to automate a test that should be automated than to automate a test that shouldn’t be.

Still thinking about automating that Google Maps / Google Docs user interface check? Then please also consider the following likely outcomes:

  • You’ll probably introduce a lot of required maintenance effort (the Google UI’s change often and are notoriously hard to automate)
  • You’re not testing your actual product (unless you work for Google, and in that case, you probably know about their APIs which are much easier and faster to automate)
  • You’re throwing away money.

Please, don’t become the world’s best automator of useless checks.