API testing skills: why you need them

This is the first post in a three-part series on API testing. This post is an introduction on APIs, API testing and its relevance to the testing world. The second part of this series will feature some best practices for everybody involved in API testing. The third and final post will contain some useful code example for those of you looking to build your own automated API testing framework.

As information systems become more and more distributed and systems and devices become ever more interconnected, the use of APIs has seen exponential growth in the past couple of years. Where traditional (or old-fashioned) computer systems were monolithic in nature, nowadays they are often made up of reusable components that communicate and exchange information with one another through various APIs.

The figure below depicts the growth in number of publicly accessible APIs, as published by ProgrammableWeb:
Growth in publicly accessible APIs in recent years

Estimates for the number of publicly available APIs for the coming years range from 300.000 in 2016 to around a million by 2017.

With APIs becoming more and more common and important, proper testing of these APIs has become a hot issue with both API providers and consumers. As a provider, you wouldn’t want to be associated with an API of poor quality. As a consumer, you wouldn’t want your software system and/or your business to rely on a buggy API.

However, as APIs are designed for computer-to-computer interaction, rather than computer-to-user interaction, they do not have a user interface through which the tester can access the API. Moreover, to properly assess whether the output as given by the API is correct, a tester would need to know at least something about the internal workings of the API (i.e., perform white-box testing rather than traditional black-box testing). This might make API testing seem ‘hard’ or ‘difficult’ for some testers.

What makes API testing even more important is that in the current wave of layered information systems, business rules and business logic is often coded and enforced within the API layer (and not in the user interface or the database layer, for example). This is yet another reason for every development project that features the development or consumption of APIs to pay sufficient attention to API testing.

In the next part of this series, I will present some pointers for those of you who are involved in API testing, are looking to do so, or have been asked to do so. Many of my tips are also featured on the API Testing Dojo. There, you can also find some exercises to test and sharpen your API testing skills. A highly recommended resource.

Four people in your organization that will benefit from test automation

The introduction of test automation within your organization can, if done correctly, affect your testing process in a number of positive ways. In this post, we will look at the ways different stakeholders within your software development process might benefit from automated testing.

Test automation benefits the software testing team in a number of ways, of which some of the most important are:

  • Automation of regression tests takes away the repetitive effort of executing the same test scripts by hand over and over again, every time new functionality is introduced. Instead of having to spend time on this often tedious activity, software testers can focus on testing the changes that are introduced in the software, while the automated test tool takes care of the regression testing.
  • Not having to spend time on regression testing means that testers can perform more tests in the same amount of time.
  • Testers with an interest in programming and tools have the opportunity to learn valuable new skills by diving into the world of automated testing.

The development team might also benefit from the introduction of test automation:

  • As automated tests can generally be run faster and more often compared to when the same test scripts have to be executed manually, developers receive feedback on the quality of their code sooner.
  • Testers involved in automated testing often work closely together with developers to set up tests and analyse test results, resulting in a bigger appreciation for what used to be ‘the other side’. Of course, this appreciation works both ways (and this point could have been under ‘Testers’ just as easily).

Project managers
Automated testing has several benefits for project management too:

  • With automated testing, more tests can be performed within the duration of the project. On the one hand, today’s dynamic market requires fast-paced software delivery and therefore ever shorter development cycles. On the other hand, the same dynamic market expects high quality software as for a lot of products many alternatives are available, making it easy for an unsatisfied customer to change products or services. Automated testing is the only way for an organization to deliver high quality software quickly.
  • Automated testing enables tests to be executed sooner and more often, giving project managers continuous insight in the quality of the software, as well as in trends in this quality. For example, the availability of daily test results gives a project manager insight in whether the introduction of a new development method – or a new development team – has the desired effect on the quality of the product to be delivered.

End users
Finally, the end user of the software delivered also benefits from the introduction of automated testing, albeit in a more indirect way:

  • Automated testing, when implemented correctly, can greatly increase the speed with which software is delivered, meaning that the end user receives a release of the product he or she has asked for quicker than would be possible with traditional manual testing. As automated testing is able to shorten development cycles, any bugs reported or updates requested by the user can be dealt with quicker as well.
  • As the test team is able to achieve higher test coverage within the given amount of time, the quality of the end product will generally also be higher as more bugs can be resolved before delivery, meaning that the user will likely be happier with this product.

How did the introduction of automated testing benefit your organization or (for the contractors) your client?

The test automation pyramid

In his book Succeeding with Agile, Mike Cohn describes the concept of a test automation pyramid, describing three levels of test automation, their relation and their relative importance. As an advocate of minimizing user interface-based test automation I wholeheartedly support this pyramid, which is why I decided to share it with you at ontestautomation.com.

Graphically, the test automation pyramid as proposed by Mike Cohn looks like this:
The test automation pyramid
Base layer: unit tests
Unit tests form the base layer of every solid automated testing approach. They can be written relatively quickly and give the programmer very specific information about the origins of a bug, up to the exact line of code where a failure occurs. Compare this to a bug report from a tester, who would usually be more like ‘function X, Y and / or Z are not working when I enter A or B, now go fix it’. This often requires more analysis (reproduction, debugging) and therefore more time from the developer to fix things.

Another advantage of unit tests is that not only can they be written quickly, test execution is also very fast, giving the developer immediate feedback on code quality.

Possible drawbacks of unit tests are that they mostly focus on small pieces of code (methods, classes) and are therefore unable to detect integration or system level bugs. Also, as they are written in code, they are written mostly by developers and not by testers. Ideally, unit tests should be written by someone other than the developer of the code that is being tested.

Top layer: user interface-level tests
Let’s skip the middle layer for a moment and go right to the top of the pyramid, where the UI-level automated tests reside. Ideally, you would want to do as little of this as possible, as they are often the most brittle and take the longest time both in test case development and in test execution. In my opinion, this form of test automation should only be used when the UI is actually being tested rather than the underlying system functionality, or when there is no viable alternative. Such an alternative is available more often than you’d think, by the way..

Middle layer: service and API tests
For those tests that exceed the scope of unit tests it is strongly advised to use tests that communicate with the application under test at the service or API level. Most modern applications offer some sort of API (either through an actual programming interface or through a web service exposing functionality to the outside world) that can be used by the tester to test those applications. These tests are often far less brittle (as service interfaces and APIs tend to change far less often than user interfaces) and execute far quicker with less false negatives.

The inverted test automation pyramid
It is no coincidence that Mike Cohn calls this middle layer the forgotten layer of test automation. All too often, test cases that cannot be covered by developers in unit tests are directly automated on the user interface level, resulting in big sets of UI level automated tests that take eons to execute and maintain. This phenomenon is represented by an inverted test automation pyramid:
The inverted test automation pyramid
In extreme cases, the middle layer doesn’t even exist in the overall test automation approach. It doesn’t need explanation that in most test automation projects that resemble this inverted pyramid a lot of money is wasted unnecessarily on development and maintenance of the automated test cases..

My advice
I’d therefore advice any test automation specialist that would like to make his or her project a success (and who doesn’t?) to do a couple of things:

  • Get familiar with unit testing, its benefits and the way the developers in your project use it. Try to understand what they test and what coverage is achieved in unit testing. Try and work with your developers to see where this coverage can be increased further and what you can do to achieve that.
  • For those tests that are not covered by unit tests, try to find out whether the application you are testing offers an API that you can use to base your automated tests on. Initially, testing ‘under the hood’ without a UI to drive tests might seem challenging, but the benefits are definitely worth it.

Following these two suggestions will help you greatly in getting your test automation pyramid in the right shape. What does your pyramid look like?

Final note: Although some articles on the Internet even go as far as to suggest an ‘ideal’ mix of the pyramid layers (80% unit tests, 15% API tests and 5% UI tests, for example), I think there is no one ‘ideal’ mix. It depends a lot on the type of application under test and the skill sets of your developers and especially your testers. However, the shape of the test automation pyramid should be roughly the same in any case.