Choose wisely

In a recent article that was published on TechBeacon, I argued that writing tests at the API level often hits the sweet spot between speed of execution, stability (both in terms of execution and maintenance required) and test coverage. What I didn’t write about in this article is what motivated me to write the article in the first place, so I thought it might be a good idea to dedicate a blog post to the reason behind the piece.

There really was only a single reason for me to suggest the topic to the people at TechBeacon: because I see things go wrong too often when people start creating automated tests. I’m currently working with a number of clients in two separate projects, and what a lot of them seem to have in common is that they revert instantly to end-to-end tests (often using a tool like Selenium or Protractor) to create the checks that go beyond the scope of unit tests.

As an example, I’m working on a project where we are going to create automated checks for a web shop that sells electronic cigarettes and related accessories in the United States. There are several product categories involved, several customer age groups to be considered (some products can be purchased if you’re over 18, some over 21, some are fit for all ages, etc.), and, this being the US, fifty different states, each with their own rules and regulations. In short, there’s a massive amount of possible combinations (I didn’t do the math yet, but it’s easily in the hundreds). Also, due to the strict US regulations, and more importantly the fines associated with violating these rules, they want all relevant combinations included in the automated test.

Fair enough, but the problem started when they suggested we write an automated end-to-end test case for each of the possible combinations. That means creating an order for every combination of product group, age group and state, and every order involves filling out three or four separate forms and some additional more straightforward web page navigation. In other words, this would result in a test that would be slow to execute (we’re talking hours here) and possibly quite hard to maintain as well.

Instead, I used Fiddler to analyze what exactly it was that the web application did in order to determine if a customer could order a given product. Lo and behold.. it simply called an API that exposed the business logic used to make this decision. So, instead of creating hundreds of UI-driven test cases, I suggested to create API-level tests that would verify the business logic configuration, and add a couple of end-to-end tests to verify that a user can indeed place an order successfully, as well as receive an error message in case he or she tries to order a product that’s not available for a specific reason.

We’re still working on this, but I think this case illustrates my point fairly well: it often pays off big time to look beyond the user interface when you’re creating automated tests for web applications:

  • Only use end-to-end tests to verify whether a user of your web application can perform certain sequences of actions (such as ordering and paying for a product in your web shop).
  • See (ask!) whether business logic hidden behind the user interface can be accessed, and therefore tested, at a lower (API or unit) level, thereby increasing both stability and speed of execution.

For those of you familiar with the test automation pyramid, this might sound an awful lot like a stock example of the application of this model. And it is. However, in light of a couple of recent blog posts I read (this one from John Ferguson Smart being a prime example) I think it might not be such a good idea to relate everything to this pyramid anymore. Instead, I agree that what it comes down to (as John says) is to get clear WHAT it is that you’re trying to test and then write tests on the right level. If that leads to an ice cream cone, so be it. If only because I like ice cream..

This slightly off-topic remark about the test automation pyramid notwithstanding, I think the above case illustrates the key point I’m trying to get across fairly well. As I’ve said before, creating the most effective automated tests comes down to:

  • First, determining why you want to automate those tests in the first place. Although that’s not really the subject of this blog post, it IS the first question that should be asked. In the example in this post, the why is simple: because the risk and impact of fines imposed in case of the sale of items to groups of people that should not be allowed to is high enough to warrant thorough testing.
  • Then, deciding what to test. In this case, it’s the business logic that determines whether or not a customer is allowed to purchase a given item, based on state of residence, product ID and date of birth.
  • Finally, we get to the topic of this blog post, the question of how to test a specific application or component. In this case, the business logic that’s the subject of our tests is exposed at the API level, so it makes sense to write tests at that level too. I for one don’t feel like writing scenarios for hundreds of UI-level tests, let alone run, monitor and maintain them..

I’m sure there are a lot of situations in your own daily work where reconsidering the approach taken in your automated tests might prove to be beneficial. It doesn’t have to be a shift from UI to API (although that’s the situation I most often encounter), it could also be writing unit tests instead of end-to-end user interface-driven tests. Or maybe in some cases replacing a large number of irrelevant unit tests with a smaller number of more powerful API-level integration tests. Again, as John explained in his LinkedIn post, you’re not required to end up with an actual pyramid, as long as you end up with what’s right for your situation. That could be a pyramid. But it could also not be a pyramid. Choose (and automate) wisely.

Three reasons to start improving your API test automation skills

Modern applications and software development methods have changed the requirements for testers and the skills they need to possess to add real value to their clients and projects. One of these emerging and sought after skills is the ability to design and execute automated tests for APIs. In this post, I will give you three reasons why it might be useful for you to start improving your API test automation skills.

API word cloud

APIs are everywhere
The first reason why you should invest in your API test automation skills is a simple question of demand and supply: APIs are becoming ever more present in current IT solutions. From mobile applications to the Internet of Things, many modern systems and applications expose their data, their business logic or both through APIs. Whether you’re building an application that uses APIs to expose data or logic to the outside world, or you’re on the other side as an API consumer, you need to be able to perform tests on APIs. Otherwise, how are you going to ensure that an API and its integration with the outside world function as expected?

Oh, and if you’re testing an application that consumes a third-party API, please don’t fall into the trap of assuming that this API you’re using works perfectly as designed, or that integration with your own application will be seamless. Can anyone really assure you that that business-critical third party API you’re relying on has been tested for your specific situation and requirements? Thought not. Make sure it does and do some proper testing on it!

API test automation hits the sweet spot between speed and coverage
The second reason why API test automation can be very useful is that automated checks on the API level hit the sweet spot between speed of execution and coverage of application features. Compared to the two other types of automated tests in the test automation pyramid, API-level tests tend to:

  • execute faster than user interface-driven tests. User interface-driven automated tests, such as those written in Selenium WebDriver, need to fire up a browser and render several web pages every time a test is executed. When your tests go through a lot of different pages, execution time skyrockets. API-level tests, on the other hand, have to wait for a server responding to HTTP calls only. The only client-side processing that needs to be done is parsing the response and performing validation checks, for example on specific elements in the response. This is a lot faster than sending (possibly many) HTTP requests to a web server to fetch all objects required for a web page and then waiting until your browser has finished rendering the page.
  • cover more business logic than unit tests. Yes, you should have unit tests. And they should cover as much of the internal workings of your application that make up the business logic. However, there’s only so much unit tests can do. For example, unit tests can check whether the salary of a given employee is calculated correctly in the back end. They cannot guarantee that the same salary is correctly sent out to the front end layer of your application (or to the IRS) upon request, though. For this, you will need to perform tests at a higher level in your application, and API tests are usually perfect for that.

Automated API tests tend to be more reliable
Apart from having the right mixture of speed of execution and coverage of functional aspects, API-level automated checks have another big advantage over user interface-driven automated tests: they’re usually far more reliable. User interface-driven tests constantly have to walk the wobbly rope of synchronization, ever changing (or ‘improving’) user interface designs, dynamic element identification methods, etc. API definitions and interfaces on the other hand are amongst the most stable parts of an application: they follow standardized specification formats (such as WSDL for SOAP or WADL, RAML or Swagger for REST) and once agreed upon, an API does not usually change all that much. This applies especially to outward-facing APIs. For example, Google can easily change its Gmail user interface without this impacting end users (apart from annoyance, maybe, but that’s a different story altogether), but sudden radical changes to its Gmail API would render a lot of third-party applications that have Gmail integration useless. Therefore, changes to an API will usually be a lot fewer and further between, resulting in less maintenance required for your API-level automated tests.

A sample API test in REST Assured

Further reading
I hope this blog post has given you some insight into why I think API test automation skills are a valuable asset for any tester with an interest in test automation. If you want to read more on API testing and test automation, I highly recommend the API Testing Dojo on the SoapUI website. Additionally, you can also check out my other posts on API testing here.

Do you check your automated checks?

Quis custodiet ipsos custodes? (Who will guard the guards?)

– Juvenal (Roman poet), from Satires (Satire VI, lines 347–8)

You’ve been coding away on your automated tests for weeks or even months on end, creating a test suite that you think is up there with the best of them. Worthy of a place in the test automation Champions League, should such a competition exist. But is your work really that good? While you may think it is, do you have the proof to back it up? In this post, I would like to explain why you should make a habit of testing your tests, or, in light of the checking vs. testing debate, of checking your checks.

Why would you want to check your checks?
To argument the need for checking your checks, let’s take a look at what might happen when you don’t do so regularly. Essentially, there are two things that can (and will) occur:

1. Your checks will go bad
Checks that are left unattended once created, might – either right from the get-go or over time as the application under test evolves – start to:

  • Fail consistently. This will probably be detected and fixed as soon as your build monitor will start turning red. When you do Continuous Delivery or Continuous Deployment, this is even more likely as consistently failing checks will stall the delivery process. And people will start to notice that!
  • Fail intermittently. Also known as ‘flakiness’. Incredibly annoying as it’s usually a lot harder to pinpoint the root cause compared to checks that fail all the time. Still, the value of flaky checks can be considered about the same as for consistently failing checks: zero.
  • Always return true. This one might turn out to be very hard to detect, because your runs will keep being green and therefore chances are that nobody will feel the urge to review the checks that are performed. They are passing, so everything must be fine, right? Right?
  • Check the wrong thing. This one may or may not be easy to spot, depending on what they ARE checking. If it fails and the root cause of the failure is being analyzed, one of two things might happen: either the check is fixed – for example by changing the expected value – but the wrong check keeps being executed, or deeper analysis reveals that the wrong check has been performed all along and the check itself is being fixed (or even removed).

No matter what the root cause is, the end result of any of these types of check defects is the same: your checks will go bad and will therefore be worthless.

High quality software or just bad checks?

And I bet that’s not what you had in mind when you created all of these automated scripts! Also, this problem isn’t solved by simply looking at the output report for your automated test runs, as doing so won’t tell you anything about the effectiveness of your checks. It will only tell you what percentage of your checks passed, which tells only half the story.

2. Your check coverage will go bad
Another problem with the effectiveness of your checks can be seen when we look at it from another perspective:

  • You may have implemented too many checks. Yes, this is actually possible! This increases maintenance efforts and test run (and therefore feedback) time. This is especially significant for UI-driven checks performed by tools such as Selenium WebDriver.
  • You may have implemented too few checks. This has a negative impact on coverage. Not that coverage is the end all and be all of it (far from that), but I’m sure we can agree that not doing enough checks can’t be a good thing when you want to release a quality product..

I think both of these reasons warrant a periodic check validation. In such a validation, all checks performed by your automated test suite should be evaluated on the characteristics mentioned above. Checks that are no longer useful should be rewritten or removed, and any shortcomings in your test coverage should be addressed.

Check your automated checks!

Automating the check validation
Of course, since we’re here to talk about test automation, it would be even better to perform such a validation in an automated manner. But is such a thing even possible?

Unit test level checks
Turns out that for checks at the unit test level (which should make up a significant part of your checks anyway) checking your checks can be done using mutation testing. Even better, one of the main purposes of mutation testing IS to check the validity and usefulness of your checks. A perfect match, so to say. I am still pretty new to the concept of mutation testing, but so far I have found it to be a highly fascinating and potentially very powerful technique (if applied correctly, of course).

API and UI level checks
Unfortunately, mutation testing is mainly geared towards unit and (to a lesser extent) unit integration tests. I am not aware of any tools that actively support check validation for API and UI tests. If anyone reading this knows of a tool, please do let me know!

One reason such a tool might not exist is that it would be much harder to apply the basic concept of mutation testing to API and UI-level checks, if only because you would need to:

  1. Mutate the compiled application or the source code (any subsequently compile it) to create a mutant
  2. Deploy the mutant
  3. Run tests to see whether the mutant can be killed

for every possible mutant, or at least a reasonable subset of all possible mutants. As you can imagine, this would require a lot of time and computation power, which would have a negative effect on the basic principle of fast feedback for automated checks.

One possible approach that covers at least part of the objectives of mutation testing would be to negate all of your checks and see if all of them fail. This might give you some interesting information about the defect finding capabilities of your checks. However, it doesn’t tell you as much about the coverage of your checks though, so this very crude approach would be only part of the solution.

The above does give yet another reason for implementing as much of the checks as you can in unit tests, though. Not only would they be easier to create and faster to run, but their usefulness can also be proven much more precisely in an automated manner (i.e., using mutation testing).

So, as a wrap-up, do you check your automated checks? If so, how? I’m very curious to read new insights on this matter!

P.S.: Richard Bradshaw wrote a great blog post on the same topic recently, I highly suggest you read it as well over here.