Choose wisely

In a recent article that was published on TechBeacon, I argued that writing tests at the API level often hits the sweet spot between speed of execution, stability (both in terms of execution and maintenance required) and test coverage. What I didn’t write about in this article is what motivated me to write the article in the first place, so I thought it might be a good idea to dedicate a blog post to the reason behind the piece.

There really was only a single reason for me to suggest the topic to the people at TechBeacon: because I see things go wrong too often when people start creating automated tests. I’m currently working with a number of clients in two separate projects, and what a lot of them seem to have in common is that they revert instantly to end-to-end tests (often using a tool like Selenium or Protractor) to create the checks that go beyond the scope of unit tests.

As an example, I’m working on a project where we are going to create automated checks for a web shop that sells electronic cigarettes and related accessories in the United States. There are several product categories involved, several customer age groups to be considered (some products can be purchased if you’re over 18, some over 21, some are fit for all ages, etc.), and, this being the US, fifty different states, each with their own rules and regulations. In short, there’s a massive amount of possible combinations (I didn’t do the math yet, but it’s easily in the hundreds). Also, due to the strict US regulations, and more importantly the fines associated with violating these rules, they want all relevant combinations included in the automated test.

Fair enough, but the problem started when they suggested we write an automated end-to-end test case for each of the possible combinations. That means creating an order for every combination of product group, age group and state, and every order involves filling out three or four separate forms and some additional more straightforward web page navigation. In other words, this would result in a test that would be slow to execute (we’re talking hours here) and possibly quite hard to maintain as well.

Instead, I used Fiddler to analyze what exactly it was that the web application did in order to determine if a customer could order a given product. Lo and behold.. it simply called an API that exposed the business logic used to make this decision. So, instead of creating hundreds of UI-driven test cases, I suggested to create API-level tests that would verify the business logic configuration, and add a couple of end-to-end tests to verify that a user can indeed place an order successfully, as well as receive an error message in case he or she tries to order a product that’s not available for a specific reason.

We’re still working on this, but I think this case illustrates my point fairly well: it often pays off big time to look beyond the user interface when you’re creating automated tests for web applications:

  • Only use end-to-end tests to verify whether a user of your web application can perform certain sequences of actions (such as ordering and paying for a product in your web shop).
  • See (ask!) whether business logic hidden behind the user interface can be accessed, and therefore tested, at a lower (API or unit) level, thereby increasing both stability and speed of execution.

For those of you familiar with the test automation pyramid, this might sound an awful lot like a stock example of the application of this model. And it is. However, in light of a couple of recent blog posts I read (this one from John Ferguson Smart being a prime example) I think it might not be such a good idea to relate everything to this pyramid anymore. Instead, I agree that what it comes down to (as John says) is to get clear WHAT it is that you’re trying to test and then write tests on the right level. If that leads to an ice cream cone, so be it. If only because I like ice cream..

This slightly off-topic remark about the test automation pyramid notwithstanding, I think the above case illustrates the key point I’m trying to get across fairly well. As I’ve said before, creating the most effective automated tests comes down to:

  • First, determining why you want to automate those tests in the first place. Although that’s not really the subject of this blog post, it IS the first question that should be asked. In the example in this post, the why is simple: because the risk and impact of fines imposed in case of the sale of items to groups of people that should not be allowed to is high enough to warrant thorough testing.
  • Then, deciding what to test. In this case, it’s the business logic that determines whether or not a customer is allowed to purchase a given item, based on state of residence, product ID and date of birth.
  • Finally, we get to the topic of this blog post, the question of how to test a specific application or component. In this case, the business logic that’s the subject of our tests is exposed at the API level, so it makes sense to write tests at that level too. I for one don’t feel like writing scenarios for hundreds of UI-level tests, let alone run, monitor and maintain them..

I’m sure there are a lot of situations in your own daily work where reconsidering the approach taken in your automated tests might prove to be beneficial. It doesn’t have to be a shift from UI to API (although that’s the situation I most often encounter), it could also be writing unit tests instead of end-to-end user interface-driven tests. Or maybe in some cases replacing a large number of irrelevant unit tests with a smaller number of more powerful API-level integration tests. Again, as John explained in his LinkedIn post, you’re not required to end up with an actual pyramid, as long as you end up with what’s right for your situation. That could be a pyramid. But it could also not be a pyramid. Choose (and automate) wisely.

10 thoughts on “Choose wisely

  1. Good post Bas. At our company we have migrated extensively in the last few years from being largely focused on UI tests to API tests instead.

    Agree with your points on the pyramid. I think the most important thing for us to do is to keep asking questions about what and how we are testing. The key thing to ask ourselves is – “are we mitigating the most risk possible?”

  2. Very Nice article. After reading it for a novice person like me it is enriching and I always learn new things and approach. Thanks.

  3. Great post as usual Bas. I think it balls down to understanding the application you are working with and where process be improved as far as automation is concerned. I believe as testers, we should always be interested on what is happening before the UI. For instance, how is the data collected by the UI used?

    At our company, we needed to create lots of orders for a performance test and using UI to create the orders was never an option for me as it will take a long time to do.

    I decided to write code that insert the the data into the database directly and then make an API call that fully create the order. Doing this, we were able to create hundreds of orders in no time. Using the UI would have taken a lot longer to achieve in my opinion. even developers now use the same code to create orders locally as setting up environment locally to create orders sometimes takes longer than it should.

    A lot of people tend to think automation is only done at the UI level (Webdriver) but there is a lot than can be done without the UI than people realise.

    Bas, do you think testers are required to be more technical to survive in the field? i.e. how can a non-technical tester grow (apart from being SME)

    • Hey Olu,

      thanks so much for sharing your own experience, sounds like you’ve seen some quite familiar stuff!

      And yes, I think testers do need to have a sufficient amount of technical insight and knowhow to be able to thrive in the field. That does NOT mean they have to become automation engineers or developers, by the way! The only exception may be a role where the tester is more of a QA coach to development teams, but even then, you wouldn’t survive or be taken seriously without at least some technical background..

  4. Pingback: Java Web Weekly, Issue 159 | Baeldung

  5. Pingback: Testing Bits – 1/8/17 – 1/14/17 | Testing Curator Blog

  6. Pingback: Java Testing Weekly 3 / 2017

  7. Pingback: Weekly Links List – Interview with Andrew Ng; Choosing automated tests carefully; Avoiding empathy fatigue in your organisation – Querydog – Weekly Links List and Research Log

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.