On the test automation pyramid (again), opposing views and TestBash Manchester

Short blog post this time, since I’m smack in the middle of three projects while also preparing for my talk at TestBash Manchester, some training stuff and reviewing a book. It can’t always be long form!

With this blog post, I’d like to draw your attention to episode #31 of the Test and Code podcast, where Brian Okken (the host) replayed parts of the recordings of him being on another podcast, hosted by Paul Merrill. Paul’s a great guy and someone whose opinions and view I hold in high esteem, so I knew this was about to be good. In the interviel Paul and Brian did, they shared and discussed their opposing views on the test automation pyramid. By the way, more news about a collaboration (of sorts) between Paul and myself to follow soon!

It seems like Paul roughly shares my own views on the pyramid (read this recent blog post to see what I think of it and how I still use it, whereas Brian thinks the pyramid is falsely promoting the fact that unit tests are the most important tests you can write. Instead, in the recording, he advocates that there’s no value in software until it’s useful to an end user (be it a human being or another system) and that therefore UI tests are the most important and should be treated that way. I deliberately didn’t use the word ‘graphical’ here, by the way, since Brian talks mostly about writing tests for software where the GUI isn’t the interface that’s most extensively used.

You can listen to the podcast episode here. I thought it refreshing to listen to someone that has such an alternative view on a well-worn concept such as the test automation pyramid. Even though my own opinions on the model and its use remain unchanged (for now), I’ll try and make it a habit to listen to other voices and opposing views more often, rather than staying in my comfort zone and listening to podcasts and reading blogs on topics I’m already familiar with. Who knows I might learn a thing or two!

On another note, as I said before, I’m quite busy putting the finishing touches to my upcoming TestBash Manchester talk, and it’s both a talk and an event I’m very, very excited about. To those of you with Ministry of Testing Dojo Pro accounts, my talk will be available shortly after the event. For those of you that haven’t got such an account, you’ll have to wait for next week’s blog post, which will highly likely be a review of the event.

See you next week!

3 thoughts on “On the test automation pyramid (again), opposing views and TestBash Manchester

  1. Thing is, the test pyramid makes no statement about what is the most ‘important’ test. To me it conveys two things:

    – Unit tests are faster & cheaper, ui/acceptance is slower & expensive (pretty uncontroversial?)

    – Advice for the ideal number of tests you should have at each level

    To me, the pyramid is reasonably neutral on the topic of importance of different types of test. If anything, to me, it emphasises that they work in harmony with one another, and provide different functions. Yes – your user tests do catch big user problems, which is great – but they’re supported by the massive coverage of lower level tests.

    This is because UI tests cannot test all user stories. if you’re using your UI tests to test unusual unicode characters or other edge cases, you’re doing it wrong – and your test suite will take days to complete (and harm your feedback cycle, and ultimately lead to slow/poor quality software). These edge cases are still use cases for many of your users. Your acceptance suite should triage the most important use cases, and offload all the more unusual cases to the faster tiers.

  2. Bas,
    I think something that needs to be considered is the “granularity” or “atomicity” of the tests. At the code level for Unit Tests you are doing a lot more in isolation of code (within a method or function) and/or related chunks of code (methods/classes & functions with stubs/drivers around them). Thus you will produce a lot more of them and thus the bottom layer of the pyramid. Even Mike Cohn has stipulated this. But the reason for the emphasis is to test “pieces” individually to make sure they are stable and robust, and that any simple defects are caught early on. These are simplistic tests. As you move up the pyramid the complexity of the test increases and so they become less granular/atomic. You are starting to put the pieces together into larger functional ones. And so on and so on as you go up the layers.

    Typically though a lot of companies/projects are not using a straight forward approach on all of this. They get too heavy on the top side and not enough in the middle so you get the hour glass effect, or they are flipped (the Ice Cream Cone effect).

    And there isn’t a fixed percentage (number) for each layer. It’s what make sense.

    Finally, I do agree with the principle of it. Test early and often, and in smaller pieces to prove the whole. It leads to more reliable and robust code, catches defects early on, helps reduce rework (and its costs), and helps to provide a stable and usable product. That is what we are all doing this for.

  3. “The pyramid is falsely promoting the fact that unit tests are the most important tests you can write” – 100% agree!

    “therefore UI tests are the most important and should be treated that way” – now we are just replacing one falsely promoted fact with another…

    There is no binary answer. Nothing is most important. Unit tests are written by programmers and for programmers.
    Tests at the top of the pyramid are written for the programmers, but also for business. Business is also involved in writing them.

    So in this context, unit and UI are both “most important”, but for different kind of people.

    Now hopefully these different kind of people do not measure their work by the number and type of tests they write.

Leave a Reply

Your email address will not be published. Required fields are marked *