On preventing your test suite from becoming too user interface-heavy

In August of last year, I published a blog post talking about why I don’t like to think of automation in terms of frameworks, but rather in terms of solutions. I’ve softened a little since then (this is probably a sign of me getting old..), but my belief that building a framework might lead to automation engineers subsequently trying to fit every test left, right and center into that framework still stands. One example of this phenomenon in particular I still see too often: engineers building a feature-rich end-to-end automation framework (for example using Selenium) and then automating all of their tests using that framework.

This is what I meant in the older post by ‘framework think’: because the framework has made it so easy for them to add new tests, they skip the step where they decide what would be the most efficient approach for a specific test and blindly add it to the test suite run by that very framework. This might not lead to harmful side effects in the short term, but as the test suite grows, chances are high that it becomes unwieldy, that the time it takes to complete a full test run becomes unnecessarily long and that maintenance efforts are not being outweighed by the added value of having the automated tests any more.

In this post, I’d like to take the practical approach once more and demonstrate how you can take a closer look at your application and decide if there might be a more efficient way to implement certain checks. We’re going to do this by opening up the user interface and see what happens ‘under the hood’. I’m writing this post as an addendum to my ‘Building great end-to-end tests with Selenium and Cucumber / SpecFlow‘ course, by the way. Yes, that’s right, one of the first things I talk about during my course on writing tests with Selenium is when not to do so. I firmly believe that’s the on of the very first steps towards creating a solid test suite: deciding what should not be in it.

The application under test
The application we’re going to write tests for is an online mortgage orientation tool, provided by a major Dutch online bank. I’ve removed all references to the client name, just to be sure, but it’s not like we’re dealing with sensitive data here. The orientation tool is a sequence of three forms, in which people that are interested in a mortgage fill in details about their financial situation, after which the orientation tool gives an indication of whether or not the applicant is eligible for a mortgage, as well as an estimate of the maximum amount of the mortgage, the interest rate and the monthly installments payable.

Our application under test - the mortgage orientation tool

What are we going to automate?
Now that we know what our application under test does, let’s see what we should automate. We’ll assume that there is a justified need for automated checks in the first place (otherwise this would have been a very short blog post!). We’ll also assume that, maybe for tests on some other part of the bank’s website, there is already a solid automation framework written around Selenium in place. So, this being a website and all, it makes sense to write some additional checks and incorporate them into the existing framework.

First of all, let’s try and make sure that the orientation tool can be used and completed, and that it displays a result. I’d say, that would be a good candidate for an automated test written using Selenium, since it confirms that the application is working from an end user perspective (there is value in the test) and I can’t think of a lower level test that would give me the same feedback. Since there are a couple of different paths through the orientation tool (you can apply for a mortgage alone or with someone else, some people have a house to sell while others have not, there are different types of contracts, etc.), I’d even go as far as to say you’ll need more than one Selenium-based test to be able to properly claim that all paths can be traversed by an end user.

Next, I can imagine that you’d want to make sure that the numbers that are displayed are correct, so your customers aren’t misinformed when they complete the orientation tool. This would lead to some massive issues of distrust later on in the mortgage application process, I’d assume.. Since we’ve been able to add the previous tests so easily to our existing framework, it makes sense to add some more tests that walk through the forms, add the data required to trigger a specific expected outcome and verify that the result screen we saw in the screenshot above displays the expected numbers. Right?

No. Not right.

It’s highly likely that the business logic used to perform the calculation and serve the numbers displayed on screen isn’t actually implemented in the user interface. Rather, it’s probably served up by a backend service containing the business logic and rules required to perform the calculations (and with mortgages, there are quite a few of those business rules, I’ve been told..). The user interface takes the values entered by the end user, sends them to a backend service that performs calculations and returns the values indicating mortgage eligibility, interest rate, height of monthly installment, etc., which are then interpreted and displayed again by that same user interface.

So, since the business logic that we’re verifying isn’t implemented in the user interface, why use the UI to verify it in the first place? That would highly likely only lead to unnecessarily slow tests and shallow feedback. Instead, let’s look if there’s a different hook we can use to write tests.

I tend to use on of two different tactics to find out if there are better ways to write automated tests in cases like these:

  1. Talk to a developer. They’re building the stuff, so they’ll probably know more about the architecture of your application and will likely be happy to help you out.
  2. Use a network analyzing tool such as Fiddler or WireShark. Tools like these two let you see what happens ‘under water’ when you’re using the user interface of a web application.

Normally, I’ll use a combination of both: find out more about the architecture of an application by talking to developers, then using a network analyzer (I prefer Fiddler myself) to see what API calls are triggered when I perform a certain action.

Analyzing API calls using Fiddler
So, let’s put my assumption that there’s a better way to automate the tests that will verify the calculations performed by the mortgage orientation tool to the test. To do so, I’ll fire up Fiddler and have it monitor the traffic that’s being sent back and forth between my browser and the application server while I interact with the orientation tool. Here’s what that looks like:

Traffic exchanged between client and server in our mortgage orientation tool

As you can see, there’s a mortgage orientation API with a Calculate operation that returns exactly those numbers that appear on the screen. See the number I marked in yellow? It’s right there in the application screenshot I showed previously. This shows that pretty much all that the front end does is performing calls to a backend API and presenting the data returned by it in a manner attractive to the end user. This means that it would not make sense to use the UI to verify the calculations. Instead, I’d advise you to mimic the API call (or sequence of calls) instead, as this will give you both faster and more accurate feedback.

To take things even further, I’d recommend you to dive into the application even deeper and see if the calculations can be covered with a decent set of unit tests. The easiest way to do this is to start talking to a developer and see if this is a possibility, and if they haven’t already done so. No need to maintain two different sets of automated checks that cover the same logic, and no need to cover logic that can be tested through unit tests with API-level checks..

Often, though, I find that writing tests like this at the API level hits the sweet spot between coverage, effort it takes to write the tests and speed of execution (and as a result, length of the feedback loop). This might be because I’m not too well versed in writing unit tests myself, but it has worked pretty well for me so far.

Deciding what to automate where: a heuristic
The above has just been one example where it would be better (as well as easier) to move specific checks from the UI level to the API level. But can we make some more generic statements about when to use UI-level checks and when to dive deeper?

Yes, we can. And it turns out, someone already did! In a recent blog post called ‘UI Test Heuristic: Don’t Repeat Your Paths‘, Chris McMahon talked about this exact subject, and the heuristic he presents in his blog post applies here perfectly:

  • Check that the end user can complete the mortgage orientation tools and is shown an indication of mortgage eligibility and associated figures > different paths through the user interface > user interface-level tests
  • Check that the figures served up by the mortgage orientation tool are correct > repeating the same paths multiple times, but with different sets of input data and expected output values > time to dive deeper

So, if you want to prevent your automated test suite from becoming too bloated with UI tests, this is a rule of thumb you can (and frankly, should) apply. As always, I’d love to hear what you think.

On quality over quantity and my career journey

As you might have read in last week’s blog post, TestBash Manchester, the talks I’ve heard there and the discussions I had around the event with other speakers and attendees, left me with a lot to think about. Especially Martin Hynie’s talk on tester craftsmanship, apprentices, journeymen and masters of the craft led to me asking a lot of questions to myself on where I am now, how I ended up where I am today, where I want to go and, most importantly, if the things that I am doing at the moment contribute to, or maybe hinder me, in my own journey towards who I want to become.

Martin’s talk and how he described masters of a craft confirmed me that that, for me, is what I do want to become: a master in the craft of automation. Someone that others turn to when they need help, and someone that is able to help and guide others on their way to becoming a master -or at least a better craftsperson- themselves. I also immediately realized that I’m nowhere near that point yet.

I might be on my way, possibly (hopefully!) even on the right way, but having thought about this for a bit now, it once more occurred to me that there is so much more to learn. Some aspects that I need to improve are directly tied to automation and testing, others are skills that are more broadly applicable (public speaking, teaching, communication skills, to name just a few), but all in all, there’s a lot of learning left to do.

I am very much looking forward to taking the next steps on my path towards mastery, but I also realize that I need to get rid of some superfluous baggage at the moment, consisting mostly of activities that take up a lot of my time yet aren’t contributing (enough) to my journey. In the words of the German designer and academic Dieter Rams, it’s time for ‘less but better’, or ‘weniger aber besser’ as he puts it himself, being a German and all..

Anyway, there are a couple of work-related activities that I will need to get rid of -or at least change significantly- in order to carve out the time required to work on the important. Starting with the projects I’m working on. I’ve just wrapped up one, but I’m still working on two different projects in parallel.

Where I used to think this was the ideal situation to be in (I do get bored quickly if I’m working on the same thing for too long), I’ve slowly started to come to the realization that all this context switching is driving down the quality of my work. Believe, no matter how hard you try dedicating specific days to specific projects, there will always be overlap in the form of emails, phone calls and other seemingly urgent, and sometimes even important, interruptions. Just like with other forms of multitasking, I lose a lot of time moving my mind from one project to the other and back again, sometimes multiple times a day.

What doesn’t help is that not all of the projects I’ve been working on lately have been equally satisfying (and in specific cases, that’s putting it mildly..). Doing only one project at a time should allow me to think more clearly about whether or not the project is, in fact, a good fit for me. So, effective as soon as I wrap up my current projects, I’ll start committing myself to working on just a single client project (meaning by-the-hour consulting work) at a time. Ideally, that would take up 3 (maybe sometimes 4, maybe sometimes 2) days of my working week, ensuring that I am both set with regards to my financial commitments (gotta feed the kids!) as well as have enough time left to dedicate to the other things I want and/or feel the need to work on. Most of those things revolve around training courses, workshops and a bit of public speaking, by the way.

Committing to less but better also means that I’m, at least for the moment, giving up on writing weekly blog posts for this site. Even though it is a highly rewarding activity, it takes up a lot of time to plan, write and review blog posts. I’ll leave the discussion on whether or not my blog posts look like significant time has been put into it to you.. Instead, I’ll shift towards writing at least one blog post per month.

The good news is that this will leave me more time to do research and thinking for my blog posts, which (at least theoretically) should lead to higher quality output. Again, less but better.. I might post more often than once a month, in case I’ve read a good book related to testing or automation, a conference experience I want to share or anything else I feel like writing about, since those posts take less effort in my case. However, I think I need to stop pressurizing myself to write a weekly blog post, since it might start to affect the quality soon. If it hasn’t started doing so already.

Lastly, I am considering looking for a mentor who can help me take the next steps on my journey towards mastery. The above measures I’m taking should help freeing up time to do the things I feel are important (e.g., more time for learning, more time to invest in teaching and developing courses), but I am by now quite convinced that I might benefit from a mentor that helps me to navigate the career and life path that’s ahead of me. I’d love to hear from others who either have been on roughly the same point in their career and have (or have not) benefited from having a mentor, or who can help me find a good mentor. All input is greatly appreciated.

So, in short, you’ll hear less from me from this moment on, but hopefully also more. And better. I’m looking forward to the next stage of my journey.

On TestBash Manchester

I’m writing this blog post sitting in the Manchester airport departure area, waiting for my flight home to Amsterdam. I’m tired, but in the best possible way. Yesterday, I’ve delivered a talk of one of the best conferences I’ve been part of so far: TestBash Manchester.

The story begins about three quarters of a year ago with me deciding (on a whim, to be honest) to submit a talk on trusting your automation and how automation can deceive you to the Continuous Call for Papers that the Ministry of Testing has for their TestBash events. I chose Manchester and Philadelphia as the target events for my talk, because those were the places I’d most wanted to go to. Some time later an email arrived bearing the message that my talk was accepted for Manchester. Needless to say, I was very happy with that! I’m by no means an expert speaker (yet?) but I do think I’ve got something to say, plus I like to travel (within reason), so this was an excellent opportunity to combine practicing my public speaking skills and revisit a country that I somehow had managed to avoid for the last 12 years, even though it’s only an hour’s flight away.

Fast forward to October 26th (skipping a lot of tinkering with my slides in between) and I find myself in Manchester. Spent most of the Thursday morning travelling from home to the airport, flying and on the tram to my hotel. There I introduced myself to the brilliant concept of Ziferblat where I spent a couple of hours working on some other stuff (living the glamorous life right there!) before heading into town for some R&R catching a film.

Since the conference was already well on its way with a Rapid Software Testing course and a day of workshops (in which I did not take part this time), MoT organized a pre-conference meetup on Thursday evening. That was a great opportunity to finally put faces to a lot of the names I’ve been seeing flying around on Slack and social media in the weeks leading up to the event. It was also great meeting a couple of people again that I’ve met at other conferences before (Rick, Beren, Ash, Marianne, Richard and I might have forgotten one or two). Funny, by the way, how you only meet some fellow Dutchies at conferences abroad, yet never get to meet and talk to them in your own country..

After dinner, some good chats and a couple of drinks it was time to head back to my hotel and get some sleep before the big day. This turned out to be a harder challenge than I thought, the travel efforts, nerves and just enough drinks ensured I had quite a restless night and found myself wide awake at 5 AM. Made the most (yeah right) of it by tinkering with my slides some more before having breakfast and finding my way to The Lowry, which served as the conference venue. There I met some more great people, including the Master Of Disaster Ceremony, Leigh Rathbone. He’s a character, alright! In all seriousness, though, he did an awesome job introducing all of the speakers and keeping energy high throughout the day, something that is not an easy task. Well done, Leigh!

After what can only be described as a series of amazing talks by Anne-Marie Charrett, Göran Kero (thank you for introducing me to ‘automation bias’ and providing a label for a good part of the content of my talk) and Gem Hill it was time for my own talk. Despite being quite nervous -which I tend to see as a good thing, if I’m not nervous it’s a sign I don’t care- it went over really well, and to my relief there were plenty of questions during the Q&A section. I’ve delivered talks before where there were no questions at all afterwards, and believe me, that’s not a nice position to be in as a speaker.

Me during my talk

Me during my talk. Photo courtesy of Cassandra Leung

The rest of the day was filled with excellent talks as well. Coming down from the rush of a talk, I generally tend to hide in a corner for a while and process all of the impressions and the feedback, but that just wasn’t an option here, since I really wanted to see all of the talks. Zoning out had to wait until afterwards! After all of the talks were done and the 99 second talks (a part of every TestBash event) were over too, it was suddenly over. But then it was not, because drinks!

Got some great feedback on my talk during the after-conference drinks and spent some more time talking to other speakers and attendees. Then we headed out for dinner with a smaller group, during which fatigue really kicked in and I decided to call it a day.

If I had to choose, my personal favourite talk was the one from Martin Hynie. It perfectly reflected the point I find myself on in my career: an apprentice of the craft on his way to hopefully one day become a master. I loved Martin’s personal story and the way he reflected on his own journey towards being the best possible version of himself, something that I am aspiring to do as well. I’ll have what he talked about running through my head for a while, I’m sure. More on that probably next week.

Reflecting on it all, this has been a fantastic event for me, some more experience as a conference speaker under my belt, and more importantly I learned so much from the other talks I’m still busy processing it all. Thank you so much, Ministry of Testing, Richard, Rosie, Claire, Gem, Beren, Neil, Hugh, James, Heather, Rick, Marianne, Marcel, Vera, Andrew, Matthew, Patrick, Michael, Robyn, Stephen, Anne-Marie, Vernon, Göran, Cassandra and all the other wonderful people I met and talked to during the event. I’m looking forward to my next TestBash already, which will be in the Netherlands!

I’ll be hosting a workshop on creating an automation strategy that works there, together with Ard Kramer. I am looking forward to seeing you all there!

Here are the slides from my talk, in case you’re interested:

The talk itself has been recorded and will be made available through the Ministry of Testing Dojo. Also, here’s an awesome sketchnote of my talk made by Constance:

Sketchnote of my talk by Constance