On supporting Continuous Testing with FITR test automation

Last week I had the chance to participate as a contributor to my very first webinar. The people at SeaLights, an Israeli company that offers a management platform for Continuous Testing, asked me to come on the webinar and share my views on test automation and continuous testing. In this post, I’ll share some of the thoughts and opinions I talked about there.

Test automation is everywhere, however…
Test automation is everywhere. That’s probably nothing new.

A lot of organizations are adopting Continuous Integration and Continuous Delivery. Also nothing new.

To be able to ‘do’ CI/CD, a lot of organizations are relying on their automated tests to help safeguard quality thresholds while increasing release speed. Again, no breaking news here.

However, to safeguard quality in CI and CD you’ll need to be able to do Continuous Testing (CT). Here’s my definition of CT, which I used in the webinar:

Continuous Testing is a process that allows you to gauge the quality of your software on demand. No matter if you’re building and deploying once a month or once a minute, CT allows you to get insight into application quality at all times.

It won’t come as a surprise to you that automated tests often form a big part of an organization’s CT strategy. However, just having automated tests is not enough to be able to support CT. Your automated test approach should not just cover application functionality and coverage, but also:

  • A solid test data management strategy
  • Effective test environment management
  • Informative reporting, targeted towards all relevant audiences

And probably many more things that I’m forgetting here..

In order to be able to leverage your automated tests successfully for supporting CT, I’ve come up with a model based on four pillars that need to be in place:

From AT to CT with FITR tests

Let’s take a quick look at each of these FITR pillars and how they are necessary in CT.

Focused
Automated tests need to be focused to effectively support CT. ‘Focused’ has two dimensions here.

First of all, your tests should be targeted at the right application component and/or layer. It does not make sense to use a user interface-driven test to test application logic that’s exposed through an API (and subsequently presented through the user interface), for example. Similarly, it does not make sense to write API-level tests that validate the inner workings of a calculation algorithm if unit tests can provide the same level of coverage. By now, most of you will be familiar with the test automation pyramid. While I think it’s not to be used as a guideline, I find that the pyramid does provide a good starting point for discussion when it comes to focusing your tests at the right level. Use the model to your advantage.

The second aspect of focused automated tests is that your tests should test what they can do effectively. This boils down to sticking to what your test solution and tools in it do best, and leaving the rest either to other tools or to testers, depending on what’s there to be tested. Don’t try and force your tool to do things it isn’t supposed to (here’s an example).

If your tests are unfocused, they are far more likely to be slow to run, to have high maintenance costs and to provide inaccurate or shallow feedback on application quality.

Informative
Touching upon shallow or inaccurate feedback, automated tests also need to be informative to effectively support CT. ‘Informative’ also has two separate dimensions.

Most importantly, the results produced and the feedback provided by your automated tests should allow you, or the system that’s doing the interpretation for you (such as an automated build tool), make important decisions based on that feedback. Make sure that the test results and reporting provided contain clear results, information and error messages, targeted towards the intended audience. Keep in mind that every audience has its own requirements when it comes to this information. Developers likely want to see stack trace, whereas managers don’t. Find out what the target audience for your reporting and test results is, what their requirements are, and then cater to them as best as you can. This might mean creating more that one report (or source of information in general) for a single test run. That’s OK.

Another important aspect of informative automated tests is that it should be clear what they do (and what they don’t do). You can make your tests themselves be more informative through various means, including (but not limited to) using naming conventions, using a BDD tool such as Cucumber or SpecFlow to create living documentation for your tests (blasphemy? maybe.. but if it works, it works), and following good programming practices to make your code better readable and maintainable.

When automated test solutions and the results they produce are not informative, valuable time is wasted analyzing shallow feedback, or gathering missing information, which evidently breaks the ‘continuous’ part of CT.

Trustworthy
When you’re relying on your automated tests to make important decisions in your CT activities, you’d better make sure they’re trustworthy. As I described in more detail in previous posts, automated tests that cannot be trusted are essentially worthless. Make sure to eliminate false positives (tests that fail when they shouldn’t), but also false negatives (tests that pass when they shouldn’t).

Repeatable
The essential idea behind CT (referring to the definition I gave at the beginning of this blog post) is that you’re able to determine application quality on demand. Which means you should be able to run your automated tests on demand. Especially when you’re including API-level and end-to-end tests, this is often not as easy as it sounds. There are two main factors that can hinder the repeatability of your tests:

  • Test data. This is in my opinion one of the hardest ones to get right, especially when talking end-to-end tests. Lots of applications I see and work with have (overly) complex data models or share test data with other systems. And if you’re especially lucky, you’ll get both. A solid test data strategy should be put in place to do CT, meaning that you’ll either have to create fresh test data at the start of every test run or have the ability to restore test data before every test run. Unfortunately, both options can be quite time consuming (if at all attainable and manageable), drawing you further away from the ‘C’ in CT instead of bringing you closer.
  • Test environments. If your application communicates with other components, applications or systems (and pretty much all of them do nowadays), you’ll need suitable test environments for each of these dependencies. This is also easier said than done. One possible way to deal with this is by using a form of simulation, such as mocking or service virtualization. Mocks or virtual assets are under your full control, allowing you to speed up your testing efforts, or even enable them in the first place. Use simulation carefully, though, since it’s yet another thing to be managed and maintained, and make sure to test against the real thing periodically for optimal results.

Having the above four pillars in place does not guarantee that you’ll be able to perform your testing as continuously as your CI/CD process requires, but it will likely give it a solid push in the right direction.

Also, the FITR model I described here is far from finished. If there’s anything I forgot or got wrong, feel free to comment or contact me through email. I’d love to get feedback.

Finally, if you’re interested in the webinar I talked about earlier, but haven’t seen it, it’s freely available on YouTube:

Romanian Testing Conference 2017 was a blast!

Last week I had the pleasure of taking part in the 2017 edition of the Romanian Testing Conference. I was contacted by Andrei from the organizing committee in August of last year, initially to host a workshop at what would be the first edition of a spin-off conference of the main RTC event. That conference unfortunately had to be cancelled, but Andrei from the organizing committee was kind enough to extend the invitation to this year’s edition of the original event. And what an excellent couple of days they’ve been!

Wednesday: Cluj
Wednesday saw a very early start to the day, with my alarm set at 3.45. My plane to Munich set off at 7.00, and after a quick and easy transfer I suddenly found myself in Romania! After getting into the country through customs I was faced with the first sign of how excellently organized this whole event would be: there was a car with a driver waiting for me at the arrivals hall to drive me from the airport to the hotel. I felt spoiled already!

The official RTC 2017 car

After checking in to the luxurious Grand Hotel Italia I decided to go and see the city for a bit, as this day would be the only day where I’d have a little time to do so. I’m not really a city person (I spent an afternoon in NYC and thought that was enough..) but I’m making a habit of seeing more of the area I’m visiting than just an airport, a hotel and a conference venue. Luckily, the weather was gorgeous and there’s some really good coffee stalls to be found on the streets of Cluj, so it was time well spent.

Upon returning to the hotel, I met some of the other speakers, as well as Rob, the conference chairman. The rest of the day was fairly uneventful, with dinner in my room, watching Office Space for the umpteenth time and an early night. The day had been long enough, plus I thought it might be a good idea to be fresh and well rested in the morning for my workshop.

Thursday: workshop day
Thursday was show time for me, the day of my workshop on REST Assured (mostly) and WireMock (a bit). I heard in advance that my workshop was fully booked, which meant that there were 30 people that registered for it. Normally, when I do training, I’ll try and get no more than 12-15 people, but since this was the fourth or fifth time I’d be giving this workshop and I received exactly 0 emails from attendees that had trouble completing the preparation instructions I’d sent them a couple of weeks in advance, I wasn’t too uncomfortable with that.

Attendees hard at work during my workshop

I was pleasantly surprised that all participants were fully prepared, which doesn’t happen regularly. A great start to the day, because that means no time lost setting up people’s laptops. Instead, we were able to dive into REST Assured directly. I felt the workshop went rather well, the only thing I had a bit of trouble with is getting the interaction going. People asked me enough questions one-on-one when I was walking around when they were working on the exercises I provided, but I wasn’t able to get a lot of plenary discussion going. As a result, it was a bit hard to gauge whether or not people were engaged and interested, or bored and distracted. They seemed to be happy enough with the way I delivered the workshop, though. This was reflected in the ratings I received afterwards:

Ratings for my workshop

For those of you who are interested in what I covered in the workshop, you can find all of the slides, the exercises and the answers on my GitHub page here. Feel free to review, steal and otherwise use them for your own fun and profit. Or book me to deliver it to a place near you 😉

After the workshops were over, it was time for the official speakers dinner. We took taxis to a nice restaurant (the name of it escapes me for now) and I spent a great couple meeting new people (Keith, Beren, Nicola, Elizabeth, Kamila, Viktor and so many others) and catching up with others I met before (Ard, Huib, Rick and others as well). One of the highlights of the whole event for me, even though I felt somewhat knackered after a full day of teaching. After dinner, it was time for a last couple of drinks in the hotel lobby (not a bad place to spend some time either, as you can see below) and off to bed.

The Grand Hotel Italia lobby

Friday: the conference
Because the hard part was over for me after delivering my workshop, I got to enjoy the conference day without the stress that comes with having to do a talk or anything else. This meant I could pick and attend the talks I liked, spend some time wandering around and talking to people, or just zoning out whenever I felt like it. The programme that was put together by the organizing committee was of very high quality, so most of the time there was at least one talk that was worth attending.

During the day, I enjoyed talks about finding and holding on to your passion (Santhosh), AI and Machine Learning (Cristina), introversion (Elizabeth), not talking about testing (Keith), bitter truths in test automation (Viktor, who seems to be able to read my mind), wrapping up projects and moving on (Nicola) and a closing keynote about youngsters and game testing by the awesome Harry (and yes, he’s really only 12).

All in all, another great day, but an exhausting one too. I wasn’t planning on attending the conference after party, but in the end I spent a couple of hours there anyway, talking some more to other speakers and attendees and reflecting on what was simply a wonderful event and an experience I’ll be remembering for a long time.

Saturday: back home
Unfortunately, the plane was scheduled to take off quite early on Saturday morning (my own fault!), but the flights home were uneventful and in the end I was happy to see my wife and kids again. When I’m writing this, I’m still feeling somewhat tired, but it was all more than worth it in the end.

If you’re ever considering attending (or better: speaking at) the Romanian Testing Conference yourself, I can only really recommend it. The organizing committee have put together a wonderful, high quality event and both speakers and attendees are taken care of in the best possible manner. And even though I’m trying to visit events in as many different countries as possible, I’m already considering going again next year!

Remember what your tests are trying to verify

Lately, I’ve been working a lot on Selenium-based test automation solutions. And even though I’m still not overly enthusiastic about creating lots of user interface-driven automated tests, now that I’m getting more skilled at creating robust and stable Selenium tests, I am starting to appreciate the tool and what you can do with it more and more. As everybody working with Selenium can tell you, there are situations where things can get, well, interesting. And by interesting, I mean tricky. Dealing with dynamic front end frameworks and unpredictable modals and overlays can ask a lot of you in terms of debugging and exception handling skills.

Not yet being fluent in Selenese, I find myself on Google and (subsequently) StackOverflow a lot, trying to find suitable solutions for problems I encounter. While doing so, there’s one thing that I see a lot of people do, yet strikes me as really odd, given what I think is the purpose of these user interface-driven tests:

Forcing your tests to do something your users can’t.

From what I’ve understood, studying and working in the test automation field for a while now, user interface-driven tests, such as these Selenium tests, should be used to verify that the user of your user interface is able to complete a sequence of predefined actions. If you’re working in a hip and happening environment, these are often called ‘customer journeys’ or ‘user journeys’. But that’s not my point. What my point is, is that quite often I see workarounds suggested that go beyond what a regular user could do with his or her keyboard and / or mouse.

For example, take an element that is invisible until you hover over it with your mouse. If you just try to do a click(), your test will probably throw an exception stating that the element was not visible. Now, there are (at least) two ways to deal with this situation:

  1. Use the Actions class to simulate a mouseover, then click the element.
  2. Use a JavaScriptExecutor to perform the click.

While both approaches might result in a passing test, I am of the opinion that one is useful and the other is a horrible idea. Based on what I’ve written so far, can you guess which of the two options I’d suggest?

Indeed, option #1 is the way to go, for two reasons:

  • User interface-driven tests should mimic actual user interaction as closely as possible. I’ve never seen a user execute some JavaScript on a page to make an element visible. Either that, or I’m hanging around the wrong group of users..
  • What happens if a front-end developer makes a mistake (I know, they never do, but let’s assume so anyway) which causes the element not to become visible, even on a mouseover? With option #1, your test will fail, for the right reason. With #2, hello false negative!

There are some exceptions to the rule, though. The prime example I can think of is handling file uploads by directly sending the absolute path of the file to be uploaded to the input element responsible for the file upload using sendKeys(), instead of clicking on it and handling the file dialog. I’ve tried the latter before, and it’s a pain, first because you can’t do it with the standard Selenium API (because the file dialog is native to the operating system), second because different browsers use different file dialog layouts, resulting in a lot of pesky code that easily breaks down. In this case, I prefer to bypass the file dialog altogether (it’s probably not the subject of my test anyway).

In (almost) all other cases though, I’d advise you to stick to simulating your end user behavior as closely as possible. The job of your user interface-driven tests, and therefore of you as its creator, is not to force a pass, but to simulate end user interaction and see if that leads to a successfully executed scenario. Don’t fool yourself and your stakeholders by underwater tricks that obscure potential user interface issues.

Remember what your tests are trying to verify.