Defining Continuous Testing for myself

On a couple of recent occasions, I found myself talking about Continuous Testing and how test automation related to this phenomenon (or buzzword, if you prefer that term). However, to this day I didn’t have a decent answer to the question of what Continuous Testing (CT) is and how exactly it relates to test automation. Now that I’m busy preparing another webinar (this time together with the people at Testim, but more on that probably in another post), and we find ourselves again talking about CT, I thought it was due time to start carving out a definition for myself. This blog post is much like me thinking out loud, so bear with me.

To start, CT is definitely not equal to test automation, not even to test automation on steroids (i.e., test automation that is actually fast, stable and repeatable. Instead, I see CT as an integrated part of the Continuous Delivery (CD) life cycle:

Continuous Testing as part of the Continuous Delivery life cycle

You could also say that CT is a means of that teams can adopt to support CD while aiming to deliver quality software.

Let’s take a closer look and dissect the term ‘Continuous Testing’. The first part, ‘Continuous’, to me is a term that means two things in the CD context:

  1. The software that is being created is continuously tested (or, more likely, continually) in a given environment. In other words, any version of the software that enters an environment is immediately subjected to the tests that are associated with that environment. This environment can be a local development environment, a test environment, or even a production environment (where testing is often done in the form of monitoring).
  2. The software that is being created is continuously tested as it passes through environments. In other words: there’s no deploy that is not being followed by some form of testing, and in case of the local development environment, tests are run before any deployment (or better, commit and build) is being done.

Continuous Testing in two dimensions

This is not necessarily different from any other software delivery method, but what makes CT (and CD) stand out is that the time between deployments is typically very short. And that’s where the second part of the term Continuous Testing comes into play: ‘Testing’. How is this testing being done? This is where automation often comes into play, as an enabler of fast feedback and software moving through the pipeline fast, yet in a controlled manner.

Teams that want to ‘do’ CT and CD simply cannot be blocked by testing as an activity tacked on at the end. Instead, CT requires a shift of mind from traditional testing as an afterthought to testing being ingrained throughout the pipeline. Formalized handoffs and boundaries between environments will have to be replaced by testing activities that act as gatekeepers and safety nets. And where necessary and useful, this testing is supported by tools. In this respect, test automation can be an enabler of Continuous Testing. But (as is so often the case), only if that automation makes sense. Again, I refer to this blog post if you want to know what I mean by ‘making sense’ in a CT context.

I still don’t have a one line, encyclopedia-style definition for Continuous Testing, but at least the thought process I went through to write this (short) blog post helped me put some things into place. Next to Katrina Clokie’s book on testing in DevOps, the following articles have been a source of information for me (while being much better written than these ramblings):

What is continuous testing? Know the basics of this core safety net for DevOps teams
A real-world guide to continuous testing
What is Continuous Testing?
Continuous Testing vs. test automation (whitepaper)
The Great Debate: Automated Testing vs Continuous Testing

What is your definition of Continuous Testing? Do you have one?

Why I think automation education is broken (and what I’ll try and do about it)

I’ve written various blog posts about test automation craftsmanship recently, a topic that is becoming dearer to me every time I see people posing automation-related statements or questions that are, at the very least, of questionable quality. Like in ye olde times, craftsmanship isn’t something that is easily attained, or can be attained at all, without proper education and mentorship. And that’s where I think the test automation world is still lacking. Or, to put it in positive terms: there’s room for improvement in this respect.

And I’m not alone in this. I had a couple of good discussions on Twitter the last couple of weeks (yes, this is possible!), most notably an insightful exchange of messages with Matt Heusser (not sure if you’re reading this, but anyway, thanks Matt!), on the current state of automation training and how it is advertised. The gist of it (and note that this is my take on it):

  1. There is an (over)abundance of tool-centered training out there. This is not necessarily a problem, but there is definitely room for more broader training on the fundamentals of test automation and how it should be applied.
  2. A lot of this tool-centered training is advertised as ‘Become an expert in tool XYZ in just three days’. This IS a problem. First of all, I don’t think it is possible to become an expert in any significant tool, approach or anything in the test automation space in just a couple of days. It’s possible to become familiar with the API and features of a tool, but that hardly makes you an expert. Expertise comes with application, failing, studying, learning, etc. It takes months, sometimes years, not days.

The second point is also dangerous in that it can lead to an army of self proclaimed ‘experts’ that are really nothing more than people with hammers that see only nails on their path. Not an image I have in mind when I think about what constitutes being a test automation expert.

What is lacking, in my opinion, is something that gives people involved in test automation a solid foundation of knowledge about the field, its challenges and its place in the larger software development space. Something that goes beyond the specifics of individual tools. Something that talks some sense into the people crying ‘automate all the things’, so to say. And by ‘people’, I don’t just mean automation engineers, but developers, scrum masters, POs, managers, CxO-level people, everybody that is a test automation stakeholder and should therefore care about what applying automation in a sensible way can bring to software development.

So, what to do? Ranting about how things are broken is one thing (and I must admit that it DOES feel good to me), but I’ve been thinking about and saying the above for a while now. So maybe it’s time to start to do something about it. That’s why I’ve started to outline a course that I think should be able to fill the void when it comes to education around test automation. Call it ‘Test automation awareness’, call it ‘Automation 101’, call it whatever you like, I’m still open to suggestions as to the name of the course. Point is, it’s time to put my money where my mouth is. I’ve already reached out to some people and received some awesome feedback (thanks guys, you know who you are). Funny thing, a couple of people I reached out to said they were working on something similar. Which is even better, as this confirms my view that there is a need for a course like this.

I’m not sure at the moment when this will go live, and in what form exactly, but as soon as there’s more to disclose, I’ll do it here. If you’d like to give input, constructive criticism and/or contribute in some other way, please send me a note at bas@ontestautomation.com and I’ll get back to you. I’m very much looking forward to making this a thing, although not so much to the work that’s ahead of me. But I feel it’s important enough to get done.

On a not totally unrelated note, I’ve also recently had a very fruitful discussion with someone from an academic research facility, and if it’ll all work out, it looks like I’ll be somewhat closer involved in one of their projects as well. This might also be a good place to start infiltrating the education system and see that test automation earns a better place in higher education as well. I don’t have the illusion that I’ll change the world overnight in this respect, but you have to start somewhere, right? And if anything it’ll be a good opportunity for me to step a little outside of my comfort zone again.

I’ll keep you posted.

P.S.: Most of you will have heard or read about the fact that Katrina Clokie’s book ‘A Practical Guide To Testing In DevOps’ has been released through LeanPub. I’ve just finished reading it, and the only thing I can say is that if you’re even remotely interested in testing or DevOps, I’d highly recommend you to buy a copy. It’s chock full of tips and case studies for everybody, tester or not, facing the challenge of keeping up with DevOps and with the rapidly increasing speed of software delivery in general, without forgetting to keep an eye on software quality.

On including automation in your Definition of Done

Working with different teams in different organizations means that I’m regularly faced with the question of whether and how to include automation in the Definition of Done (DoD) that is used in Agile software development. I’m not an Agilist myself per se (I’ve seen too many teams get lost in overly long discussions on story points and sticky notes), but I DO like to help people and teams struggling with the place of automation in their sprints. As for the ‘whether’ question: yes, I definitely think that automation should be included in any DoD. The answer to the ‘how’ of including, a question that could be rephrased as the ‘what’ to include, is a little more nuanced.

For starters, I’m not too keen on rigid DoD statements like

  • All scenarios that are executed during testing and that can be automated, should be automated
  • All code should be under 100% unit test coverage
  • All automated tests should pass at least three consecutive times, except on Mondays, when they should pass four times.

OK, I haven’t actually seen that last one, but you get my point. Stories change from sprint to sprint. Impact on production code, be it new code that needs to be written, existing code that needs to be updated or refactored or old code that needs to be removed (my personal favorite) will change from story to story, from sprint to sprint. Then why keep statements regarding your automated tests as rigid as the above examples? Doesn’t make sense to me.

I’d rather see something like:

Creation of automated tests is considered and discussed for every story and their overarching epic and applied where deemed valuable. Existing automated tests are updated where necessary, and removed if redundant.

You might be thinking ‘but this cannot be measured, how do we know we’re doing it right?’. That’s a very good question, and one that I do not have a definitive answer for myself, at least not yet. But I am of the opinion that knowing where to apply automation, and more importantly, where to refrain from automation, is more of an art than a science. I am open to suggestions for metrics and alternative opinions, of course, so if you’ve got something to say, please do.

Having said that, one metric that you might consider when deciding whether or not to automate a given test or set of tests is whether or not your technical debt increases or decreases. The following consideration might be a bit rough, but bear with me. I’m sort of thinking out loud here. On the one hand, given that a test is valuable, having it automated will shorten the feedback loop and decrease technical debt. However, automating a test takes time in itself and increases the size of the code base to be maintained. Choosing which tests to automate is about finding the right balance with regards to technical debt. And since the optimum will likely be different from one user story to the next, I don’t think it makes much sense to put overly generalizing statements with regards to what should be automated in a DoD. Instead, for every story, ask yourself

Are we decreasing or increasing our technical debt when we automate tests for this story? What’s the optimum way of automating tests for this story?

The outcome might be to create a lot of automated tests, but it might also be to not automate anything at all. Again, all depending on the story and its contents.

Another take on the question whether or not to include automated test creation in your DoD might be to discern between the different scope levels of tests:

  • Creating unit tests for the code that implements your user story will often be a good idea. They’re relatively cheap to write, they run fast and thereby, they’re giving you fast feedback on the quality of your code. More importantly, unit tests act as the primary safety net for future development and refactoring efforts. And I don’t know about you, but when I undertake something new, I’d like to have a safety net just in case. Much like in a circus. I’m deliberately refraining from stating that both circuses and Agile teams also tend to feature a not insignificant number of clowns, so forget I said that.
  • You’ll probably also want to automate a significant portion of your integration tests. These tests, for example executed at the API level, can be harder to perform manually and are relatively cheap to automate with the right tools. They’re also my personal favorite type of automated tests, because they’re at the optimum point between scope and feedback loop length. It might be harder to write integration tests when the component you’re integrating with is outside of your team’s control, or does not yet exist. In that case, simulation might need to be created, which requires additional effort that might not be perceived as directly contributing to the sprint. This should be taken into account when it comes to adding automated integration tests to your DoD.
  • Finally, there’s the end-to-end tests. In my opinion, adding the creation of this type of tests to your DoD should be considered very carefully. They take a lot of time to automate (even with an existing foundation), they often use the part of the application that is most likely to change in upcoming sprints (the UI), and they contribute the least to shortening the feedback loop.

The ratio between tests that can be automated and tests for which it make sense to be automated in sprint can be depicted as follows. Familiar picture?

Should you include automated tests in your Definition of Done?

Please note that like the original pyramid, this is a model, not a guideline. Feel free to apply it, alter it or forget it.

Jumping back to the ‘whether’ of including automation in your DoD, the answer is still a ‘yes’. As can be concluded from what I’ve talked about here, it’s more of a ‘yes, automation should have been considered and applied where it provides direct value to the team for the sprint or the upcoming couple of sprints’ rather than ‘yes, all possible scenarios that we’ve executed and that can be automated should have been automated in the sprint’. I’d love to hear how other teams have made automation a part of their DoD, so feel free to leave a comment.

And for those of you who’d like to see someone else’s take on this question, I highly recommend watching this talk by Angie Jones from the 2017 Quality Jam conference: