Defining Continuous Testing for myself

On a couple of recent occasions, I found myself talking about Continuous Testing and how test automation related to this phenomenon (or buzzword, if you prefer that term). However, to this day I didn’t have a decent answer to the question of what Continuous Testing (CT) is and how exactly it relates to test automation. Now that I’m busy preparing another webinar (this time together with the people at Testim, but more on that probably in another post), and we find ourselves again talking about CT, I thought it was due time to start carving out a definition for myself. This blog post is much like me thinking out loud, so bear with me.

To start, CT is definitely not equal to test automation, not even to test automation on steroids (i.e., test automation that is actually fast, stable and repeatable. Instead, I see CT as an integrated part of the Continuous Delivery (CD) life cycle:

Continuous Testing as part of the Continuous Delivery life cycle

You could also say that CT is a means of that teams can adopt to support CD while aiming to deliver quality software.

Let’s take a closer look and dissect the term ‘Continuous Testing’. The first part, ‘Continuous’, to me is a term that means two things in the CD context:

  1. The software that is being created is continuously tested (or, more likely, continually) in a given environment. In other words, any version of the software that enters an environment is immediately subjected to the tests that are associated with that environment. This environment can be a local development environment, a test environment, or even a production environment (where testing is often done in the form of monitoring).
  2. The software that is being created is continuously tested as it passes through environments. In other words: there’s no deploy that is not being followed by some form of testing, and in case of the local development environment, tests are run before any deployment (or better, commit and build) is being done.

Continuous Testing in two dimensions

This is not necessarily different from any other software delivery method, but what makes CT (and CD) stand out is that the time between deployments is typically very short. And that’s where the second part of the term Continuous Testing comes into play: ‘Testing’. How is this testing being done? This is where automation often comes into play, as an enabler of fast feedback and software moving through the pipeline fast, yet in a controlled manner.

Teams that want to ‘do’ CT and CD simply cannot be blocked by testing as an activity tacked on at the end. Instead, CT requires a shift of mind from traditional testing as an afterthought to testing being ingrained throughout the pipeline. Formalized handoffs and boundaries between environments will have to be replaced by testing activities that act as gatekeepers and safety nets. And where necessary and useful, this testing is supported by tools. In this respect, test automation can be an enabler of Continuous Testing. But (as is so often the case), only if that automation makes sense. Again, I refer to this blog post if you want to know what I mean by ‘making sense’ in a CT context.

I still don’t have a one line, encyclopedia-style definition for Continuous Testing, but at least the thought process I went through to write this (short) blog post helped me put some things into place. Next to Katrina Clokie’s book on testing in DevOps, the following articles have been a source of information for me (while being much better written than these ramblings):

What is continuous testing? Know the basics of this core safety net for DevOps teams
A real-world guide to continuous testing
What is Continuous Testing?
Continuous Testing vs. test automation (whitepaper)
The Great Debate: Automated Testing vs Continuous Testing

What is your definition of Continuous Testing? Do you have one?

Why I think automation education is broken (and what I’ll try and do about it)

I’ve written various blog posts about test automation craftsmanship recently, a topic that is becoming dearer to me every time I see people posing automation-related statements or questions that are, at the very least, of questionable quality. Like in ye olde times, craftsmanship isn’t something that is easily attained, or can be attained at all, without proper education and mentorship. And that’s where I think the test automation world is still lacking. Or, to put it in positive terms: there’s room for improvement in this respect.

And I’m not alone in this. I had a couple of good discussions on Twitter the last couple of weeks (yes, this is possible!), most notably an insightful exchange of messages with Matt Heusser (not sure if you’re reading this, but anyway, thanks Matt!), on the current state of automation training and how it is advertised. The gist of it (and note that this is my take on it):

  1. There is an (over)abundance of tool-centered training out there. This is not necessarily a problem, but there is definitely room for more broader training on the fundamentals of test automation and how it should be applied.
  2. A lot of this tool-centered training is advertised as ‘Become an expert in tool XYZ in just three days’. This IS a problem. First of all, I don’t think it is possible to become an expert in any significant tool, approach or anything in the test automation space in just a couple of days. It’s possible to become familiar with the API and features of a tool, but that hardly makes you an expert. Expertise comes with application, failing, studying, learning, etc. It takes months, sometimes years, not days.

The second point is also dangerous in that it can lead to an army of self proclaimed ‘experts’ that are really nothing more than people with hammers that see only nails on their path. Not an image I have in mind when I think about what constitutes being a test automation expert.

What is lacking, in my opinion, is something that gives people involved in test automation a solid foundation of knowledge about the field, its challenges and its place in the larger software development space. Something that goes beyond the specifics of individual tools. Something that talks some sense into the people crying ‘automate all the things’, so to say. And by ‘people’, I don’t just mean automation engineers, but developers, scrum masters, POs, managers, CxO-level people, everybody that is a test automation stakeholder and should therefore care about what applying automation in a sensible way can bring to software development.

So, what to do? Ranting about how things are broken is one thing (and I must admit that it DOES feel good to me), but I’ve been thinking about and saying the above for a while now. So maybe it’s time to start to do something about it. That’s why I’ve started to outline a course that I think should be able to fill the void when it comes to education around test automation. Call it ‘Test automation awareness’, call it ‘Automation 101’, call it whatever you like, I’m still open to suggestions as to the name of the course. Point is, it’s time to put my money where my mouth is. I’ve already reached out to some people and received some awesome feedback (thanks guys, you know who you are). Funny thing, a couple of people I reached out to said they were working on something similar. Which is even better, as this confirms my view that there is a need for a course like this.

I’m not sure at the moment when this will go live, and in what form exactly, but as soon as there’s more to disclose, I’ll do it here. If you’d like to give input, constructive criticism and/or contribute in some other way, please send me a note at bas@ontestautomation.com and I’ll get back to you. I’m very much looking forward to making this a thing, although not so much to the work that’s ahead of me. But I feel it’s important enough to get done.

On a not totally unrelated note, I’ve also recently had a very fruitful discussion with someone from an academic research facility, and if it’ll all work out, it looks like I’ll be somewhat closer involved in one of their projects as well. This might also be a good place to start infiltrating the education system and see that test automation earns a better place in higher education as well. I don’t have the illusion that I’ll change the world overnight in this respect, but you have to start somewhere, right? And if anything it’ll be a good opportunity for me to step a little outside of my comfort zone again.

I’ll keep you posted.

P.S.: Most of you will have heard or read about the fact that Katrina Clokie’s book ‘A Practical Guide To Testing In DevOps’ has been released through LeanPub. I’ve just finished reading it, and the only thing I can say is that if you’re even remotely interested in testing or DevOps, I’d highly recommend you to buy a copy. It’s chock full of tips and case studies for everybody, tester or not, facing the challenge of keeping up with DevOps and with the rapidly increasing speed of software delivery in general, without forgetting to keep an eye on software quality.

How I continuously hone my skills and why you should too

As a consultant, and even more so as a freelance consultant, it is imperative to stay relevant and be able to land your next project every time. Or, and that’s the position I prefer to be in, have your next project(s) find you. I thought it would be a nice idea to share with you how I go about trying to keep up in the ever changing world of test automation and testing and software development in general.

Before I dive into the strategies I employ to remain relevant, though, I’d like to stress once more that test automation, like testing and software development, is a craft. I’ve said this so many times already, but I’ll just press ‘repeat’ and say it once more: being a good test automation crafts(wo)man requires specific skills that need to be constantly sharpened it you want to remain relevant. And I don’t think I’m going out on a limb here when I assume that you actually want to remain relevant.

Also, I’m writing this post partly as preparation for a potential mentee that contacted me with some questions regarding test automation career development. If this comes through it’ll be the first time for me in a mentor role, and that’s something I’m both very much looking forward to and scared as hell of..

So, what is it that I do to try and stay on the forefront of the test automation field as best as I can?

Discover your way to provide value
I’m no proponent of the ‘jack of all trades, master of none’, excuse me, ‘generalist’ approach to career development. I think in order to be truly valuable to any team or organization, it’s important to pick your superpower and grow it as best you can. For me, that’s helping organizations define and implement the best possible way for their test automation efforts. For others, that might be writing the best possible Selenium tests, or being the best possible software tester, or … It doesn’t matter WHAT your superpower is, as long as it provides value to the software development process you’re likely to remain relevant for the duration of your career. You will need to monitor whether or not you’re providing value constantly, though.

Get hired for the role you want to grow into
This applies especially to freelance consultants. Teams and organizations that hire you tend to do that because you know how to do something well, simply because you’ve done it before. That’s all fine and dandy in the short term, of course (there’s always the next mortgage installment or child care bill that needs to be paid), but there’s a real risk of becoming a one trick pony in the longer term. Personally, I’m always evaluating a project offer for things I can learn myself and whether those are things that I actually want to learn, i.e., whether the things I’ll be learning will contribute to me being a slightly better consultant or trainer after the project has come to an end. I tend to get bored on projects quite easily, and that’s especially the case when I’m repeating the same stuff over and over for a couple of months. And that doesn’t help me nor my client.

For all of you that are employees, it might be a little easier, simply because organizations are often willing to invest in your professional and personal development. Still, I’d advise you to always ask as many questions on possibilities to grow and explore new things when in an interview. Being in a job that allows you to explore, learn and grow is much, much more important than an extra couple of hundreds bucks every month. Really, it is. Also, grow as a crafts(wo)man and that pay rise will come, if not at your current employer, then in the form of an offer from another one. At which point you’re advised to look for the professional and personal development options THEY provide, of course.

Honing your skills

Attend and speak at conferences
Next to my day to day work, I find that one of the best ways to grow as a consultant is by attending and contributing to conferences. And the talks and workshops offered by the organizational committee are but one of the ways you can learn and benefit from being at a conference. For me, meeting peers, discussing the craft with them, as well as some good old fashioned networking often proves to be even more valuable in the longer term.

And attending is just one side of the whole conference thing: actually speaking at one might be an even better way to improve your craftsmanship. Yes, preparing a talk takes a lot of time, but it is an excellent way to organize your thoughts and experiences and to ‘find your voice’. And yes, going up on stage is nerve-wracking, but remember: that’s what every speaker feels, even those that are way more experienced. Not convinced? Just read up on the #SpeakerConfessions hashtag on Twitter to see what I mean.

Reading blogs and listening to podcasts
There’s a wider testing and automation world out there, outside of the confines of your office (or wherever you do your work). I’ve been gaining access to the thoughts, experiences, successes and failures of fellow automation engineers and consultants for a couple of years now, simply by reading their blogs, learning from them and applying what I’ve learned in my own work.

Instead of singling out individual blogs that I read (which is quite a long list anyway), I’ll mention two of the greatest starting points for blog reading here: the Testing Curator Blog by Matt Hutchison and the Ministry of Testing blog feed. Both are excellent sources for all things testing and automation to read for your pleasure and learning.

Another way to learn from others is by listening to testing and automation podcasts. Since I spend a significant amount of time in the car (it’s been getting less and less, though, since I started reshaping my career, which is something I do like), I like to spend that time in a useful way, and one of the best is by listening to podcasts. There’s a large amount of software testing podcasts out there, so I’d suggest you to do a search on iTunes and find something you like. As for me, I almost never miss an episode of Joe Colantonio’s TestTalks or Keith Klain’s Quality Remarks podcast. There are some other podcasts that I listen to on and off again as well.

Writing your own blog
There’s no learning from other people’s experiences through blog posts without there being people that actually write those blog posts, obviously. I’d recommend you considering to start your own. To me, there’s no better way to organize my thoughts and express how I feel about certain topics than writing a blog post (or a series of blog posts) on those topics. They’re also a great way to showcase your projects, thoughts, insights and experiences to the wider world, although it must be said that writing a blog post or two and expecting the work and the world to come to you might lead to a little bit of disappointment. Instead, you should remember that you’re writing for yourself in the first place, anyone else reading or reacting on your posts is a bonus. Having said that, when you’re consistently (and consistency IS key) putting out decent content and showing it to the outside world, the interaction and feedback will come.

It’s just not going to happen overnight. As an example: it took me three years to get that little bit of traction I’m having at the moment. I’m very glad I stuck with it, though, because so much good has come out of it! Being given the opportunity to write and publish an ebook, being offered speaking opportunities abroad, writing articles for industry websites, none of it wouldn’t have happened (or at least it would have taken me a lot longer) without this blog. I’m grateful for that every day, and apart from the interaction with readers, it’s one of the main reasons I keep at it.

Wrapping this up, I’d like to repeat that continuously honing your skills as a testing or automation engineer, consultant or whatever it is that you’re doing is imperative to remaining relevant and staying ahead in the competitive business we’re in. Hopefully the above has given you some motivation or pointers to start being an even better crafts(wo)man than the one you already are. Again, I’d love to hear your thoughts and feedback.