What does good test automation training look like?

As I’m moving away from by-the-hour work and more towards a hybrid of consulting, training, writing and speaking, one of the things I’m working on is slowly building up a workshop and training portfolio around topics I think I’m qualified to talk about. I have created a couple of workshops already, but so far, they are centered around a specific tool that I am interested in and enthusiastic about. This, however, has the downside that they’re probably targeted towards a relatively small group of interested people (not in the least because these tools are only available for a specific programming language, i.e., Java).

To extend my options with regards to delivering training and workshops, I am currently looking at developing workshops and training material that contain higher level and more generic material, while still offering practical insights and hands-on exercises. There are a lot of different approaches and possible routes that can be taken to achieve this, especially since there is no specific certification trajectory around test automation (nor do I think there should be, but that’s a wholly different discussion that I’ll probably cover in another blog post in time). So far, I haven’t figured out the ideal contents and delivery format, but ideas have been taking shape in my head recently.

Here are some subjects I think a decent test automation training trajectory should cover:

Test automation 101: the basics
Always a good approach: start with the basics. What is test automation? What is it not (here’s a quote I love from Jim Hazen)? What role does automation play in current development and testing processes and teams? Why is it attracting the interest it does? To what levels and what areas can you apply test automation and what is that test automation pyramid thing you keep hearing about?

Test automation implementation
So, now that you know what test automation (sorta kinda) is, how to apply it to your software development process? How are you going to involve stakeholders? What information or knowledge do you want to derive from test automation? How does it fit into trends such as Agile software development, BDD, Continuous Integration and Continuous Delivery?

Test automation, the good the bad and the ugly
It’s time to talk about patterns. Not about best practices, though, I don’t like that term. But there are definitely lessons to be learned from the past on what works and what doesn’t. Think data driven. Think maintainability. Think code review. Think (or rather, forget) code-free test automation. Think reporting. Think some more.

Beyond functional test automation: what else could automation be used for?
Most of what we’ve seen so far covers functional test automation: automated checks that determine whether or not some part of the application under test functions as specified or desired (or both, if you’re lucky). However, there’s a lot more to testing than mere functional checks. Of course there’s performance testing, security testing, usability testing, accessibility testing, all kinds of testing where smart application of tools might help. But there’s more: how about automated parsing of logs generated during an exploratory testing session? Automated test data creation / generation / randomization? Automated report creation? All these are applications of test automation, or better put, automation in testing (thanks, Richard!), and all these are worth learning about.

Note that nowhere in the topics above I am focusing on specific tools. As far as I’m concerned, getting comfortable with one or more tools is one of the very last steps in becoming a good test automation engineer or consultant. I am of the opinion that it’s much more important to answer the ‘why?’ and the ‘what?’ of test automation before focusing on the ‘how?’. Unfortunately, most training offerings I’m seeing focus solely on a specific tool. I myself am quite guilty of doing the same, as I said in the first paragraph of this post.

One thing I’m still struggling with is how to make the attendants do the work. It’s quite easy to present the above subjects as a (series of) lecture(s), but there’s no better way to learn than by doing. Also, I think hosting workshops is much more fun than delivering talks, and there’s no ‘workshop’ without actual ‘work’. But it has to be meaningful, relevant to the subject covered, and if possible, fun..

So, now that I’ve shared my thoughts on what ingredients would make up a decent test automation education, I’d love to hear what you think. What am I missing (I’m pretty sure the list above isn’t complete). Do you think there’s an audience for training as mentioned above? If not, why not? What would you do (or better, what are you doing) differently? This is a topic that’s very dear to me, so I’d love to hear your thoughts on the subject. Your input is, as always, much appreciated.

In the meantime, I’ve started working on a first draft of training sessions and workshops that cover the topics above, and I’m actively looking for opportunities to deliver these, be it at a conference or somewhere in-house. I’ve got a couple of interesting opportunities lined up already, which is why I’m looking forward to 2017 with great anticipation!

On becoming a test automation craftsman

As an automation engineer, it’s hard not to get carried away by the latest tools, frameworks (I don’t like that word, but hey) and other gizmos when you’re dutifully automating away. New tools that promise to make your test automation even snazzier than it already is are popping up on an almost daily basis. But are you really missing out when you’re not including these into your daily work? I’d like to think that more often than not, you’re not. Most true craftsmen have become what they are because they’ve become exceptionally good at doing one or two things, whereas the ‘jack of all trades, master of none’ types are often quickly forgotten. So, if you want to become a test automation craftsman (or craftswoman, of course), how should you go about doing that?

Instead of frantically trying to keep up with all of the tools that are flying around, I would advise you to:

  1. select a couple of them that will likely do the trick in most situations you encounter,
  2. get really good at using them,
  3. and then provide your client or your boss with the best possible solution using this tool set.

Select your ‘go to’ tool set (and do it wisely)
The first step in becoming a craftsman is to choose a set of tools that you’re sure will go a long way in allowing you to provide maximum value to your clients or your employer. This will over time become ‘your’ tool set, and maybe, given you provide excellent work and aren’t afraid of some personal branding, you’ll become known as the go to guy or girl for questions related to a specific tool or tool set. But even if you don’t want to become a source of knowledge for other people (although I have a hard time imagining why you wouldn’t), being exceptionally skilled in one or two things will likely advance your career faster and in better ways than a shotgun approach will ever do.

As an example, I was asked a couple of months ago by Joe Colantonio (the guy behind the excellent TestTalks podcast) to contribute to his Automation Guild online conference initiative. He recognized me as someone that is knowledgeable on a specific tool (in this case it’s REST Assured) and asked me to do a session on just that specific topic. I’d like to believe he invited me because I’ve written quite a few blog posts on the tool on here in the past (it can’t be my good looks..). Had I just written bland introductory posts on a variety of tools instead of focusing on one or two quality tools, I’m not sure this opportunity would have come by.

On a side note, you should really check out Automation Guild. Not because I’m a speaker there, but because I am really enthusiastic about the concept and because I think it’s a great way for you to further hone your craftsmanship from the comfort of your own home. There are so many great crafts(wo)men on the list!

Note that you should be careful when choosing what goes into your test automation tool belt. It does not make sense to pick a tool just because there’s nobody else on the web that’s specialized in it, for example. Usually there’s a good reason for such a lack of online presence: it’s highly likely that there’s not enough market demand for the tool, and/or the tool just isn’t all that useful. Instead, it would make more sense to pick something that’s reasonably established and well supported.

Select the contents of your tool belt wisely!

Learn everything there is to know about ‘your’ tools
Now that you have selected what goes into your personal test automation tool belt, it’s time to learn the heck out of it. To be considered a craftsman, you should try and learn everything there is to know about your tool(s), both the positive and the negative sides. Or at the very least, you should know exactly where to get the information required to do your work in the best possible way. Showcase what you know, be it at your day job, online, or at conferences, to build a following and get your name out there. Connect with fellow craftsmen to exchange knowledge and further hone your skills.

For example, check if there are any classes, workshops or online courses you can take that are related to your tools of choice. They’re often chock full of goodies, examples and exercises that will help you learn even more than you already think you knew. Plus, this too is a great way of meeting like-minded folks and grow your network. You’ll never know how and when you meet your next client, coworker or employer.

Workshop at Testworks Conf

Provide your stakeholders with the best solution using your tools
Finally, after you have selected and sharpened your tools and your skills, it’s time to put them to good work. Even though I’ve mostly focused on tools in this blog post so far, in the end, it’s not about them, it’s about what you do with the contents of your tool belt. The solutions you build using your tools are what will ultimately provide value to your stakeholders. You wouldn’t pay a carpenter if all he did was give you a couple of pieces of wood and a box of nails, right?

Updating your tool belt
Of course, in the case that the contents of your tool belt no longer fit the job you’re asked or choose to do, then you’re free (and even required) to look for additions to the contents of your test tool tool belt. Acting like everything is a nail when all you’ve got is a hammer is not the approach that will lead to success. Instead, in that case, it’s probably time to look for new additions to your tool set, and possibly also a good moment to throw out some of the stuff that is no longer of use to you. But until that time, I’d stick with what you know best and keep focusing on providing as much value as you can using your tools. Be(come) a craftsman.

Monitoring the quality of your test automation code

As part of an Agile software development team, sometimes you have to (or have the opportunity to, depending on your perspective) pick up some tasks that are a little out of your comfort zone, or at least different from your usual tasks. Because I had some spare time in our current sprint, and there was a considerable amount of technical debt with regards to code quality, we as a team decided it was a good idea to put some effort into fixing some of this debt. I’m certainly no developer, but I know a bit about writing code, (in the same way that I’m not a chef, but I know how to prepare dinner), so I took this as an opportunity to get some more experience reading, interpreting and fixing code. In our current project, we use SonarQube as a quality management platform. I never worked with this tool before starting on this project, so I had no prior expectations, but I must say I like it so far.

While diligently working on removing unnecessary pieces of code and rewriting statements, the thing that struck me was that our test automation code, which is pretty closely related to our production code, was not measured against the code quality rule set we use for the production code. In this post, I’d like to discuss whether that’s a good thing or not.

Code quality

Pro argument: test automation is software development
Successful implementation of test automation requires that you treat it similar to any other software development project: it requires planning, a sound design and test automation developers that know what they’re doing. Why shouldn’t taking care of your test automation code in the same vein as your production code extend to measuring code quality? Like any other piece of code that’s developed as part of the software development life cycle, your test automation code should be readable (understandable) and maintainable. And by maintainable I mean that other people should be able to fix and extend things once the test automation ‘genius’ responsible for writing the initial code has moved on, up or out. Again, not much different from production code.

Pro argument: your test automation code safeguards production code
To some extent, depending on the context and the type of project, your test automation code could be considered even more important than your production code. It is responsible for assessing whether the production code lives up to standards and expectations, just like a driving instructor is responsible for delivering people who can drive in a decent manner, obeying the traffic rules and regulations. Wouldn’t it make sense to hold the test automation code against at least the same strictness of standards as you would expect is being done for driving instructors? Although judging from the way some people drive, I wonder what kind of driving instructor allowed them to pass the exam, but that’s a different story…

Con argument: releasing new features has priority (at the moment)
Of course, new features needing to be developed, tested and delivered is not a viable long term argument for sacrificing the code quality of your test automation. However, as we all know, reality and deadlines sometimes force us to be pragmatic, in which case building up some amount of technical debt (of which less-than-desired code quality is a form) is perfectly justifiable. However, always make sure that these improvements do not fall off the radar, for example by implementing automated code quality monitoring with a tool such as the aforementioned SonarQube. When the pressure to deliver lowers (or when test automation code quality issues exceed a certain threshold), the team can get back to improving the existing code.

Con argument: your test automation code is throwaway or end of life code
Another case where monitoring and actively improving the quality of your test automation code can be considered overkill is when you know for sure that your test automation code will be short-lived. This can be the case for proof of concept projects, where functionality has priority over code quality. Another situation where the time invested in code improvements might be better spent elsewhere is when the test automation solution reaches its end of life in the foreseeable future. In all other cases (i.e., you have a solid test automation solution that is expected to stay around for a while), test automation code quality should definitely be on the team radar.

As for my current project? At the moment, we’re still not holding our test automation code against the same standards as our production code. It’s definitely up for discussion, but there’s a lot of technical debt in our production code with regards to code quality, so we’ve decided to tackle that first.

Do you have any experience with measuring test automation code quality yourself? I’d love to hear more insights on this, as I think it’s yet another often overlooked aspect of test automation. Is it useful? Do you have any stories where adequately measuring test automation code quality has been or would have been beneficial to your product or organization? Please share your story in the comments!