On creating reasonable expectations in test automation – A TestBash Netherlands workshop

Last week, I had the incredible pleasure of co-facilitating (with Ard Kramer) a workshop on creating reasonable expectations in test automation at the second edition of TestBash Netherlands. In this post, I’d like to tell you a little more about how we got there, what the workshop was all about and, of course, how it all went down on the big day.

The build up
The adventure started when I was contacted by Huib Schoots, who has been responsible for organizing the conference, with the question if I’d be interested to host a pre-conference workshop. We discussed several subjects back and forth and, in the end, decided upon a workshop that would help people create better test automation strategies. Since I do have a lot of experience with hands-on test automation workshops where people work on exercises on their machines, but much less so with coming up with and organizing workshops where facilitating group discussions plays a big role, we thought it was a good idea to bring someone on board that has much more experience with this type of sessions. Ard was very high on both of our lists, and luckily, he was up for it as well.

Our combined experience in test automation, testing and facilitating workshops turned out to be a great match. During a number of preparation sessions (for those of you thinking of hosting similar workshops: it sure takes a lot of time to prepare!) we came up with a series of exercises, carried out either individually or in smaller groups, that ultimately would result in people coming up with three or four actionable items that they could take back to their jobs on Monday, answering tangible problems and addressing real issues that they faced in their test automation efforts.

The workshop
As said, the workshop consisted of a number of exercises that would help people identify and address gaps and opportunities in their test automation strategy. It would take way too long to describe the entire workshop, but here’s the gist of it..

We started out by having the attendees come up -individually- with strengths, weaknesses, opportunities and threats (indeed, a SWOT analysis) in their current test automation efforts. To help them on their way, we presented them with six aspects of a test automation strategy, with some example questions that you could ask to help identify strengths or pain points. These six categories (or aspects that we think make up a solid test automation strategy) are:

  • Technical
  • Knowledge and experience
  • Means and resources
  • Process and methodology
  • Organization
  • Business value

As you can see, in our opinion there’s a lot more to creating and implementing a successful test automation strategy than throwing tools at problems!

We then had the attendees discuss the challenges that resulted from the SWOT analysis in five different rounds, organized in groups based on the aforementioned categories. Each participant had the opportunity to address a subject in four different categories. For the fifth round, they had to play the role of facilitator on a subject they felt knowledgeable or comfortable about. These discussion rounds made up the larger part of the day.

Cards and forms for the discussion rounds

Finally, we had the attendees pick and pitch their most interesting improvement point and create a 99 second pitch for it, which they presented to their group of 5-6 people. Each group would then pick the best or otherwise most interesting pitch, which would be presented to the entire group, on stage. The intention behind this (and basically, behind the setup of the entire workshop) was to have people discuss with and learn as much from as many other attendees as possible.

The big day
For the day, we had a total of 27(ish, I’m not 100% sure tbh) attendees, which made for perfect group sizes. It’s always exciting to see how a new workshop or training course turns out in practice, you can think of a lot of things that might happen – and we sure did think of a lot of scenarios – but in the end, you never know what’s going to happen on game day!

As the day unfolded, Ard and I were very happy to see that the group went about our exercises with enthusiasm. Of course, there are always things that could have gone better, but all in all, discussions were going strong throughout the day and we didn’t have to correct course much.

Participants hard at work during the workshop

It does help that the general audience of TestBash conferences is made up of people that are willing to open up to, discuss with and learn from their peers. In this, these conferences are of a truly high quality, and we as facilitators learned just as much as the participants.

The part of the day I am probably most proud of is that at the end, we had some great 99 second pitches presented on stage, and at least two of the people presenting their pitches to the workshop participants repeated their talk on conference day, in front of an audience of 150-200. We sort of hoped that this would happen, but you never know how it turns out. It was truly rewarding to see this unfold in the way we intended it to do. The only downside is that I wasn’t there in person as I wasn’t able to make it to the conference day, but I can assure I lived it vicariously through my Twitter feed!

The aftermath
The workshop day flew by, and at the end, we asked the people to do a ‘dot vote’ and give us some honest feedback on what they liked, what they were indifferent about and what we could have done better. As you can see in the picture below, I think we did a decent job overall..

Feedback on our workshop

For me personally, preparing and delivering this workshop has been a great learning experience as well. As I said, the exercises in my training courses are mostly completed individually, on laptops. I’ve learned a lot about this type of workshop from Ard in the process, something that I’m sure will be of great value to me in the future (thanks again, Ard!).

I’m already looking forward to facilitating this workshop many more times in the future!

What does good test automation training look like?

As I’m moving away from by-the-hour work and more towards a hybrid of consulting, training, writing and speaking, one of the things I’m working on is slowly building up a workshop and training portfolio around topics I think I’m qualified to talk about. I have created a couple of workshops already, but so far, they are centered around a specific tool that I am interested in and enthusiastic about. This, however, has the downside that they’re probably targeted towards a relatively small group of interested people (not in the least because these tools are only available for a specific programming language, i.e., Java).

To extend my options with regards to delivering training and workshops, I am currently looking at developing workshops and training material that contain higher level and more generic material, while still offering practical insights and hands-on exercises. There are a lot of different approaches and possible routes that can be taken to achieve this, especially since there is no specific certification trajectory around test automation (nor do I think there should be, but that’s a wholly different discussion that I’ll probably cover in another blog post in time). So far, I haven’t figured out the ideal contents and delivery format, but ideas have been taking shape in my head recently.

Here are some subjects I think a decent test automation training trajectory should cover:

Test automation 101: the basics
Always a good approach: start with the basics. What is test automation? What is it not (here’s a quote I love from Jim Hazen)? What role does automation play in current development and testing processes and teams? Why is it attracting the interest it does? To what levels and what areas can you apply test automation and what is that test automation pyramid thing you keep hearing about?

Test automation implementation
So, now that you know what test automation (sorta kinda) is, how to apply it to your software development process? How are you going to involve stakeholders? What information or knowledge do you want to derive from test automation? How does it fit into trends such as Agile software development, BDD, Continuous Integration and Continuous Delivery?

Test automation, the good the bad and the ugly
It’s time to talk about patterns. Not about best practices, though, I don’t like that term. But there are definitely lessons to be learned from the past on what works and what doesn’t. Think data driven. Think maintainability. Think code review. Think (or rather, forget) code-free test automation. Think reporting. Think some more.

Beyond functional test automation: what else could automation be used for?
Most of what we’ve seen so far covers functional test automation: automated checks that determine whether or not some part of the application under test functions as specified or desired (or both, if you’re lucky). However, there’s a lot more to testing than mere functional checks. Of course there’s performance testing, security testing, usability testing, accessibility testing, all kinds of testing where smart application of tools might help. But there’s more: how about automated parsing of logs generated during an exploratory testing session? Automated test data creation / generation / randomization? Automated report creation? All these are applications of test automation, or better put, automation in testing (thanks, Richard!), and all these are worth learning about.

Note that nowhere in the topics above I am focusing on specific tools. As far as I’m concerned, getting comfortable with one or more tools is one of the very last steps in becoming a good test automation engineer or consultant. I am of the opinion that it’s much more important to answer the ‘why?’ and the ‘what?’ of test automation before focusing on the ‘how?’. Unfortunately, most training offerings I’m seeing focus solely on a specific tool. I myself am quite guilty of doing the same, as I said in the first paragraph of this post.

One thing I’m still struggling with is how to make the attendants do the work. It’s quite easy to present the above subjects as a (series of) lecture(s), but there’s no better way to learn than by doing. Also, I think hosting workshops is much more fun than delivering talks, and there’s no ‘workshop’ without actual ‘work’. But it has to be meaningful, relevant to the subject covered, and if possible, fun..

So, now that I’ve shared my thoughts on what ingredients would make up a decent test automation education, I’d love to hear what you think. What am I missing (I’m pretty sure the list above isn’t complete). Do you think there’s an audience for training as mentioned above? If not, why not? What would you do (or better, what are you doing) differently? This is a topic that’s very dear to me, so I’d love to hear your thoughts on the subject. Your input is, as always, much appreciated.

In the meantime, I’ve started working on a first draft of training sessions and workshops that cover the topics above, and I’m actively looking for opportunities to deliver these, be it at a conference or somewhere in-house. I’ve got a couple of interesting opportunities lined up already, which is why I’m looking forward to 2017 with great anticipation!

Looking forward to the 2017 conference season

Only a relatively short blog post this week, but it’s something that’s kept me busy for a while..

As we’re slowly moving towards 2017, and as I’m starting to reflect on the shift I’ve been making in how I fill my days and make a living for the last year, I think this is a good time to take a look at the conferences I would like to attend and contribute to in 2017. Delivering talks, or even better, workshops (I’ll get to that at the end of this post), has earned its place as one of the activities that nowadays are part of my work(day/week) on a fairly regular basis. So, what conferences to attend next year?

The ‘definitely’ category
There are a couple of conferences I’ll definitely attend or contribute to, either because they’ve proven their value or simply because I’m already listed to deliver a talk or workshop there:

  • AutomationGuild (January) – I’ll deliver a talk on testing RESTful APIs using REST Assured at the first edition of this online conference completely dedicated to test automation.
  • Romania Testing Conference (May) – I was supposed to deliver a full-day workshop on REST Assured at the first edition of a spin-off conference of RTC, but unfortunately the organization had to postpone the event. They were extremely kind, however, to extend their invitation to the 2017 edition of the original event, which is held in Cluj in May, so I’m Romania bound by then! Very much looking forward to that, as you can imagine, especially after the wonderful experience of my first talk abroad.
  • TestWorksConf (October) – The first two editions have been awesome and I’m already eager to see what the Xebia guys can come up with for the third edition. Maybe I’ll be a contributor again, but if not, I’ll definitely be there as a delegate.
  • Test Automation Day (June) – The other big conference in the Netherlands dedicated to test automation, this has been a ‘must’ for me for the last four or five years, and I’ll highly likely be there again next year. Hopefully as a speaker this time, something that’s still on my to-do list.

The ‘hopefully’ category
There is one European conference outside of The Netherlands that I’d like to attend or contribute to one day. And why shouldn’t that day be in 2017?

  • Software Testing Forum (June) – I must admit that the location of this conference is what first put it on my radar. I love Italy, its culture, its language and not to forget its food and wine, but the event itself has started to get my attention as well over the last couple of years. I’ve recently been in touch with the organizing committee to see if I can apply to deliver a workshop there, so here’s to hoping that turns out positively!

The ‘I wish’ category
Some day, some day, I’d like to attend (or even better, speak at) one of these conferences:

  • StarCanada (October) – I love Canada, plus I’ve always wanted to attend one of the big Star* conferences, even if only once. Travel and lodging aren’t exactly cheap, though, plus it would cost me at least a whole week of not being home, which isn’t exactly ideal with small children, so this will probably not happen for the next couple of years.
  • WeTest (twice a year) – Pretty much the same reason as the previous one: I love New Zealand. Plus from what I’ve read on the web and social media, the organization and quality of this conference is of a pretty high level. Unfortunately, so is the amount of hours I’d have to spend on the plane, so this will probably not happen in 2017 either..

What else?
So, my dear readers, is there a conference I might have forgotten? Some hidden gem you definitely think I should attend or even apply to contribute to? I’d love to hear your thoughts!

Oh, and coming back to my preference on hosting workshops over delivering talks: I love to give talks on topics I care about, but from my experience, I love hosting workshops even more. The smaller crowd and longer time frame allows me to make better contact with and pay more attention to the attendees, and to get some interesting discussions going. This to me provides more value than the larger (yet somewhat more fleeting) exposure I get when delivering a talk.. Plus I think participants get more out of a workshop too because of the hands-on learning experience, as opposed to attending a talk and picking up a couple of interesting and thought-provoking nuggets of wisdom. So if you know of an interesting conference that does workshops too, please let me know!