What does good test automation training look like?

As I’m moving away from by-the-hour work and more towards a hybrid of consulting, training, writing and speaking, one of the things I’m working on is slowly building up a workshop and training portfolio around topics I think I’m qualified to talk about. I have created a couple of workshops already, but so far, they are centered around a specific tool that I am interested in and enthusiastic about. This, however, has the downside that they’re probably targeted towards a relatively small group of interested people (not in the least because these tools are only available for a specific programming language, i.e., Java).

To extend my options with regards to delivering training and workshops, I am currently looking at developing workshops and training material that contain higher level and more generic material, while still offering practical insights and hands-on exercises. There are a lot of different approaches and possible routes that can be taken to achieve this, especially since there is no specific certification trajectory around test automation (nor do I think there should be, but that’s a wholly different discussion that I’ll probably cover in another blog post in time). So far, I haven’t figured out the ideal contents and delivery format, but ideas have been taking shape in my head recently.

Here are some subjects I think a decent test automation training trajectory should cover:

Test automation 101: the basics
Always a good approach: start with the basics. What is test automation? What is it not (here’s a quote I love from Jim Hazen)? What role does automation play in current development and testing processes and teams? Why is it attracting the interest it does? To what levels and what areas can you apply test automation and what is that test automation pyramid thing you keep hearing about?

Test automation implementation
So, now that you know what test automation (sorta kinda) is, how to apply it to your software development process? How are you going to involve stakeholders? What information or knowledge do you want to derive from test automation? How does it fit into trends such as Agile software development, BDD, Continuous Integration and Continuous Delivery?

Test automation, the good the bad and the ugly
It’s time to talk about patterns. Not about best practices, though, I don’t like that term. But there are definitely lessons to be learned from the past on what works and what doesn’t. Think data driven. Think maintainability. Think code review. Think (or rather, forget) code-free test automation. Think reporting. Think some more.

Beyond functional test automation: what else could automation be used for?
Most of what we’ve seen so far covers functional test automation: automated checks that determine whether or not some part of the application under test functions as specified or desired (or both, if you’re lucky). However, there’s a lot more to testing than mere functional checks. Of course there’s performance testing, security testing, usability testing, accessibility testing, all kinds of testing where smart application of tools might help. But there’s more: how about automated parsing of logs generated during an exploratory testing session? Automated test data creation / generation / randomization? Automated report creation? All these are applications of test automation, or better put, automation in testing (thanks, Richard!), and all these are worth learning about.

Note that nowhere in the topics above I am focusing on specific tools. As far as I’m concerned, getting comfortable with one or more tools is one of the very last steps in becoming a good test automation engineer or consultant. I am of the opinion that it’s much more important to answer the ‘why?’ and the ‘what?’ of test automation before focusing on the ‘how?’. Unfortunately, most training offerings I’m seeing focus solely on a specific tool. I myself am quite guilty of doing the same, as I said in the first paragraph of this post.

One thing I’m still struggling with is how to make the attendants do the work. It’s quite easy to present the above subjects as a (series of) lecture(s), but there’s no better way to learn than by doing. Also, I think hosting workshops is much more fun than delivering talks, and there’s no ‘workshop’ without actual ‘work’. But it has to be meaningful, relevant to the subject covered, and if possible, fun..

So, now that I’ve shared my thoughts on what ingredients would make up a decent test automation education, I’d love to hear what you think. What am I missing (I’m pretty sure the list above isn’t complete). Do you think there’s an audience for training as mentioned above? If not, why not? What would you do (or better, what are you doing) differently? This is a topic that’s very dear to me, so I’d love to hear your thoughts on the subject. Your input is, as always, much appreciated.

In the meantime, I’ve started working on a first draft of training sessions and workshops that cover the topics above, and I’m actively looking for opportunities to deliver these, be it at a conference or somewhere in-house. I’ve got a couple of interesting opportunities lined up already, which is why I’m looking forward to 2017 with great anticipation!

Open sourcing my workshop on WireMock

For those of you that want to jump to the good stuff directly, you can find the workshop slides, exercises and everything related here.

A couple of weeks ago I was given the opportunity to deliver another workshop before TestNet, the Dutch software testing community, as part of their fall conference. Half a year ago, I did a similar workshop for their spring conference. That workshop was on RESTful API testing using REST Assured, which I decided to make open source a little later on. I’ve received some positive feedback on that, so why not do it again?

This time, the conference was centered around test automation. The subject of my workshop this time was closely related to the main theme: stubbing test environment dependencies using WireMock.

The title slide for my workshop

As with most workshops that day, mine was set in a classroom-style space. I had somewhere between 15 and 20 participants, which I think is pretty much the ideal group size for a hands-on workshop that involves writing code. Not so many that I can’t give proper attention to all questions asked, but not a group so small you start to doubt whether the upfront investment has been worth it. As those of you who have prepared and delivered workshops before, you know that preparing them takes a lot of time. Spending all those hours and then only having two people turn up, one of whom is a coworker and the other one seems a little lost, is a bit of a bummer. Fortunately, this wasn’t the case for me, at least not this time..

The levels of experience of the participants (semi-pro tip: know your audience) ranged from people having some prior experience with stub and mock development to people that had never in their life written a single line of code before. An interesting mix, to say the least!

Workshop contents
As said, the main subject of the workshop was WireMock. I started out by telling people a little about the difficulties with regards to keeping test environments up and running and properly configured in these times of parallel development, Continuous Delivery and Testing and distributed applications. I then introduced WireMock, which I’ve done here on this site before as well. Then came four cycles of me presenting a WireMock feature, followed by exercises where the participants could try this feature out for themselves. I chose to highlight the following WireMock features in the workshop:

  • Writing a first, basic stub
  • Request matching options
  • Fault simulation
  • Creating stateful mocks

For each set of exercises, I prepared REST Assured tests that the participants could run to see if their stub implementation was correct. Call it test driven stub development if you like. This approach worked remarkably well, it definitely saved me a lot of time answering questions in the ‘is this correct?’ vein. Test green = stub good, test red = stub needs improvement. It doesn’t get any clearer than that.

Here’s an example of an exercise and the test that determines the correctness of the solution:

public void setupStubExercise101() {

	 * Create a stub that listens at path
	 * /exercise101
	 * and responds to all GET requests with HTTP status code 200

public void testExercise101() {

How it turned out
Fine, I think. Time flew by and I didn’t experience any major faults, missing or incorrect slides or things like that. The experience I’m slowly gathering by doing this more often is starting to pay off, I think. I received almost exclusively positive feedback on the workshop, so I’m a happy guy. Also, everybody seemed to have learned at least something new and enjoyed the process too, no matter whether they had prior stubbing or even programming experience or not, and that has been the most important result of the morning to me. I’m looking forward to the next opportunity for delivering this workshop.

Having said that…
All workshop contents, that’s:

  • The complete set of slides
  • All workshop exercises and their answers
  • Matching REST Assured tests to verify the stubs created

can be found on my GitHub page. As with the previous workshop I’ve published in this manner, feel free to download, adapt, extend and then deliver the workshop to your liking. I look forward to hearing your experiences.

And in case you’re interested in following a WireMock workshop, but do not want to deliver this yourself, don’t hesitate to contact me. I’ll be happy to discuss options. Also, this workshop can easily be combined with a workshop on REST Assured for a full day of API testing and stubbing goodness.

Review: Specification by Example workshop with Gojko Adzic

For a consultant wanting to stay on top of their game (or at the very least hireable) it’s a good idea to take a step back from the daily work of consulting and writing every once in a while and get inspired by a conference or a course or workshop. When the WannaFlex team announced they invited Gojko Adzic to bring his Specification by Example workshop to Amsterdam, it didn’t take me long to decide to sign up. I have come across the approach in previous projects, but found myself lacking experience and background information, and this workshop looked to be an excellent opportunity to get some of both.

What is Specification by Example?
In the booklet we received as a handout, Specification by Example (SbE) is defined as:

A set of process patterns that facilitate change in software products to ensure the right product is delivered efficiently

In my own words, SbE is a set of techniques that help software development teams to reach a shared understanding of the software about to be built, in order to eliminate (or at least greatly reduce) the possibility of mismatched expectations leading to incorrect software or software that will not be accepted and used to satisfaction by the end user.

Organization and setup
The SbE workshop is a two-day workshop where theory and practice around the creation and validation of software specifications take turns. First some theory, then a practical exercise, mostly conducted in small groups of four or five people. This group size is large enough to not come to an agreement on specifications too soon, but small enough to avoid the risk of endless discussions.

Specification by Example workshop announcement

Day one: creating specifications
The first day of the workshop focused mainly on the importance of creating clear specifications and how to get to that point. For example, we were given a set of ‘traditional’ requirements for a piece of software and 20 minutes to select a subset of the requirements and a pen and paper implementation. In other words, we simulated what could be sprint 1 of a development process, based on requirements that turned out to be vague, ambiguous and often plain incorrect. This is a learning technique that I have quickly come to appreciate now that I’ve experienced it first-hand: tell people just enough to get them started, let them make mistakes and discuss these afterwards. This is far more effective than simply showing and telling how something needs to be done!

We were also introduced to the process of how to get to good (or at least significantly better) specifications. In short (for the full story I highly recommend you take the workshop yourself if given the opportunity):

  1. Come up with examples of what the system should do. Any example will do as long as its valid, since every example tells you something about the desired behavior of the system you’re writing specifications for.
  2. Try and detect the pattern that’s behind the examples. What are the input parameters? What is the output? And so on.. This will help you see the bigger picture and get it clear what the system is supposed to do.
  3. Work on the boundary cases. This is where the interesting stuff happens, where questions mostly arise and where defects occur most often. Therefore, it’s a good idea to give them enough attention in the specification process.

Day two: application
Day two started with ways to apply the specifications in practice. Patterns such as Given / When / Then were introduced and the link to test automation was made. As a test automation consultant, this was especially valuable to me. I’ve seen and worked with most of the tools that were discussed (such as Cucumber, SpecFlow and FitNesse), but the link to SbE theory was very useful information. We also spent some time analysing a number of specifications from real-world projects and seeing where they could be improved (if they weren’t beyond saving..). Very useful as well.

Unfortunately, due to other commitments, I couldn’t stay until the end of the workshop so I don’t know exactly what has been discussed in the afternoon of the second day. Still, the first day and a half provided me with lots of useful information and tips on how to apply SbE in practice and how to combine this with test automation in my projects.

Lessons learned and recommendation
Too many to mention here! I’ve learned a lot in this workshop, both with regards to SbE as a technique as well as on how to facilitate a workshop. If you’re faced with SbE in your project or organization, or if you just want to know more about the technique and how it can help you to create better specifications and therefore better software, I highly recommend attending the SbE workshop. And if you like to learn from the best, you simply can’t go wrong with Gojko.

Thanks too to the people at Wannaflex for making this a very smooth and pleasant workshop.