On delivering automation training online

Posted on by

Recently (as in, over the last year and a half or so) I regularly receive questions about providing online training in addition to my in-house, in-person training offerings. Until now, I put those requests on the back burner as I was of the opinion that teaching online (either live or through prerecorded video instructions) would never be a replacement for ‘live’ training.

And then something struck me: why would it have to be an exact replacement? Why not just try it, see how it goes, learn from it and see if it’s a suitable way to conduct training?

So, when I got in touch with a test consultancy firm in the UK that was looking for training for their employees, I decided to give it a try. After some discussion, we agreed that I would deliver the first day of training in house (meaning: in Manchester), while the following modules would be delivered online, saving me a couple of trips back and forth and cutting down on overhead costs for airfare and hotels. And so it was done.

Note: I am aware that having met the students in person before delivering training online to them is a big plus. However, I believe that the lessons and the pros and cons I talk about in this blog post equally apply when you’ve never met the students in real life just as well.

So, what did I learn in the process? Let’s see.

Preparation
I could write a whole separate article about how to properly prepare for a technical training course or workshop. In fact, I’ll be doing just that in the near future, in an article that will be published on another platform.

I won’t go into too many details here, but by far the most important thing to do when you’re about to conduct training online is to make sure that the participants are ready from the start. My preferred way of doing this is by sending detailed preparation instructions (a step-by-step guide, screenshots and all) to them at least a whole week in advance, so the participants have some time to set up their device. Additionally, I make myself available for questions and troubleshooting in case something goes wrong.

I was afraid to do this in the beginning, fearing I’d be overwhelmed with questions, but it turns out that’s not the case. For all the workshops I’ve given in the last few years, I’ve only had a couple (as in: three or four) people asking for help. That doesn’t mean that everybody else is ready to go when the workshop or training starts, but that’s a whole different kettle of fish…

The reason this is extra important when delivering training online is that you cannot just walk over to the participants and look over their shoulder to see what’s going wrong. You can do screen sharing, of course, but that’s not as efficient as taking over the controls for a bit.

So, long story short, overdo it on the preparation instructions. Be very clear in them and make sure they’re unambiguous. Have them tested by somebody else if you’re not sure everything’s clear (heck, do this even if you ARE sure).

Organization
With regards to how the training days should be organized, here are some key lessons I’ve learned from the two days of training I’ve hosted so far:

  • Group size: Where I can take around 12 people for a class that involves programming when they’re in the same room, I am glad I had only 4 participants for my online training. I think I can handle up to 6 people, but no more. Keeping track of how everybody is doing takes more effort when you see them through a webcam only, and there will probably be more questions (also because participants can’t help eachother out), so it’s only fair to limit the number of attendees to make sure everybody gets the attention and the answers they deserve.
  • Type of course: Live online training works well for hands-on automation training, but probably much less so (for obvious reasons) for training courses that involve a lot of group work, discussions and presentations. I wouldn’t even know how to facilitate that online…
  • Location and connection: Make sure the participants (and you yourself as well) are in a room with good lighting and that their webcam is on, because reading facial expressions will tell you a lot about their level of engagement. Also make sure they’re in a location with a good Internet connection. Videoconferencing takes bandwidth, yet you want both video and audio to be of the highest possible quality to make sure the participants can hear and see you well.

Engagement
The hardest part about delivering training online is keeping your audience engaged. Taking training is hard enough on your energy levels when you’re in the same room as the trainer, looking at a webcam and listening to somebody who’s potentially very far away is orders of magnitude harder. Here are some tips that might help you (they worked for me!):

  • Ask the participants how they’re doing often, to the point of being annoying. Don’t lose them, don’t give them a chance to start drifting off. Make sure they are awake and engaged. In the pre-course instructions, point out that they should be well rested, and that taking a training course online is even more demanding than ‘live’ training, for both parties.
  • Consider shortening the training days (for example, teach for 6 hours instead of 8 for a day of training). Chances are high that they won’t take in anything in those last two hours anyway, simply because their energy levels are too low. Additionally, take breaks often. Even just a five minute break for a leg stretch or a bathroom visit can help keep energy levels up. I took breaks every hour, which definitely wasn’t overdoing it.
  • Involve them. Instead of just broadcasting information all day, ask them lots of questions. When you’re doing programming exercises, ask them to share their screen and talk the rest of the audience through their solution and thinking process. Again, this helps keep them engaged. Don’t let them fall asleep!

Pros and cons
As I said earlier, online training isn’t a replacement for in-person training, at least not on a 1-on-1 basis. It’s a whole different ball game. Both have their pros and cons. Some of the benefits of delivering training online for me are:

  • It allows me to work from home. Big plus. I like driving my car, but I hate wasting time commuting. With online training, I can teach from the comfort of my own home.
  • My potential client base is many times larger. I am quite limited in the amount of travel I can do in a year, and the Netherlands is a small country, which means my client base isn’t all that large. With the possibilities of online training, though, I can deliver my courses to the entire world, potentially. Added bonus: meeting and talking to people from other countries and cultures, plus it does wonders for my English.

Sure, there are some downsides as well:

  • Not being able to walk up to people and see how they’re doing. I do this a lot when teaching in-person, but that’s not an option when online. Even with a webcam, people can hide behind their screen easier and pretend all is well. Their loss, of course, but I take pride in keeping everybody engaged.
  • It isn’t suited for every course I offer. I do more and more courses where people work in groups and have discussions, and as I said earlier, that’s not really an option when teaching online.

Having said all of this, I will definitely start offering live online training more often in the future, probably starting after the summer holidays. It’s definitely a valuable addition to the services I can offer. If you’re interested in taking one of my online courses, keep an eye out on this site for future announcements.

Oh, and in case you were wondering: so far I didn’t need any dedicated virtual classroom software to conduct the training. I used the pro version of appear.in, which requires no software to be installed at all on the client side, is a breeze to work with, allows everybody to share their screen effortlessly, has a chat to share links and stuff, basically everything I need.

On ending the regression automation fixation

Posted on by

Note: in my observation, scripted test execution and the type of regression test scripts I’m referring to are slowly going away, but a lot of organizations I work with still use them. Not every organization is full of testers working in a context-driven and exploratory way while applying CI/CD and releasing multiple times per day. If you’re working in one, that’s fine. This blog post probably is not for you. But please keep in mind that there are still many organizations that apply a more traditional, script-based approach to testing.

In the last couple of months, I’ve been talking regularly about some of the failures I’ve made (repeatedly!) during my career so far. My talk at the Romanian Testing Conference, for example, kicked off with me confessing that in retrospect, a lot of the work I’ve done until all too recently has been, well, inefficient at best, and plain worthless in other cases. Only slowly am I now learning what automation really is about, and how to apply it in a more useful and effective manner than the ‘just throw more tools at it’ approach I’ve been supporting for too long.

Today, I’d like to show you another example of things that, in hindsight, I should have been doing better for longer.

One of my stock answers to the question ‘Where should we start when we’re starting with automation?’ would be to ‘automate your existing regression tests first’. This makes sense, right? Regression tests are often performed at the end of a delivery cycle to check whether existing functionality aspects have not been impacted negatively as a result of new features that were added to the product. These tests are often tedious – new stuff is exciting to test, while existing features are so last Tuesday – and often take a long time to perform, and one thing there often isn’t left is time at the end of a delivery cycle. So, automating away those regression tests is a good thing. Right?

Well, maybe. But maybe not so much.

To be honest, I don’t think ‘start with automating your regression tests’ isn’t a very good answer anymore, if it has ever been (again, hindsight is 20/20…). It can be a decent answer in some situations, but I can think of a lot of situations where it might not be. Why not? Well, for two reasons.

Regression scripts are too long
The typical regression test scripts I’ve seen are looong. As in, dozens of steps with various checkpoints along the way. That’s all well and good if a human is performing them, but when they are turned into an automated script verbatim, things tend to fall apart easily.

For example, humans are very good at finding a workaround if the application under test behaves slightly differently than is described in the script. So, say you have a 50-step regression script (which is not uncommon), and at step 10 the application does something similar to what is expected, but not precisely the same. In this case, a tester can easily make a note, find a possible way around and move on to collect information regarding the remaining steps.

Automation, on the other hand, simply says ‘f*ck you’ and exits with a failure or exception, leaving you with no feedback at all about the behaviour to be verified in steps 11 through 50.

So, to make automation more efficient by reducing the risk of early failure, the regression scripts need to be rewritten and shortened, most of the times by breaking them up in smaller, independently executed sections. This takes time and eats away the intended increase in speed expected from the introduction of automation. And on top of that, it may also frustrate people unfamiliar to testing and automation, because instead of 100 scripts, you now have to automate 300. Or 400. And that sounds like more work!

Regression scripts are written from an end user perspective
The other problem with translating regression scripts verbatim is that these scripts are often written from an end user perspective, operating on the user interface of the application under test. Again, that’s all well and fine when you’re a human, but for automation it might not be the most effective way to gain information about the quality of your application under test. User interface-driven automation is notoriously hard to write and maintain, hard to stabilize, slow to execute and relatively highly prone to false positives.

Here too, in order to translate your existing regression scripts into effective and efficient automation, you’ll need to take a thorough look at what exactly is verified through those scripts, find out where the associated behaviour or logic is implemented, find or develop a way to communicate with your application under test on that layer (possibly the user interface, more likely an API, a single class or method or maybe even a database table or two) and take it from there.

Sure, this is a valuable exercise that will likely result in more efficient and stable automation, but it’s a step that’s easy to overlook when you’re given a batch of regression scripts with the sole requirement to ‘automate them all’. And, again, it sounds like more work, which not everybody may like to hear.

So, what to do instead?

My advice: forget about automating your regression tests.

There. I’ve said it.

Instead, ask yourself the following three questions with regards to your testing efforts:

  1. What’s consuming my testing time?
  2. What part of my testing efforts are repetitive?
  3. What part of my testing efforts can be repeated or enhanced by a script?

The answer(s) to these questions may (vaguely) resemble that what you do during your regression testing, but it might also uncover other, much more valuable ways to apply automation to your testing. If so, would it still make sense to aim for ‘automating the regression testing’? I think not.

So, start writing your automation with the above questions in mind, and keep repeating to yourself and those around you that automation is there to make your and their life easier, to enable you and them to do your work more effectively. It’s not just there to be applied everywhere, and definitely not to blindly automate an existing regression test suite.

On choosing both/and, not either/or

Posted on by

Choices. We all make them tens of times each day. Peanut butter or cheese (cheese for me, most of the time). Jeans or slacks (jeans, definitely). Coffee or tea (decent coffee with a glass of water on the side please). And when you’re working on or learning about automation, there’s a multitude of choices you also can (and sometimes have to) make. A lot of these choices, as I see people discussing and making them, are flawed in my opinion, though. Some of them are even false dichotomies. Let’s take a look at the choices people think they need to make, and how there are other options available. Options that might lead to better results, and to being better at your job.

Do I need to learn Java or .NET? Selenium or UFT?
Creating automation often involves writing code. So, the ability to write code is definitely a valuable one. However, getting hung up on a specific programming language might limit your options as you’re trying to get ahead.

I still see many people asking what programming language they need to learn when they’re starting out or advancing in their career. If you’d ask me, the answer is ‘it doesn’t really matter’. With the abundance in tools, languages, libraries and frameworks that are available to software development teams nowadays, chances are high that your next gig will require using a different language than your current one.

As an example, I recently started a new project. So far, in most of my projects I’ve written automation in either Java or .NET. Not in this one, though. In the couple of weeks I’ve been here, I’ve created automation using PHP, Go and JavaScript. And you know what? It wasn’t that hard. Why? Because I’ve made a habit of learning how to program and of studying principles of object oriented programming instead of learning the ins and outs of a specific programming language. Those specifics can be found everywhere on Google and StackOverflow.

The same goes for automation tools. I started writing UI-level automation using TestPartner. Then QuickTest Pro (now UFT). I’ve used Selenium in a few projects. I’ve dabbled with Cypress. Now, I’m using Codecept. It doesn’t matter. The principles behind these tools are much the same: you identify objects on a screen, then you interact with them. You need to take care of waiting strategies. If you become proficient in these strategies, which tool you’re using doesn’t matter that much anymore. I’ve stopped chasing the ‘tool du jour’, because there will always be a new one to learn. The principles have been the same for decades, though. What do you think would be a better strategy to improve yourself?

Identify and learn to apply common principles and patterns, don’t get hung up on a single tool or language. Choose both/and, not either/or.

Do I stay a manual tester or become an automation engineer?
Another one of the choices I see people struggling with often is the one between staying a ‘manual tester’ (a term that I prefer not to use for all the reasons Michael Bolton gives in this blog post of his and becoming an automation engineer. If you’d ask me, this is a perfect example of a flawed choice in the testing field. It’s not a matter of either/or. It’s a matter of both/and.

Automation supports software testing, it does not replace it. If you want to become more proficient in automation, you need to become more proficient in testing, too. I’ve only fairly recently realized this myself, by the way. For years, all I did was automation, automation, automation, without thinking whether my efforts actually supported the testing that was being done. I’ve learned since that if you don’t know what testing looks like (hint: it’s much more than clicking buttons and following scripts), then you’ll have a pretty hard time effectively supporting those activities with automation.

Don’t abandon one type of role for the other one, especially when there’s so much overlap between them. Choose both/and, not either/or.

Do I learn to write tests against the user interface, or can I better focus on APIs?
So, I’ve been writing a lot about the benefits of writing tests at the API level, not only on this blog, but also in numerous talks and training courses. When I do so, I am often quite critical about the way too many people apply user interface-driven automation. And there IS a lot of room for improvement there, definitely. That does not mean that I’m saying you should abandon this type of automation at all, just that you should be very careful when deciding where to apply it.

Like in the previous examples, it is not a matter of either/or. For example, consider something as simple and ubiquitous as a login screen (or any other type of form in an application). When deciding on the approach for writing tests for it, it’s not a simple choice between tests at the UI level or tests at the API level; rather it depends on what you’re testing. writing a test that checks whether an end user sees the login form and all associated in their browser? Whether the user can interact with the form? Whether the data entered by the user is sent to the associated API correctly? Or whether the form looks like it’s supposed to? Those are tests that should be carried out at the UI level. Checking whether the data provided by the user is processed correctly? Whether incorrectly formatted data is handled in the appropriate manner? Whether the right level of access grants is given to the user upon enter a specific combination of username and password? Those tests might target a level below the UI. Many thanks, by the way, to Richard Bradshaw for mentioning this example somewhere on Slack. I owe you one more beer.

Being able to make the right decision on the level and scope to write the test on required knowing what the benefits and drawbacks and the possibilities of the alternatives are. It also requires the ability to recognize and apply principles and patterns to make the best possible decision.

Again, identify and learn to apply common principles and patterns, don’t get hung up on a single tool or language. Choose both/and, not either/or.

The point I’ve been trying to make with the examples above is that, like with so many things in life, being the best possible automation engineer isn’t a matter of choosing A over B. Of being able to do X or Y. What, in my opinion, will make you much better in your role is being able to do, or at least understand, A and B, X and Y. Then, extract their commonalities (these will often take the form of the previously mentioned principles and practices) and learn how to apply them. Study them. Learn more about them. Fail at applying them, and then learn from that.

I’m convinced that this is a much better approach to sustainable career development than running after the latest tool or hype and becoming a self-proclaimed expert at it, only to have to make a radical shift every couple of years (or even months, sometimes).

Don’t become a one trick pony. Choose both/and, not either/or.