Improving your craftsmanship through conferences

In an upcoming TechBeacon article I recently wrapped up, I’m talking about how to create a team of test automation crafts(wo-)men. One of the tips I give is, as a manager looking to build such a team, to let your craftsmen attend conferences regularly. For me, attending conferences is one of the most important and influential ways to extend my craftsmanship.

As a delegate
As a conference delegate (visitor), there are several ways to benefit from the experience:

  • Get inspired by the talks and workshops. A good conference provides a mix of keynote talks from established craftsmen, as well as talks and experience reports from less experienced speakers. These are a good way to get some fresh views on your field of work or, in some cases, on life in general. What I also like in a conference is the offering of hands-on workshops. These are a good way of getting to know or of diving deeper into a tool that might just make your life a lot easier.
  • Interact with fellow craftsmen. Conferences are an excellent opportunity to get to know people in similar roles from other organizations, or even from another country. As with life in general: you never know who you’re going to meet, or what comes out of a seemingly random encounter at a conference. I’ve met people at conferences years ago that I’m still in touch with today. And since the conference attendee list often includes representatives from other organizations, you might even land your next job after an informal first encounter at a conference!
  • See what is available on the tools market. Larger conferences often include a sponsor exhibit, where tool vendors show the latest versions of their offerings. If you’re looking for a solution for a problem you have, most of these vendors are happy to talk to you and give you a demo of what they can do for you.

As a speaker
One step up from being a conference attendee is to start presenting at a conference (or two, or ten) yourself. Even if it might be a bit daunting at first, there’s much to gain from even a single public speaking performance.

  • Building your personal brand. Everybody has a personal brand. I didn’t realize this until fairly recently, but it is a fact. Delivering talks is a great way to show people what you know, what you stand for and what your ideas on your craft are, and in that way build your brand. And when people are looking for someone to work with or for them, a well-crafted personal branding will get you to the top of their wish list.
  • Make sure you understand what you’re doing. An often underrated aspect of presenting is that you have to make sure that you know what you’re talking about. As wise men have said before ‘you don’t truly understand a subject until you’re able to explain it to your mother’ (or something to that extent). Being able to give a clear, comprehensive and nicely flowing talk on a subject is probably the best proof that you truly know what it is you’re doing.

What I’ve been up to recently
After a fairly slow winter (at least in terms of conferences and presentations), the pace is slowly starting to pick up again. Last week, I delivered my new talk on trust in test automation for the first time, to a crowd of just over a hundred people at TestNet, the Dutch organization for professional testers. For a first time, I think it went pretty well, and I’m looking forward to delivering this talk more often in the months to come. I’ve submitted the same talk to a number of other conferences, and I’m very much looking forward to the response from the respective organizing committees.

It’s also less than two months until my workshop on REST Assured and WireMock at the Romanian Testing Conference. Another event that I’m very much looking forward to! It’ll be my second time speaking abroad (and the first time hosting a workshop abroad), and I’m sure it’ll be a fantastic experience after all the good things I heard from last year’s event. I was also pleasantly surprised to hear that the workshop is already sold out, so it’ll be a full house for me.

Finally, next to my blogging efforts on this site, I’ve been steadily publishing articles for TechBeacon (see my contributor profile here) and I’ve also recently published my second article on StickyMinds (see my user profile here). If you happen to have a few spare minutes and feel like reading my articles, I’d love to hear what you think of them!

More troubles with test data

If managing test data in complex end-to-end automated test scenarios is an art, I’m certainly not an artist (yet). If it is a science, I’m still looking for the solution to the problem. At this moment, I’m not even sure which of the two it is, really..

The project
Some time ago, I wrote a blog post on different strategies to manage test data in end-to-end test automation. A couple of months down the road, and we’re still struggling. We are faced with the task of writing automated user interface-driven tests for a complex application. The user interface in itself isn’t that complex, and our tool of choice handles it decently. So far, so good.

As with all test automation projects I work on, I’d like to keep the end goal in mind. For now, running the automated end-to-end tests once every fortnight (at the end of a sprint) is good enough. I know, don’t ask, but the client is satisfied with that at the moment. Still, I’d like to create a test automation solution that can be run on demand. If that’s once every two weeks, all right. It should also be possible to run the test suite ten times per day, though. Shorten the feedback loops and all that.

The test data challenge
The real challenge with this project, as with a number of other projects I’ve worked on in the past, is in ensuring that the test data required to successfully run the tests is present and in the right state at all times. There are a number of complicating factors that we need to deal (or live) with:

  • The data model is fairly complex, with a large number of data entities and relations between them. What makes it really tough is that there is nobody available that completely understands it. I don’t want to mess around assuming that the data model looks a certain way.
  • As of now, there is no on demand back-up restoration procedure. Database back-ups are made daily in the test environment, but restoring them is a manual task at the moment, blocking us from recreating a known test data state whenever we want to.
  • There is no API that makes it easy for us to inject and remove specific data entities. All we have is the user interface, which results in long setup times during test execution, and direct database access, which isn’t of real use since we don’t know the data model details.

Our current solution
Since we haven’t figured out a proper way to manage test data for this project yet, we’re dealing with it the easiest way available: by simply creating the test data we need for a given test at the start of that test. I’ve mentioned the downsides of this approach in my previous post on managing test data (again, here it is), but it’s all we can do for now. We’re still in the early stages of automation, so it’s not something that’s holding us back to much, but all parties involved realize that this is not a sustainable solution for the longer term.

The way forward
What we’re looking at now is an approach that looks roughly like this:

  1. A database backup that contains all test data required is created with every new release.
  2. We are given permission to restore that database backup on demand, a process that takes a couple of minutes and currently is not yet automated.
  3. We are given access to a job that installs the latest data model configuration (this changes often, sometimes multiple times per day) to ensure that everything is up to date.
  4. We recreate the test data database manually before each regression test run.

This looks like the best possible solution at the moment, given the available knowledge and resources. There are still some things I’d like to improve in the long run, though:

  • I’d like database recreation and configuration to be a fully automated process, so it can more easily be integrated into the testing and deployment process.
  • There’s still the part where we need to make sure that the test data set is up to date. As the application evolves, so do our test cases, and somebody needs to make sure that the backup we use for testing contains all the required test data.

As you can see, we’re making progress, but it is slow. It makes me realize that managing test data for these complex automation projects is possibly the hardest problem I’ve encountered so far in my career. There’s no one-stop solution for it, either. So much depends on the availability of technical hooks, domain knowledge and resources at the client side.

On the up side, last week I met with a couple of fellow engineers from a testing services and solutions provider, just to pick their brain on this test data issue. They said they have encountered the same problem with their clients as well, and were working on what could be a solution to this problem. They too realize that it’ll never be a 100% solution to all test data issues for all organizations, but they’re confident that they can provide them (and consultants like myself) with a big step forwards. I haven’t heard too many details, but I know they know what they’re talking about, so there might be some light at the end of the tunnel! We’re going to look into a way to collaborate on this solution, which I am pretty excited about, since I’d love to have something in my tool belt that helps my clients tackle their test data issues. To be continued!

On false negatives and false positives

In a recent post, I wrote about how trust in your test automation is needed to create confidence in your system under test. In this follow up post (of sorts), I’d like to take a closer look at two phenomena that can seriously undermine this trust: false negatives and false positives.

False negatives
Simply put, false negatives are test instances that fail without there being a defect in the application under test, i.e., the test itself is the reason for the failure. False negatives can occur for a multitude of reasons, including:

  • No appropriate waiting is implemented for an object before your test (for example written using Selenium WebDriver) is interacting with it.
  • You specified incorrect test data, for example a customer or an account number that is (with reason) not present in the application under test.

False negatives can be really annoying. It takes time to analyze their root cause, which wouldn’t be so bad if the root cause was in the application under test, but that would be an actual defect, not a false negative. The minutes (hours) spent getting to the root cause of tests that fail because they’ve been poorly written would almost always have been better spent otherwise, on writing stable and better performing tests in the first place, for example.

If they’re part of an automated build and deployment process, you can find yourself in even bigger trouble with false negatives. They stall your build process unnecessarily, thereby delaying deployments that your customers or other teams are waiting for.

There’s also another risk associated with false negatives: when they’re not taken care of as soon as possible, people will start taking them for granted. Tests that regularly or consistently cause false negatives will be disregarded and, ultimately, left alone to die. This is a real waste of the time it took to create that test in the first place, I’d say. I’m all for removing tests from your test base if they no longer serve a purpose, but removing them just because they fail either intermittently or every time is NOT a good reason to discard them.

And talking about tests failing intermittently, those (also known as flaky tests) are the worst, simply because the root cause for their failure often cannot be easily determined, which in turn makes it hard to fix them. As an example: I’ve seen tests that ran overnight and failed sometimes and passed on other occasions. It took weeks before I found out what caused them to fail: on some test runs this particular test (we’re talking about an end to end test that took a couple of minutes to run here) was started just before midnight, causing funny behavior in subsequent steps that were completed after midnight (when a new day started). On other test runs, either the test started after midnight or was completed before midnight, resulting in a ‘pass’. Good luck debugging that during office hours!

False positives
While false negatives can be really annoying, the true risk with regards to trust in test automation is at the other end of the unreliability spectrum: enter false positives. These are tests that pass but shouldn’t, because there IS an actual defect in the application under test, it’s just not picked up by the test(s) responsible for covering the area of the application where the defect occurs.

False positives are far more dangerous than false negatives, since they instill a false sense of confidence in the quality of your application. You think you’ve got everything covered with your test set, and all lights are green, but there’s still a defect (or two, or ten, or …) that goes unnoticed. And guess who’s going to find them? Your users. Which is exactly what you thought you were preventing by writing your automated regression tests.

Detecting false positives is hard, too, since they don’t let you know that they’re there. They simply take up their space in your test set, running without trouble, never actually catching a defect. Sometimes, these false positives are introduced at the time of writing the test, simply because the person responsible for creating the tests isn’t paying attention. Often, though, false positives spring into life over time. Consider the following HTML snippet, representing a web page showing an error message:

		<div class="error">Here's a random error message</div>

One of the checks you’re performing is that no error messages are displayed in a happy path test case:

@Test(description="Check there is no error message when login is successful")
public void testSuccessfulLogin() {
	LoginPage lp = new LoginPage();
	HomePage ep = lp.correctLogin("username", "password");

public bool hasNoErrorText() {

	return driver.findElements(By.className("error")).size() == 0;

Now consider the event that at some point in time, your developer decides to mix up class annotations (not a highly unlikely event!), which results in a new way of displaying error messages:

		<div class="failure">Here's a random error message</div>

The aforementioned test will still run without trouble after this change. The only problem is that its defect finding ability is gone, since it wouldn’t notice in case an error message IS displayed! Granted, this might be an oversimplified example, and a decent test set would have additional tests that would fail after the change (because they expect an error message that is no longer there, for example), but I hope you can see where I’m going with this: false positives can be introduced over time, without anyone knowing.

So, how to reduce (or even eliminate) the risk of false positives? If you’re dealing with unit tests, I highly recommend experimenting with mutation testing to assess the quality of your unit test set and its defect finding ability. For other types of automated checks, I recommend the practice of checking your checks regularly. Not just at the time of creating your tests, though! Periodically, set some time aside to review your tests and see if they still make sense and still possess their original defect finding power. This will reduce the risk of false positives to a minimum, keeping your tests fresh and preserving trust in your test set and, by association, in your application under test. And isn’t that what testing is all about?