Exercises from my courses - Keeping your tests simple
To give people a better idea of the concepts I’m teaching in my test automation training courses, I’ll highlight some exercises, and the lessons behind them in a series of blog posts. This is the first post in that series.
In a blog post I wrote a couple of years ago, I referred to the concept of ‘elegance’ in relation to test automation, and how keeping your tests simple is often the best way towards test automation that is readable and maintainable.
I’ve also been referring here and there to the fact that simple does not equal easy. Quite the contrary, writing simple tests is hard! To be able to write simple tests, you’ll have to:
- be able to think logically about what it is that your tests need to do, and
- be able to structure that into well-written code, which should result in
- tests that are trustworthy and can fail for one reason and one reason only
Forgetting this and proceeding to write complex tests that are hard to read and even harder to maintain is relatively easy, a fact that I was reminded of once again recently while teaching a ‘Java for testers’ course.
This course teaches the basics of programming in Java and also provides a starter for people wanting to get into Java-based test automation. In the course, I ask people to write some unit tests for a
withdraw() method in a class
Account that represents a bank account:
Shortly before this exercise, we covered concepts such as conditionals (if-then-else) and exception handling using try/catch.
The first exercise I presented to the participants is to write a unit test for the case where we start with a savings account with an initial balance of 1000, then withdraw 500 and check that we end up with a resulting balance of 500:
The next exercise is to create two other tests for some boundary cases: withdrawing 0 should result in a balance of 1000, and withdrawing 1000 should result in a remaining balance of 0:
I then point out that by adding these tests, we’ve introduced quite a bit of code duplication: we have three tests that exercise the same flow, just with different inputs and expected outputs. We can perform the same tests more efficiently by turning this into a data driven test, which is the task for exercise three:
The next exercise, number four, is where things get tricky. In this exercise, I’ll ask them to add another test (the phrasing is important here, as it provides clues on how to approach this problem), one that checks that when we try to withdraw 1100 from our savings account, the resulting balance is 1000, since we aren’t allowed to overdraw on a savings account.
Remember that I said earlier that just before this exercise, participants learned about and practiced applying concepts like conditionals (if-then-else) and exception handling using try/catch… And that’s exactly what they tried to apply while writing this test.
Just having learned about data-driven testing made them try to extend their solution from the previous step with an additional case:
Here’s one solution someone came up with:
This test essentially reimplements part of the business logic from the class in the test code, with all the risks associated with doing so. How do we ensure the correct implementation of those tests? Will we need to write tests for these tests?
Another solution that another participant came up with looked like this:
Here, too, part of the business logic from our class under test, i.e., the part where the exception is thrown, reappears in the test code. As a result, this test can now fail for multiple reasons:
- An exception was thrown under the right conditions, but its type wasn’t
ArithmeticException, or no exception was thrown at all
- The arithmetic in the
withdraw()method wasn’t implemented correctly, leading to an incorrect resulting balance
While it’s a good thing that tests can fail, it’s an even better thing when tests can fail for one reason and one reason only, as I mentioned at the beginning of this post.
Additionally, if you expect an
ArithmaticException to be thrown and the test passes, there’s no way to be sure that the
ArithmeticException actually was thrown (since it is caught inside the test), so there’s a risk of introducing a false negative here, too.
What the participants forgot in their eagerness to solve the problem I gave them, with the tools I handed them earlier in the course, was that their tests should be as simple as possible. Instead, most of them tried to cram this extra case into their existing solution, with the results (and several variations on those) you saw above.
The lesson I try to teach here is quite the opposite. When you write your tests so that they can fail for one reason only, you’ll likely end up with more tests, but also with tests that are easier to read, require less maintenance and, and this is pretty important, too, tests that are easy to diagnose when they do fail.
My proposed solution to the people in my class? Since you’re exercising a different path in your code, write another test. That’s the easiest road to simple tests.
Or, to put it differently: if you find yourself having to use an if-then-else or a try-catch construct in your tests, you’re probably better off writing another test instead.
Yes, I know there are two assertions in this test, too, and that means that technically, it can fail for two different reasons, too, but at least we’re not exercising more than one code path in a single, data-driven test anymore. If you can think of an even cleaner way to test this path, I’d love to hear it!
Oh, and of course a proper debrief was performed after these exercises to make sure that all participants understood the reasoning behind the exercises, and the lesson(s) to be learned from them."