Four people in your organization that will benefit from test automation

The introduction of test automation within your organization can, if done correctly, affect your testing process in a number of positive ways. In this post, we will look at the ways different stakeholders within your software development process might benefit from automated testing.

Test automation benefits the software testing team in a number of ways, of which some of the most important are:

  • Automation of regression tests takes away the repetitive effort of executing the same test scripts by hand over and over again, every time new functionality is introduced. Instead of having to spend time on this often tedious activity, software testers can focus on testing the changes that are introduced in the software, while the automated test tool takes care of the regression testing.
  • Not having to spend time on regression testing means that testers can perform more tests in the same amount of time.
  • Testers with an interest in programming and tools have the opportunity to learn valuable new skills by diving into the world of automated testing.

The development team might also benefit from the introduction of test automation:

  • As automated tests can generally be run faster and more often compared to when the same test scripts have to be executed manually, developers receive feedback on the quality of their code sooner.
  • Testers involved in automated testing often work closely together with developers to set up tests and analyse test results, resulting in a bigger appreciation for what used to be ‘the other side’. Of course, this appreciation works both ways (and this point could have been under ‘Testers’ just as easily).

Project managers
Automated testing has several benefits for project management too:

  • With automated testing, more tests can be performed within the duration of the project. On the one hand, today’s dynamic market requires fast-paced software delivery and therefore ever shorter development cycles. On the other hand, the same dynamic market expects high quality software as for a lot of products many alternatives are available, making it easy for an unsatisfied customer to change products or services. Automated testing is the only way for an organization to deliver high quality software quickly.
  • Automated testing enables tests to be executed sooner and more often, giving project managers continuous insight in the quality of the software, as well as in trends in this quality. For example, the availability of daily test results gives a project manager insight in whether the introduction of a new development method – or a new development team – has the desired effect on the quality of the product to be delivered.

End users
Finally, the end user of the software delivered also benefits from the introduction of automated testing, albeit in a more indirect way:

  • Automated testing, when implemented correctly, can greatly increase the speed with which software is delivered, meaning that the end user receives a release of the product he or she has asked for quicker than would be possible with traditional manual testing. As automated testing is able to shorten development cycles, any bugs reported or updates requested by the user can be dealt with quicker as well.
  • As the test team is able to achieve higher test coverage within the given amount of time, the quality of the end product will generally also be higher as more bugs can be resolved before delivery, meaning that the user will likely be happier with this product.

How did the introduction of automated testing benefit your organization or (for the contractors) your client?

The test automation pyramid

In his book Succeeding with Agile, Mike Cohn describes the concept of a test automation pyramid, describing three levels of test automation, their relation and their relative importance. As an advocate of minimizing user interface-based test automation I wholeheartedly support this pyramid, which is why I decided to share it with you at

Graphically, the test automation pyramid as proposed by Mike Cohn looks like this:
The test automation pyramid
Base layer: unit tests
Unit tests form the base layer of every solid automated testing approach. They can be written relatively quickly and give the programmer very specific information about the origins of a bug, up to the exact line of code where a failure occurs. Compare this to a bug report from a tester, who would usually be more like ‘function X, Y and / or Z are not working when I enter A or B, now go fix it’. This often requires more analysis (reproduction, debugging) and therefore more time from the developer to fix things.

Another advantage of unit tests is that not only can they be written quickly, test execution is also very fast, giving the developer immediate feedback on code quality.

Possible drawbacks of unit tests are that they mostly focus on small pieces of code (methods, classes) and are therefore unable to detect integration or system level bugs. Also, as they are written in code, they are written mostly by developers and not by testers. Ideally, unit tests should be written by someone other than the developer of the code that is being tested.

Top layer: user interface-level tests
Let’s skip the middle layer for a moment and go right to the top of the pyramid, where the UI-level automated tests reside. Ideally, you would want to do as little of this as possible, as they are often the most brittle and take the longest time both in test case development and in test execution. In my opinion, this form of test automation should only be used when the UI is actually being tested rather than the underlying system functionality, or when there is no viable alternative. Such an alternative is available more often than you’d think, by the way..

Middle layer: service and API tests
For those tests that exceed the scope of unit tests it is strongly advised to use tests that communicate with the application under test at the service or API level. Most modern applications offer some sort of API (either through an actual programming interface or through a web service exposing functionality to the outside world) that can be used by the tester to test those applications. These tests are often far less brittle (as service interfaces and APIs tend to change far less often than user interfaces) and execute far quicker with less false negatives.

The inverted test automation pyramid
It is no coincidence that Mike Cohn calls this middle layer the forgotten layer of test automation. All too often, test cases that cannot be covered by developers in unit tests are directly automated on the user interface level, resulting in big sets of UI level automated tests that take eons to execute and maintain. This phenomenon is represented by an inverted test automation pyramid:
The inverted test automation pyramid
In extreme cases, the middle layer doesn’t even exist in the overall test automation approach. It doesn’t need explanation that in most test automation projects that resemble this inverted pyramid a lot of money is wasted unnecessarily on development and maintenance of the automated test cases..

My advice
I’d therefore advice any test automation specialist that would like to make his or her project a success (and who doesn’t?) to do a couple of things:

  • Get familiar with unit testing, its benefits and the way the developers in your project use it. Try to understand what they test and what coverage is achieved in unit testing. Try and work with your developers to see where this coverage can be increased further and what you can do to achieve that.
  • For those tests that are not covered by unit tests, try to find out whether the application you are testing offers an API that you can use to base your automated tests on. Initially, testing ‘under the hood’ without a UI to drive tests might seem challenging, but the benefits are definitely worth it.

Following these two suggestions will help you greatly in getting your test automation pyramid in the right shape. What does your pyramid look like?

Final note: Although some articles on the Internet even go as far as to suggest an ‘ideal’ mix of the pyramid layers (80% unit tests, 15% API tests and 5% UI tests, for example), I think there is no one ‘ideal’ mix. It depends a lot on the type of application under test and the skill sets of your developers and especially your testers. However, the shape of the test automation pyramid should be roughly the same in any case.

Up and running with: Mockito

This is the third article in our series on new, popular or otherwise interesting tools used in test automation. You can read all posts within this series by clicking here.

What is Mockito?
Mockito is an open source testing framework for Java that allows developers to create mock objects for use in automated unit tests. It is often used in the Test Driven Development and Behavior Driven Development approaches. Using Mockito mock objects, developers can mock external dependencies and ensure their code interacts with it in the expected manner.

Where can I get Mockito?
Mockito can be downloaded from this site.

How do I install and configure Mockito?
As the Mockito framework consists of just a single .jar file, all you need to do is to add this .jar as a dependency to your Java project and you’re ready to start mocking!

Creating a first Mockito mock object
Before we can start creating mock objects using Mockito, we first need a Java class for which we can mock the behavior. To illustrate the workings of Mockito I created a simple class Login, consisting of a single constructor, two String variables username and password and a single method login(username,password). The class implementation is included in the download link at the end of this post.

Now, assume that this Login class is either not readily available when you want to perform unit tests that have this class as a dependency, or that you want to emulate a certain type of behavior. To do so, let’s create a Mockito mock for the Login class:

public static Login createLoginMock() {

	// create and return a mock instance of the Login class	
	return mock(Login.class);

Easy, right? Now, let’s see what we can do with this mock.

Using your Mockito mock
First, let’s check whether we can interact with the methods of our mock correctly. Mockito provides a verify function for this:

public static void runVerifyTestWithMockito() {
	// create mock instance of the login class
	Login loginMock = createLoginMock();
	// call methods on the mock instance
	// verify that methods are called properly
	// attempt to verify a method that hasn't been called (this generates an error)

The last call generated an error, as we try to verify an interaction that hasn’t taken place.

Now, let’s use Mockito to stub the behavior of a method of the Login class, so that we can use our mock to simulate specific behavior in our unit tests. In Mockito, this is done using a when … thenReturn … construction:

public static void runStubTestWithMockito() {
	// create mock instance of the login class
	Login loginMock = createLoginMock();
	// stub mock behavior
	// call mock method and verify result

Finally, we can also have a class method of our mock return an exception, even when the actual class implementation might not always do that. This is useful when testing exception handling in your code and is done using a when … thenThrow … construction:

public static void runExceptionTestWithMockito() {

	// create mock instance of the login class
	Login loginMock = createLoginMock();

	// have the mock return an Exception
	when(loginMock.getPassword()).thenThrow(new RuntimeException());

	// call the mock method that throws the exception

Useful features
I’ve showed you some of the basic features of Mockito. However, the framework offers loads more, such as:

  • Verification that a method is invoked exactly once / at least x times / never / …
  • Verification that methods are invoked in a particular order
  • Explicitly verify that certain mock classes have not been interacted with

An extensive list of Mockito features, along with examples, can be found here.

Further reading
There is a lot of documentation to be found on Mockito on the web, apart from the reading material which I’ve linked to above. For example, a nice tutorial, including using Mockito for Android testing, can be found on the site of Lars Vogel. For those that prefer actual books, Tomek Kaczanowski has written a book on Mockito and TestNG, which can be purchased here.

An Eclipse project including my Login class implementation and the tests I’ve demonstrated above can be downloaded here.

Happy mocking!