Getting started with: WireMock

This is the seventh article in our series on new, popular or otherwise interesting tools used in test automation. You can read all posts within this series by clicking here.

What is WireMock?
From the WireMock.org website: WireMock is a flexible library for stubbing and mocking web services. Unlike general purpose mocking tools it works by creating an actual HTTP server that your code under test can connect to as it would a real web service. It supports HTTP response stubbing, request verification, proxy/intercept, record/playback of stubs and fault injection, and can be used from within a unit test or deployed into a test environment. Although it’s written in Java, there’s also a JSON API so you can use it with pretty much any language out there.

In short, WireMock is a very handy tool for all those situations where you need to set up a mock version of a web service, for example for testing or development purposes.

Where can I get WireMock?
WireMock can be downloaded from the WireMock Maven repository.

How do I install and configure WireMock?
Installing WireMock is as simple as downloading the latest standalone version from the WireMock Maven repository and adding it as a dependency to your Java project.

Creating a first WireMock mock service
Let’s start by creating a very simple mock service that returns a plain text string and a HTTP status code.

public void setupStub() {
		
	stubFor(get(urlEqualTo("/an/endpoint"))
            .willReturn(aResponse()
                .withHeader("Content-Type", "text/plain")
                .withStatus(200)
                .withBody("You've reached a valid WireMock endpoint")));
}

This code snippet creates a simple mock service that runs at the /an/endpoint endpoint on the WireMock server (your localhost most of the time, but more about that in a bit). It returns a response with content type text/plain, HTTP status 200 and body text “You’ve reached a valid WireMock endpoint“.

Running and testing your mock service
Of course, we would also like to validate whether the mock service we created does what we told it to do. In this post, I will use JUnit-based REST Assured tests to validate the behaviour of our mocks. But first, we need to get our mock service up and running before we start testing. This can be done very easily with WireMock using a JUnit @Rule:

@Rule
public WireMockRule wireMockRule = new WireMockRule(8090);

This rule starts and stops the mocks defined in your JUnit test class for every test in it. Next, we can define a couple of REST Assured tests to see whether the mock service is doing what we expect:

@Test
public void testStatusCodePositive() {
		
	setupStub();
		
	given().
	when().
		get("http://localhost:8090/an/endpoint").
	then().
		assertThat().statusCode(200);
}
	
@Test
public void testStatusCodeNegative() {
	
	setupStub();
		
	given().
	when().
		get("http://localhost:8090/another/endpoint").
	then().
		assertThat().statusCode(404);
}
	
@Test
public void testResponseContents() {
		
	setupStub();
	
	String response = get("http://localhost:8090/an/endpoint").asString();
	Assert.assertEquals("You've reached a valid WireMock endpoint", response);
}

When we run these tests, they will all pass, indicating our mock service behaves as expected:
WireMock JUnit test results

Request-response matching
WireMock also allows you to define a stub that returns a particular answer based on (part of) the request message using the matching matcher method:

public void setupStub() {
		
	stubFor(post(urlEqualTo("/pingpong"))
			.withRequestBody(matching("<input>PING</input>"))
            .willReturn(aResponse()
                .withStatus(200)
                .withHeader("Content-Type", "application/xml")
                .withBody("<output>PONG</output>")));
}

This service only returns a status code 200 and a response XML message when the request body equals a given value. Again, we can validate this easily using REST Assured:

@Test
public void testPingPongPositive() {
		
	setupStub();
		
	given().
		body("<input>PING</input>").
	when().
		post("http://localhost:8090/pingpong").
	then().
		assertThat().
		statusCode(200).
		and().
		assertThat().body("output", org.hamcrest.Matchers.equalTo("PONG"));
}

WireMock also allows you to use regular expressions for more flexible request matching.

Creating stateful mocks
As a final example, I’ll show how to create a stateful mock. This is done in WireMock using scenarios:

public void setupStub() {
		
	stubFor(get(urlEqualTo("/todolist"))
			.inScenario("addItem")
			.whenScenarioStateIs(Scenario.STARTED)
			.willReturn(aResponse()
                .withStatus(200)
                .withHeader("Content-Type", "application/xml")
                .withBody("<list>Empty</list>")));
		
	stubFor(post(urlEqualTo("/todolist"))
			.inScenario("addItem")
			.whenScenarioStateIs(Scenario.STARTED)
			.willSetStateTo("itemAdded")
			.willReturn(aResponse()
				.withHeader("Content-Type", "application/xml")
                .withStatus(201)));
		
	stubFor(get(urlEqualTo("/todolist"))
			.inScenario("addItem")
			.whenScenarioStateIs("itemAdded")
			.willReturn(aResponse()
                .withStatus(200)
                .withHeader("Content-Type", "application/xml")
                .withBody("<list><item>Item added to list</item></list>")));	
}

So, when we first perform a GET on this mock, an empty to-do list is returned. Then, when we do a POST, the mock service state is changed from Scenario.STARTED (a default in WireMock) to “itemAdded“. Then, when we do another GET, we get a to-do list with a single item on it:

@Test
public void testStatefulMock() {
		
	setupStub();
		
	given().
	when().
		get("http://localhost:8090/todolist").
	then().
		assertThat().
		statusCode(200).
		and().
		assertThat().body("list", org.hamcrest.Matchers.equalTo("Empty"));
		
	given().
	when().
		post("http://localhost:8090/todolist").
	then().
		assertThat().
		statusCode(201);
		
	given().
	when().
		get("http://localhost:8090/todolist").
	then().
		assertThat().
		statusCode(200).
		and().
		assertThat().body("list", org.hamcrest.Matchers.not("Empty")).
		and().
		assertThat().body("list.item", org.hamcrest.Matchers.equalTo("Item added to list"));
}

A trivial example, perhaps, but nevertheless it shows you how can create stateful mocks easily using WireMock.

Further reading
WireMock provides many more useful features for creating just the mock you want. A complete reference guide can be found on the WireMock website.

An Eclipse project including the mock services and REST Assured tests I’ve used in this post can be downloaded here.

On designing a self-diagnosing and self-healing automated test framework – part 1

This is the first post in a short series on designing robust, self-diagnosing and, if possible, even self-healing tests. You can read all articles in this series by clicking here.

I like automating tests. So much that I have even made my profession out of it. What I don’t like though is wasting a lot of time trying to find out why an automated test failed and whether the failure was a genuine bug, a flaky test or some sort of test environment issue.

A quick Google search showed me that there are a couple of commercial test tools that claim they come with some sort of self diagnosing and even a self healing mechanism, for example this one. I’m very interested to see how such a mechanism works. Skeptic as I am, I don’t really believe it until I’ve seen it. Nonetheless, I think it would be interesting to see what can be done to make existing automated test frameworks more robust by improving their ability to self-diagnose and possible even self-heal in case an unexpected error occurs. So that’s what I’ll try to do in the next couple of posts.

The step by step approach I want to follow and the considerations to be made along the way are (very) loosely inspired on a scientific article titled ‘A Self-Healing Approach for Object-Oriented Applications’, which can be downloaded from here. This article presents an approach and architecture for fault diagnosis and self-healing of interpreted object-oriented applications. As the development and maintenance of automated tests should be treated as any other software development project, I see no reason why the principles and suggestions presented in the article could not apply to a test automation framework, at least to some extent…

When is a system self-diagnosing and self-healing?
Let’s start with the end in mind, as Stephen Covey so eloquently put it. What are we aiming for? In order for a system to be self-diagnosing, it should:

  1. Be self-aware, or more concretely, always have a deterministic state
  2. Recognize the fact that an error has occurred
  3. Have enough knowledge to stabilize itself
  4. Be able to analyze the problem situation
  5. Make a plan to heal itself
  6. Suggest healing solutions to the system administrator (in this case, the person responsible for test automation)

If we want our system not only to be self-diagnosing but also self-healing, the system should also:

  1. Heal itself without human intervention

In this post – and probably some future posts as well – I will try and see whether it is possible to design a generic approach for creating robust, self-diagnosing and self-healing test automation frameworks. I’ll try and include meaningful examples based on a relatively simple Selenium test framework wherever possible.

Self-aware tests
The most straightforward way to make any piece of software self-aware is to introduce the concept of state. A robust program, and therefore an automated test as well, should always be in a specific state when it is executing. For robustness, we will assume that the state model for an automated test is deterministic, i.e., a test can never be in more than one state, and an event that triggers a state transition should always result in the test ending up in a single new state. Let’s say we identify the following states that an automated test can be in:

  • Not running
  • Initialization
  • Running
  • Error
  • Teardown

The state model or state transition diagram could then look like this:

A first state model for our automated test

A sample implementation of this state model (also known as a finite state machine or FSM) can be created using a Java enum:

public enum State {
	
	NOT_RUNNING {
        @Override
        State doTransition(String input) {
            System.out.println("Going from State.NOT_RUNNING to State.INITIALIZATION");
            return INITIALIZATION;
        }
    },
    INITIALIZATION {
        @Override
        State doTransition(String input) {
        	if (input.equals("error")) {
        		System.out.println("Going from State.INITIALIZATION to State.ERROR");
        		return ERROR;
        	} else {
        		System.out.println("Going from State.INITIALIZATION to State.RUNNING");
        		return RUNNING;
        	}
        }
    },
    // The RUNNING and TEARDOWN states are implemented in the same way as INITIALIZATION state
    ERROR {
        @Override
        State doTransition(String input) {
        	if (input.equals("ok")) {
        		System.out.println("Going from State.ERROR to State.NOT_RUNNING");
        		return NOT_RUNNING;
        	} else {
        		System.out.println("Remaining in State.ERROR");
        		return this;
        	}
        }
    };
 
    abstract State doTransition(String input);
}

In a next post, I will show how we can apply this state model to a simple Selenium WebDriver test to make it more robust. I will also demonstrate how this state model helps us in letting our tests fail gracefully and in determining what exactly constitutes an error (versus a failed check, for example).

Up and running with: XMLUnit

This is the sixth article in our series on new, popular or otherwise interesting tools used in test automation. You can read all posts within this series by clicking here.

What is XMLUnit?
From the XMLUnit.org website: XMLUnit provides you with the tools to verify the XML you emit is the one you want to create. It provides helpers to validate against an XML Schema, assert the values of XPath queries or compare XML documents against expected outcomes. The most important part is a diff-engine that provides you with full control over what kind of difference is important to you and which part of the generated document to compare with which part of your reference document.

Where can I get XMLUnit?
XMLUnit can be downloaded from the XMLUnit GitHub site. Snapshot binaries for XMLUnit 2.0 (the version that is covered in this post) can also be obtained from the Maven repository.

How do I install and configure XMLUnit?
Installing XMLUnit is as simple as downloading the latest xmlunit-core snapshot version and adding it as a dependency to your Java project.

Creating a first XMLUnittest
Let’s start with a very simple test that validates the value of a specific element of a predefined XML message:

@Test
public void aFirstTest() {
		
	Source source = Input.fromString("<foo>bar</foo>").build();
	XPathEngine xpath = new JAXPXPathEngine();
	String content = xpath.evaluate("/foo/text()", source);
	Assert.assertEquals(content, "bar");
}

Note that I have created this XMLUnit test as a TestNG test to make running the tests easier and to have TestNG automatically generate a test report.

Useful features
Using XMLUnit, XML validations can not only be done based on predefined strings containing XML, but also directly on the XML result of a call to a REST service. This is done in a very similar way:

@Test
public void restCallTest() {
		
	Source source = Input.fromURI("http://parabank.parasoft.com/parabank/services/bank/customers/12212").build();
	XPathEngine xpath = new JAXPXPathEngine();
	String content = xpath.evaluate("//city/text()", source);
	Assert.assertEquals(content, "Beverly Hills");
}

The only thing that is changed is the use of Input.fromURI instead of Input.fromString. Similar methods are available to perform XPath-based validations directly from an InputStream, a File, or a Node object, amongst others.

XMLUnit can not only perform element-based validation, it can also do a full compare of two XML messages. In this example, the XML returned by the same REST service call used in the previous example is compared against a predefined XML messages stored on the local file system (as response.xml in the messages subdirectory):

@Test
public void compareTest() {
		
	DiffBuilder db = DiffBuilder.compare(Input.fromFile("messages/response.xml")).withTest(Input.fromURI("http://parabank.parasoft.com/parabank/services/bank/customers/12212").build());
	Diff d = db.build();
		
	if(d.hasDifferences()) {
		Iterator<Difference> diffs = d.getDifferences().iterator();
		while(diffs.hasNext()) {
			Reporter.log(diffs.next().toString());
		}
	}
		
	Assert.assertEquals(d.hasDifferences(), false);
}

Any differences between the two XML files are written to the TestNG report file using the default Reporter:

XMLUnit differences written to a TestNG report

Finally, XMLUnit can also validate XML messages against an XSD document:

@Test
public void validationTest() {
		
	Validator v = Validator.forLanguage(Languages.W3C_XML_SCHEMA_NS_URI);
	v.setSchemaSources(Input.fromFile("messages/orderschema.xsd").build());
	ValidationResult result = v.validateInstance(Input.fromFile("messages/order.xml").build());
		
	Iterator<ValidationProblem> problems = result.getProblems().iterator();
		
	while (problems.hasNext()) {
		Reporter.log(problems.next().toString());
	}
		
	Assert.assertEquals(result.isValid(), true);
}

Again, any issues that occur during schema validation can be written to the TestNG report using the default Reporter:

XMLUnit schema issues in TestNG report

Further reading
An Eclipse project including the tests I’ve demonstrated above and the reports that have been generated can be downloaded here.