Three things everybody should know about test automation

This article was originally published on LinkedIn Pulse. Read the full article here.

With ever shorter development and release cycles and a need to continuously deliver high quality software in an increasingly competitive market, fast feedback on software quality has become something that most software-producing organizations just can’t do without. Back in ‘the days’, automated tests written to obtain this fast feedback used to consist of either unit tests written and executed by developers, or end-to-end test scripts created and maintained by test automation engineers – all too often using the maintainability and reusability horror that is record and playback.

Since then, test automation has evolved from something considered an ‘extra’ into an indispensable factor in the software development and testing process. However, far though the field has come, there are still a number of fundamental concepts that are unknown to those involved in or relying on test automation. In this post, I’d like to try and shed a little light on some of them.

Continue reading

Why service virtualization is like a wind tunnel

This week I attended the second edition of the Continuous Delivery Conference here in the Netherlands. It was a very interesting day with some good keynotes and presentations, plus I had a lot of interesting discussions with both old and new acquaintances, something that really adds value to a conference for me as an independent consultant. But that’s not what this post is about.. Rather it’s about something that struck me when listening to one of the presentations. I’m not sure who was talking as I made notes only much later, but if it was you, please reveal yourself and come get your credits! The presenter briefly discussed service virtualization as a tool to use in the continuous delivery pipeline and compared it to using a wind tunnel for investigating the aeodynamic properties of cars or airplanes. This struck me as a very solid analogy, so much that I would like to share it with you here.

Testing in a wind tunnel

So why do I think service virtualization IS like wind tunnel testing, exactly?

It allows for executing tests in a controlled environment
Testing car or airplane aerodynamics only yields valuable results if you know exactly what conditions applied and which input parameters (wind speed, angle, variation, etc.) were used to obtain the results. Similarly, when you’re testing any distributed system with external dependencies, you can only safely rely on the test results when you know exactly how those dependencies are behaving. With modern-day highly distributed applications – this especially applies to applications built using microservice architectures – not all dependencies are necessarily under your control anymore. If you want to have full control over the behaviour of those dependencies for stable and reliable testing, service virtualization is an excellent option.

Tests can be repeated under the exact same circumstances
Wind tunnels enable test teams to rerun specific tests over and over again, using the same conditions every time. This allows them to exactly determine the effect of any change on the aerodynamics of their car or plane under test. In software testing, this is exactly what you want when you need to reproduce or analyze a defect or any other suspect behaviour. Unfortunately, when dependencies are outside your circle of control, this might not be easy, if possible at all. When you’re using virtual assets instead of external dependencies, it’s far easier to recreate the exact conditions that applied when the defect occurred.

It can be used to test for highly improbable situations
150mph wind gusts, wind coming from three directions at the same time, … Situations that might be really hard – if not impossible – to reproduce when road testing, but made possible by using a wind tunnel. It’s those corner cases where interesting behaviour of your test object might surface, so they are really worth looking at. With service virtualization, it’s possible, for example, to prepare highly improbable responses for a virtualized third party dependency and see how your application handles these. This is a great way to improve the resilience of and trust in the application you’re developing and testing.

There’s always a need for real life road testing
As with testing cars and planes, you can go a long way using simulated test environments, but there’s no place like the road to really see how your software holds up. So never trust on virtualization alone when testing any application that uses dependencies, because there’s always a situation or two you didn’t think of when virtualizing.. Instead, use your software wind tunnel wisely and your testing process will see the benefits.

Testing RESTful webservices

With the world of IT systems and applications becoming more and more distributed, testing the interfaces between system components becomes ever more important. In a previous post I have introduced a very basic way to test a SOAP web service interface. However, nowadays, in the world of APIs, the REpresentational State Transfer or REST standard becomes increasingly popular due to its lightweight nature. This article introduces a way to test RESTful APIs in a simple yet effective manner, and in a way that integrates smoothly with your existing Java-based testing frameworks, such as those based on Selenium Webdriver.

RESTful APIs come in several varieties. In this post, we are going to see two of them: one returning XML responses (I used this one) and one returning JSON responses (I used this one). You’re welcome to click on either link to see the difference in response format.

Also, I have used some external libraries to get the code to work:

Verify the HTTP status code
The first thing we are going to test for our services is whether they respond to our requests at all. This means we are going to validate whether the HTTP response code for the REST service is correct. There way to do this is equal for both XML and JSON REST APIs:

public static void testStatusCode(String restURL) throws ClientProtocolException, IOException {

	HttpUriRequest request = new HttpGet(restURL);
	HttpResponse httpResponse = HttpClientBuilder.create().build().execute(request);

Pretty simple, right? As you probably know, calling a RESTful API is done by either sending a HTTP POST or a HTTP GET to a specific URL. This in contrast with SOAP web services, where you send a specific XML message to an endpoint. Then, retrieve the HTTP status code from the response and check whether this is OK (i.e., equal to HTTP 200).

Verify the response content type
Next, we are going to verify whether our web service sends us the content type we expect (either XML or JSON). Again, this can be done with just a couple of lines of code:

public static void testMimeType(String restURL, String expectedMimeType) throws ClientProtocolException, IOException {
	HttpUriRequest request = new HttpGet(restURL);
	HttpResponse httpResponse = HttpClientBuilder.create().build().execute(request);

Again, just call the RESTful API URL and retrieve the information you need from the response. In this case, we are interested in the MIME type, which should be either ‘application/xml’ or ‘application/json’, depending on the URL we use.

Verify the response content
Now that we have verified whether our services return correct responses with the correct MIME type, it’s time to look at the actual content of the response. This is where testing XML RESTful APIs differs from testing JSON RESTful APIs, since we need to parse the response to extract the elements and element values we’re interested in.

For XML, we can do this using the following piece of code:

public static void testContent(String restURL, String element, String expectedValue) throws ClientProtocolException, IOException, SAXException, ParserConfigurationException {
	Document doc = DocumentBuilderFactory.newInstance().newDocumentBuilder().parse(restURL);
	NodeList nodelist = doc.getElementsByTagName(element);

Note that we use a different way to retrieve the result from the REST URL as we need to parse it as actual XML. After we’ve done that, we simply look for the XML element we’re interested in and compare its value to the value we expect.

For JSON, we can verify element values as follows:

public static void testContentJSON(String restURL, String element, String expectedValue) throws ClientProtocolException, IOException, SAXException, ParserConfigurationException, JSONException {

	HttpUriRequest request = new HttpGet(restURL);
	HttpResponse httpResponse = HttpClientBuilder.create().build().execute(request);

	// Convert the response to a String format
	String result = EntityUtils.toString(httpResponse.getEntity());

	// Convert the result as a String to a JSON object
	JSONObject jo = new JSONObject(result);

	Assert.assertEquals(expectedValue, jo.getString(element));

Here, we need to take an intermediate step to parse the response to a sensible format. First, we convert the HttpResponse to a String, then we convert this String to a JSONObject. From this JSONObject we can then extract the value for a specific element and compare it to an expected value.

The above examples should get you started nicely when you’re asked to include testing RESTful web services in your automated testing solutions.

Again, happy testing!

An example Eclipse project where I use the code above can be downloaded here.