Data driven testing in C# with NUnit and RestSharp

In a previous post, I gave some examples of how to write some basic tests in C# for RESTful APIs using NUnit and the RestSharp library. In this post, I would like to extend on that a little by showing you how to make these tests data driven.

For those of you that do not know what I mean with ‘data driven’: when I want to run tests that exercise the same logic or flow in my application under test multiple times with various combinations of input values and corresponding expected outcomes, I call that data driven testing.

This is especially useful when testing RESTful APIs, since these are all about sending and receiving data as well as exposing business logic to other layers in an application architecture (such as a graphical user interface) or to other applications (consumers of the API).

As a starting point, consider these three tests, written using RestSharp and NUnit:

[TestFixture]
public class NonDataDrivenTests
{
    private const string BASE_URL = "http://api.zippopotam.us";

    [Test]
    public void RetrieveDataForUs90210_ShouldYieldBeverlyHills()
    {
        // arrange
        RestClient client = new RestClient(BASE_URL);
        RestRequest request = 
            new RestRequest("us/90210", Method.GET);

        // act
        IRestResponse response = client.Execute(request);
        LocationResponse locationResponse =
            new JsonDeserializer().
            Deserialize<LocationResponse>(response);

        // assert
        Assert.That(
            locationResponse.Places[0].PlaceName,
            Is.EqualTo("Beverly Hills")
        );
    }

    [Test]
    public void RetrieveDataForUs12345_ShouldYieldSchenectady()
    {
        // arrange
        RestClient client = new RestClient(BASE_URL);
        RestRequest request =
            new RestRequest("us/12345", Method.GET);

        // act
        IRestResponse response = client.Execute(request);
        LocationResponse locationResponse =
            new JsonDeserializer().
            Deserialize<LocationResponse>(response);

        // assert
        Assert.That(
            locationResponse.Places[0].PlaceName,
            Is.EqualTo("Schenectady")
        );
    }

    [Test]
    public void RetrieveDataForCaY1A_ShouldYieldWhiteHorse()
    {
        // arrange
        RestClient client = new RestClient(BASE_URL);
        RestRequest request = 
            new RestRequest("ca/Y1A", Method.GET);

        // act
        IRestResponse response = client.Execute(request);
        LocationResponse locationResponse =
            new JsonDeserializer().
            Deserialize<LocationResponse>(response);

        // assert
        Assert.That(
            locationResponse.Places[0].PlaceName,
            Is.EqualTo("Whitehorse")
        );
    }
}

Please note that the LocationResponse type is a custom type I defined myself, see the GitHub repository for this post for its implementation.

These tests are a good example of what I wrote about earlier: I’m invoking the same logic (retrieving location data based on a country and zip code and then verifiying the corresponding place name from the API response) three times with different sets of test data.

This quickly gets very inefficient when you add more tests / more test data combinations, resulting in a lot of duplicated code. Luckily, NUnit provides several ways to make these tests data driven. Let’s look at two of them in some more detail.

Using the [TestCase] attribute

The first way to create data driven tests is by using the [TestCase] attribute that NUnit provides. You can add multiple [TestCase] attributes for a single test method, and specify the combinations of input and expected output parameters that the test method should take.

Additionally, you can specify other characteristics for the individual test cases. One of the most useful ones is the TestName property, which can be used to provide a legible and useful name for the individual test case. This name also turns up in the reporting, so I highly advise you to take the effort to specify one.

Here’s what our code looks like when we refactor it to use the [TestCase] attribute:

[TestFixture]
public class DataDrivenUsingAttributesTests
{
    private const string BASE_URL = "http://api.zippopotam.us";

    [TestCase("us", "90210", "Beverly Hills", TestName = "Check that US zipcode 90210 yields Beverly Hills")]
    [TestCase("us", "12345", "Schenectady", TestName = "Check that US zipcode 12345 yields Schenectady")]
    [TestCase("ca", "Y1A", "Whitehorse", TestName = "Check that CA zipcode Y1A yields Whitehorse")]
    public void RetrieveDataFor_ShouldYield
        (string countryCode, string zipCode, string expectedPlaceName)
    {
        // arrange
        RestClient client = new RestClient(BASE_URL);
        RestRequest request =
            new RestRequest($"{countryCode}/{zipCode}", Method.GET);

        // act
        IRestResponse response = client.Execute(request);
        LocationResponse locationResponse =
            new JsonDeserializer().
            Deserialize<LocationResponse>(response);

        // assert
        Assert.That(
            locationResponse.Places[0].PlaceName,
            Is.EqualTo(expectedPlaceName)
        );
    }
}

Much better! We now only have to define our test logic once, and NUnit takes care of iterating over the values defined in the [TestCase] attributes:

NUnit test results for the data driven tests using [TestCase] attributes

There are some downsides to using the [TestCase] attributes, though:

  • It’s all good when you just want to run a small amount of test iterations, but when you want to / have to test for larger numbers of combinations of input and output parameters, your code quickly gets messy (on a side note, if this is the case for you, try looking into property-based testing instead of the example-based testing we’re doing here).
  • You still have to hard code your test data in your code, which might give problems with scaling and maintaining your tests in the future.

This is where the [TestCaseSource] attribute comes in.

Using the [TestCaseSource] attribute

If you want to or need to work with larger numbers of combinations of test data and/or you want to be able to specify your test data outside of your test class, then using [TestCaseSource] might be a useful option to explore.

In this approach, you specify or read the test data in a separate method, which is then passed to the original test method. NUnit will take care of iterating over the different combinations of test data returned by the method that delivers the test data.

Here’s an example of how to apply [TestCaseSource] to our tests:

[TestFixture]
public class DataDrivenUsingTestCaseSourceTests
{
    private const string BASE_URL = "http://api.zippopotam.us";

    [Test, TestCaseSource("LocationTestData")]
    public void RetrieveDataFor_ShouldYield
        (string countryCode, string zipCode, string expectedPlaceName)
    {
        // arrange
        RestClient client = new RestClient(BASE_URL);
        RestRequest request =
            new RestRequest($"{countryCode}/{zipCode}", Method.GET);

        // act
        IRestResponse response = client.Execute(request);
        LocationResponse locationResponse =
            new JsonDeserializer().
            Deserialize<LocationResponse>(response);

        // assert
        Assert.That(
            locationResponse.Places[0].PlaceName,
            Is.EqualTo(expectedPlaceName)
        );
    }

    private static IEnumerable<TestCaseData> LocationTestData()
    {
        yield return new TestCaseData("us", "90210", "Beverly Hills").
            SetName("Check that US zipcode 90210 yields Beverly Hills");
        yield return new TestCaseData("us", "12345", "Schenectady").
            SetName("Check that US zipcode 12345 yields Schenectady");
        yield return new TestCaseData("ca", "Y1A", "Whitehorse").
            SetName("Check that CA zipcode Y1A yields Whitehorse");
    }
}

In this example, we specify our test data in a separate method LocationTestData(), and then tell the test method to use that method as the test data source using the [TestDataSource] attribute, which takes as its argument the name of the test data method.

For clarity, the test data is still hard coded in the body of the LocationTestData() method, but that’s not mandatory. You could just as easily write a method that reads the test data from any external source, as long as the test data method is static and returns an object of type IEnumerable, or any object that implements this interface.

Also, since the [TestCase] and [TestCaseSource] attributes are features of NUnit, and not of RestSharp, you can apply the principles illustrated in this post to other types of tests just as well.

Beware, though, before you use them for user interface-driven testing with tools like Selenium WebDriver. Chances are that you’re falling for a classic case of ‘just because you can, doesn’t mean you should’. I find data driven testing with Selenium WebDriver to be a test code smell: if you’re going through the same screen flow multiple times, and the only variation is in the test data, there’s a high chance that there’s a more efficient way to test the same underlying business logic (for example by leveraging APIs).

Chris McMahon explains this much more eloquently in a blog post of his. I highly recommend you reading that.

For other types of testing (API, unit, …), data driven testing could be a very powerful way to make your test code better maintainable and more powerful.

All example code in this blog post can be found on this GitHub page.

Creating data driven API tests with REST Assured and TestNG

As you’ve probably read in previous posts on this site, I am a big fan of the REST Assured library for writing tests for RESTful web services. In this post, I will demonstrate how to create even more powerful tests with REST Assured by applying the data driven principle, with some help from TestNG.

Combining REST Assured and TestNG
REST Assured itself is a Domain Specific Language (DSL) for writing tests for RESTful web services and does not offer a mechanism for writing data driven tests (i.e., the ability to write a single test that can be executed multiple times with different sets of input and validation parameters). However, REST Assured tests are often combined with JUnit or TestNG, and the latter offers an easy to use mechanism to create data driven tests through the use of the DataProvider mechanism.

Creating a TestNG DataProvider
A TestNG DataProvider is a method that returns an object containing test data that can then be fed to the actual tests (REST Assured tests in this case). This data can be hardcoded, but it can also be read from a database or a JSON specification, for example. It’s simply a matter of implementing the DataProvider in the desired way. The following example DataProvider creates a test data object (labeled md5hashes) that contains some example text strings and their md5 hash:

@DataProvider(name = "md5hashes")
public String[][] createMD5TestData() {
		
	return new String[][] {
			{"testcaseOne", "4ff1c9b1d1f23c6def53f957b1ed827f"},
			{"testcaseTwo", "39738347fb533d798aca9ae0f56ca126"},
			{"testcaseThree", "db6b151bb4bde46fddb361043bc3e2d9"}
	};
}

Using the DataProvider in REST Assured tests (with query parameters)
Using the data specified in the DataProvider in REST Assured tests is fairly straightforward. In the example below, we’re using it in combination with a RESTful web service that uses a single query parameter:

@Test(dataProvider = "md5hashes")
public void md5JsonTest(String originalText, String md5Hash) {
		
	given().
		parameters("text", originalText).
	when().
		get("http://md5.jsontest.com").
	then().
		assertThat().
		body("md5", equalTo(md5Hash));
}

Notice how the data in the DataProvider can be used for specification of expected results just as easy as for specifying input parameters. When we run this test, we can see in the output that the test is run three times, once for every set of input and validation parameters:

Console output for our data driven REST Assured test

Using the DataProvider in REST Assured tests (with query parameters)
The other parameter approach commonly used with RESTful web services is the use of path parameters. Let’s create another DataProvider, this one specifying Formula 1 circuits and the country they are situated in:

@DataProvider(name = "circuitLocations")
public String[][] createCircuitTestData() {
		
	return new String[][] {
			{"adelaide","Australia"},
			{"detroit","USA"},
			{"george","South Africa"}
	};
}

The Ergast Developer API provides motor racing data through a public API (check it out, it’s really cool). Here’s how to parameterize the call to get specific circuit data and check the country for each circuit with REST Assured, again using our DataProvider:

@Test(dataProvider = "circuitLocations")
public void circuitLocationTest(String circuitId, String location) {
		
	given().
		pathParameters("circuitId",circuitId).
	when().
		get("http://ergast.com/api/f1/circuits/{circuitId}.json").
	then().
		assertThat().
		body("MRData.CircuitTable.Circuits[0].Location.country",equalTo(location));
}

The only real difference is in how to set up path parameter use in REST Assured (see also here), otherwise making your tests data driven is just as easy as with query parameters.

When we run this second test, we can see again that it’s run three times, once for every circuit specified:

Console output for our data driven REST Assured test (with path parameters_

A Maven Eclipse project containing the code demonstrated in this blog post can be downloaded here.

Data driven JavaScript tests using Jasmine

In my previous post I introduced Jasmine as a tool for testing JavaScript code. Now that I’ve had some more time to play around with it, combining it with Protractor for user interface-driven tests for one of my client’s websites, I’ve noticed that one feature was sorely missing: built-in support for data driven testing. All BDD tools (or tools that say they support BDD) that I have worked with so far, such as Cucumber, SpecFlow and JGiven, support the creation of data driven scenarios. However, Jasmine doesn’t, at least not as far as I have seen in the limited time I have been using it so far.

As I think data driven testing is key when creating maintainable and reusable scenarios, I fired up Google to check whether someone had already found a solution. From what I’ve read, there are (at least) two different ways to do data-driven testing with Jasmine. Let’s have a look at both.

Specifying your test data in an array variable
I found the first approach in the comments of this StackOverflow post. The comment submitted by peterhendrick suggests the use of a JSON file containing the test data. I slightly simplified this to use a simple array of key-value pairs and have the Jasmine spec loop through the array. You can simply populate this array any way you like, for example from an external .json file as suggested, but in any other way as well. In this example, I hard-coded it in the spec itself:

describe("Calculator (data driven tests with test data specified in array)", function() {
  var calc;
  var testdata = [{"x":1,"y":1,"sum":2},{"x":2,"y":3,"sum":5},{"x":4,"y":2,"sum":6}];
  
  beforeEach(function() {
    calc = new Calculator();
  });

	for(var i in testdata) {
		it("should be able to add two numbers, using (" + testdata[i].x + "," + testdata[i].y + "," + testdata[i].sum + ")", function() {
			
			expect(calc.add(testdata[i].x, testdata[i].y)).toEqual(testdata[i].sum);
		});	
	};
});

I particularly like this example because it enables you to add the actual values used to the test output. You can see this when you run your spec:

Test results from driving test data through an array

Data driven testing with a custom using function
This blog post from JP Castro presents an alternative way to do data driven testing in Jasmine. His solution seemed easy enough for me to apply directly to my own specs. Let’s see what it looks like.

The basic idea behind his solution is to wrap each it block in a spec with a using block that passes the parameter values to the it block:

describe("Calculator (data driven tests with using() function)", function() {
  var calc;
  
  beforeEach(function() {
    calc = new Calculator();
  });

  using('parameters',[[1,1,2],[2,3,5],[4,2,6]], function(parameters){
	it("should be able to add two numbers, using (" + parameters[0] + "," + parameters[1] + "," + parameters[2] + ")", function() {
		expect(calc.add(parameters[0],parameters[1])).toEqual(parameters[2]);
	});
  });
});

The using function he presents is implemented as follows:

function using(name, values, func) {
	for(var i = 0, count = values.length; i < count; i++) {
		if(Object.prototype.toString.call(values[i]) !== '[Object Array]') {
			values[i] = [values[i]];
		}
		func.apply(this,values[i]);
	}
}

When we run the spec above, the output looks like this:

Test results from driving test data through a using function

So, now we have two simple (and admittedly similar) workarounds for what I think is still a fundamental shortcoming in Jasmine. The code demonstrated in this blog post can be downloaded here.