RESTful API testing in C# with RestSharp

Since my last blog post that involved creating tests at the API level in C#, I’ve kept looking around for a library that would fit all my needs in that area. So far, I still haven’t found anything more suitable than RestSharp. Also, I’ve found out that RestSharp is more versatile than I initially thought it was, and that’s the reason I thought it would be a good idea to dedicate a blog post specifically to this tool.

The examples I show you in this blog post use the Zippopotam.us API, a publicly accessible API that resolves a combination of a country code and a zip code to related location data. For example, when you send an HTTP GET call to

http://api.zippopotam.us/us/90210

(where ‘us’ is a path parameter representing a country code, and ‘90210’ is a path parameter representing a zip code), you’ll receive this JSON document as a response:

{
	post code: "90210",
	country: "United States",
	country abbreviation: "US",
	places: [
		{
			place name: "Beverly Hills",
			longitude: "-118.4065",
			state: "California",
			state abbreviation: "CA",
			latitude: "34.0901"
		}
	]
}

Some really basic checks
RestSharp is available as a NuGet package, which makes it really easy to add to your C# project. So, what does an API test written using RestSharp look like? Let’s say that I want to check whether the previously mentioned HTTP GET call to http://api.zippopotam.us/us/90210 returns an HTTP status code 200 OK, this is what that looks like:

[Test]
public void StatusCodeTest()
{
    // arrange
    RestClient client = new RestClient("http://api.zippopotam.us");
    RestRequest request = new RestRequest("nl/3825", Method.GET);

    // act
    IRestResponse response = client.Execute(request);

    // assert
    Assert.That(response.StatusCode, Is.EqualTo(HttpStatusCode.OK));
}

If I wanted to check that the content type specified in the API response header is equal to “application/json”, I could do that like this:

[Test]
public void ContentTypeTest()
{
    // arrange
    RestClient client = new RestClient("http://api.zippopotam.us");
    RestRequest request = new RestRequest("nl/3825", Method.GET);

    // act
    IRestResponse response = client.Execute(request);

    // assert
    Assert.That(response.ContentType, Is.EqualTo("application/json"));
}

Creating data driven tests
As you can see, creating these basic checks is quite straightforward with RestSharp. Since APIs are all about sending and receiving data, it would be good to be able to make these tests data driven. NUnit supports data driven testing through the TestCase attribute, and using that together with passing the parameters to the test method is really all that it takes to create a data driven test:

[TestCase("nl", "3825", HttpStatusCode.OK, TestName = "Check status code for NL zip code 7411")]
[TestCase("lv", "1050", HttpStatusCode.NotFound, TestName = "Check status code for LV zip code 1050")]
public void StatusCodeTest(string countryCode, string zipCode, HttpStatusCode expectedHttpStatusCode)
{
    // arrange
    RestClient client = new RestClient("http://api.zippopotam.us");
    RestRequest request = new RestRequest($"{countryCode}/{zipCode}", Method.GET);

    // act
    IRestResponse response = client.Execute(request);

    // assert
    Assert.That(response.StatusCode, Is.EqualTo(expectedHttpStatusCode));
}

When you run the test method above, you’ll see that it will run two tests: one that checks that the NL zip code 3825 returns HTTP 200 OK, and one that checks that the Latvian zip code 1050 returns HTTP 404 Not Found (Latvian zip codes are not yet available in the Zippopotam.us API). In case you ever wanted to add a third test case, all you need to do is add another TestCase attribute with the required parameters and you’re set.

Working with response bodies
So far, we’ve only written assertions on the HTTP status code and the content type header value for the response. But what if we wanted to perform assertions on the contents of the response body?

Technically, we could parse the JSON response and navigate through the response document tree directly, but that would result in hard to read and hard to maintain code (see for an example this post again, where I convert a specific part of the response to a JArray after navigating to it and then do a count on the number of elements in it. Since you’re working with dynamic objects, you also don’t have the added luxury of autocomplete, because there’s no way your IDE knows the structure of the JSON document you expect in a test.

Instead, I highly prefer deserializing JSON responses to actual objects, or POCOs (Plain Old C# Objects) in this case. The JSON response you’ve seen earlier in this blog post can be represented by the following LocationResponse class:

public class LocationResponse
{
    [JsonProperty("post code")]
    public string PostCode { get; set; }
    [JsonProperty("country")]
    public string Country { get; set; }
    [JsonProperty("country abbreviation")]
    public string CountryAbbreviation { get; set; }
    [JsonProperty("places")]
    public List<Place> Places { get; set; }
}

and the Place class inside looks like this:

public class Place
{
    [JsonProperty("place name")]
    public string PlaceName { get; set; }
    [JsonProperty("longitude")]
    public string Longitude { get; set; }
    [JsonProperty("state")]
    public string State { get; set; }
    [JsonProperty("state abbreviation")]
    public string StateAbbreviation { get; set; }
    [JsonProperty("latitude")]
    public string Latitude { get; set; }
}

Using the JsonProperty attribute allows me to map POCO fields to JSON document elements without names having to match exactly, which in this case is especially useful since some of the element names contain spaces, which are impossible to use in POCO field names.

Now that we have modeled our API response as a C# class, we can convert an actual response to an instance of that class using the deserializer that’s built into RestSharp. After doing so, we can refer to the contents of the response by accessing the fields of the object, which makes for far easier test creation and maintenance:

[Test]
public void CountryAbbreviationSerializationTest()
{
    // arrange
    RestClient client = new RestClient("http://api.zippopotam.us");
    RestRequest request = new RestRequest("us/90210", Method.GET);

    // act
    IRestResponse response = client.Execute(request);

    LocationResponse locationResponse =
        new JsonDeserializer().
        Deserialize<LocationResponse>(response);

    // assert
    Assert.That(locationResponse.CountryAbbreviation, Is.EqualTo("US"));
}

[Test]
public void StateSerializationTest()
{
    // arrange
    RestClient client = new RestClient("http://api.zippopotam.us");
    RestRequest request = new RestRequest("us/12345", Method.GET);

    // act
    IRestResponse response = client.Execute(request);
    LocationResponse locationResponse =
        new JsonDeserializer().
        Deserialize<LocationResponse>(response);

    // assert
    Assert.That(locationResponse.Places[0].State, Is.EqualTo("New York"));
}

So, it looks like I’ll be sticking with RestSharp for a while when it comes to my basic C# API testing needs. That is, until I’ve found a better alternative.

All the code that I’ve included in this blog post is available on my Github page. Feel free to clone this project and run it on your own machine to see if RestSharp fits your API testing needs, too.

On handling processing time in your integration tests with Awaitility, and on using the Gmail API

For someone that writes and talks about REST Assured a lot, it has taken me a lot of time to find an opportunity to use it in an actual client project. Thankfully, my current project finally gives me the opportunity to use it on a scale broader than the examples and exercises from my workshop. Being able to do so has made me realize that some of the concepts and features I teach in the workshop deserve a more prominent spot, while others can be taught later on, or more briefly.

But that’s not what I wanted to talk about today. No, now that I actually use REST Assured for integration testing, as in: the integration between a system and its adjacent components, there’s something I have to deal with that I haven’t yet had to tackle before: processing time, or the time it takes for a message triggered using the REST API of system A to reach and be processed by system B, and being able to verify its result through consuming the REST API of system B.

Unlike writing tests for a single API, which is how I have been using and demonstrating REST Assured until now, I need a mechanism that helps me wait exactly long enough for the processing to be finished. Similar to user interface-driven automation, I could theoretically solve this by using calls to Thread.sleep(), but that’s ugly and vile and… just no.

Instead, I needed a mechanism that allowed me to poll an API until its response indicated a certain state change had occurred before performing the checks I needed to perform. In this case, and this is the example I’ll use in the remainder of this blog post, I invoked the REST API of system A to trigger a password reset for a given user account, and wanted to check if that resulted in a ‘password reset’ email message arriving in system B, system B being a specific Gmail inbox here.

Triggering the password reset
Triggering the password reset is done by means of a simple API call, which I perform (as expected) using REST Assured:

given().
    spec(identityApiRequestSpecification)
and().
    body(new PasswordForgottenRequestBody()).
when().
    post("/passwordreset").
then().
    assertThat().
    statusCode(204);

Waiting until the password reset email arrives
As stated above, I could just include a fixed waiting period of, say, 10 seconds before checking Gmail and seeing whether the email message arrived as expected. But again, Thread.sleep() is evil and dirty and… and should be avoided at all times. No, I wanted a better approach. Preferably one that didn’t result in unreadable code, both because I use my code in demos and I’d like to spend as little time as possible explaining my tests to others, and therefore want to keep it as readable as possible. Looking for a suitable library (why reinvent the wheel.. ) I was pointed to a solution that was created by Johan Haleby (not coincidentally also the creator of REST Assured), called Awaitility. From the website:

Awaitility is a DSL that allows you to express expectations of an asynchronous system in a concise and easy to read manner.

I’m not going to write about all of the features provided by Awaitility here (the usage guide does that way better than I ever could), but to demonstrate its expression power, here’s how I used it in my test:

await().
    atMost(10, TimeUnit.SECONDS).
with().
    pollInterval(1, TimeUnit.SECONDS).
    until(() -> this.getNumberOfEmails() == 1);

This does exactly what it says on the tin: it executes a method called getNumberOfEmails() once per second for a duration of 10 seconds, until the result returned by that method equals 1 (in which case my test execution continues) or until the 10 second timeout period has been exceeded, resulting in an exception being thrown. All with a single line of readable code. That’s how powerful it is.

In this example, the getNumberOfEmails() is a method that retrieves the contents for a specific Gmail mailbox and returns the number of messages in it. Before the test starts, I empty the mailbox completely to make sure that no old messages remain there and cause false positives in my test. Here’s how it looks:

private int getNumberOfEmails() {

    return given().
        spec(gmailApiRequestSpec).
    when().
        get("/messages").
    then().
        extract().
        path("resultSizeEstimate");
}

This method retrieves the number of emails in a Gmail inbox (the required OAuth2 authentication details, base URL and base path are specified in the gmailApiRequestSpec RequestSpecification) by means of a GET call to /messages and extracting and returning the value of the resultSizeEstimate field of the JSON response returned by the API. If you want to know more about the Gmail API, its documentation can be found here, by the way.

Checking the content of the password reset message
So, now that we know that an email message has arrived in the Gmail inbox, all that’s left for us to do is check whether it is a password reset message and not any other type of email message that might have arrived during the execution of our test. All we need to do is to once more retrieve the contents of the mailbox, extract the message ID of the one email message in it, use that to retrieve the details for that message and check whatever we want to check (in this case, whether it has landed in the inbox and whether the subject line has the correct value):

String messageID =
    given().
        spec(gmailApiRequestSpec).
    when().
        get("/messages").
    then().
        assertThat().
        body("resultSizeEstimate", equalTo(1)).
    and().
        extract().
        path("messages.id[0]");

    // Retrieve email and check its contents
    given().
        spec(gmailApiRequestSpec).
    and().
        pathParam("messageID", messageID).
    when().
        get("/messages/{messageID}").
    then().
        assertThat().
        body("labelIds",hasItem("INBOX")).
    and().
        body("payload.headers.find{it.name=='Subject'}.value",equalTo("Password reset"));

Gmail authentication
It took me a little while to figure out how to consume the Gmail API. In the end, this proved to be quite simple but I spent a couple of hours fiddling with OAuth2, authentication codes, OAuth2 access and refresh tokens and the way Google has implemented this. Describing how all this goes is beyond the scope of this blog post, but you can find instructions here. Once you’ve obtained a refresh token, store it, because that’s the code you can use to generate a new access token through the API, without having to deal with the pesky authentication user interface. For those of you more experienced with OAuth2, this might sound obvious, but it took me a while to figure it out. Still, it’s far better than writing automation against the Gmail user interface, though (seriously, DON’T DO THAT).

So, to wrap things up, there are two lessons here.

One, if you’re looking for a library that helps you deal with processing times in integration tests in a flexible and readable manner, and you’re writing your tests in Java, I highly recommend taking a look at Awaitility. I’ve only recently discovered it but I’m sure this one won’t leave my tool belt anytime soon.

Two, if you want to include checking email into your integration or possibly even your end-to-end tests, skip the user interface and go the API route instead. Alternatively, you could try an approach like Angie Jones presents in a recent blog post, leveraging the JavaMail API, instead. Did I say don’t use the Gmail (or Outlook, or Yahoo, or whatever) user interface?

On preventing your test suite from becoming too user interface-heavy

In August of last year, I published a blog post talking about why I don’t like to think of automation in terms of frameworks, but rather in terms of solutions. I’ve softened a little since then (this is probably a sign of me getting old..), but my belief that building a framework might lead to automation engineers subsequently trying to fit every test left, right and center into that framework still stands. One example of this phenomenon in particular I still see too often: engineers building a feature-rich end-to-end automation framework (for example using Selenium) and then automating all of their tests using that framework.

This is what I meant in the older post by ‘framework think’: because the framework has made it so easy for them to add new tests, they skip the step where they decide what would be the most efficient approach for a specific test and blindly add it to the test suite run by that very framework. This might not lead to harmful side effects in the short term, but as the test suite grows, chances are high that it becomes unwieldy, that the time it takes to complete a full test run becomes unnecessarily long and that maintenance efforts are not being outweighed by the added value of having the automated tests any more.

In this post, I’d like to take the practical approach once more and demonstrate how you can take a closer look at your application and decide if there might be a more efficient way to implement certain checks. We’re going to do this by opening up the user interface and see what happens ‘under the hood’. I’m writing this post as an addendum to my ‘Building great end-to-end tests with Selenium and Cucumber / SpecFlow‘ course, by the way. Yes, that’s right, one of the first things I talk about during my course on writing tests with Selenium is when not to do so. I firmly believe that’s the on of the very first steps towards creating a solid test suite: deciding what should not be in it.

The application under test
The application we’re going to write tests for is an online mortgage orientation tool, provided by a major Dutch online bank. I’ve removed all references to the client name, just to be sure, but it’s not like we’re dealing with sensitive data here. The orientation tool is a sequence of three forms, in which people that are interested in a mortgage fill in details about their financial situation, after which the orientation tool gives an indication of whether or not the applicant is eligible for a mortgage, as well as an estimate of the maximum amount of the mortgage, the interest rate and the monthly installments payable.

Our application under test - the mortgage orientation tool

What are we going to automate?
Now that we know what our application under test does, let’s see what we should automate. We’ll assume that there is a justified need for automated checks in the first place (otherwise this would have been a very short blog post!). We’ll also assume that, maybe for tests on some other part of the bank’s website, there is already a solid automation framework written around Selenium in place. So, this being a website and all, it makes sense to write some additional checks and incorporate them into the existing framework.

First of all, let’s try and make sure that the orientation tool can be used and completed, and that it displays a result. I’d say, that would be a good candidate for an automated test written using Selenium, since it confirms that the application is working from an end user perspective (there is value in the test) and I can’t think of a lower level test that would give me the same feedback. Since there are a couple of different paths through the orientation tool (you can apply for a mortgage alone or with someone else, some people have a house to sell while others have not, there are different types of contracts, etc.), I’d even go as far as to say you’ll need more than one Selenium-based test to be able to properly claim that all paths can be traversed by an end user.

Next, I can imagine that you’d want to make sure that the numbers that are displayed are correct, so your customers aren’t misinformed when they complete the orientation tool. This would lead to some massive issues of distrust later on in the mortgage application process, I’d assume.. Since we’ve been able to add the previous tests so easily to our existing framework, it makes sense to add some more tests that walk through the forms, add the data required to trigger a specific expected outcome and verify that the result screen we saw in the screenshot above displays the expected numbers. Right?

No. Not right.

It’s highly likely that the business logic used to perform the calculation and serve the numbers displayed on screen isn’t actually implemented in the user interface. Rather, it’s probably served up by a backend service containing the business logic and rules required to perform the calculations (and with mortgages, there are quite a few of those business rules, I’ve been told..). The user interface takes the values entered by the end user, sends them to a backend service that performs calculations and returns the values indicating mortgage eligibility, interest rate, height of monthly installment, etc., which are then interpreted and displayed again by that same user interface.

So, since the business logic that we’re verifying isn’t implemented in the user interface, why use the UI to verify it in the first place? That would highly likely only lead to unnecessarily slow tests and shallow feedback. Instead, let’s look if there’s a different hook we can use to write tests.

I tend to use on of two different tactics to find out if there are better ways to write automated tests in cases like these:

  1. Talk to a developer. They’re building the stuff, so they’ll probably know more about the architecture of your application and will likely be happy to help you out.
  2. Use a network analyzing tool such as Fiddler or WireShark. Tools like these two let you see what happens ‘under water’ when you’re using the user interface of a web application.

Normally, I’ll use a combination of both: find out more about the architecture of an application by talking to developers, then using a network analyzer (I prefer Fiddler myself) to see what API calls are triggered when I perform a certain action.

Analyzing API calls using Fiddler
So, let’s put my assumption that there’s a better way to automate the tests that will verify the calculations performed by the mortgage orientation tool to the test. To do so, I’ll fire up Fiddler and have it monitor the traffic that’s being sent back and forth between my browser and the application server while I interact with the orientation tool. Here’s what that looks like:

Traffic exchanged between client and server in our mortgage orientation tool

As you can see, there’s a mortgage orientation API with a Calculate operation that returns exactly those numbers that appear on the screen. See the number I marked in yellow? It’s right there in the application screenshot I showed previously. This shows that pretty much all that the front end does is performing calls to a backend API and presenting the data returned by it in a manner attractive to the end user. This means that it would not make sense to use the UI to verify the calculations. Instead, I’d advise you to mimic the API call (or sequence of calls) instead, as this will give you both faster and more accurate feedback.

To take things even further, I’d recommend you to dive into the application even deeper and see if the calculations can be covered with a decent set of unit tests. The easiest way to do this is to start talking to a developer and see if this is a possibility, and if they haven’t already done so. No need to maintain two different sets of automated checks that cover the same logic, and no need to cover logic that can be tested through unit tests with API-level checks..

Often, though, I find that writing tests like this at the API level hits the sweet spot between coverage, effort it takes to write the tests and speed of execution (and as a result, length of the feedback loop). This might be because I’m not too well versed in writing unit tests myself, but it has worked pretty well for me so far.

Deciding what to automate where: a heuristic
The above has just been one example where it would be better (as well as easier) to move specific checks from the UI level to the API level. But can we make some more generic statements about when to use UI-level checks and when to dive deeper?

Yes, we can. And it turns out, someone already did! In a recent blog post called ‘UI Test Heuristic: Don’t Repeat Your Paths‘, Chris McMahon talked about this exact subject, and the heuristic he presents in his blog post applies here perfectly:

  • Check that the end user can complete the mortgage orientation tools and is shown an indication of mortgage eligibility and associated figures > different paths through the user interface > user interface-level tests
  • Check that the figures served up by the mortgage orientation tool are correct > repeating the same paths multiple times, but with different sets of input data and expected output values > time to dive deeper

So, if you want to prevent your automated test suite from becoming too bloated with UI tests, this is a rule of thumb you can (and frankly, should) apply. As always, I’d love to hear what you think.