Writing tests for RESTful APIs in Python using requests – part 1: basic tests

Recently, I’ve delivered my first ever three day ‘Python for testers’ training course. One of the topics that was covered in this course is writing tests for RESTful APIs using the Python requests library and the pytest unit testing framework.

In this short series of blog posts, I want to explore the Python requests library and how it can be used for writing tests for RESTful APIs. This first blog post is all about getting started and writing our first tests against a sample REST API.

Getting started
To get started, first we need a recent installation of the Python interpreter, which can be downloaded here. We then need to create a new project in our IDE (I use PyCharm, but any decent IDE works) and install the requests library. The easiest way to do this is using pip, the Python package manager:

 pip install -U requests 

Don’t forget to create and activate a virtual environment if you prefer that setup. We’ll also need a unit testing framework to provide us with a test runner, an assertion library and some basic reporting functionality. I prefer pytest, but requests works equally well with other Python unit testing frameworks.

 pip install -U pytest 

Then, all we need to do to get started is to create a new Python file and import the requests library using

import requests

Our API under test
For the examples in this blog post, I’ll be using the Zippopotam.us REST API. This API takes a country code and a zip code and returns location data associated with that country and zip code. For example, a GET request to http://api.zippopotam.us/us/90210

returns an HTTP status code 200 and the following JSON response body:

{
     "post code": "90210",
     "country": "United States",
     "country abbreviation": "US",
     "places": [
         {
             "place name": "Beverly Hills",
             "longitude": "-118.4065",
             "state": "California",
             "state abbreviation": "CA",
             "latitude": "34.0901"
         }
     ]
 }

A first test using requests and pytest
As a first test, let’s use the requests library to invoke the API endpoint above and write an assertion that checks that the HTTP status code equals 200:

def test_get_locations_for_us_90210_check_status_code_equals_200():
     response = requests.get("http://api.zippopotam.us/us/90210")
     assert response.status_code == 200

What’s happening here? In the first line of the test, we call the get() method in the requests library to perform an HTTP GET call to the specified endpoint, and we store the entire response in a variable called response. We then extract the status_code property from the response object and write an assertion, using the pytest assert keyword, that checks that the status code is equal to 200, as expected.

That’s all there is to a first, and admittedly very basic, test against our API. Let’s run this test and see what happens. I prefer to do this from the command line, because that’s also how we will run the tests once they’re part of an automated build pipeline. We can do so by calling pytest and telling it where to look for test files. Using the sample project referenced at the end of this blog post, and assuming we’re in the project root folder, calling

 pytest tests\01_basic_tests.py 

results in the following console output:

Console output showing a passing test.

It looks like our test is passing. Since I never trust a test I haven’t seen fail (and neither should you), let’s change the expected HTTP status code from 200 to 201 and see what happens:

Console output showing a failing test.

That makes our test fail, as you can see. It looks like we’re good to go with this one.

Extending our test suite
Typically, we’ll be interested in things other than the response HTTP status code, too. For example, let’s check if the value of the response content type header correctly identifies that the response body is in JSON format:

def test_get_locations_for_us_90210_check_content_type_equals_json():
     response = requests.get("http://api.zippopotam.us/us/90210")
     assert response.headers['Content-Type'] == "application/json"

In the response object, the headers are available as a dictionary (a list of key-value pairs) headers, which makes extracting the value for a specific header a matter of supplying the right key (the header name) to obtain its value. We can then assert on its value using a pytest assertion and the expected value of ‘application/json‘.

How about checking the value of a response body element? Let’s first check that the response body element country (see the sample JSON response above) is equal to United States:

def test_get_locations_for_us_90210_check_country_equals_united_states():
     response = requests.get("http://api.zippopotam.us/us/90210")
     response_body = response.json()
     assert response_body["country"] == "United States"

The requests library comes with a built-in JSON decoder, which we can use to extract the response body from the response object and turn it into a proper JSON object. It is invoked using the json() method, which will raise a ValueError if there is no response body at all, as well as when the response is not valid JSON.

When we have decoded the response body into a JSON object, we can access elements in the body by referring to their name, in this case country.

To extract and assert on the value of the place name for the first place in the list of places, for example, we can do something similar:

def test_get_locations_for_us_90210_check_city_equals_beverly_hills():
     response = requests.get("http://api.zippopotam.us/us/90210")
     response_body = response.json()
     assert response_body["places"][0]["place name"] == "Beverly Hills"

As a final example, let’s check that the list of places returned by the API contains exactly one entry:

def test_get_locations_for_us_90210_check_one_place_is_returned():
     response = requests.get("http://api.zippopotam.us/us/90210")
     response_body = response.json()
     assert len(response_body["places"]) == 1

This, too, is straightforward after we’ve converted the response body to JSON. The len() method that is built into Python returns the length of a list, in this case the list of items that is the value of the places element in the JSON document returned by the API.

In the next blog post, we’re going to explore creating data driven tests using pytest and requests.

Using the examples for yourself
The code examples I have used in this blog post can be found on my GitHub page. If you download the project and (given you have installed Python properly) run

 pip install -r requirements.txt 

from the root of the python-requests project to install the required libraries, you should be able to run the tests for yourself. See you next time!

Data driven testing in C# with NUnit and RestSharp

In a previous post, I gave some examples of how to write some basic tests in C# for RESTful APIs using NUnit and the RestSharp library. In this post, I would like to extend on that a little by showing you how to make these tests data driven.

For those of you that do not know what I mean with ‘data driven’: when I want to run tests that exercise the same logic or flow in my application under test multiple times with various combinations of input values and corresponding expected outcomes, I call that data driven testing.

This is especially useful when testing RESTful APIs, since these are all about sending and receiving data as well as exposing business logic to other layers in an application architecture (such as a graphical user interface) or to other applications (consumers of the API).

As a starting point, consider these three tests, written using RestSharp and NUnit:

[TestFixture]
public class NonDataDrivenTests
{
    private const string BASE_URL = "http://api.zippopotam.us";

    [Test]
    public void RetrieveDataForUs90210_ShouldYieldBeverlyHills()
    {
        // arrange
        RestClient client = new RestClient(BASE_URL);
        RestRequest request = 
            new RestRequest("us/90210", Method.GET);

        // act
        IRestResponse response = client.Execute(request);
        LocationResponse locationResponse =
            new JsonDeserializer().
            Deserialize<LocationResponse>(response);

        // assert
        Assert.That(
            locationResponse.Places[0].PlaceName,
            Is.EqualTo("Beverly Hills")
        );
    }

    [Test]
    public void RetrieveDataForUs12345_ShouldYieldSchenectady()
    {
        // arrange
        RestClient client = new RestClient(BASE_URL);
        RestRequest request =
            new RestRequest("us/12345", Method.GET);

        // act
        IRestResponse response = client.Execute(request);
        LocationResponse locationResponse =
            new JsonDeserializer().
            Deserialize<LocationResponse>(response);

        // assert
        Assert.That(
            locationResponse.Places[0].PlaceName,
            Is.EqualTo("Schenectady")
        );
    }

    [Test]
    public void RetrieveDataForCaY1A_ShouldYieldWhiteHorse()
    {
        // arrange
        RestClient client = new RestClient(BASE_URL);
        RestRequest request = 
            new RestRequest("ca/Y1A", Method.GET);

        // act
        IRestResponse response = client.Execute(request);
        LocationResponse locationResponse =
            new JsonDeserializer().
            Deserialize<LocationResponse>(response);

        // assert
        Assert.That(
            locationResponse.Places[0].PlaceName,
            Is.EqualTo("Whitehorse")
        );
    }
}

Please note that the LocationResponse type is a custom type I defined myself, see the GitHub repository for this post for its implementation.

These tests are a good example of what I wrote about earlier: I’m invoking the same logic (retrieving location data based on a country and zip code and then verifiying the corresponding place name from the API response) three times with different sets of test data.

This quickly gets very inefficient when you add more tests / more test data combinations, resulting in a lot of duplicated code. Luckily, NUnit provides several ways to make these tests data driven. Let’s look at two of them in some more detail.

Using the [TestCase] attribute

The first way to create data driven tests is by using the [TestCase] attribute that NUnit provides. You can add multiple [TestCase] attributes for a single test method, and specify the combinations of input and expected output parameters that the test method should take.

Additionally, you can specify other characteristics for the individual test cases. One of the most useful ones is the TestName property, which can be used to provide a legible and useful name for the individual test case. This name also turns up in the reporting, so I highly advise you to take the effort to specify one.

Here’s what our code looks like when we refactor it to use the [TestCase] attribute:

[TestFixture]
public class DataDrivenUsingAttributesTests
{
    private const string BASE_URL = "http://api.zippopotam.us";

    [TestCase("us", "90210", "Beverly Hills", TestName = "Check that US zipcode 90210 yields Beverly Hills")]
    [TestCase("us", "12345", "Schenectady", TestName = "Check that US zipcode 12345 yields Schenectady")]
    [TestCase("ca", "Y1A", "Whitehorse", TestName = "Check that CA zipcode Y1A yields Whitehorse")]
    public void RetrieveDataFor_ShouldYield
        (string countryCode, string zipCode, string expectedPlaceName)
    {
        // arrange
        RestClient client = new RestClient(BASE_URL);
        RestRequest request =
            new RestRequest($"{countryCode}/{zipCode}", Method.GET);

        // act
        IRestResponse response = client.Execute(request);
        LocationResponse locationResponse =
            new JsonDeserializer().
            Deserialize<LocationResponse>(response);

        // assert
        Assert.That(
            locationResponse.Places[0].PlaceName,
            Is.EqualTo(expectedPlaceName)
        );
    }
}

Much better! We now only have to define our test logic once, and NUnit takes care of iterating over the values defined in the [TestCase] attributes:

NUnit test results for the data driven tests using [TestCase] attributes

There are some downsides to using the [TestCase] attributes, though:

  • It’s all good when you just want to run a small amount of test iterations, but when you want to / have to test for larger numbers of combinations of input and output parameters, your code quickly gets messy (on a side note, if this is the case for you, try looking into property-based testing instead of the example-based testing we’re doing here).
  • You still have to hard code your test data in your code, which might give problems with scaling and maintaining your tests in the future.

This is where the [TestCaseSource] attribute comes in.

Using the [TestCaseSource] attribute

If you want to or need to work with larger numbers of combinations of test data and/or you want to be able to specify your test data outside of your test class, then using [TestCaseSource] might be a useful option to explore.

In this approach, you specify or read the test data in a separate method, which is then passed to the original test method. NUnit will take care of iterating over the different combinations of test data returned by the method that delivers the test data.

Here’s an example of how to apply [TestCaseSource] to our tests:

[TestFixture]
public class DataDrivenUsingTestCaseSourceTests
{
    private const string BASE_URL = "http://api.zippopotam.us";

    [Test, TestCaseSource("LocationTestData")]
    public void RetrieveDataFor_ShouldYield
        (string countryCode, string zipCode, string expectedPlaceName)
    {
        // arrange
        RestClient client = new RestClient(BASE_URL);
        RestRequest request =
            new RestRequest($"{countryCode}/{zipCode}", Method.GET);

        // act
        IRestResponse response = client.Execute(request);
        LocationResponse locationResponse =
            new JsonDeserializer().
            Deserialize<LocationResponse>(response);

        // assert
        Assert.That(
            locationResponse.Places[0].PlaceName,
            Is.EqualTo(expectedPlaceName)
        );
    }

    private static IEnumerable<TestCaseData> LocationTestData()
    {
        yield return new TestCaseData("us", "90210", "Beverly Hills").
            SetName("Check that US zipcode 90210 yields Beverly Hills");
        yield return new TestCaseData("us", "12345", "Schenectady").
            SetName("Check that US zipcode 12345 yields Schenectady");
        yield return new TestCaseData("ca", "Y1A", "Whitehorse").
            SetName("Check that CA zipcode Y1A yields Whitehorse");
    }
}

In this example, we specify our test data in a separate method LocationTestData(), and then tell the test method to use that method as the test data source using the [TestDataSource] attribute, which takes as its argument the name of the test data method.

For clarity, the test data is still hard coded in the body of the LocationTestData() method, but that’s not mandatory. You could just as easily write a method that reads the test data from any external source, as long as the test data method is static and returns an object of type IEnumerable, or any object that implements this interface.

Also, since the [TestCase] and [TestCaseSource] attributes are features of NUnit, and not of RestSharp, you can apply the principles illustrated in this post to other types of tests just as well.

Beware, though, before you use them for user interface-driven testing with tools like Selenium WebDriver. Chances are that you’re falling for a classic case of ‘just because you can, doesn’t mean you should’. I find data driven testing with Selenium WebDriver to be a test code smell: if you’re going through the same screen flow multiple times, and the only variation is in the test data, there’s a high chance that there’s a more efficient way to test the same underlying business logic (for example by leveraging APIs).

Chris McMahon explains this much more eloquently in a blog post of his. I highly recommend you reading that.

For other types of testing (API, unit, …), data driven testing could be a very powerful way to make your test code better maintainable and more powerful.

All example code in this blog post can be found on this GitHub page.

Writing API tests in Python with Tavern

So far, most of the blog posts I’ve written that covered specific tools were focused on either Java or C#. Recently, though, I got a request for test automation training for a group of data science engineers, with the explicit requirement to use Python-based tools for the examples and exercises.

Since then, I’ve been slowly expanding my reading and learning to also include the Python ecosystem, and I’ve also included a couple of Python-based test automation courses in my training offerings. So far, I’m pretty impressed. There are plenty of powerful test tools available for Python, and in this post, I’d like to take a closer look at one of them, Tavern.

Tavern is an API testing framework running on top of pytest, one of the most popular Python unit testing frameworks. It offers a range of features to write and run API tests, and if there’s something you can’t do with Tavern, it claims to be easily extensible through Python or pytest hooks and features. I can’t vouch for its extensibility yet, thought, since all that I’ve been doing with Tavern so far was possible out of the box. Tavern has good documentation too, which is also nice.

Installing Tavern on your machine is easiest when done through pip, the Python package installer and manager using the command

pip install -U tavern

Tests in Tavern are written in YAML files. Now, you either love it or hate it, but it works. To get started, let’s write a test that retrieves location data for the US zip code 90210 from the Zippopotam.us API and checks whether the response HTTP status code is equal to 200. This is what that looks like in Tavern:

test_name: Get location for US zip code 90210 and check response status code

stages:
  - name: Check that HTTP status code equals 200
    request:
      url: http://api.zippopotam.us/us/90210
      method: GET
    response:
      status_code: 200

As I said, Tavern runs on top of pytest. So, to run this test, we need to invoke pytest and tell it that the tests we want to run are in the YAML file we created:

As you can see, the test passes.

Another thing you might be interested in is checking values for specific response headers. Let’s check that the response content type is equal to ‘application/json’, telling the API consumer that they need to interpret the response as JSON:

test_name: Get location for US zip code 90210 and check response content type

stages:
  - name: Check that content type equals application/json
    request:
      url: http://api.zippopotam.us/us/90210
      method: GET
    response:
      headers:
        content-type: application/json

Of course, you can also perform checks on the response body. Here’s an example that checks that the place name associated with the aforementioned US zip code 90210 is equal to ‘Beverly Hills’:

test_name: Get location for US zip code 90210 and check response body content

stages:
  - name: Check that place name equals Beverly Hills
    request:
      url: http://api.zippopotam.us/us/90210
      method: GET
    response:
      body:
        places:
          - place name: Beverly Hills

Since APIs are all about data, you might want to repeat the same test more than once, but with different values for input parameters and expected outputs (i.e., do data driven testing). Tavern supports this too by exposing the pytest parametrize marker:

test_name: Check place name for multiple combinations of country code and zip code

marks:
  - parametrize:
      key:
        - country_code
        - zip_code
        - place_name
      vals:
        - [us, 12345, Schenectady]
        - [ca, B2A, North Sydney South Central]
        - [nl, 3825, Vathorst]

stages:
  - name: Verify place name in response body
    request:
      url: http://api.zippopotam.us/{country_code}/{zip_code}
      method: GET
    response:
      body:
        places:
          - place name: "{place_name}"

Even though we specified only a single test with a single stage, because we used the parametrize marker and supplied the test with three test data records, pytest effectively runs three tests (similar to what @DataProvider does in TestNG for Java, for example):

Tavern output for our data driven test, run from within PyCharm

So far, we have only performed GET operations to retrieve data from an API provider, so we did not need to specify any request body contents. When, as an API consumer, you want to send data to an API provider, for example when you perform a POST or a PUT operation, you can do that like this using Tavern:

test_name: Check response status code for a very simple addition API

stages:
  - name: Verify that status code equals 200 when two integers are specified
    request:
      url: http://localhost:5000/add
      json:
        first_number: 5
        second_number: 6
      method: POST
    response:
      status_code: 200

This test will POST a JSON document

{'first_number': 5, 'second_number': 6}

to the API provider running on localhost port 5000. Please note that for obvious reasons this test will fail when you run it yourself, unless you built an API or a mock that behaves in a way that makes the test pass (great exercise, different subject …).

So, that’s it for a quick introduction to Tavern. I quite like the tool for its straightforwardness. What I’m still wondering is whether working with YAML will lead to maintainability and readability issues when you’re working with larger test suites and larger request or response bodies. I’ll keep working with Tavern in my training courses for now, so a follow-up blog post might see the light of day in a while!

All examples can be found on this GitHub page.