An experiment in creating better tool-centered automation training

Last week, I delivered the second part of a two-evening API testing training at my former employer. They contacted me a while ago to see if I could help them in offering test automation training for their employees, as well as for their clients and other contacts. When I was still working with them, I used to deliver automation training as well, and it felt really great to be asked back, even though I have left them almost three years ago now.

But that’s not what this post is about.

This API testing training was in some ways an experiment I have wanted to conduct for a while now. I see a lot of individuals and organizations offering automation training, and most of it is specific to one single tool. In itself, this isn’t inherently a bad thing, but I’ve got one big problem with a lot of these tool-centered courses: instead of teaching you how to create a sound automation approach around one or more tools, they simply go through the most important features of a single tool and teach the participants some (sometimes useful) tricks. I know. I’ve delivered those courses in the past as well. If you’re a tool vendor / creator, I can understand why you would want to do that. But I think there’s more to good automation education than teaching you all the ins and outs of a specific tool. Let’s call them the 3 C’s:

  • Context – A tool is often only useful in a specific context. This context includes the skills of the people that will use the tool, the development and delivery process that the tool is to be made part of, and much more. Without context, it’s very hard to decide if a specific tool is the right one for the job.
  • Competition – For nearly all tools out there, there’s at least one competitor on the market (but often much more) that can be used to complete the same task. It is therefore essential for good training to introduce more than one option, have the participants get some hands on experience with all of them and let them decide what would work best for their tasks and team.
  • Cutting the crap – Tool-specific training might give the idea that the tool the participants are being trained in is the best thing since sliced bread. Which in turn leads people to try and automate anything and everything with a single tool. Which in turn all too often leads to crap. In other words: what’s the point in knowing all different types of waits available in Selenium if you don’t know how to decide what is a good scenario to automate using Selenium in the first place?

So, instead of delivering my API testing training around REST Assured alone (which I’ve done a number of times in the past), I decided to introduce three different tools to the participants: REST Assured, the open source version of SmartBear SoapUI and Parasoft SOAtest. After an introduction into what constitutes API testing, why it is useful and what you can test using APIs, I let the participants create a number of basic API tests with each of these tools (pretty much the same tests three times over), so they could experience firsthand how the features provided by each of the tools compare. Moreover, since I chose tools that are at opposite ends of the API test tool spectrum (REST Assured is a Java library for RESTful APIs, SOAtest is a commercially licensed enterprise-grade tool that supports a wide variety of protocols and message types, with SoapUI somewhere in between), participants get a much broader view of API testing than they’d get by learning REST Assured alone.

The feedback I received afterwards confirmed what I hoped to achieve with my experiment: all the attendees thought it was great to see more than a single tool, and since I gave them pointers to material for further exploration, they could decide for themselves in which direction their further education will take them.

I recently launched another course in which I try to do something similar, although in another fashion: instead of teaching people how to use Selenium WebDriver (i.e., teaching them the API and some useful Selenium-only tricks), I explain what types of tests should be created using Selenium, and I teach them how I would approach creating readable, maintainable and reliable tests with Selenium, Cucumber/SpecFlow, JUnit/NUnit and ExtentReports. Again: providing context and cutting the crap (you can argue about whether or not I’m covering ‘competition’ in this one) instead of teaching people all of the methods and features of a single tool.

I hope to deliver this type of automation training much more often in the future, and I’d love to see other automation training providers follow suit. For those of you who’d like some more details on the training objectives and subjects covered in this API testing training, please click here.

As always, I’d love to hear your thoughts. What to you constitutes good automation training?

Extending my solution with API testing capabilities, and troubles with open source projects

In last week’s blog post, I introduced how I would approach creating a solution for creating and executing automated user interface-driven tests. In this blog post, I’m going to extend the capabilities of my solution by adding automated tests that exercise a RESTful API.

Returning readers might know that I’m a big fan of REST Assured. However, since that’s a Java-based DSL, and my solution is written in C#, I can’t use REST Assured. I’ve spent some time looking at alternatives and decided upon using RestSharp for this example. Why RestSharp? Because it has a clean and fairly readable API, which makes it easy to use, even for non-wizardlike programmers, such as yours truly. This is a big plus for me when creating test automation solutions, because, as a consultant, there will always come a moment where you need to hand over your solution to somebody else. And that somebody might be just as inexperienced when it comes to programming as myself, so I think it’s important to use tools that are straightforward and easy to use while still powerful enough to perform the required tasks. RestSharp ticks those boxes fairly well.

A sample feature and scenario for a RESTful API
Again, we start with the top level of our test implementation: the feature and scenario(s) that describe the required behaviour. Here goes:

Feature: CircuitsApi
	In order to impress my friends
	As a Formula 1 fan
	I want to know the number of races for a given Formula 1 season

@api
Scenario Outline: Check the number of races in a season
	Given I want to know the number of Formula One races in <season>
	When I retrieve the circuit list for that season
	Then there should be <numberOfCircuits> circuits in the list returned
	Examples:
	| season | numberOfCircuits |
	| 2017   | 20               |
	| 2016   | 21               |
	| 1966   | 9                |
	| 1950   | 8                |

Note that I’ve added a tag @api to the scenario. This is so I can prevent my solution from starting a browser for these API tests as well (which would just slow down test execution) by writing dedicated setup and teardown methods that execute only for scenarios with a certain tag. This can be done real easy with SpecFlow. See the GitHub page for the solution for more details.

The step definitions
So, how are the above scenario steps implemented? In the Given step, I handle creating the RestClient that is used to send the HTTP request and intercept the response, as well as setting the path parameter specifying the specific year for which I want to check the number of races:

private RestClient restClient;
private RestRequest restRequest;
private IRestResponse restResponse;

[Given(@"I want to know the number of Formula One races in (.*)")]
public void GivenIWantToKnowTheNumberOfFormulaOneRacesIn(string season)
{
    restClient = new RestClient(Constants.ApiBaseUrl); //http://ergast.com/api/f1

    restRequest = new RestRequest("{season}/circuits.json", Method.GET);

    restRequest.AddUrlSegment("season", season);
}

The When step is even more straightforward: all is done here is executing the RestClient and storing the response in the IRestResponse:

[When(@"I retrieve the circuit list for that season")]
public void WhenIRetrieveTheCircuitListForThatSeason()
{
    restResponse = restClient.Execute(restRequest);
}

Finally, in the Then step, we parse the response to get the JSON field value we’re interested in and check whether it matches the expected value. In this case, we’re not interested in a field value, though, but rather in the number of times a field appears in the response (in this case, the length of the array of circuits). And, obviously, we want to report the result of our check to the ExtentReports report we’re creating during test execution:

[Then(@"there should be (.*) circuits in the list returned")]
public void ThenThereShouldBeCircuitsInTheListReturned(int numberOfSeasons)
{
    dynamic jsonResponse = JsonConvert.DeserializeObject(restResponse.Content);

    JArray circuitsArray = jsonResponse.MRData.CircuitTable.Circuits;

    OTAAssert.AssertEquals(null, test, circuitsArray.Count, numberOfSeasons, "The actual number of circuits in the list is equal to the expected value " + numberOfSeasons.ToString());
}

Basically, what we’re doing here is deserializing the JSON response and storing it into a dynamic object. I wasn’t familiar with the dynamic concept before, but it turns out to be very useful here. The dynamic type can be used for objects of which you don’t know the structure until runtime, which holds true here (we don’t know what the JSON response looks like). Then, we can simply traverse the dynamic jsonResponse until we get to the field we need for our check. It may not be the best or most reusable solution, but it definitely shows the power of the C# language here.

The trouble with RestSharp
As you can see, with RestSharp, it’s really easy to write tests for RESTful APIs and add them to our solution. There’s one problem though, and that’s that RestSharp no longer seems to be actively maintained. The most recent version of RestSharp was released on August 26 of 2015, more than a year and a half ago. There’s no response to the issues posted on GitHub, either, which also doesn’t bode very well for the liveliness of the project. For me, when deciding whether or not to use an open source project, this is a big red flag.

One alternative to RestSharp I found was RestAssured.Net. This project looks like an effort to port the original REST Assured to C#. It looks useful enough, however, it suffers from the same problem that RestSharp does: no activity. For me, that’s enough reason to discard it.

Just before writing this post, I was made aware of yet another possible solution, called Flurl. This does look like a promising alternative, but unfortunately I didn’t have the time to try it out for myself before the due date of this blog post. I’ll check it out during the week and if it lives up to its promising appearance, Flurl stands a good chance of being the topic for next week’s blog post. Until then, you can find my RestSharp implementation of the RESTful API tests on the GitHub page of my solution.

Three reasons to start improving your API test automation skills

Modern applications and software development methods have changed the requirements for testers and the skills they need to possess to add real value to their clients and projects. One of these emerging and sought after skills is the ability to design and execute automated tests for APIs. In this post, I will give you three reasons why it might be useful for you to start improving your API test automation skills.

API word cloud

APIs are everywhere
The first reason why you should invest in your API test automation skills is a simple question of demand and supply: APIs are becoming ever more present in current IT solutions. From mobile applications to the Internet of Things, many modern systems and applications expose their data, their business logic or both through APIs. Whether you’re building an application that uses APIs to expose data or logic to the outside world, or you’re on the other side as an API consumer, you need to be able to perform tests on APIs. Otherwise, how are you going to ensure that an API and its integration with the outside world function as expected?

Oh, and if you’re testing an application that consumes a third-party API, please don’t fall into the trap of assuming that this API you’re using works perfectly as designed, or that integration with your own application will be seamless. Can anyone really assure you that that business-critical third party API you’re relying on has been tested for your specific situation and requirements? Thought not. Make sure it does and do some proper testing on it!

API test automation hits the sweet spot between speed and coverage
The second reason why API test automation can be very useful is that automated checks on the API level hit the sweet spot between speed of execution and coverage of application features. Compared to the two other types of automated tests in the test automation pyramid, API-level tests tend to:

  • execute faster than user interface-driven tests. User interface-driven automated tests, such as those written in Selenium WebDriver, need to fire up a browser and render several web pages every time a test is executed. When your tests go through a lot of different pages, execution time skyrockets. API-level tests, on the other hand, have to wait for a server responding to HTTP calls only. The only client-side processing that needs to be done is parsing the response and performing validation checks, for example on specific elements in the response. This is a lot faster than sending (possibly many) HTTP requests to a web server to fetch all objects required for a web page and then waiting until your browser has finished rendering the page.
  • cover more business logic than unit tests. Yes, you should have unit tests. And they should cover as much of the internal workings of your application that make up the business logic. However, there’s only so much unit tests can do. For example, unit tests can check whether the salary of a given employee is calculated correctly in the back end. They cannot guarantee that the same salary is correctly sent out to the front end layer of your application (or to the IRS) upon request, though. For this, you will need to perform tests at a higher level in your application, and API tests are usually perfect for that.

Automated API tests tend to be more reliable
Apart from having the right mixture of speed of execution and coverage of functional aspects, API-level automated checks have another big advantage over user interface-driven automated tests: they’re usually far more reliable. User interface-driven tests constantly have to walk the wobbly rope of synchronization, ever changing (or ‘improving’) user interface designs, dynamic element identification methods, etc. API definitions and interfaces on the other hand are amongst the most stable parts of an application: they follow standardized specification formats (such as WSDL for SOAP or WADL, RAML or Swagger for REST) and once agreed upon, an API does not usually change all that much. This applies especially to outward-facing APIs. For example, Google can easily change its Gmail user interface without this impacting end users (apart from annoyance, maybe, but that’s a different story altogether), but sudden radical changes to its Gmail API would render a lot of third-party applications that have Gmail integration useless. Therefore, changes to an API will usually be a lot fewer and further between, resulting in less maintenance required for your API-level automated tests.

A sample API test in REST Assured

Further reading
I hope this blog post has given you some insight into why I think API test automation skills are a valuable asset for any tester with an interest in test automation. If you want to read more on API testing and test automation, I highly recommend the API Testing Dojo on the SoapUI website. Additionally, you can also check out my other posts on API testing here.