Remember what your tests are trying to verify

Lately, I’ve been working a lot on Selenium-based test automation solutions. And even though I’m still not overly enthusiastic about creating lots of user interface-driven automated tests, now that I’m getting more skilled at creating robust and stable Selenium tests, I am starting to appreciate the tool and what you can do with it more and more. As everybody working with Selenium can tell you, there are situations where things can get, well, interesting. And by interesting, I mean tricky. Dealing with dynamic front end frameworks and unpredictable modals and overlays can ask a lot of you in terms of debugging and exception handling skills.

Not yet being fluent in Selenese, I find myself on Google and (subsequently) StackOverflow a lot, trying to find suitable solutions for problems I encounter. While doing so, there’s one thing that I see a lot of people do, yet strikes me as really odd, given what I think is the purpose of these user interface-driven tests:

Forcing your tests to do something your users can’t.

From what I’ve understood, studying and working in the test automation field for a while now, user interface-driven tests, such as these Selenium tests, should be used to verify that the user of your user interface is able to complete a sequence of predefined actions. If you’re working in a hip and happening environment, these are often called ‘customer journeys’ or ‘user journeys’. But that’s not my point. What my point is, is that quite often I see workarounds suggested that go beyond what a regular user could do with his or her keyboard and / or mouse.

For example, take an element that is invisible until you hover over it with your mouse. If you just try to do a click(), your test will probably throw an exception stating that the element was not visible. Now, there are (at least) two ways to deal with this situation:

  1. Use the Actions class to simulate a mouseover, then click the element.
  2. Use a JavaScriptExecutor to perform the click.

While both approaches might result in a passing test, I am of the opinion that one is useful and the other is a horrible idea. Based on what I’ve written so far, can you guess which of the two options I’d suggest?

Indeed, option #1 is the way to go, for two reasons:

  • User interface-driven tests should mimic actual user interaction as closely as possible. I’ve never seen a user execute some JavaScript on a page to make an element visible. Either that, or I’m hanging around the wrong group of users..
  • What happens if a front-end developer makes a mistake (I know, they never do, but let’s assume so anyway) which causes the element not to become visible, even on a mouseover? With option #1, your test will fail, for the right reason. With #2, hello false negative!

There are some exceptions to the rule, though. The prime example I can think of is handling file uploads by directly sending the absolute path of the file to be uploaded to the input element responsible for the file upload using sendKeys(), instead of clicking on it and handling the file dialog. I’ve tried the latter before, and it’s a pain, first because you can’t do it with the standard Selenium API (because the file dialog is native to the operating system), second because different browsers use different file dialog layouts, resulting in a lot of pesky code that easily breaks down. In this case, I prefer to bypass the file dialog altogether (it’s probably not the subject of my test anyway).

In (almost) all other cases though, I’d advise you to stick to simulating your end user behavior as closely as possible. The job of your user interface-driven tests, and therefore of you as its creator, is not to force a pass, but to simulate end user interaction and see if that leads to a successfully executed scenario. Don’t fool yourself and your stakeholders by underwater tricks that obscure potential user interface issues.

Remember what your tests are trying to verify.

Data driven JavaScript tests using Jasmine

In my previous post I introduced Jasmine as a tool for testing JavaScript code. Now that I’ve had some more time to play around with it, combining it with Protractor for user interface-driven tests for one of my client’s websites, I’ve noticed that one feature was sorely missing: built-in support for data driven testing. All BDD tools (or tools that say they support BDD) that I have worked with so far, such as Cucumber, SpecFlow and JGiven, support the creation of data driven scenarios. However, Jasmine doesn’t, at least not as far as I have seen in the limited time I have been using it so far.

As I think data driven testing is key when creating maintainable and reusable scenarios, I fired up Google to check whether someone had already found a solution. From what I’ve read, there are (at least) two different ways to do data-driven testing with Jasmine. Let’s have a look at both.

Specifying your test data in an array variable
I found the first approach in the comments of this StackOverflow post. The comment submitted by peterhendrick suggests the use of a JSON file containing the test data. I slightly simplified this to use a simple array of key-value pairs and have the Jasmine spec loop through the array. You can simply populate this array any way you like, for example from an external .json file as suggested, but in any other way as well. In this example, I hard-coded it in the spec itself:

describe("Calculator (data driven tests with test data specified in array)", function() {
  var calc;
  var testdata = [{"x":1,"y":1,"sum":2},{"x":2,"y":3,"sum":5},{"x":4,"y":2,"sum":6}];
  beforeEach(function() {
    calc = new Calculator();

	for(var i in testdata) {
		it("should be able to add two numbers, using (" + testdata[i].x + "," + testdata[i].y + "," + testdata[i].sum + ")", function() {
			expect(calc.add(testdata[i].x, testdata[i].y)).toEqual(testdata[i].sum);

I particularly like this example because it enables you to add the actual values used to the test output. You can see this when you run your spec:

Test results from driving test data through an array

Data driven testing with a custom using function
This blog post from JP Castro presents an alternative way to do data driven testing in Jasmine. His solution seemed easy enough for me to apply directly to my own specs. Let’s see what it looks like.

The basic idea behind his solution is to wrap each it block in a spec with a using block that passes the parameter values to the it block:

describe("Calculator (data driven tests with using() function)", function() {
  var calc;
  beforeEach(function() {
    calc = new Calculator();

  using('parameters',[[1,1,2],[2,3,5],[4,2,6]], function(parameters){
	it("should be able to add two numbers, using (" + parameters[0] + "," + parameters[1] + "," + parameters[2] + ")", function() {

The using function he presents is implemented as follows:

function using(name, values, func) {
	for(var i = 0, count = values.length; i < count; i++) {
		if([i]) !== '[Object Array]') {
			values[i] = [values[i]];

When we run the spec above, the output looks like this:

Test results from driving test data through a using function

So, now we have two simple (and admittedly similar) workarounds for what I think is still a fundamental shortcoming in Jasmine. The code demonstrated in this blog post can be downloaded here.

Getting started with JavaScript testing using Jasmine

New project, new environment, new tools to explore! I just started working as a freelance test engineer on a new project involving some of Holland’s most visited job sites. An environment where a lot of PHP and JavaScript is used, as opposed to the familiar Java and C#. This also means I get to learn a lot of new stuff (I’ve not worked much in web development projects before) and a lot of new and interesting test automation tools and concepts to explore and apply. Which I can then use to bore you.

The first tool I get to explore is Jasmine, a BDD-style tool for testing JavaScript code. As with most other BDD tools, it’s only truly a BDD tool if you use it as such and I’m not yet convinced that we’re doing that right in this project. However, Jasmine offers enough interesting features for a blog post, so here we go..

What is Jasmine and where can I get it?
Simply put, Jasmine is a stand-alone JavaScript testing tool that supports writing your tests in a BDD-like manner. It does not require a DOM and therefore a browser to run, which makes tests run incredibly fast. Its syntax is also pretty clear and simple, as we will see in this post.

You can download the latest Jasmine release from here, after which you can unzip it. The package contains the Jasmine source (also written in JavaScript) and some sample code and tests.

The code to be tested
My example code that will be tested is a simple Car object that supports a two methods, one to add a driver to a car and one to start driving the car:

function Car() {

Car.prototype.addDriver = function() {
  if (this.hasDriver) {
	  throw new Error("Someone's already driving this car");
  if (this.isDriving) {
	  throw new Error("Can't add a driver when the car is driving");
  this.hasDriver = true;

Car.prototype.startDriving = function() {
	this.isDriving = true;

A first Jasmine test
Let’s say we want to write a simple test to see whether we can actually add a driver to a car. In Jasmine, this is done as follows:

describe("A Car", function() {

  var car;
  var person;

  beforeEach(function() {
    car = new Car();
    person = new Person();

  it("should be able to be assigned a driver", function() {

The describe keyword starts a new test suite and takes two parameters: the first is a string providing a description for the test suite, the second is a method that implements the test suite. A test suite consists of a number of tests (or specs in Jasmine terminology), each defined using the it keyword. Just as with Java testing frameworks such as JUnit and TestNG, you have the option to perform setup and teardown actions, in this case using beforeEach.

Assertions are performed using the expect keyword and a matcher, in this case toBeTruthy(). This matcher checks whether the value of a variable evaluates to true (note that values such as 1 and ‘test’ also evaluate to true in JavaScript). Jasmine comes with a rich set of matchers, which are described here.

Running your tests
Running Jasmine tests can be done by creating an HTML file that includes both the source code and the specs files in the and opening it in a browser. The HTML runner file for my demo project looks like this:

<!DOCTYPE html>
  <meta charset="utf-8">
  <title>Jasmine Car Demo Spec Runner v2.4.1</title>

  <link rel="shortcut icon" type="image/png" href="lib/jasmine-2.4.1/jasmine_favicon.png">
  <link rel="stylesheet" href="lib/jasmine-2.4.1/jasmine.css">

  <!-- include Jasmine -->
  <script src="lib/jasmine-2.4.1/jasmine.js"></script>
  <script src="lib/jasmine-2.4.1/jasmine-html.js"></script>
  <script src="lib/jasmine-2.4.1/boot.js"></script>

  <!-- include source files -->
  <script src="src/Car.js"></script>
  <script src="src/Person.js"></script>

  <!-- include spec files -->
  <script src="spec/SpecHelper.js"></script>
  <script src="spec/CarSpec.js"></script>

When we open this HTML file in our browser, the specs are run and we can see the following result:

Results of first Jasmine test run


By the way, if the assertion would fail, the Jasmine output would look like this:

Example of a failing Jasmine check

Checking for expected errors
Returning to the code for the Car object, we can see that some methods throw errors. If you want to check that these errors are indeed thrown, you can do that in Jasmine like this:

it("should throw an error when assigned a second driver", function() {
  expect(function() {
  }).toThrowError("Someone's already driving this car");

Adding and using custom matchers
Even though Jasmine includes a variety of matchers to perform all kinds of checks, sometimes you might have to perform very specific checks not covered by one of the standard matchers. In this case, it’s also possible to write a custom matcher (which is called toBeDriving in this case):

beforeEach(function () {
    toBeDriving: function () {
      return {
        compare: function (actual, expected) {
          var car = actual;

          return {
            pass: car.isDriving

This custom matcher can then be used in Jasmine specs just like any other matcher:

it("should be able to start driving", function() {

Jasmine offers lots more features, such as the ability to write custom reporters. Check it out!

The example project I have used in this blog post can be downloaded here. It includes Jasmine 2.4.1 and a working HTML runner, so you can start experimenting with Jasmine right away.