Defining test steps and object properties in an Excel Data Source for Selenium

Recently, one of the readers of this blog sent me an email asking whether it was possible to define not only input and validation parameters in an external data source (such as an Excel file), but also the test steps to be taken and the object properties on which these steps need to be performed by Selenium WebDriver. As it happens, I have been playing around with this idea for a while in the past, but until now I’ve never gotten round to writing a blog post on it.

The Excel Data Source
As an example, we are going to perform a search query on Google and validate the number of results we get from this search query. This scripts consists of the following steps:

  1. Open a new browser instance
  2. Navigate to the Google homepage
  3. Type the search query in the text box
  4. Click on the search button
  5. Validate the contents of the web element displaying the number of search results against our expected value
  6. Close the browser instance

For each step, we are going to define (if applicable):

  • A keyword identifying the action to be taken
  • A parameter value that is used in the action (note that for example a type action takes a parameter, a click action does not)
  • The attribute type that uniquely identifies the object on which the action is performed
  • The corresponding attribute value

For our example test script, the Excel Data Source to be used could look something like this:
Excel Data Source
Reading the Data Source
Similar to the concept explained in a previous post, first we need to read the values from the Excel sheet in order to execute the test steps defined. We need to be a little extra careful here as some cells might be empty and the methods we use do not particularly like reading values from cells that do not exist.

public static void main (String args[]) {

		String action = "";
		String value = "";
		String attribute = "";
		String attrval = "";
		
		try {
			// Open the Excel file for reading
			FileInputStream fis = new FileInputStream("C:\\Tools\\testscript.xls");
			// Open it for writing too
			FileOutputStream fos = new FileOutputStream("C:\\Tools\\testscript.xls");
			// Access the required test data sheet
			HSSFWorkbook wb = new HSSFWorkbook(fis);
			HSSFSheet sheet = wb.getSheet("steps");
			// Loop through all rows in the sheet
			// Start at row 1 as row 0 is our header row
			for(int count = 1;count<=sheet.getLastRowNum();count++){
				HSSFRow row = sheet.getRow(count);
				System.out.println("Running test step " + row.getCell(0).toString());

				// Run the test step for the current test data row
				if(!(row.getCell(1) == null || row.getCell(1).equals(Cell.CELL_TYPE_BLANK))) {
					action = row.getCell(1).toString();
				} else {
					action = "";
				}

				if(!(row.getCell(2) == null || row.getCell(2).equals(Cell.CELL_TYPE_BLANK))) {
					value = row.getCell(2).toString();
				} else {
					value = "";
				}

				if(!(row.getCell(3) == null || row.getCell(3).equals(Cell.CELL_TYPE_BLANK))) {
					attribute = row.getCell(3).toString();
				} else {
					attribute = "";
				}

				if(!(row.getCell(4) == null || row.getCell(4).equals(Cell.CELL_TYPE_BLANK))) {
					attrval = row.getCell(4).toString();
				} else {
					attrval = "";
				}

				System.out.println("Test action: " + action);
				System.out.println("Parameter value: " + value);
				System.out.println("Attribute: " + attribute);
				System.out.println("Attribute value: " + attrval);
				
				String result = runTestStep(action,value,attribute,attrval);
				
				// Write the result back to the Excel sheet
				row.createCell(5).setCellValue(result);
				
			}
			
			// Save the Excel sheet and close the file streams
			wb.write(fos);
			fis.close();
			fos.close();
			
		} catch (Exception e) {
			System.out.println(e.toString());
		}
}

We do not read the value from the Result column, but rather we are going to execute the test steps using the values from the other columns, determine the result and write this back to the Excel sheet. In this way, we have a rudimentary logging function built in directly into our framework. Neat, right?

Defining and executing test steps
Next, we need to implement the generic runTestStep method we use to execute the test steps we have defined in our Data Source. This can be done pretty straightforward by looking at the current Action keyword and then executing the necessary steps for that keyword.

public static String runTestStep(String action, String value, String attribute, String attrval) throws Exception {

		switch(action.toLowerCase()) {
		case "openbrowser":
			switch(value.toLowerCase()) {
			case "firefox":
				driver = new FirefoxDriver();
				driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
				return "OK";
			case "htmlunit":
				driver = new HtmlUnitDriver();
				driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
				return "OK";
			default:
				return "NOK";
			}
		case "navigate":
			driver.get(value);
			return "OK";
		case "type":
			try {
				WebElement element = findMyElement(attribute,attrval);
				element.sendKeys(value);
				return "OK";
			} catch (Exception e) {
				System.out.println(e.toString());
				return "NOK";
			}
		case "click":
			try {
				WebElement element = findMyElement(attribute,attrval);
				element.click();
				return "OK";
			} catch (Exception e) {
				System.out.println(e.toString());
				return "NOK";
			}
		case "validate":
			try {
				WebElement element = findMyElement(attribute,attrval);
				if (element.getText().equals(value)) {
					return "OK";
				} else {
					System.out.println("Actual value: " + element.getText() + ", expected value: " + value);
					return "NOK";
				}
			} catch (Exception e) {
				System.out.println(e.toString());
			}
		case "closebrowser":
			driver.quit();
			return "OK";
		default:
			throw new Exception("Unknown keyword " + action);
		}
}

This setup makes it very easy to add other keywords (e.g., for selecting a value from a dropdown list), both for standardized browser actions as well as for custom keywords. Every step returns a result (here either OK or NOK) to indicate whether the action has been executed successfully. This is written back to the Excel sheet in our main method, so we can see whether our test steps have been executed correctly.

Finding the required object
I’ve also used a helper method findMyElement that, given an attribute and an attribute value, returns a WebElement that corresponds to these values. Its implementation is quite rudimentary, but it should be easy for you to extend it and make it more fail safe. I’m lazy sometimes. I haven’t even implemented all types of selectors that Selenium can handle!

public static WebElement findMyElement(String attribute, String attrval) throws Exception {

		switch(attribute.toLowerCase()) {
		case "id":
			return driver.findElement(By.id(attrval));
		case "name":
			return driver.findElement(By.name(attrval));
		case "xpath":
			return driver.findElement(By.xpath(attrval));
		case "css-select":
			return driver.findElement(By.cssSelector(attrval));
		default:
			throw new Exception("Unknown selector type " + attribute);
		}
}

Now, when I run the test, my code nicely reads all rows from Excel, determines what test action needs to be performed, executes it and reports the result back to Excel. After running the test, the Excel sheet looks like this:
Excel Data Source including test results
The only test step that fails does so because it Google doesn’t always return the same number of test results in the same amount of time, obviously. Nevertheless, it illustrates the added value of writing back the result of every step to the Excel sheet. Of course, you can very easily extend this to include a useful error message and even a link to a screenshot.

The Eclipse project containing the code I’ve used to create this example can be downloaded here.

Up and running with: JBehave

This is the fourth article in our series on new, popular or otherwise interesting tools used in test automation. You can read all posts within this series by clicking here.

What is JBehave?
From the JBehave.org website: JBehave is a framework for Behaviour-Driven Development (BDD). BDD is an evolution of test-driven development (TDD) and acceptance-test driven design, and is intended to make these practices more accessible and intuitive to newcomers and experts alike. It shifts the vocabulary from being test-based to behaviour-based, and positions itself as a design philosophy.

Using JBehave follows these five simple steps, which we will follow in the examples provided in this article as well:

  1. Write your user story
  2. Map the steps in the user story to Java code
  3. Configure your user stories
  4. Run your JBehave tests
  5. Review the test results

Where can I get JBehave?
JBehave can be downloaded from this site.

How do I install and configure JBehave?
JBehave consists of a number of .jar files. All you need to do is to add these .jar files as a dependency to your Java project and you’re ready to start using JBehave. You can also set up and configure JBehave using Maven. See for more details the extensive JBehave reference guide here.

Creating a first JBehave user story and corresponding test
Before we start writing our first JBehave test, we need a Java class that we can test. For this purpose, I have written a very simple POJO class Flipper with two variables: a String state (which is either ‘even’ or ‘odd’) and an Integer value. The source code for this class is included in the download at the end of this post. Among the methods for this class is a method flipState, which flips the state from ‘even’ to ‘odd’ (or the other way round) and increases the value by 1. Useless, maybe, but it does the trick for this example!

Step 1: Write your user story
A user story is written in the ‘given-when-then’ format that is characteristic for the BDD approach. In our example, we want to make sure (i.e., test) that given a fresh instance of the Flipper class, when we call the flipState method, then the value variable of the Flipper class is assigned the value 2. See how I used the ‘given-when-then’ strcuture to describe my test case? Translated to a user story that is JBehave-readable, our test case looks like this:

Scenario: when a user flips a flipper, its value is increased by 1

Given a flipper
Given the flipper has value 1
When the user flips the flipper
Then the value of the flipper must become 2

Step 2: Map the steps in the user story to Java code
One of the nicer features of JBehave is that you can run your tests as JUnit tests, which makes for easy integration with the rest of your testing and / or continuous integration framework. To do this, your test class containing the implementation of your story steps should extend the JUnitStories class. JBehave uses the @Given, @When and @Then annotations to identify story steps and their nature. For example, our test class and the implementation of the first ‘given’ step might look like this:

public class JBehaveTest extends JUnitStories {

	private Flipper flipper;

	@Given("a flipper")
	public void aFlipper() {

		flipper = new Flipper();
}

Now, whenever JBehave encounters the line ‘Given a flipper’ in one of the stories that is executed, it will automatically execute the aFlipper() method.

We can do the same for the ‘when’ and ‘then’ clauses of our story, using the appropriate annotations:

@When("the user flips the flipper")
public void whenTheUserFlipsTheFlipper() {

	flipper.flipState();
}

@Then("the value of the flipper must become 2")
public void valueOfFlipperMustBecomeTwo() {

	Assert.assertEquals(2, flipper.getValue());
}

Step 3: Configure your user stories
In order for our stories to be run, we need to tell JBehave where our stories are located, and which configuration should be used to execute them. For now, we are going to use the default configuration.

@Override
public Configuration configuration() {
	return new MostUsefulConfiguration().useStoryLoader(new LoadFromClasspath(getClass().getClassLoader())).useStoryReporterBuilder(new StoryReporterBuilder().withFormats(Format.CONSOLE));
}

@Override
public List<CandidateSteps> candidateSteps() {
	return new InstanceStepsFactory(configuration(), this).createCandidateSteps();
}
	
@Override
protected List<String> storyPaths() {
	return Arrays.asList("com/ontestautomation/jbehave/demo/test_value.story");
}

@Override
@Test
public void run() throws Throwable {
	super.run();
}

The configuration method loads the default configuration for running our JBehave tests, known as the MostUsefulConfiguration. The candidateSteps method loads the story steps defined in the current class (‘this’). The storyPaths method is used to define which stories are going to be executed. I included my sample story in the Eclipse project I used, but you can load story definitions from everywhere, including from an URL. Finally, the run method allows us to run our class as a test.

Step 4: Run your JBehave tests
Since we defined our JBehave tests as JUnit tests, running them is as easy as running regular JUnit tests. For example, in Eclipse, you can just right-click on your JBehave test class and select Run As > JUnit Test.

Step 5: Review the test results
Again, since our JBehave tests are just another type of JUnit tests, we can review the results for our tests like we do for any other JUnit test. Again, in Eclipse, a new tab displaying our test results opens automatically after test execution:
JBehave test results
In the Eclipse console, we can also see that our story has been processed successfully:

Processing system properties {}
Using controls EmbedderControls[batch=false,skip=false,generateViewAfterStories=true,ignoreFailureInStories=false,ignoreFailureInView=false,verboseFailures=false,verboseFiltering=false,storyTimeoutInSecs=300,threads=1]

(BeforeStories)

Running story com/ontestautomation/jbehave/demo/test_value.story

(com/ontestautomation/jbehave/demo/test_value.story)
Scenario: when a user flips a flipper, its value is increased by 1
Given a flipper
Given the flipper has value 1
When the user flips the flipper
Then the value of the flipper must become 2

Useful features
JBehave offers a lot of useful features, all of which are described in the online reference manual which can be found here. Some of the most notable features are:

Further reading
An Eclipse project including my Flipper class implementation and the JBehave test story I’ve demonstrated above can be downloaded here.

Happy JBehaving!

API testing best practices

This is the second post in a three-part series on API testing. The first post, which can be found here, provided a brief introduction on APIs, API testing and its relevance to the testing world. This post will feature some best practices for everybody involved in API testing. The third and final post will contain some useful code example for those of you looking to build your own automated API testing framework.

As was mentioned in the first post in this mini-series, API test execution differs from user interface-based testing since APIs are designed for communication between systems or system components rather than between a system or system component and a human being. This introduces some challenges to testing APIs, which I will try to tackle here.

API communication
Whereas a lot of testing on the user interface level is still done by hand (and rightfully so), this is impossible for API testing; you need a tool to communicate with APIs. There are a lot of tools available on the market. Some of the best known tools that are specifically targeted towards API testing are:

I have extensive experience with SOAtest and limited experience with SoapUI and can vouch for their usefulness in API testing.

Structuring tests
An API usually consists of several methods or operations that can be tested individually as well as through the setup of test scenarios. These test scenarios are usually constructed by stringing together multiple API calls. I suggest a three step approach to testing any API:

  1. Perform syntax testing of individual methods or operations
  2. Perform functional testing of individual methods or operations
  3. Construct and execute test scenarios

Syntax testing
This type of testing is performed to check whether the method or operation accepts correct input and rejects incorrect input. For example, syntax testing determines whether:

  • Leaving mandatory fields empty results in an error
  • Optional fields are accepted as expected
  • Filling fields with incorrect data types (for example, putting a text value into an integer field) results in an error

Functional testing of individual operations or methods
This type of testing is performed to check whether the method or operations performs its intended action correctly. For example:

  • Is calculation X performed correctly when calling operation / method Y with parameters A, B and C?
  • Is data stored correctly for future use when calling a setter method?
  • Does calling a getter method retrieve the correct information?

Test scenarios
Finally, when individual methods or operations have been tested successfully, method calls can be strung together to emulate business processes, For example:
API test scenarios
You see that this approach is not unlike user interface-based testing, where you first test individual components for their correct behaviour before executing end-to-end test scenarios.

API virtualization
When testing systems of interconnected components, the availability of some of the components required for testing might be limited at the time of testing (or they might not be available at all). Reasons for limited availability of a component might be:

  • The component itself is not yet developed
  • The component features insufficient or otherwise unusable test data
  • The component is shared with other teams and therefore cannot be freely used

In any of these cases, virtualization of the API can be a valuable solution, enabling testing to continue as planned. Several levels of API virtualization exist:

  • Mocking – This is normally done for code objects using a framework such as Mockito
  • Stubbing – this is used to create a simple emulation of an API, mostly used for SOAP and REST web services
  • Virtualization – This is the most advanced technique of the three, enabling the simulation of behaviour of complex components, including back-end database connectivity and transport protocols other than HTTP

Non-functional testing
As with all software components, APIs can (and should!) be tested for characteristics other than functionality. Some of the most important nonfunctional API test types that should at least be considered are:

  • Security testing – is the API accessible to those who are allowed to use it and inaccessible to those without the correct permissions?
  • Performance – Especially for web services: are the response times acceptable, even under a high load?
  • Interoperability and connectivity – can be API be consumed in the agreed manner and does it connect to other components as expected?

Most of the high-end API testing tools offer solutions for execution of these (and many other types of) nonfunctional test types.

More useful API testing best practices can again be found in the API Testing Dojo.

Do you have any additional API testing best practices you would like to share with the world?