Defining test steps and object properties in an Excel Data Source for Selenium

Recently, one of the readers of this blog sent me an email asking whether it was possible to define not only input and validation parameters in an external data source (such as an Excel file), but also the test steps to be taken and the object properties on which these steps need to be performed by Selenium WebDriver. As it happens, I have been playing around with this idea for a while in the past, but until now I’ve never gotten round to writing a blog post on it.

The Excel Data Source
As an example, we are going to perform a search query on Google and validate the number of results we get from this search query. This scripts consists of the following steps:

  1. Open a new browser instance
  2. Navigate to the Google homepage
  3. Type the search query in the text box
  4. Click on the search button
  5. Validate the contents of the web element displaying the number of search results against our expected value
  6. Close the browser instance

For each step, we are going to define (if applicable):

  • A keyword identifying the action to be taken
  • A parameter value that is used in the action (note that for example a type action takes a parameter, a click action does not)
  • The attribute type that uniquely identifies the object on which the action is performed
  • The corresponding attribute value

For our example test script, the Excel Data Source to be used could look something like this:
Excel Data Source
Reading the Data Source
Similar to the concept explained in a previous post, first we need to read the values from the Excel sheet in order to execute the test steps defined. We need to be a little extra careful here as some cells might be empty and the methods we use do not particularly like reading values from cells that do not exist.

public static void main (String args[]) {

		String action = "";
		String value = "";
		String attribute = "";
		String attrval = "";
		
		try {
			// Open the Excel file for reading
			FileInputStream fis = new FileInputStream("C:\\Tools\\testscript.xls");
			// Open it for writing too
			FileOutputStream fos = new FileOutputStream("C:\\Tools\\testscript.xls");
			// Access the required test data sheet
			HSSFWorkbook wb = new HSSFWorkbook(fis);
			HSSFSheet sheet = wb.getSheet("steps");
			// Loop through all rows in the sheet
			// Start at row 1 as row 0 is our header row
			for(int count = 1;count<=sheet.getLastRowNum();count++){
				HSSFRow row = sheet.getRow(count);
				System.out.println("Running test step " + row.getCell(0).toString());

				// Run the test step for the current test data row
				if(!(row.getCell(1) == null || row.getCell(1).equals(Cell.CELL_TYPE_BLANK))) {
					action = row.getCell(1).toString();
				} else {
					action = "";
				}

				if(!(row.getCell(2) == null || row.getCell(2).equals(Cell.CELL_TYPE_BLANK))) {
					value = row.getCell(2).toString();
				} else {
					value = "";
				}

				if(!(row.getCell(3) == null || row.getCell(3).equals(Cell.CELL_TYPE_BLANK))) {
					attribute = row.getCell(3).toString();
				} else {
					attribute = "";
				}

				if(!(row.getCell(4) == null || row.getCell(4).equals(Cell.CELL_TYPE_BLANK))) {
					attrval = row.getCell(4).toString();
				} else {
					attrval = "";
				}

				System.out.println("Test action: " + action);
				System.out.println("Parameter value: " + value);
				System.out.println("Attribute: " + attribute);
				System.out.println("Attribute value: " + attrval);
				
				String result = runTestStep(action,value,attribute,attrval);
				
				// Write the result back to the Excel sheet
				row.createCell(5).setCellValue(result);
				
			}
			
			// Save the Excel sheet and close the file streams
			wb.write(fos);
			fis.close();
			fos.close();
			
		} catch (Exception e) {
			System.out.println(e.toString());
		}
}

We do not read the value from the Result column, but rather we are going to execute the test steps using the values from the other columns, determine the result and write this back to the Excel sheet. In this way, we have a rudimentary logging function built in directly into our framework. Neat, right?

Defining and executing test steps
Next, we need to implement the generic runTestStep method we use to execute the test steps we have defined in our Data Source. This can be done pretty straightforward by looking at the current Action keyword and then executing the necessary steps for that keyword.

public static String runTestStep(String action, String value, String attribute, String attrval) throws Exception {

		switch(action.toLowerCase()) {
		case "openbrowser":
			switch(value.toLowerCase()) {
			case "firefox":
				driver = new FirefoxDriver();
				driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
				return "OK";
			case "htmlunit":
				driver = new HtmlUnitDriver();
				driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
				return "OK";
			default:
				return "NOK";
			}
		case "navigate":
			driver.get(value);
			return "OK";
		case "type":
			try {
				WebElement element = findMyElement(attribute,attrval);
				element.sendKeys(value);
				return "OK";
			} catch (Exception e) {
				System.out.println(e.toString());
				return "NOK";
			}
		case "click":
			try {
				WebElement element = findMyElement(attribute,attrval);
				element.click();
				return "OK";
			} catch (Exception e) {
				System.out.println(e.toString());
				return "NOK";
			}
		case "validate":
			try {
				WebElement element = findMyElement(attribute,attrval);
				if (element.getText().equals(value)) {
					return "OK";
				} else {
					System.out.println("Actual value: " + element.getText() + ", expected value: " + value);
					return "NOK";
				}
			} catch (Exception e) {
				System.out.println(e.toString());
			}
		case "closebrowser":
			driver.quit();
			return "OK";
		default:
			throw new Exception("Unknown keyword " + action);
		}
}

This setup makes it very easy to add other keywords (e.g., for selecting a value from a dropdown list), both for standardized browser actions as well as for custom keywords. Every step returns a result (here either OK or NOK) to indicate whether the action has been executed successfully. This is written back to the Excel sheet in our main method, so we can see whether our test steps have been executed correctly.

Finding the required object
I’ve also used a helper method findMyElement that, given an attribute and an attribute value, returns a WebElement that corresponds to these values. Its implementation is quite rudimentary, but it should be easy for you to extend it and make it more fail safe. I’m lazy sometimes. I haven’t even implemented all types of selectors that Selenium can handle!

public static WebElement findMyElement(String attribute, String attrval) throws Exception {

		switch(attribute.toLowerCase()) {
		case "id":
			return driver.findElement(By.id(attrval));
		case "name":
			return driver.findElement(By.name(attrval));
		case "xpath":
			return driver.findElement(By.xpath(attrval));
		case "css-select":
			return driver.findElement(By.cssSelector(attrval));
		default:
			throw new Exception("Unknown selector type " + attribute);
		}
}

Now, when I run the test, my code nicely reads all rows from Excel, determines what test action needs to be performed, executes it and reports the result back to Excel. After running the test, the Excel sheet looks like this:
Excel Data Source including test results
The only test step that fails does so because it Google doesn’t always return the same number of test results in the same amount of time, obviously. Nevertheless, it illustrates the added value of writing back the result of every step to the Excel sheet. Of course, you can very easily extend this to include a useful error message and even a link to a screenshot.

The Eclipse project containing the code I’ve used to create this example can be downloaded here.

How to create screenshots in your Selenium Webdriver tests

As they say, a picture often says more than a thousand words. This also applies to test execution reports – no matter the clarity of your error messages, often a screenshot of the status of your browser instance at the moment a particular error occurs is the most efficient way to record what has gone wrong. In this post, I will show you how to generate a screenshot in Selenium Webdriver, which you can then save to disk for later reference and/or include in your custom reports.

First, we need a simple test script in which a screenshot is created at a given moment. Here it is:

public static void runTest() {

	WebDriver driver = new FirefoxDriver();

	driver.get("http://parabank.parasoft.com");

	// log in
	driver.findElement(By.name("username")).sendKeys("john");
	driver.findElement(By.name("password")).sendKeys("demo");
	driver.findElement(By.xpath("//input[@value='Log In']")).click();
		
	//create screenshot
	createScreenshot(driver,"C:\\temp\\screen.png");

	driver.quit();
}

The createScreenshot method takes two arguments: the driver instance and a physical location where the created screenshot will be stored. The implementation of this method is pretty straightforward:

public static void createScreenshot(WebDriver driver, String location) {

	// generate screenshot as a file object
	File scrFile = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);
	try {
		// copy file object to designated location
		FileUtils.copyFile(scrFile, new File(location));
	} catch (IOException e) {
		System.out.println("Error while generating screenshot:\n" + e.toString());
	}
}

When we run this test, we see that a screenshot is created and stored in the location we provided (in my case, C:\temp):
Screenshot as created by our code

Creating screenshots when an alert is active
One of my readers, Kritika Gupta, nicely alerted me that the method described above does not work when there is a Javascript alert active. Instead, an UnhandledAlertException is thrown and no screenshot is created. This seems to be a known issue with Selenium, as can be read here. I have found a way to circumvent this problem by using the Java Robot class. It works as follows:

public static void createScreenshotUsingRobot(WebDriver driver, String location) {
		
	BufferedImage image = null;
	try {
		image = new Robot().createScreenCapture(new Rectangle(Toolkit.getDefaultToolkit().getScreenSize()));
	} catch (HeadlessException | AWTException e1) {
		e1.printStackTrace();
	}
	try {
		ImageIO.write(image, "png", new File(location));
	} catch (IOException e) {
		e.printStackTrace();
	}
}

When we use this method to create screenshot, it even works when Javascript alert messages are active, as can be seen in the screenshot below, which has been taken using the code above:
Screenshot taken using the Java Robot class
Please note that the code above takes a screenshot of your complete desktop (as you can see) instead of a screenshot from the active browser window only. Also, remember that this code does not take care of handling the popup. This should be done in your main test code.

The Eclipse project I used for this example can be downloaded here.

Create your own HTML report from Selenium tests

As I am learning more and more about using Selenium Webdriver efficiently (experiences I try to share through this blog), I’m slowly turning away from my original standpoint that user interface-based test automation is not for me. I am really starting to appreciate the power of Selenium, especially when you use proper test automation framework design patterns such as the Page Object pattern I wrote about earlier. However, Selenium by default lacks one vital aspect of what makes a good test automation tool to me: proper reporting options. Luckily, as Selenium is so open, there’s lots of ways to build custom reporting yourself. This post shows one possible approach.

My approach
Personally, I prefer HTML reports as they are highly customizable, relatively easy to build and can be easily distributed to other project team members. To build a nice HTML report, I use the following two step approach:

  • Execute tests and gather statistics about validations executed
  • Create HTML report from these statistics after test execution has finished

In this post I’ll use the following test script as an example. I created a page with five HTML text fields, for which I am going to validate the default text. Nothing really realistic, but remember it’s only used to illustrate my reporting concept.

public static void main (String args[]) {
		
	WebDriver driver = new HtmlUnitDriver();
	driver.get("http://www.ontestautomation.com/files/report_test.html");
		
	for (int i = 1; i <=5; i++) {
		WebElement el = driver.findElement(By.id("textfield" + Integer.toString(i)));
		Assert.assertEquals(el.getAttribute("value"), "Text field " + Integer.toString(i));
	}
		
	driver.close();	
}

When we run this script, one error is generated as text field 4 contains a different default value (go to the URL in the script to see for yourself).

Custom reporting functions
To be able to create a nice HTML report, we first need some custom reporting functions that store test results in a way we can reuse them later to generate our report. To achieve this, I created a report method in a Reporter class that stores validation results in a simple List:

public static void report(String actualValue,String expectedValue) {
	if(actualValue.equals(expectedValue)) {
		Result r = new Result("Pass","Actual value '" + actualValue + "' matches expected value");
		details.add(r);
	} else {
		Result r = new Result("Fail","Actual value '" + actualValue + "' does not match expected value '" + expectedValue + "'");
		details.add(r);
	}
}

The Result object is a simple class with three class variables: result (which is either Pass or Fail), a resultText (which is a custom description) and a URL for a screenshot (the use of which we will see later).

For every test we execute in our Selenium script, instead of using the standard TestNG / JUnit assertions, we use our own reporting function. You might want to throw an error as well when a validation fails, but I’m happy just to write it to my report and let test execution continue.

After test execution is finished, we are going to write the test results we gathered to a file. For this, I used an extremely simple HTML template (yes, I was too lazy even to indent it properly):

<html>
<head>
<title>Test Report</title>
<head>
<body>
<h3>Test results</h3>
<table>
<tr>
<th width="10%">Step</th>
<th width="10%">Result</th>
<th width="80%">Remarks</th>
</tr>
<!-- INSERT_RESULTS -->
</table>
</body>

In this template I am going to insert my test results, using a simple replace function

public static void writeResults() {
	try {
		String reportIn = new String(Files.readAllBytes(Paths.get(templatePath)));
		for (int i = 0; i < details.size();i++) {
			reportIn = reportIn.replaceFirst(resultPlaceholder,"<tr><td>" + Integer.toString(i+1) + "</td><td>" + details.get(i).getResult() + "</td><td>" + details.get(i).getResultText() + "</td></tr>" + resultPlaceholder);
		}
			
		String currentDate = new SimpleDateFormat("dd-MM-yyyy").format(new Date());
		String reportPath = "Z:\\Documents\\Bas\\blog\\files\\report_" + currentDate + ".html";
		Files.write(Paths.get(reportPath),reportIn.getBytes(),StandardOpenOption.CREATE);
			
	} catch (Exception e) {
		System.out.println("Error when writing report file:\n" + e.toString());
	}
}

Finally, we need to use these custom reporting functions in our test script:

public static void main (String args[]) {
		
	WebDriver driver = new HtmlUnitDriver();
	Reporter.initialize();
	driver.get("http://www.ontestautomation.com/files/report_test.html");
		
	for (int i = 1; i <=5; i++) {
		WebElement el = driver.findElement(By.id("textfield" + Integer.toString(i)));
		Reporter.report(el.getAttribute("value"), "Text field " + Integer.toString(i));
	}
		
	Reporter.writeResults();
	driver.close();	
}

The initialize() method simply clears all previous test results by emptying the List we use to store our test results. When we run our test, the following HTML report is generated:

The HTML test report

Here, we can clearly see that our test results are now available in a nicely readable (though not yet very pretty) format. In one of my next posts, I am going to enhance these HTML reporting functions with some additional features, such as:

  • Screenshots in case of errors
  • Use of stylesheets for eye candy
  • Test execution statistics

Hopefully the above will get you started creating nicely readable HTML reports for your Selenium tests!

The Eclipse project used in the above example can be downloaded here. The HTML report template can be downloaded here (right click, save as).