Selenium and what it is not

Please note: this post is in no way meant as a rant towards some of my fellow testers. I’ve only written this to try and clear up some of the misconceptions with regards to Selenium WebDriver that I see popping up on various message boards and social media channels.

Selenium is hot in the test automation world. This probably isn’t news to you. If it is, you’re welcome, please remember you read it here first! In all seriousness, Selenium has been widely used for the past couple of years, and I don’t see its popularity fading any time soon. Due to its popularity, Selenium has been the go-to solution for everybody looking to write automated checks for browser-based applications, even though sometimes it might not be the actual best answer to the question at hand. In the past couple of years I have seen a lot of funny, strange and downright stupid questions being asked with regards to Selenium.

In this post, I would like to look at Selenium from a slightly different perspective by answering the question ‘what is Selenium NOT?’. All of the examples in the remainder of this blog post are based on actual questions or anecdotes I have read. I’m not going to link to them, since this post is not about blaming and shaming, it’s about educating those in need of some education.

Selenium is not a test tool
You read that right, Selenium is not a test tool. A lot of people (including a staggeringly large percentage of recruiters) get this wrong, unfortunately. To quote the Selenium web site:

Selenium automates browsers. That’s it! What you do with that power is entirely up to you.

My personal number one reason why Selenium is not a test tool: it does not provide a mechanism to perform assertions. For that, you’ll have to combine Selenium with JUnit or TestNG (for Java), NUnit (for C#) or any other actual (unit) testing framework. Selenium only performs predefined actions on a web site or other application running in a browser. It does this fairly well, but that doesn’t make it a test tool.

No, Selenium is NOT a test tool

No, Selenium is NOT a test tool

Off-topic: the same applies to tools that support Behaviour Driven Development, such as Cucumber and SpecFlow. Those aren’t test tools either. They can be used as an assistant in writing automated tests (checks), but that isn’t the same thing.

Selenium is not a tool to be used in API testing
I’ve seen this one come by a number of times, even as recently as a few days ago:

How can I perform tests on APIs using Selenium?

Again, Selenium automates browsers, so it operates on the user interface level of an application. Actions performed on a user interface might invoke API calls. Selenium is completely oblivious to this API interaction, however. There’s no way to have Selenium interact with APIs directly. If you want to perform automated checks on the API level, please use a tool that is specifically created for these types of checks, such as REST Assured. It might be wise to repeatedly ask yourself ‘Am I actually testing the user interface, or am I merely testing my application THROUGH that user interface?‘.

Selenium is not a performance test tool
Seeing how easy it is to write tests using Selenium (I’m not saying it’s easy to write GOOD tests, though), a lot of people seem to think that these tests can easily be leveraged to execute load and performance tests as well, for example using Selenium Grid. This is a pretty bad idea, though. Selenium Grid is a means to execute your functional tests in parallel, thereby shortening the execution time. It is not meant to be (ab)used as a performance testing platform, if only for these two simple reasons:

  • It doesn’t scale very well, so you’ll probably have a hard time generating anything but trivial loads
  • Selenium does not offer a mechanism to measure response times (at least not without taking into account the time needed at the client side to render a page or specific elements), and Selenium Grid isn’t able to gather and aggregate these response times for each individual note and present them in a concise and useful manner.

Both of these are essential if you’re serious about your performance testing, so please use a tool that is specifically written for that purpose, such as JMeter.

Selenium can’t handle your desktop applications
I feel that I’m starting to repeat myself here: Selenium automates browsers. Anything outside the scope of that browser cannot be handled by Selenium. This means that Selenium also can’t handle alerts and dialogs that are native to the operating system, such as the Windows file upload/download dialogs. To handle these, either use a tool such as AutoIt (if you’re on Windows) or, and this approach is much preferred, bypass the user interface altogether and use a solution such as the one presented in this blog post.

I sincerely hope this clears up some of the misconceptions around Selenium. If you have any other examples, feel free to post a comment or send me an email.

Using the LoadableComponent pattern for better Page Object handling in Selenium

I’m sure that by now, everybody who is at least remotely involved in the creation of automated user interface tests using Selenium WebDriver is aware of the Page Object pattern (read more about it here and here) for introducing modularity and reusability in their automated tests. In this post I will introduce the LoadableComponent pattern, an extension to this pattern that standardizes handling the loading and verifying loading status of these page objects.

Why use the LoadableComponent pattern?
Out of the box, Selenium WebDriver does a good job of determining whether an HTML page has been loaded by using the document.readyState property, which is part of the W3C WebDriver specification. However, this property alone is not always enough to assert that all dynamic content on your page has been fully loaded and that all elements required in your test script are present on the page. This is especially the case when your web page uses a JavaScript-heavy framework such as the popular AngularJS or KnockoutJS frameworks.

So, even when Selenium tells you the page is loaded and continues to execute the next step in your test, this step might fail because the actual element needed for that step is not yet visible, clickable or otherwise ready. You can solve this problem to an extent by using suitable wrapper methods for the standard Selenium API methods, but wouldn’t it be nice to enhance our Page Objects with a generic approach to evaluate page load status? This is where the LoadableComponent pattern comes in.

Introducing the LoadableComponent pattern
To explain the concept of the LoadableComponent pattern, we will again turn to the ParaBank application. LoadableComponent is a base class in Selenium, which means that you can simply define your Page Objects as an extension of the LoadableComponent class. So, for example, we can simply define a LoginPage object as follows:

public class LoginPage extends LoadableComponent<LoginPage> {
  // class implementation will be explained later 
}

This does nothing more than defining this class as a LoadableComponent that loads the LoginPage page.

Now, by having our Page Objects extend the LoadableComponent base class, we need to implement two new methods, load() and isLoaded() (note that in C#, these are called ExecuteLoad() and EvaluateLoadedStatus() for some reason). These methods provide the added value of using the LoadableComponent pattern. The load() method contains the code that is executed to navigate to the page, while the isLoaded() method is used to evaluate whether we are on the correct page and whether page loading has finished successfully. Using LoadableComponent, our LoginPage class now looks like this:

public class LoginPage extends LoadableComponent<LoginPage> {
	
	private WebDriver driver;
	
	public LoginPage(WebDriver driver) {
		
		this.driver = driver;
		driver.get("http://parabank.parasoft.com");
	}
	
	@Override
	protected void isLoaded() throws Error {
		
		if(!PageLoad.myElementIsClickable(this.driver, By.name("username"))) {
			throw new Error("LoginPage was not successfully loaded");
		}
	}

	@Override
	protected void load() {		
	}
}

Note the use of a custom wrapper boolean method myElementIsClickable() to determine whether the page has loaded successfully. It can be used as a generic way to specify which element must be present for any given Page Object in order to consider it fully loaded:

public static boolean myElementIsClickable (WebDriver driver, By by) {
		
	try
	{
		new WebDriverWait(driver, 10).until(ExpectedConditions.elementToBeClickable(by));
	}
	catch (WebDriverException ex)
	{
		return false;
	}
	return true;		
}

Also, for most pages I don’t actually put code inside the body of the load() method. This is because in a typical application, most Page Objects are only accessed by means of navigation via other Page Objects instead of being accessed directly. Even for the LoginPage I have included the navigate() call in the constructor, because in this way we do not need to wait for the timeout in isLoaded() to be exceeded before the navigate() method is called from within load(). The example project I will link to at the end of this post shows you how I implemented the LoadableComponent pattern in my latest project.

So, now that we’ve implemented our Page Objects as LoadableComponents, we can use it in our tests simply by doing this:

new LoginPage(driver).get();

The get() method is implemented as follows:

public T get() {
	try {
		isLoaded();
		return (T) this;
	} catch (Error e) {
		load();
	}
 
	isLoaded();
 
	return (T) this;
}

In plain English: it calls isLoaded() first to see whether the page is loaded. If this is not the case, it calls load() to load the page. Afterwards, it calls isLoaded() again to see if the page is now successfully loaded.

Leveraging LoadableComponents in your tests
To ensure in your tests that your Page Objects are loaded before moving on, you can simply call get() each time a Page Object is instantiated:

@Test
public void validLoginTest() {

	// Load login page
	LoginPage loginPage = new LoginPage(driver).get();

	// Log in using valid credentials
	HomePage homePage = loginPage.correctLogin("john", "demo").get();

	// Load home page and check welcome text
	Assert.assertEquals("Welcome text is correct","Welcome John Smith", homePage.getWelcomeString());
}

The assertion in isLoaded() will fail if MyElementIsClickable() returns false, leading to the test as a whole to fail:

LoadableComponent error

Enhancing the LoadableComponent class
Finally, another interesting use of the LoadableComponent pattern is writing your own implementation of the class. In that way you can, for example, generate specific logging to the console or to an ExtentReports report in case you use it.

The source code for the original LoadableComponent implementation can be found here. A simple addition is writing the error message that is thrown by isLoaded() (which in turn is the error message thrown by the performed assertion) to the console:

public abstract class CustomLoadableComponent<T extends CustomLoadableComponent<T>> {

	@SuppressWarnings("unchecked")
	public T get() {
		try {
			isLoaded();
			return (T) this;
		} catch (Error e) {
			// This is the extra line of code 
			System.out.println("Error encountered during page load: " + e.getMessage());
			load();
		}

		isLoaded();

		return (T) this;
	}

	protected abstract void load();

	protected abstract void isLoaded() throws Error;
}

Now you can simply define your Page Objects as an extension of the CustomLoadableComponent class:

public class LoginPage extends CustomLoadableComponent<LoginPage> {
  // class implementation remains the same 
}

When we run our tests and encounter an error, we see that the error message is written to the console:
Custom LoadableComponent output

A working Eclipse project (created using Maven) containing all of the code samples included in this post can be downloaded here. Just run the tests in the tests package to see for yourself how it works.

Using wrapper methods for better error handling in Selenium

Frequent users of Selenium WebDriver might have come across the problem that when they’re using Selenium for testing responsive, dynamic web applications, timing and synchronization can be a major hurdle in creating useful and reliable automated tests. In my current project, I’m creating WebDriver-based automated tests for a web application that heavily relies on Javascript for page rendering and content retrieval and display, which makes waiting for the right element to be visible, clickable or whatever state it needs to be in significantly less than trivial.

Using the standard Selenium methods such as click() and sendKeys() directly in my script often results in failures as a result of the weg page element being invisible, disabled or reloaded (this last example resulting in a StaleElementReferenceException). Even using implicit waits when creating the WebDriver object didn’t always lead to stable tests, and I refuse to use Thread.sleep() (and so should you!). Also, I didn’t want to use individual WebDriverWait calls for every single object that needed to be waited on, since that introduces a lot of extra code that needs to be maintained. So I knew I had to do something more intelligent for my tests to be reliable – and therefore valuable as opposed to a waste of time and money.

Wrapper methods
The solution to this problem lies in using wrapper methods for the standard Selenium methods. So instead of doing this every time I need to perform a click:

(new WebDriverWait(driver, 10)).until(ExpectedConditions.elementToBeClickable(By.id("loginButton")));
driver.findElement(By.id("loginButton")).click();

I have created a wrapper method click() in a MyElements class that looks like this:

public static void click(WebDriver driver, By by) {
	(new WebDriverWait(driver, 10)).until(ExpectedConditions.elementToBeClickable(by));
	driver.findElement(by).click();
}

Of course, the 10 second timeout is arbitrary and it’s better to replace this with some constant value. Now, every time I want to perform a click in my test I can simply call:

MyElements.click(driver, By.id("loginButton");

which automatically performs a WebDriverWait, resulting in much stabler, better readable and maintainable scripts.

Extending your wrapper methods: error handling
Using wrapper methods for Selenium calls has the additional benefit of making error handling much more generic as well. For example, if you often encounter a StaleElementReferenceException (which those of you writing tests for responsive and dynamic web applications might be all too familiar with), you can simply handle this in your wrapper method and be done with it once and for all:

public static void click(WebDriver driver, By by) {
	try {
		(new WebDriverWait(driver, 10)).until(ExpectedConditions.elementToBeClickable(by));
		driver.findElement(by).click();
	catch (StaleElementReferenceException sere) {
		// simply retry finding the element in the refreshed DOM
		driver.findElement(by).click();
	}
}

This is essentially my own version of this excellent solution for handling a StaleElementReferenceException. By using wrapper methods, you can easily handle any type of exception in its own specific way, which will improve your tests significantly.

Extending your wrapper methods: logging
Your wrapper methods also allow for better logging capabilities. For example, if you’re using the ExtentReports library for creating HTML reports for Selenium tests (as I do in my current project), you can create a log entry every time an object is not clickable after the WebDriverWait times out:

public static void click(WebDriver driver, By by) {
	try {
		(new WebDriverWait(driver, 10)).until(ExpectedConditions.elementToBeClickable(by));
		driver.findElement(by).click();
	catch (StaleElementReferenceException sere) {
		// simply retry finding the element in the refreshed DOM
		driver.findElement(by).click();
	}
	catch (TimeoutException toe) {
		test.log(logStatus.Error, "Element identified by " + by.toString() + " was not clickable after 10 seconds");
	}
}

Here, test is the ExtentTest object representing the log for the current test. See how easy this is?

Other possible benefits
There’s a couple of other benefits to using these wrapper methods, some of which I use in my current project as well, such as:

  • Automatically failing the JUnit / TestNG (or in my case NUnit) test whenever a specific error occurs by including an Assert.Fail();
  • Restoring or recreating specific conditions or a specific application state after an error occurs to prepare for the next test (e.g. ending the current user session)
  • Automatically restarting a test in case of a specific error

And I’m sure there are many more.

Wrapping this up (pun intended!), using these wrapper methods for standard Selenium calls can be very beneficial to your Selenium experience and those of your clients or employers. They certainly saved me a lot of unnecessary code and frustration over brittle tests..