Getting started with: JGiven

This is the eighth article in our series on new, popular or otherwise interesting tools used in test automation. You can read all posts within this series by clicking here.

What is JGiven?
From the JGiven.org website: JGiven is a developer-friendly and pragmatic BDD tool for Java. Developers write scenarios in plain Java using a fluent, domain-specific API, JGiven generates reports that are readable by domain experts.

In short, JGiven can be used as an alternative to other BDD frameworks, such as JBehave and Cucumber. Where the latter two separate the features from the test code, JGiven does not and might therefore be considered more suitable to unit and integration tests rather than automated system and regression tests.

Where can I get JGiven?
The JGiven sources can be downloaded from the JGiven GitHub site.

How do I install and configure JGiven?
The easiest way to get started with JGiven is by creating a new Maven project and including the following dependency:

<dependency>
	<groupId>com.tngtech.jgiven</groupId>
	<artifactId>jgiven-testng</artifactId>
	<version>0.9.5</version>
	<scope>test</scope>
</dependency>

This is for use in combination with TestNG, which I will do in the rest of this post. If you prefer to user JUnit, use jgiven-junit as your artifactId. Please note that this dependency does not include TestNG (or JUnit) itself, so be sure to include it separately in your pom.xml.

Creating and running a first JGiven test
Taking my trusted ParaBank application as an example, I want to perform a test determining whether I can login successfully using a JGiven scenario. I.e., I want to execute the following scenario:

Given I am at the login page
When I login as john with password demo
Then the login action will be successful

The implementation of this scenario as a TestNG-based test in JGiven is pretty straightforward:

public class LoginTest extends ScenarioTest<GivenIAmAtTheLoginPage, WhenILoginAsJohnWithPasswordDemo, ThenTheLoginActionWillBeSuccessful> {
	
	@Test
	public void aFirstLoginTest() {
		
		given().I_am_at_the_login_page();
		when().I_login_as_john_with_password_demo();
		then().the_login_action_will_be_successful();
	}
}

The ScenarioTest class requires three parameters, each representing a stage in the Given-When-Then scenario. To make this compile, we also need to implement each of the three classes that are used as a parameter. As an example, this is what the implementation of the GivenIAmAtTheLoginPage class looks like:

public class GivenIAmAtTheLoginPage extends Stage<GivenIAmAtTheLoginPage>{
	public GivenIAmAtTheLoginPage I_am_at_the_login_page() {
		
		return self();
	}
}

The other two classes (stages) are implemented in a similar way. Now that this is done, we can run our test, which lads to the following output in the console:

Test Class: com.ontestautomation.jgiven.tests.LoginTest

 Scenario: A first login test

   Given I am at the login page
    When I login as john with password demo
    Then the login action will be successful

PASSED: aFirstLoginTest

We can see that our test passes, so we have something to build upon. No actual test actions are performed yet, so we are going to add these next. Since we are performing a Selenium WebDriver test, it’s required that we pass along the browser instance as a parameter for each of the steps. This also means we can initialize it before the test is run (using @BeforeTest) and destroy it afterwards (using @AfterTest):

public class LoginTest extends ScenarioTest<GivenIAmAtTheLoginPage, WhenILoginAsJohnWithPasswordDemo, ThenTheLoginActionWillBeSuccessful> {
	
	WebDriver driver;
	
	@BeforeTest
	public void initBrowser() {

		driver = new FirefoxDriver();
		driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
	}
	
	@Test
	public void aFirstLoginTest() {
		
		given().I_am_at_the_login_page(driver);
		when().I_login_as_john_with_password_demo(driver);
		then().the_login_action_will_be_successful(driver);
	}
	
	@AfterTest
	public void tearDown() {
		
		driver.quit();
	}
}

The WebDriver actions associated with the Given, When and Then steps can now be added to the respective classes as well, for example for GivenIAmAtTheLoginPage this results in:

public class GivenIAmAtTheLoginPage extends Stage<GivenIAmAtTheLoginPage>{
	
	WebDriver _driver;
	
	public GivenIAmAtTheLoginPage I_am_at_the_login_page(WebDriver driver) {
		
		_driver = driver;
		
		_driver.get("http://parabank.parasoft.com");
		
		return self();
	}
}

If we implement the other two Stage classes in a similar vein – where the Then stage should include an actual TestNG assertion to make it a proper test case – and rerun our test, we can see that it passes again:

Test Class: com.ontestautomation.jgiven.tests.LoginTest

 Scenario: A first login test

   Given I am at the login page FirefoxDriver: firefox on WINDOWS (6bccf261-5ce0-4378-8118-b545b6c82eca)
    When I login as john with password demo FirefoxDriver: firefox on WINDOWS (6bccf261-5ce0-4378-8118-b545b6c82eca)
    Then the login action will be successful FirefoxDriver: firefox on WINDOWS (6bccf261-5ce0-4378-8118-b545b6c82eca)

PASSED: aFirstLoginTest

Note that JGiven automatically adds a bit of information on the browser instance used to the console output.

Useful features
Some useful additional features of JGiven are:

  • Several types of test execution and results reporting, including JSON and HTML
  • Support for parameterized steps (in the above example, you could for instance parameterize the username and password in the When step)
  • Support for tags to organize scenarios

Further reading
Apart from the features mentioned above, JGiven provides some more useful features for creating useful BDD-style Java-based tests. A complete reference guide can be found on the JGiven website.

An Eclipse Maven project including the tests and test classes I’ve used in this post can be downloaded here.

Model-based testing with GraphWalker and Selenium – part 1

In this post I’d like to make a start exploring the possibilities and drawbacks that model-based testing (MBT) can offer to test automation in general and Selenium WebDriver in particular. I’ll be using GraphWalker as my MBT tool of choice, mostly because it’s open source, doesn’t have a steep learning curve and it’s available as a Java library, which makes integration with any Selenium project as simple as downloading the necessary .jar files and adding them to your test project.

I’ll use the login procedure for the Parabank demo application I’ve used so often before as an example. First, we need to model the login procedure using standard edges and nodes (which are called vertexes in Graphwalker):
A model of the Parabank login procedure
I’ve used yEd to draw this model, mostly because this tool offers the option to export the model in .graphml format, which will come in very handy at a later stage. I’ll go into that in another post.

After the browser has been started and the Parabank homepage is displayed, we can perform either a successful or an unsuccessful login. Unsuccessful logins take us to an error page, which offers us the possibility to try again, again with both positive and negative results as a possibility. A successful login takes us to the Accounts Overview page, from where we can log out. The application offers us many more options once we have performed a successful login, but for now I’ll leave these out of scope.

Next, we are going to ‘draw’, i.e. implement this model in GraphWalker. This is pretty straightforward:

private Model createModel() {

	// Create a new, empty model
	Model model = new Model();

	// Create vertexes (nodes)
	Vertex v_Start = new Vertex().setName("v_Start");
	Vertex v_HomePage = new Vertex().setName("v_HomePage");
	Vertex v_ErrorPage = new Vertex().setName("v_ErrorPage");
	Vertex v_AccountsOverviewPage = new Vertex().setName("v_AccountsOverviewPage");

	// Create edges
	Edge e_StartBrowser = new Edge()
		.setName("e_StartBrowser")
		.setSourceVertex(v_Start)
		.setTargetVertex(v_HomePage);
	Edge e_LoginFailed = new Edge()
		.setName("e_LoginFailed")
		.setSourceVertex(v_HomePage)
		.setTargetVertex(v_ErrorPage);
	Edge e_LoginFailedAgain = new Edge()
		.setName("e_LoginFailedAgain")
		.setSourceVertex(v_ErrorPage)
		.setTargetVertex(v_ErrorPage);
	Edge e_LoginSucceeded = new Edge()
		.setName("e_LoginSucceeded")
		.setSourceVertex(v_HomePage)
		.setTargetVertex(v_AccountsOverviewPage);
	Edge e_LoginSucceededAfterFailure = new Edge()
		.setName("e_LoginSucceededAfterFailure")
		.setSourceVertex(v_ErrorPage)
		.setTargetVertex(v_AccountsOverviewPage);
	Edge e_Logout = new Edge()
		.setName("e_Logout")
		.setSourceVertex(v_AccountsOverviewPage)
		.setTargetVertex(v_HomePage);

	// Add vertexes to the model
	model.addVertex(v_Start);
	model.addVertex(v_HomePage);
	model.addVertex(v_ErrorPage);
	model.addVertex(v_AccountsOverviewPage);

	// Add edges to the model
	model.addEdge(e_StartBrowser);
	model.addEdge(e_LoginFailed);
	model.addEdge(e_LoginFailedAgain);
	model.addEdge(e_LoginSucceeded);
	model.addEdge(e_LoginSucceededAfterFailure);
	model.addEdge(e_Logout);

	return model;
}

Easy, right? The only downside is that this is quite a lot of work, especially when your models get pretty large. It also leaves (a lot of) room for improvement when it comes to maintainability. We’ll get to that in a later post as well.

Theoretically, we could have GraphWalker go through this model and explore all vertexes and edges, but in order to do meaningful work we need to link them to concrete action steps performed on Parabank. A good and clean way to do this is to consider the following:

  • Edges are like state transitions in your model, so this is where the action (typing in text boxes, clicking on links, etc.) ends up.
  • Vertexes represent states in your model, so this is where the verifications (checks) need to be performed.

For example, this is the implementation of the e_StartBrowser edge:

WebDriver driver = null;

public void e_StartBrowser() {

	driver = new FirefoxDriver();
	driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);

	driver.get("http://parabank.parasoft.com");
}

By giving the method the same name as the actual edge, GraphWalker knows that it needs to execute this code snippet every time the e_StartBrowser node is encountered when running through the model. We can do the same for the vertexes, for example for the v_HomePage vertex representing the page we land on when we perform a successful login:

public void v_HomePage() {

	Assert.assertEquals(driver.getTitle(),"ParaBank | Welcome | Online Banking");
}

For simplicity, we’ll just do a check on the page title, but you’re free to add any type of check you wish, of course.

Finally, we need to tell GraphWalker to load and run the model. I prefer using TestNG for this:

@Test
public void fullCoverageTest() {

	// Create an instance of our model
	Model model = createModel();
		
	// Build the model (make it immutable) and give it to the execution context
	this.setModel(model.build());
		
	// Tell GraphWalker to run the model in a random fashion,
	// until every vertex is visited at least once.
	// This is called the stop condition.
	this.setPathGenerator(new RandomPath(new VertexCoverage(100)));
		
	// Get the starting vertex (v_Start)
	setNextElement(model.getVertices().get(0));
		
	//Create the machine that will control the execution
	Machine machine = new SimpleMachine(this);
				
	// As long as the stop condition of the path generator is not fulfilled, hasNext will return true.
	while (machine.hasNextStep()) {
			
		//Execute the next step of the model.
		machine.getNextStep();
	}
}

The most important part in this test is the stop condition, which basically tells GraphWalker how to run the test and when to stop. In this example, I want to run the model and its corresponding Selenium actions randomly (within the restrictions implied by the model, of course) until every vertex has been hit at least once (100% vertex coverage). GraphWalker offers a lot of other stop conditions, such as running the model until:

  • Every edge has been hit at least once
  • A predefined vertex is hit
  • A predefined time period has passed (this is great for reliability tests)

A complete list of options for stop conditions can be found here in the GraphWalker documentation.

When we run this test, we can see in the console output that GraphWalker walks through our model randomly until the stop condition has been satisfied:
GraphWalker console outputAs usual when using TestNG, a nice little report has been created that tells us everything ran fine:
GraphWalker TestNG report - passNote that unlike what I did in the example above, it’s probably a good idea to use soft asserts when you’re using TestNG together with GraphWalker. Otherwise, TestNG and therefore GraphWalker will stop executing at the first check that fails. Unless that’s exactly what you want, of course.

In this post we’ve seen a very basic introduction into the possibilities of MBT using GraphWalker and how it can be applied to Selenium tests. In upcoming tests I’ll dive deeper into the possibilities GraphWalker provides, how it can be used to generate executable models from .graphml models, and how to make the most of the tool while keeping your tests clean and maintainable.

The Eclipse project containing the code I’ve used in this post can be downloaded here.

Using the ExtentReports TestNG listener in Selenium Page Object tests

In this (long overdue) post I would like to demonstrate how to use ExtentReports as a listener for TestNG. By doing so, you can use your regular TestNG assertions and still create nicely readable ExtentReports-based HTML reports. I’ll show you how to do this using a Selenium WebDriver test that uses the Page Object pattern.

Creating the Page Objects
First, we need to create some page objects that we are going to exercise during our tests. As usual, I’ll use the Parasoft Parabank demo application for this. My tests cover the login functionality of the application, so I have created Page Objects for the login page, the error page where you land when you perform an incorrect login and the homepage (the AccountsOverviewPage) where you end up when the login action is successful. For those of you that have read my past post on Selenium and TestNG, these Page Objects have been used in that example as well. The Page Object source files are included in the Eclipse project that you can download at the end of this post.

Creating the tests
To demonstrate the reporting functionality, I have created three tests:

  • A test that performs a successful login – this one passes
  • A test that performs an unsuccessful login, where the check on the error message returns passes – this one also passes
  • A test that performs an unsuccessful login, where the check on the error message returns fails – this one fails

The tests look like this:

public class LoginTest {
	
 WebDriver driver;
     
    @BeforeSuite
    public void setUp() {
	         
        driver = new FirefoxDriver();
        driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
    }
	     
    @Parameters({"incorrectusername","incorrectpassword"})
    @Test(description="Performs an unsuccessful login and checks the resulting error message (passes)")
    public void testFailingLogin(String username, String incorrectpassword) {
	         
        LoginPage lp = new LoginPage(driver);
        ErrorPage ep = lp.incorrectLogin(username, incorrectpassword);
        Assert.assertEquals(ep.getErrorText(), "The username and password could not be verified.");
    }
	    
    @Parameters({"incorrectusername","incorrectpassword"})
    @Test(description="Performs an unsuccessful login and checks the resulting error message (fails)")
    public void failingTest(String username, String incorrectpassword) {
	         
        LoginPage lp = new LoginPage(driver);
        ErrorPage ep = lp.incorrectLogin(username, incorrectpassword);
        Assert.assertEquals(ep.getErrorText(), "This is not the error message you're looking for.");
    }
	    
    @Parameters({"correctusername","correctpassword"})
    @Test(description="Performs a successful login and checks whether the Accounts Overview page is opened")
    public void testSuccessfulLogin(String username, String incorrectpassword) {
	         
        LoginPage lp = new LoginPage(driver);
        AccountsOverviewPage aop = lp.correctLogin(username, incorrectpassword);
        Assert.assertEquals(aop.isAt(), true);
    }
	     
    @AfterSuite
    public void tearDown() {
         
        driver.quit();
    }
}

Pretty straightforward, right? I think this is a clear example of why using Page Objects and having the right Page Object methods make writing and maintaining tests a breeze.

Creating the ExtentReports TestNG listener
Next, we need to define the TestNG listener that creates the ExtentReports reports during test execution:

public class ExtentReporterNG implements IReporter {
    private ExtentReports extent;
 
    @Override
    public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
        extent = new ExtentReports(outputDirectory + File.separator + "ExtentReportTestNG.html", true);
 
        for (ISuite suite : suites) {
            Map<String, ISuiteResult> result = suite.getResults();
 
            for (ISuiteResult r : result.values()) {
                ITestContext context = r.getTestContext();
 
                buildTestNodes(context.getPassedTests(), LogStatus.PASS);
                buildTestNodes(context.getFailedTests(), LogStatus.FAIL);
                buildTestNodes(context.getSkippedTests(), LogStatus.SKIP);
            }
        }
 
        extent.flush();
        extent.close();
    }
 
    private void buildTestNodes(IResultMap tests, LogStatus status) {
        ExtentTest test;
 
        if (tests.size() > 0) {
            for (ITestResult result : tests.getAllResults()) {
                test = extent.startTest(result.getMethod().getMethodName());
 
                test.getTest().startedTime = getTime(result.getStartMillis());
                test.getTest().endedTime = getTime(result.getEndMillis());
 
                for (String group : result.getMethod().getGroups())
                    test.assignCategory(group);
 
                String message = "Test " + status.toString().toLowerCase() + "ed";
 
                if (result.getThrowable() != null)
                    message = result.getThrowable().getMessage();
 
                test.log(status, message);
 
                extent.endTest(test);
            }
        }
    }
 
    private Date getTime(long millis) {
        Calendar calendar = Calendar.getInstance();
        calendar.setTimeInMillis(millis);
        return calendar.getTime();        
    }
}

This listener will create an ExtentReports report called ExtentReportTestNG.html in the default TestNG output folder test-output. This report lists all passed tests, then all failed tests and finally all tests that were skipped during execution. There’s no need to add specific ExtentReports log statements to your tests.

Running the test and reviewing the results
To run the tests, we need to define a testng.xml file that enables the listener we just created and runs all tests we want to run:

<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="Parabank login test suite" verbose="1">
	<listeners>
		<listener class-name="com.ontestautomation.extentreports.listener.ExtentReporterNG" />
	</listeners>
	<parameter name="correctusername" value="john" />
	<parameter name="correctpassword" value="demo" />
	<parameter name="incorrectusername" value="wrong" />
	<parameter name="incorrectpassword" value="credentials" />
	<test name="Login tests">
		<packages>
			<package name="com.ontestautomation.extentreports.tests" />
		</packages>
	</test>
</suite>

When we run our test suite using this testng.xml file, all tests in the com.ontestautomation.extentreports.tests package are run and an ExtentReports HTML report is created in the default test-output folder. The resulting report can be seen here.

More examples
More examples on how to use the ExtentReports listener for TestNG can be found on the ExtentReports website.

The Eclipse project I have created to demonstrate the above can be downloaded here.