Running your tests in a specific order

General consensus within the test automation community is that your automated tests should be able to run independently. That is, tests should be runnable in any given order and the result of a test should not depend on the outcome of one or more previous tests. This is generally a good practice and one I try and adhere to as much as possible. Nothing worse than a complete test run being wasted because some initial test data setup actions failed, causing all other tests to fail as well. Especially when you’re talking about end-to-end user interface tests that only run overnight because they’re taking so awfully long to finish.

However, in some cases, having your tests run in a specific order can be the most pragmatic or even the only (practical) option. For example, in my current project I am automating of a number of regression tests that require specific test data to be present before the actual checks can be performed. And since:

  • I don’t want to rely on test data that’s already being present in the database
  • Restoring a database snapshot is not a feasible option (at least not at the moment)
  • Creation of suitable test data takes at least a minute but closer to two for most tests. This has to be done through the user interface since a lot of data is stored in blobs, making SQL updates a challenging strategy to say the least..

the only viable option is to make the creation of test data the first step in a test run. Creating the necessary test data before each and every individual test would just take too long.

Since we’re using SpecFlow, we’re creating test data in the first scenario in every feature. All other scenarios in the feature rely on this test data. This means that the test data creation scenario needs to be run first, otherwise the subsequent scenarios will fail. Using Background is not an option, because those steps are run before each individual scenario, whereas we want to run the test data creation steps once every feature.

The above situation is just one example of a case where being able to control the execution order of your tests can come in very useful. Luckily, most testing frameworks support this in one or more ways. In the remainder of this post, we’re going to have a look at how you can define test execution order in JUnit, TestNG and NUnit.

JUnit
Before version 4.11, JUnit did not support controlling the test execution order. However, newer versions of the test framework allow you to annotate your test classes using @FixMethodOrder, which enables you to select one of various MethodSorters. For example, the tests in this class are run in ascending alphabetical order, sorted by test method name:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class JUnitOrderedTests {
	
	@Test
	public void thirdMethod() {		
	}
	
	@Test
	public void secondMethod() {		
	}
	
	@Test
	public void firstMethod() {		
	}
}

Running these tests show that they are executed in the specified order:
JUnit FixMethodOrder result

TestNG
TestNG offers no less than three ways to order your tests:

Using preserve-order in the testng.xml file
You can use the preserve-order attribute in the testng.xml file (where you specify which tests will be run) to have TestNG run the tests in the order they appear in the XML file:

<test name="OrderedTestNGTests" preserve-order="true">
	<classes>
		<class name="TestNGTestClass">
			<methods>
				<include name="testOne" />
				<include name="testTwo" />
			</methods>
		</class>
	</classes>
</test>

Using the priority attribute
You can also use the priority attribute in your @Test annotation to prioritize your test methods and determine the order in which they are run:

public class TestNGPrioritized {
	
	@Test(priority = 3)
	public void testThree() {		
	}
	
	@Test(priority = 1)
	public void testOne() {		
	}
	
	@Test(priority = 2)
	public void testTwo() {		
	}	
}

Using dependencies
In TestNG, you can have tests and test suites depend on other tests / test suites. This also implicitly defines the order in which the tests are executed: when test A depends on test B, test B will automatically be run before test A. These dependencies can be defined in code:

public class TestNGOrderedTests {
	
	@Test(dependsOnMethods = {"parentTest"})
	public void childTest() {		
	}
	
	@Test
	public void parentTest() {		
	}
}

This works on method level (using dependsOnMethods) as well as on group level (using dependsOnGroups). Alternatively, you can define dependencies on group level in the testng.xml file:

<test name="TestNGOrderedTests">
	<groups>
		<dependencies>
			<group name="parenttests" />
			<group name="childtests" depends-on="parenttests" />
		</dependencies>
	</groups>
</test>

NUnit
NUnit (the .NET version of the xUnit test frameworks) does not offer an explicit way to order tests, but you can control the order in which they are executed by naming your methods appropriately. Tests are run in alphabetical order, based on their method name. Note that this is an undocumented feature that may be altered or removed at will, but it still works in NUnit 3, which was recently released, and I happily abuse it in my current project..

At the beginning of this post, I mentioned that in my current project, we use SpecFlow to specify our regression tests. We then execute our SpecFlow scenarios using NUnit as a test runner, so we can leverage this alphabetical test order ‘trick’ by naming our SpecFlow scenarios alphabetically inside a specific feature. This gives us a way to control the order in which our scenarios are executed:

Scenario: 01 Create test data
	Given ...
	When ...
	Then ...
	
Scenario: 02 Modify data
	Given ...
	When ...
	Then ...

Scenario: 03 Remove modified data
	Given ...
	When ...
	Then ...

Again, it is always best to create your tests in such a way that they can be run independently. However, sometimes this just isn’t possible or practical. In those cases, you can employ one of the strategies listed in this post to control your test order execution.

10 thoughts on “Running your tests in a specific order

  1. Pingback: Testing Bits – 3/27/16 – 4/2/16 | Testing Curator Blog

  2. Hi Bas,

    I’m not doing much automation myself but I like to pick up useful pieces of info when I can so that I know a bit about what is possible, difficult, or problematic. I would think this can be very handy in many cases.

    However, and this might be a silly question; unless your data creation is actually part of the regression checks, can you not take care of that in a initial step before running the checks? I seem to remember some @Before, and @Beforeclass (or something like that) that together with the similar @After can take care of setup and teardown. I assume you are more familiar with those than I am so it makes me wonder. Perhaps they are not supported by your favourite framework?

    Cheers,
    Geir

    • Hi Geir, you make a very valid point. In most cases, you would do test data setup in a @BeforeXYZ method. They’re supported by most major test frameworks including those I like to use. In hindsight I think that wasn’t the best way to explain a situation where running tests in a specific order could come in handy. A better example might be when you want to test CRUD operations and setting up the initial conditions might take a lot of time and/or effort. Having four tests that do Create, Read, Update and Delete and running them in that specific order could be A (I’d never say THE) solution..

      • Cool, thanks for elaborating.
        Setting up perfect examples for explaining something can be tricky. Some people just jump in an ask about details that wasn’t really the point of the discussion 🙂
        I am sure there are many cases where forcing a specific running order makes sense even if it might not be considered a good practice in general.

  3. Pingback: Java Testing Weekly 14 / 2016

  4. Hi Bas,
    Domain: E-commerce

    I am running a test suite through a command line. In a class, i have few test methods which has priority attribute set in @Test to maintain the test order.

    My driver object I am creating in the @BeforeClass method and closes the browser in @AfterClass.

    The challenge is when I run the test suite it executes test1 in a class and then close the browser.

    after a while, it tries to execute test2, now as while executing test1 it has already closed the browser when it tries to execute test2 it is throwing Null Pointer execution as it browser object is NULL.

    public Class PLPdeptFilter extends Testbase{

    @Test(priority = 0)
    public void filterdept{
    }

    @Test(priority = 1)
    public void searchFilterdept{
    }
    }//end of class

    I am unable to understand why it executes only one test method in a class and then tries to execute another test method in another class. Then after it comes back to execute the second test.

    I shall appreciate if you show me the direction.

    • Hey Dipesh,

      from the annotations you used, I assume you’re using TestNG? Have you tried debugging your test run? I can’t think of any obvious reason for this behavior, to be honest..

  5. Hi bas

    I landing on this page out of the blue but a good to know basis.

    I currently have all my test run independent from each other, where my login is in a before scenario so every test has to go through that process.

    And logout is a test feature in its self, so that on gets rated once.

    I have been looking at using a cookie there after the first login in and use that for all other test going forward so I don’t have to call that method each time after the first successful test case, so all test would use that cookie for login to save some time.

    But as per setting priority I’ve not had a need to really go down that route due to making them not dependent on being ran after or before certain test.

    But thanks for the write up.

Leave a Reply

Your email address will not be published. Required fields are marked *