Running your tests in a specific order

General consensus within the test automation community is that your automated tests should be able to run independently. That is, tests should be runnable in any given order and the result of a test should not depend on the outcome of one or more previous tests. This is generally a good practice and one I try and adhere to as much as possible. Nothing worse than a complete test run being wasted because some initial test data setup actions failed, causing all other tests to fail as well. Especially when you’re talking about end-to-end user interface tests that only run overnight because they’re taking so awfully long to finish.

However, in some cases, having your tests run in a specific order can be the most pragmatic or even the only (practical) option. For example, in my current project I am automating of a number of regression tests that require specific test data to be present before the actual checks can be performed. And since:

  • I don’t want to rely on test data that’s already being present in the database
  • Restoring a database snapshot is not a feasible option (at least not at the moment)
  • Creation of suitable test data takes at least a minute but closer to two for most tests. This has to be done through the user interface since a lot of data is stored in blobs, making SQL updates a challenging strategy to say the least..

the only viable option is to make the creation of test data the first step in a test run. Creating the necessary test data before each and every individual test would just take too long.

Since we’re using SpecFlow, we’re creating test data in the first scenario in every feature. All other scenarios in the feature rely on this test data. This means that the test data creation scenario needs to be run first, otherwise the subsequent scenarios will fail. Using Background is not an option, because those steps are run before each individual scenario, whereas we want to run the test data creation steps once every feature.

The above situation is just one example of a case where being able to control the execution order of your tests can come in very useful. Luckily, most testing frameworks support this in one or more ways. In the remainder of this post, we’re going to have a look at how you can define test execution order in JUnit, TestNG and NUnit.

Before version 4.11, JUnit did not support controlling the test execution order. However, newer versions of the test framework allow you to annotate your test classes using @FixMethodOrder, which enables you to select one of various MethodSorters. For example, the tests in this class are run in ascending alphabetical order, sorted by test method name:

public class JUnitOrderedTests {
	public void thirdMethod() {		
	public void secondMethod() {		
	public void firstMethod() {		

Running these tests show that they are executed in the specified order:
JUnit FixMethodOrder result

TestNG offers no less than three ways to order your tests:

Using preserve-order in the testng.xml file
You can use the preserve-order attribute in the testng.xml file (where you specify which tests will be run) to have TestNG run the tests in the order they appear in the XML file:

<test name="OrderedTestNGTests" preserve-order="true">
		<class name="TestNGTestClass">
				<include name="testOne" />
				<include name="testTwo" />

Using the priority attribute
You can also use the priority attribute in your @Test annotation to prioritize your test methods and determine the order in which they are run:

public class TestNGPrioritized {
	@Test(priority = 3)
	public void testThree() {		
	@Test(priority = 1)
	public void testOne() {		
	@Test(priority = 2)
	public void testTwo() {		

Using dependencies
In TestNG, you can have tests and test suites depend on other tests / test suites. This also implicitly defines the order in which the tests are executed: when test A depends on test B, test B will automatically be run before test A. These dependencies can be defined in code:

public class TestNGOrderedTests {
	@Test(dependsOnMethods = {"parentTest"})
	public void childTest() {		
	public void parentTest() {		

This works on method level (using dependsOnMethods) as well as on group level (using dependsOnGroups). Alternatively, you can define dependencies on group level in the testng.xml file:

<test name="TestNGOrderedTests">
			<group name="parenttests" />
			<group name="childtests" depends-on="parenttests" />

NUnit (the .NET version of the xUnit test frameworks) does not offer an explicit way to order tests, but you can control the order in which they are executed by naming your methods appropriately. Tests are run in alphabetical order, based on their method name. Note that this is an undocumented feature that may be altered or removed at will, but it still works in NUnit 3, which was recently released, and I happily abuse it in my current project..

At the beginning of this post, I mentioned that in my current project, we use SpecFlow to specify our regression tests. We then execute our SpecFlow scenarios using NUnit as a test runner, so we can leverage this alphabetical test order ‘trick’ by naming our SpecFlow scenarios alphabetically inside a specific feature. This gives us a way to control the order in which our scenarios are executed:

Scenario: 01 Create test data
	Given ...
	When ...
	Then ...
Scenario: 02 Modify data
	Given ...
	When ...
	Then ...

Scenario: 03 Remove modified data
	Given ...
	When ...
	Then ...

Again, it is always best to create your tests in such a way that they can be run independently. However, sometimes this just isn’t possible or practical. In those cases, you can employ one of the strategies listed in this post to control your test order execution.

34 thoughts on “Running your tests in a specific order

  1. Pingback: Testing Bits – 3/27/16 – 4/2/16 | Testing Curator Blog

  2. Hi Bas,

    I’m not doing much automation myself but I like to pick up useful pieces of info when I can so that I know a bit about what is possible, difficult, or problematic. I would think this can be very handy in many cases.

    However, and this might be a silly question; unless your data creation is actually part of the regression checks, can you not take care of that in a initial step before running the checks? I seem to remember some @Before, and @Beforeclass (or something like that) that together with the similar @After can take care of setup and teardown. I assume you are more familiar with those than I am so it makes me wonder. Perhaps they are not supported by your favourite framework?


    • Hi Geir, you make a very valid point. In most cases, you would do test data setup in a @BeforeXYZ method. They’re supported by most major test frameworks including those I like to use. In hindsight I think that wasn’t the best way to explain a situation where running tests in a specific order could come in handy. A better example might be when you want to test CRUD operations and setting up the initial conditions might take a lot of time and/or effort. Having four tests that do Create, Read, Update and Delete and running them in that specific order could be A (I’d never say THE) solution..

      • Cool, thanks for elaborating.
        Setting up perfect examples for explaining something can be tricky. Some people just jump in an ask about details that wasn’t really the point of the discussion 🙂
        I am sure there are many cases where forcing a specific running order makes sense even if it might not be considered a good practice in general.

        • Another example is the “forced password reset” scenario.

          A user’s password is reset to password1

          User logs in with password1

          User is forced to reset password, sets to password2

          [User logs out, if neccessary]

          User logs in with password2

  3. Pingback: Java Testing Weekly 14 / 2016

  4. Hi Bas,
    Domain: E-commerce

    I am running a test suite through a command line. In a class, i have few test methods which has priority attribute set in @Test to maintain the test order.

    My driver object I am creating in the @BeforeClass method and closes the browser in @AfterClass.

    The challenge is when I run the test suite it executes test1 in a class and then close the browser.

    after a while, it tries to execute test2, now as while executing test1 it has already closed the browser when it tries to execute test2 it is throwing Null Pointer execution as it browser object is NULL.

    public Class PLPdeptFilter extends Testbase{

    @Test(priority = 0)
    public void filterdept{

    @Test(priority = 1)
    public void searchFilterdept{
    }//end of class

    I am unable to understand why it executes only one test method in a class and then tries to execute another test method in another class. Then after it comes back to execute the second test.

    I shall appreciate if you show me the direction.

    • Hey Dipesh,

      from the annotations you used, I assume you’re using TestNG? Have you tried debugging your test run? I can’t think of any obvious reason for this behavior, to be honest..

  5. Hi bas

    I landing on this page out of the blue but a good to know basis.

    I currently have all my test run independent from each other, where my login is in a before scenario so every test has to go through that process.

    And logout is a test feature in its self, so that on gets rated once.

    I have been looking at using a cookie there after the first login in and use that for all other test going forward so I don’t have to call that method each time after the first successful test case, so all test would use that cookie for login to save some time.

    But as per setting priority I’ve not had a need to really go down that route due to making them not dependent on being ran after or before certain test.

    But thanks for the write up.

    • If you’re using SpecFlow (or any other tool that supports BDD) you can use tags to group scenarios and run specific sets of them.

  6. Hi Bas,

    I am Sanjoy, currently working as a automation QA professional. I am facing several challenges while I am designing automation framework. For Ex: 1) A user can add products to there purchase list. 2) User, User admin, client side admin can remove product from a PL.

    Now if adding functionality is failing all the depends functionality is also failing. The point is how can be require test data independent from each other. I am using TestNG in my project.

    How I can avoid these dependency and make my framework more robust.

    • Hey Sanjoy, are you currently adding products through the UI? If so, can you insert them directly in the database or use an API to create them for the tests that modify / delete existing products? No need to use the UI for those cases.

  7. Hi Bas,
    I go through this blog and I am realy very happy to find this blog but I just want to know that I am using Junit 3.8.1
    How to run the feature files in specific order like 1st it will run the Login feature file than others, because generally it runs alphabetically so login feature file run after some other feature files.

    • Running tests in a specific order isn’t a good idea to be honest. If you really need specific actions to complete before other tests it’s better to put them in a setup method.

  8. This has been very helpful.
    I am using Specflow+C#+NUnit and I have struggled to order tests to take advantage of steps conditioned at the start of execution.

    Implementing this has reduced my execution cycle by 50%.

    Thank you so much.

    • You’re welcome! Always good to hear that people get value from what I write. Even when I would now probably try and argue with you why having tests depend on one another isn’t a good idea lots of times 🙂

  9. Hi Bas,

    How can we set the order of execution in a Serenity BDD Framework, which is using SerenityRunner for execution?

    Thanks in advance!

    • Hey! I’ve never really used Serenity so I don’t know. I guess I don’t have to warn you that having to rely on the order of execution is a big antipattern? 😉

  10. hey, wanted to know about the nunit order
    is the numbers is enough to make it run in the right order or also the feature name must be in alphabetical order?

    01 first
    02 second

    or its must be

    01 A first
    02 B second

    • The numbers are enough, those become part of the test name and NUnit runs them in alphabetical order. Not sure how this works on feature level to be honest but it did work on scenario level. Again, it’s best to make them independent of one another 🙂

  11. Hey, In Automation If will run all the script with one shot in browser, chances is there to fail script or is there any other way to Running a test cases in on flow.
    With out Closing the browser we run all test cases will driver will support to complete all Test cases.
    Please suggest me.
    I am using Dot net, C# coding, Nunit.

    • I’m not sure I understand your question correctly, but do you want to run all your tests within the same browser instance, so without closing the browser after a test? Why would you want to do that? What’s the added benefit of doing that? I don’t think it’s a good idea.

      • Yes, some Requirement is there it ‘s not possible Right.
        All scenario we can do it in one test cases?
        Step by step in one browser
        Eg; Opening the browser
        login in >> Add new person details>> Edit person Details>>Doing payment. Etc…
        Please Suggest me if any chances are there.

  12. When I am trying to run mutliple packages at the same time in TestNg for 308 classes in total, the order of execution is getting changed and many test cases are being run parallely. Due to this, many test cases are failing. Please help me resolve this

    • The only sensible thing to do is to make sure that your tests *can* run in parallel, i.e., they should not depend on one another.

Leave a Reply to Geir Gulbrandsen Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.