Remember what your tests are trying to verify

Lately, I’ve been working a lot on Selenium-based test automation solutions. And even though I’m still not overly enthusiastic about creating lots of user interface-driven automated tests, now that I’m getting more skilled at creating robust and stable Selenium tests, I am starting to appreciate the tool and what you can do with it more and more. As everybody working with Selenium can tell you, there are situations where things can get, well, interesting. And by interesting, I mean tricky. Dealing with dynamic front end frameworks and unpredictable modals and overlays can ask a lot of you in terms of debugging and exception handling skills.

Not yet being fluent in Selenese, I find myself on Google and (subsequently) StackOverflow a lot, trying to find suitable solutions for problems I encounter. While doing so, there’s one thing that I see a lot of people do, yet strikes me as really odd, given what I think is the purpose of these user interface-driven tests:

Forcing your tests to do something your users can’t.

From what I’ve understood, studying and working in the test automation field for a while now, user interface-driven tests, such as these Selenium tests, should be used to verify that the user of your user interface is able to complete a sequence of predefined actions. If you’re working in a hip and happening environment, these are often called ‘customer journeys’ or ‘user journeys’. But that’s not my point. What my point is, is that quite often I see workarounds suggested that go beyond what a regular user could do with his or her keyboard and / or mouse.

For example, take an element that is invisible until you hover over it with your mouse. If you just try to do a click(), your test will probably throw an exception stating that the element was not visible. Now, there are (at least) two ways to deal with this situation:

  1. Use the Actions class to simulate a mouseover, then click the element.
  2. Use a JavaScriptExecutor to perform the click.

While both approaches might result in a passing test, I am of the opinion that one is useful and the other is a horrible idea. Based on what I’ve written so far, can you guess which of the two options I’d suggest?

Indeed, option #1 is the way to go, for two reasons:

  • User interface-driven tests should mimic actual user interaction as closely as possible. I’ve never seen a user execute some JavaScript on a page to make an element visible. Either that, or I’m hanging around the wrong group of users..
  • What happens if a front-end developer makes a mistake (I know, they never do, but let’s assume so anyway) which causes the element not to become visible, even on a mouseover? With option #1, your test will fail, for the right reason. With #2, hello false negative!

There are some exceptions to the rule, though. The prime example I can think of is handling file uploads by directly sending the absolute path of the file to be uploaded to the input element responsible for the file upload using sendKeys(), instead of clicking on it and handling the file dialog. I’ve tried the latter before, and it’s a pain, first because you can’t do it with the standard Selenium API (because the file dialog is native to the operating system), second because different browsers use different file dialog layouts, resulting in a lot of pesky code that easily breaks down. In this case, I prefer to bypass the file dialog altogether (it’s probably not the subject of my test anyway).

In (almost) all other cases though, I’d advise you to stick to simulating your end user behavior as closely as possible. The job of your user interface-driven tests, and therefore of you as its creator, is not to force a pass, but to simulate end user interaction and see if that leads to a successfully executed scenario. Don’t fool yourself and your stakeholders by underwater tricks that obscure potential user interface issues.

Remember what your tests are trying to verify.

9 thoughts on “Remember what your tests are trying to verify

  1. Bas,
    General Rule of Thumb for GUI automation is to use the basic intrinsic Methods and Properties of an Object first. These would include MouseOver. Also, the property “Visible” (along with Enabled) should be checked on an object before any other type of action is performed. The tester should do that as part of the script they have built. This will build in some level of Fault Tolerance, or better yet detection & prevention. And this isn’t cheating the test, but is part of making sure you can do something or not. If not then you flag it and fail.

    And there are times when you do need to knowingly cheat. Text fields that have AJAX in them are notorious for not taking a SET method, you have to use SENDKEYS (Type in other tools) to push the information in and trigger the AJAX correctly. Sometimes you have to say screw it and do something that a “normal” user wouldn’t do.

    But your point is valid. Do it the right way first.

    • Thanks for your insights Jim! Part of those I cover by wrapping Selenium API calls in custom methods that deal with waiting until something’s clickable etc.

      Not sure in how far I agree with the ‘screw it’ thing because that’s a fine line you’re walking.. Simulating end user behavior as best as you can versus forcing a ‘green’ by having your tests do stuff your users can’t..

      • That all depends on what the test is trying to verify.

        I find it helpful to think in terms of pre-conditions, actions, and post-conditions. Ideally, each test should have three phases:

        1. Establish pre-conditions
        2. Perform one action
        3. Verify one post-condition

        If a test is modeled this way, then only phase 2 concerns user actions and should therefore be free of any activity that a user cannot perform. If black magic in phases 1 and 3 helps you to get through the day or produces more reliable tests, then I’m all for it.

        Tests designed this way bear more resemblance to unit tests than manual tests and have the distinct advantage of isolating the verification points so a failure at one has no impact on the others.

        It also forces you to do exactly what the headline of your post suggests.

        • That’s a very good point you make there, Mike. I was thinking about ‘step 2’ actions (as you call them) myself when writing this post, but failed to make that clear enough, obviously.

          By the way, your phasing looks very similar to the common arrange/act/assert or given/when/then patterns. I assume you’re familiar with those πŸ™‚

  2. Pingback: Java Web Weekly, Issue 176 | Baeldung

  3. Pingback: Testing Bits – 5/7/17 – 5/13/17 | Testing Curator Blog

  4. Pingback: Java Testing Weekly 20 / 2017

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.