Are your test automation efforts holding you back?

In case you either have lived under a rock when it comes to testing trends (I don’t say you did), you haven’t paid any attention to what I’ve written in the past (this is much more likely), or possibly both, it seems that everybody in the testing world sort of agrees that automated testing can be highly beneficial or even essential to software development project success. Especially with ever shortening release cycles and less and less time per cycle available for testing.

But why are so many hours spent on creating automated tests nothing more than a waste of time? In this post, I would like to sum up a couple of reasons that cause automated test efforts to go bad. These are reasons I’ve seen in practice, that I had to work with and improve upon. And yes, I have certainly been guilty of doing some of the things I’ll mention in this post myself. That’s how I learned – the hard way.

Your flaky tests are sending mixed signals
One of the most prominent causes of time wasted with test automation is what I like to call ‘flaky test syndrome’. These are tests that pass some of the time and fail at other times. As a result, your test results cannot be trusted upon no matter what, which effectively renders your tests useless. There is a multitude of possible reasons that can cause your tests to become flaky, the most important being timing or synchronization issues. For example: a result is not stored in the database yet when a test step tries to check it, or an object is not yet present on a web page when your test tries to access it. There are a lot of possible solutions to these problems, and I won’t go into detail on every one of them, but what is paramount to test automation success is that your test results should be 100% trustworthy. That is, if a test fails, it should be due to a defect in your AUT (or possibly to an unforeseen test environment issue). Not due to you using Thread.sleep() throughout your tests.


Your test results take too long to analyze
There is no use in having an automated test suite that is running all smoothly when your test results can’t be interpreted in a glance. If you can’t see in five seconds whether a test run is successful, you might want to update your reporting strategy. Also, if you can’t determine at least the general direction to the source of an error from your reporting in a couple of seconds, you might want to update your reporting strategy. If your general response to a failure appearing in your test reports is to rerun the test and start debugging or analyzing from there, you might want to update your reporting strategy.

You put too much trust put in your automated tests and test results
Even though I just stated that you should do everything within your power to make your automated tests as trustworthy as possible, this does by no means imply that you should trust on your automated tests alone when it comes to product quality. Automated tests (or better, checks) can only do so much in verifying whether your application meets all customer demands. So, for all of you having to deal with managers telling you to automate all of the tests, please, please do say NO.

Please don't do this!

You test everything through the user interface
I don’t think I have to tell this again, especially not to those of you that are regular readers of my blog, but I’m fairly skeptical when it comes to GUI-driven test automation. If anything, it makes your test runs take longer and it will probably result in more maintenance efforts as a result of any UI redesign. Setting up a proper UI-based automation framework also tends to take longer, so unless you’re really testing the user interface (which is not that often the case), please try and avoid using the UI to execute your tests as much as possible. It might take a little longer to figure out how to perform a test, but it will be much more efficient in the longer run.

Hopefully these points will trigger you to take a critical look at what you’re doing at any point in your test automation efforts. What also might help is to ask yourself what I think is the most important question when it comes to test automation (and to work in general, by the way):

“Am I being effective, or am I just being busy?”

Let’s make test automation better.

28 thoughts on “Are your test automation efforts holding you back?

  1. To be frank bas,i didn’t understand much.Anyway you write well πŸ™‚
    well i always had this doubt.For a website what all things we should automate?i mean if there is hardly any functionality,like a simple website just displaying images and all we might have to write automated scripts to check if all links are present whether they are active,whether the images displayed are correct etc.right?But automating such websites doesn’t make much sense right?
    On the other hand suppose if i have a website which has lot of functionalities like a login ,searching,lot of textboxes to enter information before submitting then automating by writing a function for each is really useful right?
    Also i argued with my teammates,that suppose there is a website which has lot of information to enter before submitting each time we login to that website,and if we write an automated script which takes input from an excel sheet or something that contains data for entering into each textboxes surely it saves time and sort of make things easy (since entering manually every time is difficult),but that is not the actual purpose of automated testing right?,since we have to check or test something in manual testing or automated testing rather than just inputting lot of data right?

    • Hi Sherin,

      indeed, it does not make sense to write automated test scripts for web sites that only display static information or images.

      You’re also right that testing more complex web sites profits from well-built test automation frameworks with dedicated functions to perform actions such as login, search, filling in forms, etc.

      In short, as Alan Page so nicely put it: automate 100% of the tests that need to be automated. See for an interesting interview with him here.

      As for the Excel test data issue, if you always use the same test data for your test then putting it in Excel or a database might not be necessary, but if you need to update your test data frequently I think it is a very good idea to do so. It also cleanly separates test data from test execution, which is always a good practice.

      • yes but if i take the example of your login parabank demo using extent report,what i am confused is unless we have a large number of username and password login credentials to test,automating it is not that helpful right?I mean we can just manually test it right?Also i told you i am the only tester there so i can decide what to automate and unfortunately there i am confused.since a small company there is no distributing of work like in big companies.and everyone will develop something different.sometimes wordpress sites and mostly php websites.sometimes with lot of functionalities.
        so my question is suppose if i take your parabank demo full website if you are the tester what all things you would have automated?

        • Hmm now i think i am getting somewhere.what i did is i created a dummy username and password in the parabank demo website.After i logged in,i saw many links like open new account,accounts overview,Transfer funds,Request Loan etc.So in case we need to manually test if all this link work correctly,say after one week,then after two week etc. it is better to automate all those links right?

          • But i think there is a better reason than this.suppose if the parabank wants to add a link in future.During that time if we need to check the newly implemented module has any effect on previous one,surely automation helps right?but how to design the framework-should we go for inheritence or page objects or write individual functions for everything say checking lot of links that also is a tricky part right?

          • Hi Sherin,

            yes, in the case of a web application with clearly defined functions per page (such as the Parabank demo application) the Page Objects pattern fits perfectly.

    • Hi Sherin,

      no, I don’t use Skype, I prefer to communicate via email and Twitter where possible. Or blog post comments in this case πŸ™‚ I usually work at client’s offices and I can’t (and don’t want to) hold unrelated Skype conversations there, plus there’s always the time difference which makes things more complicated.

      What do you mean by family place?

  2. Pingback: Testing Bits – 7/12/15 – 7/18/15 | Testing Curator Blog

  3. Hmm i saw that interview.Alan Page is a very famous tester?The blog in which he gives the interview,famous as well? I asked about skype to you seeing that interview πŸ™‚

      • GUI test automation refers to automating tests on the user interface level, i.e., using the user interface to manipulate the application under test. This can be a web page, but also a Windows desktop application or a terminal application, as long as the user interface is used to perform tests.

        • which means the coding and tools used won’t change instead they are hidden right?so if there is a button having name-Test Links when we click on it,our
          usual selenium webdriver code is run on the background right?so if such an application is developed can it test all websites or only the website it is developed for?

          • I think you’re confusing this with something else..

            Anyway, tests are normally specifically written for a single application or website. The test framework you’re using (which can include Selenium WebDriver) could be developed to handle more than a single application.

          • Selenium IDE? That is used to create and perform tests on the user interface (or GUI) level. It also has a user interface (GUI) itself.

            Selenium WebDriver is also used for GUI-level tests, but it doesn’t have a GUI, it’s a Java library.

          • That’s a development tool and has nothing to do with automated tests per se. You can use Eclipse to develop automated tests, though (which I do).

    • Alan Page is pretty well known in the testing world, yes. I hadn’t heard about the blog that this interview appears on before, saw it mentioned on Twitter I think.

      • Hi Bas,
        Do you have any previous automation script where you automated a full website.i mean since you worked as a test engineer you might have a sample right?if there is send to ok?i think it might be very useful for me to get a complete idea.otherwise leave it ok?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.