Why there’s no such thing as codeless automation

In today’s blog post – which, again, is really nothing more than a thinly veiled rant – I’d like to cover something that’s been covered before, just not by me: codeless test automation and why I think there isn’t and should not be such a thing.

I’ve seen numerous ‘solution’ vendors advertise their products as ‘codeless’, implying that everybody in the team will be able to create, run and maintain automated tests, without having to, well, write code. I’ve got a number of problems with selling test automation in this way.

It’s not codeless. It’s hiding code.
The first gripe I have with ‘codeless’ automation is a semantic one. These solutions aren’t codeless at all. They simply hide the code that runs the test from plain sight. There are no monkeys in the solution that magically execute the instructions that make up a test. No, those instructions are translated into actual code by the solution, then executed. As a user of such a solution, you’re still coding (i.e., writing instructions in a manner that can be interpreted by a machine), just in a different syntax. That’s not codeless.

While it might be empowering, it’s also limiting.
Sure, using codeless tools might potentially lead to more people contributing to writing automated tests (although from my experience, that’s hardly how it’s going to be in the end). The downside is: it’s also limiting the power of the automated tests. As I said above, the ‘codeless’ solution is usually nothing more than an abstraction layer on top of the test automation code. And with abstraction comes loss of detail. In this case, this might be loss of access to features of the underlying code. For example, if you’re using a codeless abstraction on top of Selenium, you might lose access to specific waiting, synchronization or error handling mechanisms (which are among the exact things that makes Selenium so powerful).

It might also be loss of access to logging, debugging or other types of root cause analysis tools, which in turn leads to shallower feedback in case something goes wrong. While the solution might show you that something has gone wrong, it loses detail on where things went wrong and what caused the failure. Not something I like.

Finally, it might also limit access to hooks in the application, or limit you to a specific type of automated tests. If such a solution makes it potentially easier to write automated tests on the user interface level, for example, there’s significant risk that all tests will be written at that level, even though that might not be the most efficient approach in the first place. If all you’ve got is a hammer…

It’s doing nothing for the hard problems in creating maintainable automation.
Let’s face it: while writing code might seem hard to people that haven’t done it before, it actually isn’t that difficult once you’ve had a couple of basic programming classes, or followed a course or two on Codecademy. What is hard is writing good, readable, maintainable code. Applying SOLID and DRY principles. Structuring your tests. Testing the right thing at the right level. Creating a solid test data and test environment strategy. Those things are hard. And codeless test automation does nothing for those problems. As I tried to make clear in the previous paragraphs, it’ll often make it even harder to solve those problems effectively.

I’m all for creating solutions that make it easier to write, run and maintain automation. I hate people selling solutions as something they’re not. Codeless test automation is not going to solve your test automation problems. People that know

  • how to decide what good automation is
  • how to write that automation, and
  • how to pick the tools that will help them achieve the goals of the team and organization

will.

15 thoughts on “Why there’s no such thing as codeless automation

  1. Excellent points. I too have shuddered at the ‘lego stones’ metaphor in relation to BPM solutions for example: just write applications with drag and drop.
    It all comes down to Brooks’ essential complexity and no fancy framework is going to take that off your hands.

    I prefer the Cucumber scripting approach: a non-programmer writes the acceptance criteria in the gherking language, and the developers then edit it and hook it into executable test code.

  2. Funny thing is is that I’ve heard people call Cucumber/Gherkin a “codeless” approach to automation. But as you have pointed out (both of you) is that there is underlying code that has to be created and maintained.

    I’ve been fighting/arguing this idea of “Codeless/Scriptless” for the past few years. As has been said it is an extrapolation layer method that these tools use. When I build BPT (Business Process) components it is a building block that a tester can use with the DSL (Domain Specific Language) for it to pull together a scenario. But it does have some limitations as stated. If I need to do something more detailed and deep then I’ll write more specific code for it.

    The real problem isn’t the tools themselves. It is both the way they are marketed/sold (Automagic anyone) and the misconceptions by unknowing & inexperienced people who look to them as a silver bullet.

    The impetus is upon us as test professionals to educate and enlighten those people. I’ve fought with a couple of vendors about how they market/sell their tools. And it has been hit and miss. I’ve also made a point of educating the other side of the equation, typically management, on the myths and misconceptions surrounding automation of testing with Codeless/Scriptless being a new one to watch out for.

    So do these tools have some benefit and merit, yes. They do have some good capabilities and functionality in them. Are they being promoted/sold the wrong way, yes. Can we fix this situation, yes and only if we can get the conversation going first.

    Because in the end it is really the computer between your ears that does the work. It’s Automation, Not Automagic!

  3. I would add one more bullet:

    – How to pick the right test cases to automate.

    Having more people to automate more test cases isn’t always the right answer. You may end up with way too many test cases and not the right coverage.

    • Hey Bill, that’s one of the hard problems in automation as well. However, to me, that’s more related to the question of ‘what?’ to automate, where this blog post mainly deals with another question: that of ‘how?’ to automate. Would you agree?

      P.S. say hi to Paul G. from me 🙂

      • Agreed. The problem of what to automate sometimes leads organizations to think they need as many people as possible creating as many tests as possible and clouds the decision making process… It’s important to have the right strategy before you choose a tool (IMHO).

        P.S. I will!

  4. Thank you for this article. I agree with it, especially that there isn’t anything really “magical” going on, and the marketing of a lot of these tools is basically over-promising and under-emphasizing the complexity.

    As creator of snaptest.io, this issue has been close to home because I’ve had to write some marketing copy recently. I chose not to say “codeless” because one of the main goals is to have people get the actual code behind the generation and use it and let it complement manually written ones. Also, I’ve encouraged writing your own custom generator to help you organize how you’d like.

    This doesn’t solve the “organization” of code issue, which is one of the hardest, but that’s where i feel a generator actually helps. I recently worked in a startup that didn’t have time to hire a QA expert (I’m a fullstack engineer and have tons to do)… So I made the tool so I didn’t HAVE to keep organizing things, especially as products change a lot. It’s saved me tons of hours of moving things around and maintaining because I can just adjust the generator and it applies to ALL generated tests (not the custom ones I write alongside them).

    • Hey Joseph, thank you for your insights from ‘the other side’ (though I don’t want to create a divide here!). It’s great to hear that you’re aware of the snake oil that comes with too many codeless solutions and that you’re actively trying to improve. Let me be clear that I don’t have a gripe with tools that give people the means to create tests more efficiently (anything but) but what I can’t stand is people selling tools by saying that anybody can create tests with their product. Maybe that’s true but the end result won’t be pretty.

  5. I believe a sweet spot can be found where the tool gives you the ability to compose an automated process with a DSL whilst also allowing small code snippets or expressions to be injected in places. But I do agree that such a tool cannot make the claim that “anyone” can do it. The ability to compose automated tasks in a useful and efficient manner that eliminates redundancy requires skill in itself. This is the bread and butter of developers but I’ve seen that testers are very good at it too.

  6. Hey Bas – AS usual a great article. While I agree with your larger point on how effective the tests will be when they are written in this way. However, on your point on,

    “t might also be loss of access to logging, debugging or other types of root cause analysis tools, which in turn leads to shallower feedback in case something goes wrong”
    What if we build this mechanism in the backend and help the user who is building a script to do course correction?

    In summary – apart from the classic problems that you have highlighted, dont you think things related to logs, error handling etc which are more technical in nature are problems which can be easily solved?

    • Hey Ameet,

      depends on how you define ‘solved’. Sure, it shouldn’t be hard to define something that keeps track of logs and exceptions, but to be able to extract the right data from those sources and translate them to meaningful information might not be so trivial.

  7. Pingback: Testing Bits – 9/24/17 – 9/30/17 | Testing Curator Blog

  8. Hi Bas,

    I enjoyed your ‘rant’ on codeless automation.

    One thing I think the test tool needs to do is be very robust and helpful, so that a tester (test developer, actually) does not get into trouble.

    The tool needs to take responsibility for everything it *can* take responsibility for.

    Over at GitHub, I’m working on something that I think does that: project RubyTest.

    Currently I’m extending a Tester Tour:

    https://github.com/BurdetteLamar/RubyTest/blob/master/examples/rest_api/TesterTour.md

    which shows how to use the example REST API framework that’s part of RubyTest.

    The project main page:

    https://github.com/BurdetteLamar/RubyTest

    P.S.: Here’s Bret Pettichord’s ‘rant’, about how test tools are sometimes sold:

    https://www.youtube.com/watch?v=s_CUPs6xAWw

Leave a Reply

Your email address will not be published. Required fields are marked *