Defining Continuous Testing for myself

On a couple of recent occasions, I found myself talking about Continuous Testing and how test automation related to this phenomenon (or buzzword, if you prefer that term). However, to this day I didn’t have a decent answer to the question of what Continuous Testing (CT) is and how exactly it relates to test automation. Now that I’m busy preparing another webinar (this time together with the people at Testim, but more on that probably in another post), and we find ourselves again talking about CT, I thought it was due time to start carving out a definition for myself. This blog post is much like me thinking out loud, so bear with me.

To start, CT is definitely not equal to test automation, not even to test automation on steroids (i.e., test automation that is actually fast, stable and repeatable. Instead, I see CT as an integrated part of the Continuous Delivery (CD) life cycle:

Continuous Testing as part of the Continuous Delivery life cycle

You could also say that CT is a means of that teams can adopt to support CD while aiming to deliver quality software.

Let’s take a closer look and dissect the term ‘Continuous Testing’. The first part, ‘Continuous’, to me is a term that means two things in the CD context:

  1. The software that is being created is continuously tested (or, more likely, continually) in a given environment. In other words, any version of the software that enters an environment is immediately subjected to the tests that are associated with that environment. This environment can be a local development environment, a test environment, or even a production environment (where testing is often done in the form of monitoring).
  2. The software that is being created is continuously tested as it passes through environments. In other words: there’s no deploy that is not being followed by some form of testing, and in case of the local development environment, tests are run before any deployment (or better, commit and build) is being done.

Continuous Testing in two dimensions

This is not necessarily different from any other software delivery method, but what makes CT (and CD) stand out is that the time between deployments is typically very short. And that’s where the second part of the term Continuous Testing comes into play: ‘Testing’. How is this testing being done? This is where automation often comes into play, as an enabler of fast feedback and software moving through the pipeline fast, yet in a controlled manner.

Teams that want to ‘do’ CT and CD simply cannot be blocked by testing as an activity tacked on at the end. Instead, CT requires a shift of mind from traditional testing as an afterthought to testing being ingrained throughout the pipeline. Formalized handoffs and boundaries between environments will have to be replaced by testing activities that act as gatekeepers and safety nets. And where necessary and useful, this testing is supported by tools. In this respect, test automation can be an enabler of Continuous Testing. But (as is so often the case), only if that automation makes sense. Again, I refer to this blog post if you want to know what I mean by ‘making sense’ in a CT context.

I still don’t have a one line, encyclopedia-style definition for Continuous Testing, but at least the thought process I went through to write this (short) blog post helped me put some things into place. Next to Katrina Clokie’s book on testing in DevOps, the following articles have been a source of information for me (while being much better written than these ramblings):

What is continuous testing? Know the basics of this core safety net for DevOps teams
A real-world guide to continuous testing
What is Continuous Testing?
Continuous Testing vs. test automation (whitepaper)
The Great Debate: Automated Testing vs Continuous Testing

What is your definition of Continuous Testing? Do you have one?

Continuous Delivery and user interface-driven test automation: does that compute?

In this post, I’d like to take a closer look at the combination of Continuous Delivery on the one hand and automated tests at the user interface level on the other hand. Is this a match made in heaven? In hell? Or is the truth somewhere out there in between? (Hint: as with so many things in life, it is..).

Continuous Delivery (CD) is an approach in which software development teams produce software and release it into the production environment in very short cycles. Automation of building, testing and deploying the software often is a vital part in achieving CD. Since this is a blog on testing and test automation, I’ll focus on that, leaving the topics of build and deployment automation to those more experienced in that field.

Automated tests on the user interface level (such as those built using Selenium) traverse your complete application from the user interface to the database and back and can therefore be considered as end-to-end tests. These tests are often:

  • relatively slow to execute, since they require firing up an actual browser instance, rendering pages, dealing with page loading times, etc.,
  • demanding in terms of maintenance, since the user interface is among the components of an application that are most prone to change during the software life cycle, and
  • brittle, because object on a web page or in an application are often dynamically generated and rendered and because wait times are not always predictable, making synchronization a tough issue to tackle correctly.

So, we have CD striving for fast and reliable test feedback (CD is hard to do properly when you’ve got flaky tests stalling your builds) one the one hand, and user interface tests and their issues on the other hand. So, is there a place for these tests in the CD pipeline? I’d like to argue there is, if they satisfy a number of criteria.

The tests are actually verifying the user interface
There’s a difference between verifying the user interface itself, and using the user interface to verify (business) logic implemented in lower layers of your application. If you’re merely using the user interface as a means to verify, for example, API or database logic, you should reconsider moving the test to that specific level. The test automation pyramid isn’t popular without reason.. On the other hand, if it IS the user interface you’re testing, then you’re on the right track. But maybe there is a better option…

The user interface cannot be tested as a unit
Instead of verifying your user interface by running end-to-end tests, it might be worthwhile to see whether you can isolate the user interface in some way and test its logic as a unit instead. If this is possible, it will likely result in significant gains in terms of time needed to execute tests. I’ve recently written a blog post about this, so you might want to check that one out too.

The tests are verifying vital application logic
This point is more about the ‘what’ of the tests than the ‘how’. If you want to include end-to-end user interface-driven tests in your CD pipeline, they should verify business critical application features or logic. In other words, ask yourself ‘if this test fails, do we need to stop deploying into production?’ If the answer is yes, then the test has earned its place in the pipeline. If not, then maybe you should consider taking it out and running it periodically outside of the pipeline (no-one says that all tests need to be in the pipeline or that no testing can take place outside of the pipeline!). or maybe removing the test from your test set altogether, if it doesn’t provide enough value.

The tests are optimized in terms of speed and reliability
Once it’s clear that your user interface-driven end-to-end tests are worthy of being part of the CD pipeline, you should make sure that they’re as fast and stable as possible to prevent unnecessarily long delivery times and false negatives (and therefore blocked pipelines) due to flaky tests. For speed, you can for example make sure that there are no superfluous waits in your tests (Thread.sleep(), anyone?), and in case you have a lot of tests to execute – and all these tests should be run in the pipeline – you can see if it’s possible to parallelize test execution and have them run on different machines. For reliability, you should make sure that your error handling is top notch. For example, you should avoid any StaleElementReferenceException occurrence in Selenium, something you can achieve by implementing proper wrapper methods.

In short, I’d say you should free up a place for user-interface driven end-to-end tests in your CD pipeline, but it should be a very well earned place indeed.

Adding service virtualization to your Continuous Delivery pipeline

Lately I’ve been working on a couple of workshops and presentations on service virtualization and the trends in that field. One of these trends – integrating virtualized test environments, containerization and Continuous Delivery – is starting to see some real traction within the service virtualization realm. Therefore I think it might be interesting to take a closer look at it in this post.

What is service virtualization again?
Summarizing: service virtualization is a method to simulate the behaviour of dependencies that are required to perform tests on your application under test. You can use service virtualization to create intelligent and realistic simulations of dependencies that are not available, don’t contain the required test data or are otherwise access-restricted. I’ve written a couple of blog posts on service virtualization in the past, so if you want to read more, check those out here.

Service virtualization

Continuous Delivery and test environment challenges
The challenges presented by restricted access to dependencies during development and testing grow stronger when organizations are moving towards Continuous Delivery. A lack of suitable test environments for your dependencies can be a serious roadblock when you want to build, test and ultimately deploy software on demand and without human intervention. You can imagine that having virtual assets that emulate the behaviour of access-restricted components at your disposition – and more importantly: under your own control – is a significant improvement, especially when it comes to test execution and moving up the pipeline.

Using virtual test environments as artefacts in your pipeline
While having virtual test environments run at some server in your network is already a big step, ‘the next level’ when it comes to service virtualization as an enabler for Continuous Delivery is treating your virtual assets as artefacts that can be created, used and ultimately destroyed at will. I’ll talk about the specific benefits later, but first let’s see how we can achieve such a thing.

One way of treating your virtual test environments as CD artefacts is by containerizing them, for example using Docker. With Docker, you can for example create an image that contains your service virtualization engine, any virtual assets you would like to use and – if applicable – data sources that contain the necessary test data to emulate the required behaviour.

Another way of creating virtual test environments on demand is by hosting them using an on-demand cloud-based service such as Microsoft Azure. This allows you to spin up a simulated test environment when required, use and abuse it and destroy it after you are (or your deployment process) is done.

At the time of writing this blog post, I’m preparing a workshop where we demonstrate – and have the participants work with – both of these approaches, and while the technology is still pretty new, I think this is an interesting and exciting way of taking (or regaining) full control of your test environments and ultimately test faster, more and better.

Adding containerized service virtualization to your continuous delivery pipeline

Benefits of containerized test environments
Using containerized test environments in your Continuous Delivery process has a couple of interesting advantages for your development and testing activities:

  • As said, test environments can be spun up on demand, so no more waiting until that dependency you need is available and configured like you need it to be.
  • You don’t need to worry about resetting the test environment to its original state after testing has completed. Just throw the test environment away and spin up a new instance ready for your next test run.
  • Having the same (virtual) test environment at your disposal every time makes it easier to reproduce and analyze bugs. No more ‘works on my machine’!

Further reading
Want to read more about containerized service virtualization? See for an example specific to Parasoft Virtualize this white paper. And keep an eye out for other service virtualization tool vendors, such as HP and CA. They’re likely working on something similar as well, if they haven’t released it already.

Also, if you’re more of a hands-on person, you can apply containerization to open source service virtualization solutions such as HoverFly yourself.