Service virtualization: open source or commercial tooling?

I’ve been talking regularly about the benefits of introducing service virtualization to your software development and testing process. However, once you’ve decided that service virtualization is the way forward, you’re not quite there yet. One of the decisions that still needs to be made is the selection of a tool or tool suite.

A note: Following the ‘why?’-‘what?’-‘how?’ strategy of implementing new tools, tool selection (which is part of the ‘how?’) should only be done when it is clear why implementing service virtualization might be beneficial and what services are to be virtualized. A tool is a means of getting something done, nothing more..

As with many different types of tools, one of the choices to be made in the tool selection process is the question of purchasing a commercial tool or taking the open source route. In this post, I’d like to highlight some of the benefits and drawbacks of either option.

Commercial off the shelf (COTS) solutions
Many major tool vendors, including Parasoft, HP and CA, offer a service virtualization platform as part of their product portfolio. These tools are sophisticated solutions that are generally rich in features and possibilities. If you’d ask me, the most important benefits of going the commercial route would be:

  • Their support for a multitude of communication protocols and message standards. When you’re dealing with a lot of different types of messages, or with proprietary or uncommon message standards or communication protocols, commercial solutions offer a much higher chance of supporting everything you need to virtualize these out-of-the-box. This potentially saves a lot of time developing and maintaining a custom solution.
  • Support for different types of testing. Most commercial tools, next to their ability to simulate request/response behaviour, offer options to simulate behavior that is necessary to perform other types of tests. These options include random message dropping (to test resilience and error handling of your appliation under test) and setting performance characteristics (when virtual assets are to be included in performance testing).
  • Seamless integration with test tools. Commercial vendors often offer API and other functional testing tools that can be combined with their corresponding service virtualization solution to create effective and flexible testing solutions.
  • Extensive product support. When you have questions with regards to using these tools, support is often just a phone call away. Conditions are often formalized in the form of support and maintenance contracts.

As with everything, there are some drawbacks too when selecting a COTS service virtualization solution:

  • Cost. These solutions tend to be very expensive, both in license fees as well as consultancy fees required for successful implementation. This is the single biggest drawback for organizations to invest in service virtualization, as heard from friends that sell and implement these tools for a living.
  • Possible vendor lock-in. A lot of organizations prefer not to work with big tool vendors because it’s hard to migrate away if over time you decided to discontinue their tools. I have no first hand experience with this drawback, however.

Open source solutions
As an alternative to the aforementioned commercially licensed service virtualization tools, more and more open source solutions are entering the field. Some good examples of these would be WireMock and Hoverfly. These open source solutions have a number of advantages over paid-for alternatives:

  • Get started quickly. You can simply download and install them and get started right away. Documentation and examples are available online to get you up and running.
  • Possibly cheaper. There are no license fees involved, which for many organizations is a reason to choose open source instead of COTS solutions. Beware of the hidden costs of open source software, though! License fees aren’t the only factor to be considered.

Open source service virtualization solutions, too, have their drawbacks:

  • Limited set of features. The range of features they offer are generally more limited compared to their commercial counterparts. For example, both WireMock and Hoverfly can virtualize HTTP-based messaging systems, but do not offer support for other types of messaging, such as JMS or MQ.
  • Narrow scope. As a result if the above, these tools are geared towards a specific situation where they might be very useful, but they might not suffice when trying to do enterprise-wide implementation. For example, a HTTP-based service virtualization tool might have been implemented successfully for HTTP-based services, but when the team wants to extend service virtualization implementation to JMS or MQ services, that’s often not possible without either developing an adapter or adding another tool to the tool set.
  • Best-effort support. When you have a question regarding an open source tool, support is usually provided on a best effort basis, either by the creator(s) of the tool or by the user community. Especially for tools that are not yet commonplace, support may be hard to get by.

So, what to choose?
When given the choice, I’d follow these steps for making a decision between open source and commercial service virtualization and, more importantly, making it a success:

  1. Identify development and testing bottlenecks and decide whether service virtualization is a possible solution
  2. Research open source solution to see whether there is a tool available that enables you to virtualize the dependencies that form the bottleneck
  3. Compare the tools found to commercial tools that are also able to do this. Make sure to base this comparison on the Total Cost of Ownership (TCO), not just on initial cost!
  4. Select the tool that is the most promising for what you’re trying to achieve, do (in case of open source) or maybe request (in case of COTS) a proof of concept and evaluate whether the selected tool is fit for the job.
  5. Continue implementation and periodically evaluate your tool set to see whether the fit is still just as good (or better!).

If you have a story to tell concerning service virtualization tool selection and implementation, please do share it here in the comments!

Adding service virtualization to your Continuous Delivery pipeline

Lately I’ve been working on a couple of workshops and presentations on service virtualization and the trends in that field. One of these trends – integrating virtualized test environments, containerization and Continuous Delivery – is starting to see some real traction within the service virtualization realm. Therefore I think it might be interesting to take a closer look at it in this post.

What is service virtualization again?
Summarizing: service virtualization is a method to simulate the behaviour of dependencies that are required to perform tests on your application under test. You can use service virtualization to create intelligent and realistic simulations of dependencies that are not available, don’t contain the required test data or are otherwise access-restricted. I’ve written a couple of blog posts on service virtualization in the past, so if you want to read more, check those out here.

Service virtualization

Continuous Delivery and test environment challenges
The challenges presented by restricted access to dependencies during development and testing grow stronger when organizations are moving towards Continuous Delivery. A lack of suitable test environments for your dependencies can be a serious roadblock when you want to build, test and ultimately deploy software on demand and without human intervention. You can imagine that having virtual assets that emulate the behaviour of access-restricted components at your disposition – and more importantly: under your own control – is a significant improvement, especially when it comes to test execution and moving up the pipeline.

Using virtual test environments as artefacts in your pipeline
While having virtual test environments run at some server in your network is already a big step, ‘the next level’ when it comes to service virtualization as an enabler for Continuous Delivery is treating your virtual assets as artefacts that can be created, used and ultimately destroyed at will. I’ll talk about the specific benefits later, but first let’s see how we can achieve such a thing.

One way of treating your virtual test environments as CD artefacts is by containerizing them, for example using Docker. With Docker, you can for example create an image that contains your service virtualization engine, any virtual assets you would like to use and – if applicable – data sources that contain the necessary test data to emulate the required behaviour.

Another way of creating virtual test environments on demand is by hosting them using an on-demand cloud-based service such as Microsoft Azure. This allows you to spin up a simulated test environment when required, use and abuse it and destroy it after you are (or your deployment process) is done.

At the time of writing this blog post, I’m preparing a workshop where we demonstrate – and have the participants work with – both of these approaches, and while the technology is still pretty new, I think this is an interesting and exciting way of taking (or regaining) full control of your test environments and ultimately test faster, more and better.

Adding containerized service virtualization to your continuous delivery pipeline

Benefits of containerized test environments
Using containerized test environments in your Continuous Delivery process has a couple of interesting advantages for your development and testing activities:

  • As said, test environments can be spun up on demand, so no more waiting until that dependency you need is available and configured like you need it to be.
  • You don’t need to worry about resetting the test environment to its original state after testing has completed. Just throw the test environment away and spin up a new instance ready for your next test run.
  • Having the same (virtual) test environment at your disposal every time makes it easier to reproduce and analyze bugs. No more ‘works on my machine’!

Further reading
Want to read more about containerized service virtualization? See for an example specific to Parasoft Virtualize this white paper. And keep an eye out for other service virtualization tool vendors, such as HP and CA. They’re likely working on something similar as well, if they haven’t released it already.

Also, if you’re more of a hands-on person, you can apply containerization to open source service virtualization solutions such as HoverFly yourself.

Creating virtual assets in service virtualization: record and playback or behaviour modeling?

When you’re implementing service virtualization, there are basically two different options for the creation of your virtual assets. In this post, I would like to take a closer look at both of these approaches and discuss why I think one should be recommended over the other. Spoiler alert: as with so many things, this is never a matter of black or white. There are always situations where the use of one approach can be preferred over using the other. You’ll see as we dive deeper into this subject.

Record and playback
The first approach for creating virtual assets is by first using your service virtualization solution as a proxy to capture request-response pairs (traffic) sent between your application under test and the dependency that is ultimately being virtualized.

Creating virtual assets using record and playback

This approach does have a number of advantages:

  • Using the record and playback approach, you can have a functional virtual asset up and running in a couple of minutes. There is no need for behaviour modeling or time-consuming response creation.
  • This approach suits situations where there is no formal specification of the structure or contents of the traffic that passes between the application under test and the dependency to be virtualized.

However, there are also a number of downsides:

  • This approach can only be used when the dependency is available for traffic recording. Often, access to the dependency is restricted or the dependency simply does not exist at all.
  • The behaviour that will be exerted by the virtual asset is restricted to the request-response pairs that were previously recorded. This puts a severe limit to the flexibility of the virtual asset. A simple example would be the fact that the virtual asset will never be able to generate unique message ID’s when you adopt a pure record and playback strategy – a blocker for a lot of systems.

When looking at these advantages and disadvantages, one can easily see that applying record and playback for service virtualization is much the same as with test automation: it is a good aproach for quickly generating virtual assets, but there are severe limits with regards to flexibility, plus maintenance will most likely be a pain in the a**. Having said that, using record and playback CAN be beneficial, for instance to discover how request and response messages are structured when you’re confronted with a lack of formal message specifications (and yes, that happens more often than you’d think…).

Modeling virtual asset behaviour
As an alternative, virtual asset behaviour can be modeled from the ground up, based on service and/or request and response message specifications. For example, any serious service virtualization solution allows you to generate virtual assets from WSDL or XSD documents (for SOAP-based web services) or WADL, RAML or Swagger specifications (for RESTful web services). To make them more flexible, data sources such as databases or Excel spreadsheets can be used to make the virtual assets data driven.

Creating virtual assets using behaviour modeling

Some advantages of creating virtual assets from scratch are:

  • The resulting virtual assets are generally easier to maintain since their structure is linked to a specification. When the service interface specifications change, updating your virtual asset to reflect the latest interface definition version can often be done with a single click of a button.
  • Since the virtual asset is created using specifications rather than a limited set of request-response pairs, theoretically all incoming requests can be processed and responded to, no matter the data they contain. In practice, there might still be some situations where an incoming request cannot be handled, at least not without some more advanced virtual asset configuration, but the number of requests that CAN be handled is much higher compared to the record and playback approach.

Of course, the behaviour modeling approach too has some disadvantages:

  • It takes longer to create the virtual assets. Where record and playback generates a working asset in minutes, modeling a virtual asset from scratch might take longer, and the time required grows with the number of message types to be supported and the size of and number of elements in the response messages.
  • As said before, sometimes message or web service specifications for the dependency to be virtualized are not readily available, for example when the dependency is still under development. This makes it hard to create virtual assets (although in this case record and playback isn’t an option either).

So, which approach should I choose?
After reading the arguments presented in this blog post, it shouldn’t be too hard to deduce that I am a big fan of creating virtual assets using the behaviour modeling approach. I firmly believe that although the initial setup of a virtual asset takes some additional work compared to record and playback, behaviour modeling results in virtual assets that are more flexible, more well-versed and better maintainable. There are some cases where using the record and playback approach may be beneficial, including vendor demos that show how easy service virtualization really is (beware of those!). In general, though, you should go for building virtual assets from the ground up.

Leveraging previously recorded traffic to create more flexible virtual assets
Again, I’m not completely writing off the use of record and playback, espeically since some interesting recent developments open up options to leverage virtual assets created from previously recorded traffic. Perhaps the most interesting option was shown to me recently by Hasan Ferit Enişer, an M.Sc. student from the Computer Engineering department at Boğaziçi University. I have been in touch with him on and off for the last year or so.

He is doing some interesting work that touches on service virtualization and he’s looking to apply some of the specification mining theories proposed in this paper to prerecorded traffic and virtual assets. This research is still in an early phase, so there’s no telling what the end result will look like and how applicable it will be to industry challenges, but it’s an interesting development nonetheless and one that I’ll keep following closely. Hopefully I’ll be able to write a follow-up post with some interesting results in the not too distant future.