Service virtualization: open source or commercial tooling?

I’ve been talking regularly about the benefits of introducing service virtualization to your software development and testing process. However, once you’ve decided that service virtualization is the way forward, you’re not quite there yet. One of the decisions that still needs to be made is the selection of a tool or tool suite.

A note: Following the ‘why?’-‘what?’-‘how?’ strategy of implementing new tools, tool selection (which is part of the ‘how?’) should only be done when it is clear why implementing service virtualization might be beneficial and what services are to be virtualized. A tool is a means of getting something done, nothing more..

As with many different types of tools, one of the choices to be made in the tool selection process is the question of purchasing a commercial tool or taking the open source route. In this post, I’d like to highlight some of the benefits and drawbacks of either option.

Commercial off the shelf (COTS) solutions
Many major tool vendors, including Parasoft, HP and CA, offer a service virtualization platform as part of their product portfolio. These tools are sophisticated solutions that are generally rich in features and possibilities. If you’d ask me, the most important benefits of going the commercial route would be:

  • Their support for a multitude of communication protocols and message standards. When you’re dealing with a lot of different types of messages, or with proprietary or uncommon message standards or communication protocols, commercial solutions offer a much higher chance of supporting everything you need to virtualize these out-of-the-box. This potentially saves a lot of time developing and maintaining a custom solution.
  • Support for different types of testing. Most commercial tools, next to their ability to simulate request/response behaviour, offer options to simulate behavior that is necessary to perform other types of tests. These options include random message dropping (to test resilience and error handling of your appliation under test) and setting performance characteristics (when virtual assets are to be included in performance testing).
  • Seamless integration with test tools. Commercial vendors often offer API and other functional testing tools that can be combined with their corresponding service virtualization solution to create effective and flexible testing solutions.
  • Extensive product support. When you have questions with regards to using these tools, support is often just a phone call away. Conditions are often formalized in the form of support and maintenance contracts.

As with everything, there are some drawbacks too when selecting a COTS service virtualization solution:

  • Cost. These solutions tend to be very expensive, both in license fees as well as consultancy fees required for successful implementation. This is the single biggest drawback for organizations to invest in service virtualization, as heard from friends that sell and implement these tools for a living.
  • Possible vendor lock-in. A lot of organizations prefer not to work with big tool vendors because it’s hard to migrate away if over time you decided to discontinue their tools. I have no first hand experience with this drawback, however.

Open source solutions
As an alternative to the aforementioned commercially licensed service virtualization tools, more and more open source solutions are entering the field. Some good examples of these would be WireMock and Hoverfly. These open source solutions have a number of advantages over paid-for alternatives:

  • Get started quickly. You can simply download and install them and get started right away. Documentation and examples are available online to get you up and running.
  • Possibly cheaper. There are no license fees involved, which for many organizations is a reason to choose open source instead of COTS solutions. Beware of the hidden costs of open source software, though! License fees aren’t the only factor to be considered.

Open source service virtualization solutions, too, have their drawbacks:

  • Limited set of features. The range of features they offer are generally more limited compared to their commercial counterparts. For example, both WireMock and Hoverfly can virtualize HTTP-based messaging systems, but do not offer support for other types of messaging, such as JMS or MQ.
  • Narrow scope. As a result if the above, these tools are geared towards a specific situation where they might be very useful, but they might not suffice when trying to do enterprise-wide implementation. For example, a HTTP-based service virtualization tool might have been implemented successfully for HTTP-based services, but when the team wants to extend service virtualization implementation to JMS or MQ services, that’s often not possible without either developing an adapter or adding another tool to the tool set.
  • Best-effort support. When you have a question regarding an open source tool, support is usually provided on a best effort basis, either by the creator(s) of the tool or by the user community. Especially for tools that are not yet commonplace, support may be hard to get by.

So, what to choose?
When given the choice, I’d follow these steps for making a decision between open source and commercial service virtualization and, more importantly, making it a success:

  1. Identify development and testing bottlenecks and decide whether service virtualization is a possible solution
  2. Research open source solution to see whether there is a tool available that enables you to virtualize the dependencies that form the bottleneck
  3. Compare the tools found to commercial tools that are also able to do this. Make sure to base this comparison on the Total Cost of Ownership (TCO), not just on initial cost!
  4. Select the tool that is the most promising for what you’re trying to achieve, do (in case of open source) or maybe request (in case of COTS) a proof of concept and evaluate whether the selected tool is fit for the job.
  5. Continue implementation and periodically evaluate your tool set to see whether the fit is still just as good (or better!).

If you have a story to tell concerning service virtualization tool selection and implementation, please do share it here in the comments!

Five considerations before choosing an automated testing solution

Selecting a tool to aid your project or organisation with the implementation of automated software testing can be a daunting task. Many tools are available, each of them with its own benefits and drawbacks. The following considerations will provide you with some much-needed guidance in the test tool selection process.

What kind of technology is to be automated?
The choice for a specific automated testing tool heavily depends on the type of application(s) your tests are written for. Many tools are specifically designed for a single type of application, such as Selenium, which is targeted towards websites and web applications running in a browser. Other tools are able to handle a lot of different types of applications, such as browser applications, Windows applications and SAP. Tools that fall into the latter category tend to be more expensive, but might be the better choice if your test scripts cover different types of applications.

Do I go for open source or for commercially licensed tooling?
At first sight, going the open source route when selecting an automated testing tool might seem the most cost-effective option, due to the lack of license fees and support contract fees that typically come with commercial test tools. However, open source tools come with necessary investments of their own, as it often takes more time and more technical (programming) knowledge to set up and maintain automated test scripts. One often overlooked advantage of open source test tooling is the fact that the more popular ones are often supported a large community on the Internet. This community can be tapped into for technical support and often offers a library of additional features and extensions for the tool in case.

Do I prefer in house implementation or do I want to outsource test automation activities?
Another consideration to make is whether to have your own team implement the automated test scripts or to leave this to a third party. Keeping automated test development in house has a number of benefits, such as:

  • The people working on test automation are likely to be very familiar with the application(s) under test.
  • Knowledge gained on test automation is retained within the organisation.
  • The opportunity to work on test automation might be a good motivator for people in the organisation willing to learn something new.

On the other hand, outsourcing test automation implementation also has certain benefits, the most important being that dedicated test (automation) service providers likely have a lot of experience with the selected tool, enabling them to implement it much more efficiently.

How much training is required for the implementation of test automation?
Some tools require more technical knowledge upfront before users can successfully automate tests with it, while others do all they can to abstract from the technical details and offer an interface that makes it possible to implement and run automated tests for everybody. The former category of tools typically requires more training for them to be used. Another factor to be considered is the programming or scripting language used by the automated test tool. If, for instance, a tool is Java-based and all you have in your company are PHP developers, more training might be required than when your software development department – and the testers included in it – works with Java on a daily basis.

Does the tool match our current software testing process?
Finally, another thing worth considering is the extent to which the tool matches your current software testing process. Questions that you might want to ask with respect to this are:

  • Do the reports produced by the tool match the information requirements from the software development team and from management?
  • Does the tool integrate nicely with my current or desired automated build or continuous integration software delivery setup?
  • Can scripts be run by everybody within the software development and testing team? Note that this differs from who is going to develop the scripts.

When there is a mismatch between the features of the tool under consideration and your current software testing process, there are two further options. One is to consider a different tool that better fits the process, the other is to adapt the process to fit around the tool. The former is generally the more preferable, but in some cases where the testing tool fits the software to be tested perfectly, it might be worthwhile to adapt your testing process in some points to get a good match between the two.