Creating virtual assets in service virtualization: record and playback or behaviour modeling?

When you’re implementing service virtualization, there are basically two different options for the creation of your virtual assets. In this post, I would like to take a closer look at both of these approaches and discuss why I think one should be recommended over the other. Spoiler alert: as with so many things, this is never a matter of black or white. There are always situations where the use of one approach can be preferred over using the other. You’ll see as we dive deeper into this subject.

Record and playback
The first approach for creating virtual assets is by first using your service virtualization solution as a proxy to capture request-response pairs (traffic) sent between your application under test and the dependency that is ultimately being virtualized.

Creating virtual assets using record and playback

This approach does have a number of advantages:

  • Using the record and playback approach, you can have a functional virtual asset up and running in a couple of minutes. There is no need for behaviour modeling or time-consuming response creation.
  • This approach suits situations where there is no formal specification of the structure or contents of the traffic that passes between the application under test and the dependency to be virtualized.

However, there are also a number of downsides:

  • This approach can only be used when the dependency is available for traffic recording. Often, access to the dependency is restricted or the dependency simply does not exist at all.
  • The behaviour that will be exerted by the virtual asset is restricted to the request-response pairs that were previously recorded. This puts a severe limit to the flexibility of the virtual asset. A simple example would be the fact that the virtual asset will never be able to generate unique message ID’s when you adopt a pure record and playback strategy – a blocker for a lot of systems.

When looking at these advantages and disadvantages, one can easily see that applying record and playback for service virtualization is much the same as with test automation: it is a good aproach for quickly generating virtual assets, but there are severe limits with regards to flexibility, plus maintenance will most likely be a pain in the a**. Having said that, using record and playback CAN be beneficial, for instance to discover how request and response messages are structured when you’re confronted with a lack of formal message specifications (and yes, that happens more often than you’d think…).

Modeling virtual asset behaviour
As an alternative, virtual asset behaviour can be modeled from the ground up, based on service and/or request and response message specifications. For example, any serious service virtualization solution allows you to generate virtual assets from WSDL or XSD documents (for SOAP-based web services) or WADL, RAML or Swagger specifications (for RESTful web services). To make them more flexible, data sources such as databases or Excel spreadsheets can be used to make the virtual assets data driven.

Creating virtual assets using behaviour modeling

Some advantages of creating virtual assets from scratch are:

  • The resulting virtual assets are generally easier to maintain since their structure is linked to a specification. When the service interface specifications change, updating your virtual asset to reflect the latest interface definition version can often be done with a single click of a button.
  • Since the virtual asset is created using specifications rather than a limited set of request-response pairs, theoretically all incoming requests can be processed and responded to, no matter the data they contain. In practice, there might still be some situations where an incoming request cannot be handled, at least not without some more advanced virtual asset configuration, but the number of requests that CAN be handled is much higher compared to the record and playback approach.

Of course, the behaviour modeling approach too has some disadvantages:

  • It takes longer to create the virtual assets. Where record and playback generates a working asset in minutes, modeling a virtual asset from scratch might take longer, and the time required grows with the number of message types to be supported and the size of and number of elements in the response messages.
  • As said before, sometimes message or web service specifications for the dependency to be virtualized are not readily available, for example when the dependency is still under development. This makes it hard to create virtual assets (although in this case record and playback isn’t an option either).

So, which approach should I choose?
After reading the arguments presented in this blog post, it shouldn’t be too hard to deduce that I am a big fan of creating virtual assets using the behaviour modeling approach. I firmly believe that although the initial setup of a virtual asset takes some additional work compared to record and playback, behaviour modeling results in virtual assets that are more flexible, more well-versed and better maintainable. There are some cases where using the record and playback approach may be beneficial, including vendor demos that show how easy service virtualization really is (beware of those!). In general, though, you should go for building virtual assets from the ground up.

Leveraging previously recorded traffic to create more flexible virtual assets
Again, I’m not completely writing off the use of record and playback, espeically since some interesting recent developments open up options to leverage virtual assets created from previously recorded traffic. Perhaps the most interesting option was shown to me recently by Hasan Ferit Enişer, an M.Sc. student from the Computer Engineering department at Boğaziçi University. I have been in touch with him on and off for the last year or so.

He is doing some interesting work that touches on service virtualization and he’s looking to apply some of the specification mining theories proposed in this paper to prerecorded traffic and virtual assets. This research is still in an early phase, so there’s no telling what the end result will look like and how applicable it will be to industry challenges, but it’s an interesting development nonetheless and one that I’ll keep following closely. Hopefully I’ll be able to write a follow-up post with some interesting results in the not too distant future.

Applying software design patterns to your test automation code

Yes, yes, yes… The creation of automated checks as a means of supporting and speeding up parts of the software development and testing process (a concept better known as ‘test automation’) should be considered software development. I’ve mentioned that a couple of times on this blog already, and today I’m doing it again. In this post, I’d like to take a look at why it’s probably a good idea to apply good software design patterns to your test automation code. Before I ramble on, please take note that I deliberately don’t call these patterns ‘best practices’. This is because sticking the ‘best practice’ label onto a software development pattern or habit usually promises something that that pattern cannot live up to. For each ‘best practice’, there’s at least one situation (but usually a lot more) where this practice just isn’t the best one to be applied. So, instead of ‘best practice’ it should be called ‘best practice for situations X and Y, but not for situation Z’. Or ‘best practice, but only on a Tuesday’. Instead, I think it’s best just to steer clear of the term altogether.

With that personal gripe out of the way, let’s look at why I think you should make use of proven software design and implementation patterns to improve your test automation code. Basically, it comes down to three aspects: maintainability, maintainability and .. maintainability. Applying patterns will not improve the quality and effectiveness of your checks, but it will do wonders for what probably is the most time-consuming task in test automation: maintaining the test code to ensure that it stays fresh, up to date and, well, running. Code that does not follow any basic software design patterns is likely to require much more effort to be read, understood and maintained. This is especially the case when you’re the lucky guy or girl who’s assigned the task of maintaining test code that you didn’t write yourself, but I definitely wouldn’t rule out the chance that you return to your own badly written test code after a couple of weeks and start thinking ‘what the … was THAT supposed to do again?’.

So, in short, you should apply good design and implementation patterns and practices. Here are some of the most common ones.

Don’t Repeat Yourself (DRY)
Being the opposite of WET (Write Everything Twice), DRY states that ‘Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.’ Or, as I like to explain DRY with regards to test automation, if any change to your application under test requires you to make the same change twice in your test code, you’re doing it wrong. Any object on any level in your test code (from a single constant all the way up to a sequence of actions required to simulate a certain business process) should be defined exactly once. Some test automation-related examples of the DRY pattern in action are:

  • The use of constants classes for anything from default timeout values to database connection strings – update once, effective everywhere
  • The Page Object pattern in Selenium – all elements, actions and checks related to a certain page or part of a page are defined in a central spot: the Page Object
  • The use of Examples in your Cucumber specs – A scenario is written once and then repeated multiple times with different input and validation values

DRY - Don't repeat yourself

Keep It Simple, Stupid (KISS)
While the DRY principle is probably the best known in software development, I think the KISS principle is just as important and just as (if not more) effective when it comes to maintainability of your test automation code. In my own words, this principle states that the solution that is easiest to implement is almost always the best one. Therefore, no matter what the problem is, you should always strive for the simplest solution possible. Some examples with regards to test automation:

  • Use predefined libraries instead of writing your own solutions. I’ve been guilty of this myself as well. For example, I still receive comments on this post regarding creating your own HTML reports for Selenium tests, even though I’ve since discovered and written multiple times about ExtentReports, a predefined library that does everything you want and more. No maintenance needed, just import it and you’re ready to go. This saves you a lot of unnecessary code to maintain as well.
  • Don’t automate everything in sight. Automating something just because you can or because your manager or (even worse) a senior test automation engineer (who should know better) is a recipe for disaster. I’m currently writing another blog post (one that will probably end up on LinkedIn Pulse rather than this blog) on why you should first ask the ‘why’, then the ‘what’, and then finally the ‘how’. This will save you from writing and maintaining a lot of useless checks, which in turn makes your solution a lot simpler.

KISS - Keep It Simple, Stupid

Choose a proper naming convention and stick to it
The last pattern that I think should definitely be applied to test automation code – or even better: to all related artifacts – is the use of a proper naming convention. I don’t really have a preference to a specific naming convention (although it took me a while to get used to C# methods starting with an uppercase character with me coming from Java), but picking one and sticking with it has definite benefits:

  • Assuming you’ve picked a naming convention that requires you to use descriptive names for your variables and methods (forget any convention that does not do this), using one and the naming convention everywhere keeps your code readable to anyone involved. By this, maintainability is also improved since it should be clear from the names used what a specific variable or method is actually representing or doing.
  • Using proper naming (and structuring) conventions for files and folders allows anybody on your team to locate that one pesky file or test automation report that’s being requested.

Use a proper naming convention

Start from the get go
To ensure that your test automation code remains readable and maintainable, you should apply good software design patterns from the very first line of code onward. Even if you’re just writing a little script to set up some test data or to perform a short and quick end user routine, you never know what those five lines of code will grow into one day. With code, it’s a bit like with pavements: it’s easy to leave behind rubbish when someone before you has done the same. However, when the pavement (or the code) is spotless, you’ll probably feel bad about leaving behind any trash after you’ve paid a visit. Always keep in mind that refactoring and improving badly written code is something no developer likes to do.

Be pragmatic
Of course, as with so many other things in life, applying patterns isn’t a case of black or white. It might not make sense to apply patterns to throw-away code (code that’s only used once). Also, with regards to the DRY principle, there’s such a thing as too DRY code. In general, though, applying good software design patterns will definitely benefit your code and those who rely on or need to work with it. As I said at the beginning of this post, test automation is software development. Let’s start treating it as such.