Step-by-step integration testing: a case study

For the last 9 months or so, I have been working as a tester on a project where we develop and deliver the supply chain suite connected to a brand new highly automated warehouse for a big Dutch retailer. As with so many modern software development projects, we have to deal with a lot of different applications and the information that is exchanged between them. For example, the process of ordering a single article on the website and then processing it all the way until the moment it is on your doorstep involves 10+ applications and multiple times that amount of XML messages.

Testing whether all these applications communicate correctly with one another is not simply a matter of placing an order and seeing what happens. It requires structured testing and a bottom-up approach, starting with the smallest level of integration and moving up until the complete set of applications involved is exercised. In this post, I will try and sketch how we have done this using three levels of integration testing.

First, here’s a highly simplified overview of what the application landscape looks like. On one side we have the supply chain suite and all other applications containing and managing necessary information. On the other side there’s the warehouse itself. I have been mostly involved in testing the former, but now that we’re into the final stages of the project, I am also involved (to an extent) in integration testing between both sides.

The application landscape

Level 1: message-level integration testing
The first level of integration testing that we perform is on the message level. On this level, we check whether message type XYZ can be sent successfully from application A to application B. Virtually all application integration is done using the Microsoft BizTalk platform. To create and perform these tests, we use BizUnit, a test tool specifically designed for testing BizTalk integrations. Every test follows the same procedure:

  1. Prepare the environment by cleaning relevant input and output queues and file locations
  2. Place the message type to be tested on the relevant BizTalk receive location (a queue or RESTful web service)
  3. Validate whether the message has been processed successfully by BizTalk and placed on the correct send location.
  4. Check whether the messages have been archived properly for auditing purposes
  5. Rinse and repeat for other message flows

Note that on this test level, no checks are performed on the contents of messages. The only checks that are performed concern message processing and routing. BizTalk does not do anything or even care for message contents, it only processes and routes messages based on XML header information. Therefore, it does not make sense to perform message content validations here.

Scope of the BizUnit message level tests

Level 2: business process-level integration
The second level of integration testing that is performed focuses on successful completion of different business processes, such as customer orders, customer returns, purchasing, etc. These business processes involve the exchange of information between multiple applications. As the warehouse management system is developed in parallel, that interface is simulated using a custom built simulation tool. On a side note: this is a form of service virtualization.

Tests on this level involve triggering the relevant business process and tracking the process instance as related messages pass through the applications involved. For example, a test on the customer order process is started by creating a new order and verifying amongst other things if the order:

  • can be successfully picked and shipped by the warehouse simulator,
  • is created, updated and closed correctly by the supply chain suite,
  • is administrated correctly in the order manager,
  • triggers the correct stock movements, and
  • successfully triggers the invoicing process

Scope of the business process tests

A number of tests on this level have been automated using FitNesse. For me, this was the first project where I had to use FitNesse, and while it does the job, I haven’t exactly fallen in love with it yet. Maybe I just don’t know enough about how to use it properly?

Level 3: integration with warehouse
The third and final level of integration testing was done on the interface between the supply chain suite and connecting applications and the actual warehouse itself. As both systems were developed in parallel, it took quite a bit of time before we finally were able to test this interface properly. And even though our warehouse simulation had been designed and implemented very carefully, and it certainly did a lot for us in speeding up the development process, the first integration tests showed that there is no substitute for the real thing. After lots of bug fixing and retesting, we were able to successfully complete this final level of integration testing.

Scope of the warehouse integration tests

For this final level of integration testing, we were not able to use automated tests due to the time required by the warehouse to physically pick and ship the created orders. It would not have made sense to build automated tests that have to wait for an hour or more before a response indicating the order has been shipped is returned from the warehouse. The test cases executed mostly follow the same steps as those in level 2 as they are also focused on executing business processes.

I hope this post has given you some ideas on how to break down the task of integration testing for a large and reasonably complex application landscape. Thinking in different integration levels has certainly helped me to determine which steps and which checks to include in a given test scenario.

7 thoughts on “Step-by-step integration testing: a case study

  1. Pingback: Testing Bits – 8/30/15 – 9/5/15 | Testing Curator Blog

  2. Bas,
    I am asking something different this time if you don’t mind ok?
    Indian Education System involves studying in school upto age 18 then to get a bachelor degree studying 3 or 4 years in from college itself big companies hire bright students(Intelligent and having high marks) mainly in the last year.since there can be students from different streams say like electronics engineering,i guess an it company if hires such a student they will give them training where i believe they teach them programming and the language they use in their project etc..
    Now other students when passed out from college they will try for interview of their own.The lucky ones get into companies.others will usually go for studying a language(obviously which has hire scope of getting job during that period) in private institutions and do a project there.
    To put in short i did such a course-php and i got into this small company now i work.But since i was weak in finding logic and do the coding they asked if i can test websites.i said i don’t have any experience.They said just paste the url in different browsers and check if there is any design breaking issue.Then slowly did some simple manual functional testing.
    Now doing little automation testing with the help of your blog and some youtube selenium webdriver tutorials.
    The point what i want to make is what all i know now on automation is not from my school or college but from internet 😛
    If you can share how was your journey into automation testing field?

    • I have a background in Computer Science. After graduation, I more or less rolled into testing by accident, and I quickly learned that test automation related most to my technical skills and background, so about six months after starting my testing career (almost ten years ago now) I was into test automation. Never really looked back since, although I have had the occasional project where I did manual testing as well.

  3. Bas,
    The testing of your project you said above seems really tough to me.if i understand your project correctly,your company implemented the things a normal shopping cart does,well in this case since this is a very large project i guess a very complex cart correct?so you guys
    also does testing in the coding side?my god that is so difficult right?so if coding is in php you know?
    or maybe no need of checking coding.maybe….
    sorry i have no idea.How you guys test this.may be the tools you said you guys used only needs the file doesn’t need to understand the code to test?

    • No, the project was far bigger than implementing a shopping cart (the website actually didn’t really change that much). What we had to implement and test was all the backend stuff, order managers, databases, supply chain management software, etc.

  4. Bas,
    if you guys need to build a simulator that works like the original,you guys should develop it using somewhat same code as original that developers use right?How you guys did it for warehousing?

    • The most important thing when you build such a simulator is to make sure that the behaviour is as close to the original as required for testing. Whether or not you achieve that by reusing code from the original application is not really relevant..

Leave a Reply

Your email address will not be published. Required fields are marked *