Do you want a framework or a solution?

Reading through recent comments on this blog, as well as questions that are being asked in some of the LinkedIn groups I am a member of, something has started to strike me as odd. It seems like that in every other comment, someone is asking ‘how do I add so and so to my automation framework?’ or ‘how do I design a framework that does X, Y and Z?’. It isn’t the question as such that is the problem (if you can call me thinking of something as odd a problem in the first place), questions are good. You learn a lot by asking questions, and even more by asking the right questions. No, what gets to me is that all of these questions refer to automation frameworks. Why does it seem that so many people are so focused on ‘designing’ a ‘framework’? In this post, I’d like to explain why I think it’s better not to focus on frameworks in test automation, but shift your attention to something much more important.

First of all, what is a framework anyway? If you’d ask me, this is a framework:

Now this is what I'd call a framework

But within the test automation realm, what does a framework constitute? Wikipedia defines a test automation framework as:

A test automation framework is an integrated system that sets the rules of automation of a specific product. This system integrates the function libraries, test data sources, object details and various reusable modules. These components act as small building blocks which need to be assembled to represent a business process. The framework provides the basis of test automation and simplifies the automation effort.

So, it’s an integrated system of libraries, modules and other stuff that sets the rules of automation. And there’s exactly the main gripe I have with talking and thinking about automation in terms of frameworks. Your framework shouldn’t be the one setting the rules for automation. Instead, your application under test, the quality of that application you want to have achieved when deploying it into production and the tests and checks that form the evidence of this quality should be the ones setting the rules. Test automation is merely one of the possible means to adhere to these rules.

When you start your test automation efforts by building a framework, either from scratch or by glueing together two or more existing tools and only then try and fit in all tests you want to perform, you’re approaching the problem from the wrong end. Trying to fit everything into an existing framework limits flexibility, as your prime concern becomes ‘how can I perform this test using my framework?’ instead of ‘what is the best way to execute this test?’. It’s a bit like buying a two-bedroom apartment and only then realizing you’ve got six children. And a horse. Sure, it might fit, but it’s not necessarily going to be comfortable. It’s the same with test automation. Building a framework (the two-bedroom apartment) and then trying to fit in all sorts of tests (your six children, plus yourself and your significant other) will probably not lead to the most efficient (comfortable) situation. And don’t get me started on that horse..

Consider the following scenario: you have created a number of automated tests for your web application using Selenium (for browser automation), Cucumber (for BDD support) and a tool such as TestNG or JUnit (for assertions and possibly for reporting). At some point in time, you’re asked to also start writing automated tests for the newly built API that exposes part of the application logic. What do you do?

  1. Do you write some more tests that exercise the user interface, which in turn invoke the API ‘under the hood’?
  2. Do you search for an API testing tool that fits best inside your framework, glue it all together, start specifying features and scenarios for the API tests and automate away?
  3. Or do you look for the tool that best supports your actual API testing needs, use that and make it part of your overall testing activities, for example by making API tests a stage in your Continuous Integration or Continuous Delivery process?

If you chose #1, then I have three words for you (well, technically, four): don’t do that. If you opted for #2, you might have thought that API testing capabilities would be a nice addition to your framework, making it more versatile than it was before. But are you sure you can specify the behavior of your API in the same manner (i.e., describing the business value with the same amount of clarity) in Given/When/Then format? What if you discarded an API testing tool that fits the task at hand much better, just because it didn’t fit in your framework?

Option #1 and especially #2 are examples of what I’d like to call ‘framework think‘. While you think you’re building something flexible and reusable, you´re really limiting yourself by discarding solutions that do not fit the framework. Option #3, on the other hand, describes what I’d call ‘solution think‘, in my opinion a far better way of thinking and talking about test automation.

Test automation efforts should serve a higher purpose: solving the problem of giving fast and reliable insight into specific aspects of application quality. Every decision that’s made when implementing test automation, from the decision whether or not to automate at all down to selecting a specific tool to perform a certain test, should be made with this problem in mind. You should always ask: does this decision move me closer to providing a solution? And to what problem? The second question is equally, if not more important, since the perfect solution to a problem that’s irrelevant is just as wasteful as a solution that’s not right in the first place.

Stop framework think, start solution think

So, in short, here’s my advice: stop ‘framework think’. Instead, start practicing ‘solution think’. It might be only a change in mindset, but sometimes that’s all it takes to get you in the right direction.

Thanks to Alan Richardson for putting me on track to write this post with a quote from his 2015 Test Automation Day keynote. He also wrote a somewhat related post recently, which I think you should all read.

Continuous Delivery and user interface-driven test automation: does that compute?

In this post, I’d like to take a closer look at the combination of Continuous Delivery on the one hand and automated tests at the user interface level on the other hand. Is this a match made in heaven? In hell? Or is the truth somewhere out there in between? (Hint: as with so many things in life, it is..).

Continuous Delivery (CD) is an approach in which software development teams produce software and release it into the production environment in very short cycles. Automation of building, testing and deploying the software often is a vital part in achieving CD. Since this is a blog on testing and test automation, I’ll focus on that, leaving the topics of build and deployment automation to those more experienced in that field.

Automated tests on the user interface level (such as those built using Selenium) traverse your complete application from the user interface to the database and back and can therefore be considered as end-to-end tests. These tests are often:

  • relatively slow to execute, since they require firing up an actual browser instance, rendering pages, dealing with page loading times, etc.,
  • demanding in terms of maintenance, since the user interface is among the components of an application that are most prone to change during the software life cycle, and
  • brittle, because object on a web page or in an application are often dynamically generated and rendered and because wait times are not always predictable, making synchronization a tough issue to tackle correctly.

So, we have CD striving for fast and reliable test feedback (CD is hard to do properly when you’ve got flaky tests stalling your builds) one the one hand, and user interface tests and their issues on the other hand. So, is there a place for these tests in the CD pipeline? I’d like to argue there is, if they satisfy a number of criteria.

The tests are actually verifying the user interface
There’s a difference between verifying the user interface itself, and using the user interface to verify (business) logic implemented in lower layers of your application. If you’re merely using the user interface as a means to verify, for example, API or database logic, you should reconsider moving the test to that specific level. The test automation pyramid isn’t popular without reason.. On the other hand, if it IS the user interface you’re testing, then you’re on the right track. But maybe there is a better option…

The user interface cannot be tested as a unit
Instead of verifying your user interface by running end-to-end tests, it might be worthwhile to see whether you can isolate the user interface in some way and test its logic as a unit instead. If this is possible, it will likely result in significant gains in terms of time needed to execute tests. I’ve recently written a blog post about this, so you might want to check that one out too.

The tests are verifying vital application logic
This point is more about the ‘what’ of the tests than the ‘how’. If you want to include end-to-end user interface-driven tests in your CD pipeline, they should verify business critical application features or logic. In other words, ask yourself ‘if this test fails, do we need to stop deploying into production?’ If the answer is yes, then the test has earned its place in the pipeline. If not, then maybe you should consider taking it out and running it periodically outside of the pipeline (no-one says that all tests need to be in the pipeline or that no testing can take place outside of the pipeline!). or maybe removing the test from your test set altogether, if it doesn’t provide enough value.

The tests are optimized in terms of speed and reliability
Once it’s clear that your user interface-driven end-to-end tests are worthy of being part of the CD pipeline, you should make sure that they’re as fast and stable as possible to prevent unnecessarily long delivery times and false negatives (and therefore blocked pipelines) due to flaky tests. For speed, you can for example make sure that there are no superfluous waits in your tests (Thread.sleep(), anyone?), and in case you have a lot of tests to execute – and all these tests should be run in the pipeline – you can see if it’s possible to parallelize test execution and have them run on different machines. For reliability, you should make sure that your error handling is top notch. For example, you should avoid any StaleElementReferenceException occurrence in Selenium, something you can achieve by implementing proper wrapper methods.

In short, I’d say you should free up a place for user-interface driven end-to-end tests in your CD pipeline, but it should be a very well earned place indeed.

An update on crafting my career

Now that it’s almost time for me to go on what I think is a well deserved holiday, I thought it would be a good idea to take some time and see where I stand with regards to reshaping my career the way I want it to look like. In the last couple of months, some interesting developments have taken place that I think are small steps in the right direction. I also discussed my ideas on how my ideal working life would look like (freedom and variety basically sums it up) with some other people, resulting in either encouragement or blank stares. I don’t know what to make of the latter, but the encouragement is nice.

So, what have I been up to? First of all, I started a new project, since the one I was previously working on was not a good fit for me. I didn’t feel I could make a valuable enough contribution there to warrant both my hourly rate and the commute (an hour one way), so I decided to end it and go look for a new one. My current project is in an enterprise environment with a very heterogeneous application landscape and lots of room for improvement in the testing and test automation area. I’ve come to realize that this kind of organization and project fits me much more than the web development organization I was at before. Also, my commute has been cut in half, which gives me much more time to spend at home with the wife and kids, and to work on other projects, which to me is huge as well. That’s one thing I love about being self employed: the ability to choose what you’ll be working on and when to stop a project that is not a good fit.

Freelance freedom

Also, I’ve been asked by O’Reilly (the media and publishing company) to write a 20-30 page report on the state of and current trends in service virtualization. At the time of publishing of this blog post, it’s up for a final review, with a planned publishing date of September of this year. I’ll probably write a separate blog post with a link to the report once it’s published, so keep an eye out for that one to see if it is interesting to you too. The report will be accompanied by a blog post on another web site as well (of which I currently do not know the URL), which is part of the package deal agreed upon. The report will be freely downloadable and sponsored by HP Enterprise. I am very excited to have been asked to do this in the first place, much more because these writing assignments are exactly the type of projects I would like to fill my ideal workday with. Also, it’s another valuable exercise in honing my technical writing and English skills.

Furthermore, a couple of weeks ago, I had the privilege to receive an invitation to deliver a test automation-related workshop at the first edition of the Iasi spin-off of Romania Testing, to be held on November 4th (thanks again, Richard, for referring the organization to me!). Needless to say I happily accepted the invitation, so I’m currently in the early stages of workshop preparation. Also, I’ve never been to Romania before, so that’s a nice bonus too for someone that wants to see as much of the world as his schedule and finances allow.

Romania Testing - Iasi edition

Next to that, I’ll be giving a presentation at Centric (an IT services provider) in November as well on a yet to be determined topic related to test automation. I was invited to do this thanks to a referral from Sara, who attended my REST Assured workshop in May. So, again, thanks for that! It’ll be a nice opportunity to meet new people, do some networking and to further practice my public speaking skills.

Another thing that has kept me busy for some time now is the idea of transitioning from billed-by-the-hour work to offering project- and value-based services. Or even a product.. I’m still not sure as to what such a service or product should look like, but I AM sure that I don’t want to be depending purely on hourly work for long anymore. It doesn’t scale and it is a restriction to the freedom of working when and where I want to that I would like to have. As you can read above, I’ve been given the opportunity to take some small steps in the right direction, but I’m not nearly there yet.

The last I’ve been thinking about, and this is the first time I’m talking about this to anyone but myself, is writing a book on test automation. I know, there are lots of those already, but a lot of them focus on specific tools and how-to’s. What I think is missing is a thoughtful and balanced overview of the current state of test automation, and a debunking of a lot of still common test automation myths. If I decide to go through with this plan (currently I’m veering towards a ‘yes’) that’s another thing I would like to start on this year. Any comments or ideas are highly welcomed!

To round things up, I’m still not sure as to how to move forward, but writing this up makes me see for myself that I am moving in the right direction. The end goal is pretty clear now, but the road towards that goal is still pretty misty. Maybe some time off work will generate new ideas that can be pursued once I get back.

Finally, I’d like to share two blog posts from Louise Stigell that pretty much describe how I’m thinking about my career at the moment: ‘Being rich versus being free‘ and ‘Unemployable and proud‘.

Oh, and if you’ve already returned from your holiday: I hope you had a good one. If you’re still going: enjoy it! If you’re currently on holiday: what the &*%^ are you doing here?