Test strategy is critical to the success of any testing effort – but where do you start? How does your strategy conform to different development methodologies? What should you tackle in-house, and what should you outsource? How you involve real users in beta testing?
In our interview with Adam Satterfield, director of Testing and Quality at Anthem, we dive deeper into these questions and the importance of test strategy on the whole. We’ll also look at how beta testing fits into that and the value it brings to both testing professionals and the organizations they work for.
Before we start, can you give a quick intro of your testing background and the specific industries you have experience in?
Adam: Sure! I have 20 years of experience in the testing industry. I started out at a semiconductor company, doing a mix of hardware and software testing on Linux and Unix. Then I moved on to telecom, military simulation software, cloud SaaS, consulting and finally healthcare. I’ve enjoyed the mix of backgrounds because you can pull from your experiences to help solve some of the complex testing problems of our day.
At a high level, how do you define test strategy and why is it important?
Adam: First, we need to think about what the word “strategy” means. It is a plan of action to achieve an overall aim. So that means that our testing strategy is a plan of action for how we will test functionality to achieve the overall business or company goals or aim.
A test strategy creator should strive to be succinct and ensure the information is valuable. For each point or section, think to yourself “Is this valuable to my stakeholders? Can they act on this information?”
A test strategy creator should strive to be succinct and ensure the information is valuable. Think to yourself, “Is this valuable to my stakeholders?”
Can you give us a preview of how test strategy relates to the development methodology used by each organization?
Adam: I think people get too caught up in the fact that a test strategy does not belong in Agile because we want to “limit documentation.” Part of what we are going to talk about is how to create test strategies for an Agile world. The great thing that Agile brings is the usefulness of many new tools, such as Confluence.
I believe you should embrace these tools and adjust your strategy to best fit what your organization is using. For Waterfall, we will discuss breaking the habit of having excessive documentation just because we have the time to create it. You should focus on what is valuable first.
How about in-house versus outsourced? What are the main areas QA leaders need to be aware of with test planning and strategy?
Adam: Great question! This is where the soft skills component comes in. You can’t simply send a strategy document across the pond or across the country and assume that the person reading it will completely understand it. You must have a review session to discuss the points of the strategy document. In Agile, this could be done during Sprint Planning. In Waterfall, this can be done as part of the official QA review.
Let’s transition to test strategy as it pertains to beta testing. What’s your prediction on companies leveraging customers to test their products – will it continue on?
Adam: I love beta testing in its many forms and believe it will continue on. We are seeing it grow with the capabilities of technology and companies implementing A/B, Canary or Crowdsourced testing via platforms like Centercode. It can be an important part of any strategy and one that should definitely be documented.
Testers can learn a lot by how real customers are using the functionality in production and real-world environments. Or by conducting blind tests to learn from strangers and customers of competitive products. I think these are often overlooked in many companies.
Testers can learn a lot by how real customers are using the functionality in production and real-world environments… I think this is often overlooked in many companies.
What’s your advice for testing effectively in today’s world of interconnected products and apps?
Adam: Keep learning! Technology is changing very rapidly and along with it, the capabilities for us to do better and different production releases. This, along with capabilities like Docker, allows a tester to truly make sure that exactly what they tested makes it to production.
Another great tool for testers is virtualization. This allows you to test various things on your own with little risk to your PC or laptop.
And finally, coming back to beta testing. Get your products in the hands of real users in real-world scenarios, thus enabling continuous product improvements before they launch.
Lastly, how can we stay up to date with upcoming presentations you’re delivering this year?
Adam: My LinkedIn page is the best place to follow my posts and talks!
Thank you for these great insights, Adam. Look forward to following your upcoming talks and new posts on LinkedIn.
Learn how agile software development teams ensure quality through Customer Validation despite faster release cycles with the Software Testing in the Age of Iterative Development infographic.
