Is more testers always the best solution when you’re experiencing a dip in feedback? In this Ask the Expert series, Centercode’s Chris Rader weighs in on large tester teams: when they work, when they don’t, and how to manage when your stakeholders want to keep adding testers.

Got a question you’d like to see on the Centercode Blog? Shoot an email to help@centercode.com with the subject line, “Ask the Expert”!

How Do You Manage Large Tester Teams?

Hey Experts,

I’m a test manager at a company that makes mobile apps. I started running beta tests as part of my job last year, but basically, as soon as quarantine started, my company needed someone full time so I volunteered as tribute. 🙂

Needless to say, I’ve had to learn a lot in a short period. Even though I feel like I’ve made really good progress, I’m still having a hard time getting quality feedback from projects with 500+ testers.

Whenever there isn’t enough feedback coming in, my stakeholders tell me to add more testers. But even though adding testers does bring in more feedback, it’s harder to get the details I need to send it to Engineering. I spend so much time just following up with testers that it eats into the time I need to triage and build reports. Overall, I feel like the project quality goes down the more testers I add.

What should I do?

Thanks!
Liz R

Meet Today’s Expert

Chris Rader

Director of Product

Chris Rader is a product director at Centercode. He collaborates with our customers to interpret their beta test data, conduct market and user research, and dream up creative product solutions for the Centercode team. His superpower is the ability to paint a picture with customer data that explains the why and how behind complex problems.

Hey LR,

Happy to shed some light on this topic!

Adding more testers is a pretty common solution to needing more feedback or survey responses, especially when it’s coming from people like your stakeholders, who aren’t often spending time on the project themselves.

The Pros and Cons of Big Tester Teams

A larger tester pool does have its upsides. It increases the quantity of feedback (but not necessarily the percentage of testers who participate). It’s also great for gathering product analytics in mobile apps. The downside is that if your goal is to improve product quality, identifying feedback you can actually act on is a lot more difficult and time-consuming with larger tester teams.

Imagine there’s a crowd of people (socially distanced and armed with masks, of course), and you want all of them to download your app. If there’s 50 of them, you’ll be able to provide support and communicate with them in detail fairly easily. But with 500 people, it’s significantly harder to get everyone on the same page. The same goes for your test project.

For most tests, more testers = more time. The more testers you have to manage, the more time it takes to get everyone to download the app, respond to feedback, communicate weekly activities, etc. This all means that even though you’re pulling in more feedback,  identifying what’s actionable will likely take a hit, making the results less valuable overall.

Deciding What To Do

Before you decide what to do, take a look at your goals. If the goal of your test is to improve specific areas of your product, adding more testers may not help you in the long run. The best tactic I can give for getting actionable, high-quality feedback from your testers is to look at how you communicate with them.

Testers respond well to frequent, personal communication. Setting clear expectations, responding to feedback, and engaging with testers (in tester forums, for example) are all surefire ways to increase feedback and responses around the areas that are important to you and your stakeholders.

Difficulty engaging your testers? Learn how to navigate roadblocks with the comprehensive best practices and troubleshooting guidelines in the Tester Engagement Pocket Map.

If your stakeholders are primarily concerned with gathering product analytics and soliciting feedback to validate popular issues, then adding more testers is probably your best bet. Since mobile apps aren’t limited by constraints like the number of beta units available, you shouldn’t have to worry about that.

But in your case, you might need to focus your project on specific priorities (instead of trying to take on the whole enchilada) to collect and maintain quality feedback. You can do this by reducing the number of objectives or focusing your surveys and weekly topics around your stakeholders’ highest priorities. Combine this with communicating with your testers and you’ll get exactly what you need to know around your product priorities.

And of course, the Centercode Platform handles all that feedback without sacrificing quality. Because it automates tasks like separating issues from feature requests, minimizing duplicates with voting, and prioritizing feedback by severity and popularity, it will save you tons of time. If you want to learn more about automating feedback management, don’t hesitate to hop on a demo!

Hope this helps!

Happy Testing,
Chris

Discover Your Ideal Test Size

So, how many testers do you need to get product coverage and achieve statistical significance? Find the ideal test size number for your current or next project with the Test Size Calculator. This free resource will quickly and accurately calculate how many testers you need based on your specific project. Go on, take it for a spin.