Building a great tester pool is like assembling a championship NBA team. As the Beta Program Manager, you’re the GM, creating a group of end users who work better together than they would alone. Your goal: a 2023–24 Celtics-level team firing on all cylinders—not a flashy 2024–25 76ers roster that underperforms.

In my experience, there are four key types of testers you need to include in your pool, each contributing to a successful product launch in their own unique way. To continue the basketball analogy…

High-Usage Testers - Jayson Tatum and Jaylen Brown

These testers are the backbone of your program, much like Jayson Tatum and Jaylen Brown are the core of the Celtics’ lineup—reliable, high-performing players who consistently deliver results. High-usage testers should make up at least 60%* of your pool. While building out the Beta Tester Score framework at Sonos, I categorized anyone with at least twice the PlayHours of the typical user as a high-usage tester (i.e., they listened to twice as much music or watched twice as much TV as the typical user). These testers are often highly engaged and may also overlap with other tester categories.

Even if no bugs are reported or survey participation is low during your test, high-usage testers give you confidence that your product works from a technical standpoint. Their usage data, analyzed through your company’s data platform, provides critical insights. They’re especially valuable in fast-paced software development environments where you might only have 5–7 days to validate a new user experience (UX).

_{*While 60% might feel low, remember that your goal is to validate the product for all customers, not just power users. When rounding out your tester pool, think about users like your parents—they might not be as tech-savvy or frequent in their usage, but they represent real customers. Trust me, there are people out there who look, act, and think just like your parents, and including them ensures your product works for everyone, not just the superusers.}

Bug Reporters - Derrick White and Jrue Holiday

In basketball, players like Derrick White and Jrue Holiday are known for their hustle and ability to clean up the play—whether that means locking down the defense or making smart, unselfish moves to set up their teammates. In the world of beta testing, Bug Reporters play a similar role by identifying and diagnosing issues that might otherwise slip through the cracks.

Not everyone will encounter a bug during a user test, so historical bug report counts aren’t the most reliable metric when recruiting testers. Instead, focus on testers who are more technically adept or those who frequently jump into discussions and bug reports to offer help and troubleshooting tips for the rest of the pool.

These testers are your support team’s MVPs (shoutout to Andy P, Ken G, and Rich H!) because their unique superpower is helping your team quickly diagnose and resolve issues in the field. They’re the ones who do the dirty work, reproducing other testers’ issues to provide your engineering teams with more actionable data.

Ideally, Bug Reporters make up 10–20% of your pool, but in practice, a handful of highly qualified super users often emerge over time. These all-stars become so valuable to your program that they’re carried from test to test because of the impact they bring to the table (shoutout again to Andy P, Ken G, and Rich H!).

Community Engagement - Kristaps Porzingis, Al Horford, Luke Kornet

This is the bucket that often requires a second look, as it only takes a bit of discipline to log into the Community a few times each week, check in on what’s going on, and add thoughtful contributions. As a Beta PM, you can track metrics like weekly logins, posts liked, discussions or ideas created, and comments made to identify who is using the Community as it’s meant to be used—a space for like-minded customers to collaborate and improve the end-user experience for millions around the globe.

Within this group, you’ll find testers who embody a variety of valuable roles, much like key players on the Celtics:

• Kristaps Porzingis: Testers who bring versatility and fresh ideas, sparking engaging discussions and submitting unique contributions to the Community.
• Al Horford: Veteran testers with experience and insight, offering guidance and connecting the dots for others, much like Horford's leadership on the court.
• Luke Kornet: Testers who create a positive atmosphere and foster camaraderie, using humor and teamwork to build a strong sense of community.

Community engagement testers are the glue that holds the Community together. Their metrics can also serve as a valuable filter when populating other buckets or as a tiebreaker when deciding between tester A and tester B.

Survey (Task) Responsiveness - Payton Pritchard and Sam Hauser

Payton Pritchard and Sam Hauser are known for their ability to provide instant offense for the Celtics—they step in, execute with precision, and make a quick impact when called upon. Similarly, testers in this group are dependable and responsive, consistently completing tasks and providing feedback promptly. They understand the importance of a snappy feedback loop, often replying to your asks within 24 hours, which gives you a jumpstart on reviewing initial feedback with your stakeholders.

As you approach launch, these testers become even more critical. When working in two-week sprints, you can’t always afford to give testers a full week to complete a survey. Your team needs confirmation within 1–2 days that the product is working as intended and that users are satisfied. Without that quick turnaround, you won’t have enough time to implement changes or fix bugs.

This group isn’t about representing a specific percentage of your testing pool but rather ensuring you reach a statistically significant confidence level. For example, if only 5 out of 25 testers report satisfaction, it likely indicates the product isn’t ready for launch—but it’s also a reminder to evaluate the sample size before making significant decisions.

What Kind of Tester Are You?

Every program has a unique set of goals, and the way you curate your testing pool should align with those goals. There’s no one-size-fits-all approach to recruiting for a beta program—just as there’s no universal formula for building an NBA team. By mixing and matching testers from the groups above and understanding the unique value each type brings, you’ll be well-equipped to provide stakeholders with accurate insights and actionable feedback. This sets the stage for a successful launch of your new feature or experience.

For me, I’d say I’m a mix of Payton Pritchard and Luke Kornet—quick to respond but always keeping the mood light. In terms of NBA players from your team, who are you as a tester?

—

About the Author - Chip Duval

Chip Duval is a Senior Program Manager with over nine years of experience driving new software features and products from incubation to launch. He has managed both hardware and software programs of all sizes and led some of the most impactful partner integrations at Sonos, Latch, Peloton, and, most recently, at a robotic massage company called Aescape. Outside of work, he spends his free time exploring with his wife, son, and dog or sneaking in some golf.

You can hear more from Chip on the Centercode podcast by tapping the button below!