We often get emails from upset applicants who weren’t selected for a beta test despite impressive credentials. And sometimes we see companies that only want to select tech-savvy beta testers. In both cases, there’s a simple misunderstanding about beta tests that we’d like to clear up. A tester team needs to reflect the product’s target market, which usually means recruiting testers with all levels of expertise — even low levels. When you rely solely on expert beta testers, your results can be at best incomplete and at worst completely misleading.

Expert-only results are flawed because of a phenomenon called the curse of knowledge. According to this idea, once you become an expert at something, it’s very hard to imagine what it’s like to not know it. For example, tech-savvy beta testers can’t just pretend to know less about computers or gadgets and replicate the feedback you’d receive from your whole target market. Their knowledge will creep back in and cloud those attempts. At best, you’re getting an incomplete picture of your product’s performance, but acting on it as if it were comprehensive.

What do you miss out on that’s so important? Glad you asked.

Novices Don’t Go Rogue

When an expert encounters a bug, they sometimes have an impulse (and more importantly, an aptitude) for coming up with workarounds. They might not even think about it, but just proceed in a new direction to achieve the desired outcome. Then, if they’re successful, it’s “All’s well that ends well.” Now, hopefully you’ll have conscientious beta testers who log anything out of the ordinary. But if some testers do and others don’t, that can skew your understanding of a bug’s severity and causes. When novices encounter a bug, you can expect to hear about it more reliably. They often think they’ll make it worse by trying other things or just don’t know what else to try, so they seek help.

Novices Voice Critical Opinions on User Experience

Everyone has witnessed a frustrated user say something like, “I just want to print. Why won’t this work the way I want it to?” There’s a simple answer. It’s the curse of knowledge again, applying this time to product designers. You may have more experience trying to think like a user, but it’s still very difficult to account for how someone without your insight will understand and relate your product. If the subtleties of your product are lost on your users, you need to know that before release. But when you pack your tester team with experts instead of your target market, you may not know there are problems until the damage is done. They tend to think more like you, which isn’t always a good thing.

Novices Will Read Instructions

Someone who doesn’t instinctively know how to use your product will probably turn to the directions for help. And if this is done during beta, because you’ve recruited for your target market, you’ll find out right away if your documentation is up to the task. Experts will read the documentation if you ask them to, which isn’t uncommon during a beta test. However, they’re better equipped to understand poor documentation despite flaws, skipped steps, and overly technical explanations. You won’t really get the feedback you need if you go that route.

Novices are a Divining Rod for Support Planning

With novices, you get more than just a better idea of what needs improvement in your product before release. Their feedback on bugs, usability problems, and documentation issues can also clue you in to areas that might need bolstered support once your product hits the market. Sometimes there are tasks and processes that don’t come naturally to all users even after you’ve improved them. Identifying those hot spots and getting resources in place to help overcome the challenges will minimize problems. If your beta test consists only of experts, though, you might find yourself unprepared for support issues at launch.

Novices Have Different Priorities

Experts are quick to judge a product on criteria like specs, performance, and reputation. Novices, on the other hand, are often influenced by other things like color, design, and product descriptions. A properly recruited beta test is an excellent tool for gauging user acceptance and the effectiveness of marketing materials across your entire target market. If you gather insights that can be leveraged for a more successful launch, the opinion of your beta program within your company will get a corresponding boost in reputation.

As you’ve seen, getting the right mix of expertise in your beta test helps ensure that the product you’ve designed can actually be used as intended. If you focus only on a small segment of your target market, you can’t fully evaluate if your product is ready for release. The highly experienced, tech savvy demographics certainly have a place in beta testing (we’ll discuss how you leverage their skills in a future post), but not at the expense of other segments. You miss out on too much that way.

Recruit the right beta testers with our software

Image courtesy of Flickr user Nigelmaine.