For our third installment of the Beta Tips series we’re covering tester selection. At this point you’ve planned your beta test and you’ve recruited a large pool of people interested in testing your product. You know that your testers can make or break your beta, so how do you make sure the testers you select are top notch?
1. Be Selective
Don’t let just anyone into your beta. Public beta tests sound like a seductive idea because you don’t have to spend time reviewing and selecting testers, and an open beta test has to result in more feedback, right? Except, it doesn’t really work that way. Public betas are hard to manage, and you end up with a larger beta that gives you less feedback. With a public beta, people generally join out of curiosity, which is a much different mind-set than you get in closed beta tests. In addition, it offers a much wider group of potential customers an early look at an unfinished (and most likely buggy) product that can present a very negative impression, ultimately hurting sales.
2. Look for Detail and Care
Select candidates who complete your test application with great detail, limited spelling and grammatical errors, and a good argument for why they should be in the beta test. These people are likely to deliver similarly detailed results during your beta project. On the other hand, PEOPLE WHO WRITE LIKE THSI SHULD B AVOIDED (unless, of course, they’re your market).
3. Consider Applicant Environments
Most technology relies on adjacent or parent hardware to function properly (like the PC that runs your software or the smartphone that runs your mobile app). These environments can cause issues and bugs for your product and are one of best reasons to run a beta. So, when it comes to selecting your participants, you want to know about more than just personal characteristics. It’s also important to know about the environments (we call them “test platforms”) that they’ll be using during the test. Ideally, you want to be testing your product in environments that resemble what it will face when customers start taking it home (or to work). Then you can analyze the feedback you get in light of that test platform. This is one of the primary (and most well understood) benefits of beta that you don’t get in the controlled quality lab environments.
4. Seek Out Passion
If someone who meets your basic requirements is hounding you to get into a beta test, you should give that person a chance. Enthusiasm and persistence are two excellent traits in beta testers, and that person will likely be one of your more active participants.
5. Avoid Friends and Family
The ideal beta participants are those who are unbiased and truly represent your target audience. This effectively eliminates friends, family, and most employees from being great candidates. Sometimes their loyalty compels them to hold back, while other times they don’t feel obligated to participate like other testers. Either way, your best move is to rely on testers that you don’t already know well.
6. Avoid Sales Leads as Testers
It’s common for sales to ask a beta manager to let a lead into a beta test. There are at least two problems with this. First, sales leads usually aren’t interested in testing your product, but rather they’re interested in evaluating your product. They’re looking at it relative to their immediate needs, and you can’t rely on their participation because the motivations are completely different. Second, beta testers are more willing to forgive bugs and other quality problems than a customer who’s focused on evaluation. Besides, the last thing you want is an angry call from sales insisting that their lead didn’t buy the product because it was too buggy during the beta test.
7. Make Selections Quickly
The availability of your testers can change, so we recommend keeping the window between recruitment and beta tester selection very short (days, if possible). This will help ensure that those who volunteer are still excited for the project and still capable of committing the time necessary. Otherwise, you might find yourself reopening recruitment to find replacements for the testers you lost, further delaying your project.
8. Don’t Trust Email
If you identify a highly qualified candidate, but they don’t respond to your invitation to test, it’s likely they missed your email. Often, a simple phone call will reveal that your message was missed or ended up in a spam folder. People generally don’t express interest in being in a beta test and then immediately turn around and change their minds.
9. Always Keep a Pool of Alternates
There can be many reasons for losing a tester. Some realize they don’t have the time to commit, while others have to be removed for disciplinary reasons. Whatever the reason, it’s a good idea to have a pool of alternates available. You don’t want to go through the hassle of reopening recruitment in the middle of your beta test. Having alternates also gives you the option of adding extra testers if you aren’t hitting your participation goals or want to extend your test.
10. Notify Selected Testers Only
You may have the urge to let the applicants who weren’t selected know that selection is complete. We’ve found that this is usually a bad idea. More often than not, those who weren’t selected will react negatively, feeling snubbed because they weren’t selected while they feel they’re perfectly qualified. It’s generally a better idea to set expectations ahead of the recruitment, with a message along the lines of “You will be notified within 1 week if you’re selected to test.”
Have a tip to share? Be sure to leave it in the comments below!
