This is a guest post contribution by James McKey, a beta program manager at Symantec, a company with a rich history in successful beta testing (and a long time user of Centercode). After speaking with James regarding his creative methods of incentivizing his beta testers, I felt our broader audience would really benefit from the successful strategy he’s developed. – Luke Freiler, CEO @ Centercode.
As anyone who has run a beta program will tell you, it’s not easy motivating someone to help you out free of charge, in a short time frame, and still produce quality results. For instance, getting a user to provide a report on a bug that: a) crashes your program, b) can be reproduced at will through specific steps and c) provides adequate logs, may seem like a semi-miracle when being done at no benefit to the beta tester. We’ve got to make it worth their while.
There are several classic ways to motivate beta testers, including giving each tester individual attention, allowing testers to have significant influence upon future features of the products, and awarding incentive items to anyone who provides feedback. The last option, physical incentives, seems to be more popular and can be done with little forethought while providing at least some decent results. The problem is that it becomes more likely that users will game the system as the program becomes more popular. In addition, the feedback can, at times, be of low value in nature (e.g. spelling error on page 93; sorry to you copy editors out there reading this).
Giving a random “big” prize doesn’t actually help much since (in the USA at least) doing so means legally having to open up the contest to anyone, whether they actually test the software or not (don’t ask me why, this is not the legal 101 blog). Plus it still doesn’t guarantee valuable bug finds. However, in the United States and most of Canada (minus Quebec — again, don’t ask) you can outline a contest based on skill. Think of those halftime shooting contests they have at basketball games where if someone sinks a ball in from the half court they get a big cash prize.
For a beta program, a skill based contest would involve awarding geek accredited “cool” prizes (e.g. in Q1/2011: an iPad, Kindle, or Xbox 360 Kinect) for the best bug find of the week. I’ve done this myself with amazing results. I found not only an increase in valuable bug finds but also demonstrated there was a clear increase in the number of bug submissions as the contest deadline approached. You can simply evaluate all submissions by the set deadline and judge the bugs by weighing half on severity and half on probability. Chances are you are already doing this and reporting at the end of the beta what the “best” bug finds were in the program. And don’t worry, it’s easy enough to find pre-existing “contest” legal documents online for sale (Google is your friend here), though you may need to request tweaking of the doc for your specific needs.
As an aside, Centercode’s agreement system makes it easy to implement a legal document with an electronic signature required to proceed (plug, plug, I know… seriously though, I really do like that methodology; no paperwork to deal with).
It’s worth mentioning that bringing a competitive element of any kind to a beta test can backfire if handled incorrectly. This proves especially true when competitive information (e.g. a leader list) is shared throughout the beta while all of the testers are working toward a single long-term goal. In these scenarios, users who pull ahead quickly may discourage others who feel they’ll never catch up, and therefore achieving the reward is too difficult or unrealistic. It’s likely that they’ll simply give up testing at that point. This can be avoided by simply offering a new reward each week (similar to the example above), acting as a reset. This gives everyone the opportunity to “win”, thus encouraging consistent participation.
In the end, you’ll find your beta testers appreciate the effort even if they don’t win and you might even have fun yourself as you make those phone calls saying “I’m pleased to inform you…” in your best Ed McMahon voice.
Have any of your own creative tester incentive strategies? Please comment!
