Each year around this time, right as it seems that the world of business is stepping into its short winter slumber, we inevitably receive the same call: “I’d like to get a beta test started immediately!”. Undoubtedly, they’re working on a product for release early in the new year, it’s most likely consumer facing, and they naturally assume that since so many people have much of the holiday period off, it’s a great time to begin a beta .
Tip: This is generally not a great idea.
While we’re happy that they’re thinking about beta (and just as important, calling us for help!), their assumption about their customer’s free time is generally amiss. While it’s true that the holidays are a time without work and school for many, they’re also a time of great unpredictability.
Holidays Are Too Unpredictable
This includes family visits, holiday sporting events, pre-planned trips, and quite often shiny new holiday gifts that sway their attention, abruptly lowering participation and risking the success of your entire beta test (quickly followed by your entire product launch) as a result. Given that steady participation is already among the most challenging aspects of nearly any beta, this is far too serious a problem to ignore.
You also have your own staff to consider. While they may be willing to take advantage of the holiday period, they obviously have personal lives as well. The same unpredictability that harms your participation will likely reduce your own team’s effectiveness to support your project. Unfortunately, if bugs and email aren’t being responded to, or forums aren’t being continuously moderated, beta testers will feel neglected and quickly move on to other things (which during the holiday are aplenty). On the other hand, if your staff isn’t enthusiastic about working during the holiday in the first place (understandable), their lowered morale (aka lack of “holiday cheer”) could likewise have an equally negative effect.
Rescheduling Your Beta Test
With that in mind, it’s a good idea to alter your schedule to avoid the holidays. Moving the project to just shortly after the holidays (a few days) is optimal. While this may appear difficult to “sell” up the chain (betas commonly have tight deadlines already), it should help to point out that 2 weeks of testing after the holiday will likely produce far better results than a 3-week test period that includes a holiday, but with substantially less risk and a lot more predictability.
If you can’t delay until after the holiday, it’s often best to pre-set expectations very clearly. Let your testers know that they’re not expected to test during or immediately after the holiday itself. Some may want to contribute (they may not celebrate the same holiday or prefer your product to their own gifts), and that’s ok – but it’s important that they clearly understand what’s expected of them as well as when support will be available to them. The same goes for your own staff.
When Holiday Testing Works
There are of course exceptions. If you’re working on Diablo 3, a 4096p HDTV, or <<insert sci-fi gadget much cooler than anything they’re getting this year>>, and/or your audience is young and likely out of school for a couple of weeks, then there’s a good chance you’ll be able to get enough attention to achieve successful results (at least initially). In these cases, it’s extremely important that your own team is willing and available to support the effort, or the previous problems will still apply. That said, while these exceptions can get away with holiday testing, it’s still not the preferred option.
There Are Holidays You Haven’t Heard Of
I’d like to note that while we’re US based and the Christmas/New Year’s season is the most impactful event on our own calendar, these problems persist across all geographies and cultures. We’ve experienced the same problems at Easter and Thanksgiving (to a lesser extent), so likewise these and other local holidays are best avoided as well.
This becomes even more important when testing internationally, where testers may be celebrating unfamiliar major holiday periods that are unaligned with your own. This makes it worth checking into major holidays for any region that you’re testing in during the planning phase, as getting unlucky in running into a holiday that you’re unaware of could drastically harm your results.
Beta Tests Are Not Gifts
As a final tip, I’d like to point out that beta products make terrible holiday gifts. Unlike real gifts they come with expectations that are important to meet. If your beta testers feel like the product is a present then they’ll lack any real motivation to provide great feedback, regardless of how much they actually use it.
Happy Holidays from Centercode!
