When you encounter a critical issue in a beta test, things can get out of hand quickly. Your testers are demanding results, your developers are demanding data, and your marketing people are demanding reassurance. What you need is a reliable series of steps to help you maintain control of your beta and reduce the heat you’re feeling. So, we’ve developed something called the ICEGAP protocol for just this situation.
The six steps that make up the ICEGAP protocol are designed to keep things orderly and contained at each stage, so testers and colleagues trust your leadership and know you haven’t succumbed to the chaos. The first step is to (I)nterrupt the test, then (C)ollect pertinent data, (E)nlist help from select testers, (G)enerate a plan, (A)djust your test schedule, and finally (P)ush forward.
Interrupt the Test
After a show-stopping bug occurs, beta testers can find it hard to focus on anything else. You need to intervene somehow. If you don’t step in and break their fixation, all you’re likely to get is redundant and increasingly frustrated feedback.
What’s the best way to interrupt the test? It depends on your needs. The most obvious way is to just temporarily stop the test altogether. You put your project on hold and ask participants to wait until you have more data or a solution. If you have a tight schedule or fear losing engagement, you could also focus on goals that don’t require the product, like gathering feedback on documentation, packaging, or marketing campaigns.
Collect Pertinent Data
Once your testers are on hold or working on other assignments, you need to do some investigating. Whatever helpful information you can uncover now will make it easier to get your beta test back on track. If you happen to be using Centercode, this is a good time to utilize its search and test platform features. They’ll make the data collection process much easier.
When you review the bug reports that you’ve received, are there any similarities or trends that emerge? Does the bug only occur on specific platforms or when the product is used in certain ways? It’s also important to review all types of feedback you collect. You might find that additional testers experienced the bug, but only mentioned it in a survey, journal, or forum post. Finally, make sure your collection efforts aren’t focused on your beta alone. Communicate with people from other teams, since this problem might have been foreseen but underestimated or ignored.
Enlist Help From Select Testers
Having developed some hypotheses regarding the bug’s scope and root causes, it’s time to enlist help from select testers to confirm. Depending on what your investigation uncovered, you might select these testers based on platform, earlier experience with the bug, or perhaps feedback quality. Whatever the criteria, emphasize that this is a special test team and your testers will be that much more eager to help.
Your focus now should be on targeted experimentation, collecting detailed system reports, and identifying steps to replicate the issue. Using surveys and assigned tasks during this process is a good way to ensure your testers are giving you the specific feedback you need.
Generate a Plan
By this point, you should be able to deliver very specific information to your engineers that they can use to fix the bug. Not only will they appreciate the help, but it will also make it easier for them to estimate how long the fix should take. From there, you can begin to make your plan for restarting the beta test.
There’s more than than the bug fix to consider, though. In addition to engineering time, your plan has to account for any internal testing that needs to be done, efforts to prepare the next build, marketing and product release concerns, as well as whatever steps you need to take to prepare your testers to start back up.
Adjust Your Schedule
The time it takes to fix a bug depends on a lot of factors. It’s possible your issue could be fixed in a few days or weeks. Then again, it’s possible it could take months. Even a relatively quick fix, though, can cause friction when your testers are left in the dark. As soon as you can make a reasonable time estimate, you need to adjust your beta test schedule and let your participants know what to expect.
Beta testers will usually stay interested for three to six weeks total. If your beta will be interrupted for a long time, you might have to consider ending the current test and recruiting new participants when you’re ready to begin again. What you don’t want to lose sight of here is incentives. Give the interrupted testers something for their time and apologize that things went badly. If you’re lucky and your original beta test only runs a little longer as a result of the delay, include a little something extra with your incentives to thank testers for their patience.
Push Forward
Once you’ve executed your plan and gotten your participants back on board, it’s time to set their attention to regression testing. Make sure everyone has updated to the new code. Then, for those who submitted a bug, give them specific instructions to review their bug and verify the fix works. For testers who did not report that specific bug, encourage them to note any changes with the update. It might be invisible to them, but then again, it may create a new bug. Either way, it gets them engaged again after the delay.
A bug crisis protocol is the sort of thing we’d prefer you never have to use. Then again, it’s in the nature of beta for these things to occur. And it’s much better that they happen now, instead of during product launch. Hopefully, with ICEGAP, you’ll be able to stay calm focused and help your team do the same.
What strategies do you use to handle critical bugs in a beta test?
