Alpha Testing

Alpha testing is a method of software and user acceptance testing used to evaluate the early stability and quality of a product. It does this by gathering feedback from employees and/or more technical members of your target market.‍

Other terms for alpha tests (particularly in the context of delta testing) include: dogfooding, employee testing, and customer zero testing.

This type of testing helps uncover show-stopping defects, major usability problems, critical feature gaps, and in-the-wild interoperability problems. What makes an alpha test unique is that it can leverage both real customers and your company’s employees. But outside of finding and improving the quality of the product, there are other benefits to running an alpha test.

• Paving the way for more accurate beta tests. Alpha testing clears the path of major technical issues so a product is ready for beta. A common mistake companies make is skipping alpha and leaving the real-world technical bug hunt to the beta phase. But without the buffer of alpha testing, they risk giving testers a product with critical errors and blocking issues that make it impossible to continue using the product. This drains time and resources and causes painful delays in the back and forth between testers, test managers, and the development team. The presence of show-stopping errors also kills the potential to get the valuable insights about product satisfaction you would normally gain from beta testing.‍
• Increasing employee knowledge and fostering collaboration. Alpha testing with employees (or dogfooding) offers technical insights about your product from people who understand what it's supposed to do and how it's supposed to work. At the same time, it encourages different departments to experience the product and gain practical insight on the user experience, which will improve alignment within your organization. You’re also giving your customer-facing teams a chance to prepare for questions and concerns they might receive once the product has been released.‍
• Shift-left customer influence. While testing with employees reveals valuable natural use cases, they aren't the only group who can provide these insights. Innovators and early adopters are also great candidates for testing your product in the alpha stage. Their enthusiasm for technology and technical aptitude help you identify gaps and gain a better understanding of your current testing processes. And since they're closer to your target market, they're less likely to introduce bias into your test results.

The alpha phase of software development usually begins when the product has enough completed features to test it outside of the lab. For most products, this comes after smoke testing or automation testing.

For teams working with hardware, alpha phases typically have EVT (Engineering Validation Testing) or DVT (Design Validation Testing) units that will need to be ordered or collected in order to get them out to alpha testers.

Pro tip: A good rule of thumb is entering the alpha phase at 60% feature-complete and entering the early beta phase at 80-100% feature-complete.

Alpha testing is typically managed by QA (quality assurance) or software development teams because they have access to the early builds. Since alpha testing relates directly to their primary department initiatives, these teams are able to quickly improve the product with the feedback collected.

In connection with the testing and development teams, product managers are another likely candidate for managing or influencing alpha testing. The insights gained during alpha enhance the unique position product managers play as they make product decisions, collaborate with many teams throughout an organization, and engage directly with the target audience.

Other common departments that manage alpha testing are:

• Customer Support
• Operations Management
• Project or Program management
• User Experience

Like other forms of user testing, alpha tests follow a standard multi-phase approach. These phases are: planning, recruitment, testing (engagement, feedback, results), and wrap-up. This structure allows teams to account for the pace of project activity and plan when to expect results.

Time is usually tight in the later phase of development, so maintaining a schedule is key. Your project manager will thank you for having a clear understanding of the different phases of your alpha test.

Planning phase: Every project starts with a plan. This is a simple and concise document that outlines the objectives or intent, schedule, and ideal tester profile. During the planning phase, gathering input from stakeholders from Product Management, Quality Assurance, Engineering, User Experience, and Project Management is ideal as a jumping off point to discuss test plan details. Our free planning kit lays out everything you need to run a successful alpha test.

Recruitment phase: During the recruitment phase, the team managing the alpha test will start identifying ideal candidates for testing and invite them to join the project. While the process can differ depending on your organization and test audience (employees and/or technical users), the general stages of recruitment are: application, tester selection, and tester onboarding. Learn how to build a foolproof recruitment plan with the resources in this free recruitment kit.

Engagement management: Communicating with testers about which product areas to cover and activities to complete is essential to surfacing issues and smoothing out usability during an alpha test. To keep feedback flowing in, setting expectations is key: provide testers with a project schedule and send out reminders about what needs to be tested and when. For tips on maintaining engagement, check out this tester engagement pocket map.

Feedback management: Engaged testers will provide ample product feedback. But in order to make the most out of that feedback, it needs to be triaged and cleaned up. This means: organizing it by product area, ensuring it has enough actionable detail, and prioritizing it based on frequency and severity. This ensures stakeholders can understand what needs to happen and start improving the product as early as possible. Here are some tips for handling tester feedback.

Results distribution: Giving stakeholders the results of alpha testing on a consistent basis is key to optimizing its usefulness. Daily and weekly updates are the most common types of distributions for this data. The results from testing typically include: tester feedback overviews, surveys, and tester engagement outcomes. One to two week sprints generally allow enough time to provide results to your team members while still giving them time to complete necessary tasks.

Wrap-up: During the wrap-up phase, the team or individual managing the alpha test will tie up any loose ends and bring the test to a close. These activities include: announcing the test closure, distributing incentives (if applicable), retrieving products from testers (if applicable), and distributing the comprehensive results from the project. Here are some resources to help with the closure of your test.

If you've ever dabbled in alpha testing management, you know that it can be time-consuming and stressful. But equipping yourself with the right tools greatly reduces the time you spend managing project tasks.

Here are some useful tools for managing an alpha test:

• Centercode: The Centercode Platform supports every aspect of your user testing program, including alpha testing. With powerful automation to maximize your resources and smart analytics to keep your stakeholders connected to your project, the platform ensures your alpha tests have the greatest possible impact on your product.
• Google Sheets or Microsoft Excel: If you are really pressed on budget, a spreadsheet tool can be a lifesaver. While it does require a lot of manual work and isn't well-integrated into your other business systems, they can help you organize your testers, feedback, and results. 
• Your Inbox: Email is a cheap and useful way to invite and communicate with testers, announce activities, and share your results with stakeholders. This can be unwieldy depending on the size of your test, so stay organized. For example, using filters can help prevent follow-up feedback from falling through the cracks.
• Jira: If you do have employee testers, a Jira project can be an easy way to collect issues from your testers. Jira has a lot of integration capabilities, so linking Jira between other systems like Centercode can dramatically increase the usefulness of your data. If you're inviting external users to submit feedback through Jira, be very careful about who has permissions and access to what.
• TestFairy or TestFlight: These are useful tools for distributing your mobile app to Android and/or iOS devices. Once you have a list of alpha testers, it's pretty easy to upload a build and send it to your list of testers.
• LaunchDarkly: For those teams that  leverage feature flagging or feature toggling, LaunchDarkly is a great tool to test new features with a group of testers in live environments.‍
• SurveyMonkey or TypeForm: One critical, time-saving tool to have is something to collect feedback from testers beyond email (which is more difficult to organize). Here's where survey tools can come in handy. It's relatively easy to collect feedback through SurveyMonkey or TypeForm. However, these tools are disconnected and don't enable collaboration from testers. This makes it difficult to prioritize what to fix, improve, or promote in your product. You'll have to analyze the data collected from survey tools in spreadsheets and put them into Jira and/or a slidedeck to get your results in front of your team.

We have an entire blog post covering the differences between alpha and beta testing. But here's a summary of the three primary differences between these two user testing methods.

Product state: Alpha testing happens earlier in the product development life cycle than beta testing — alpha testing generally starts when the product is 60% feature-complete, versus 80-100% feature complete for beta testing. Therefore, products in alpha are likely to be a little less developed at this initial testing stage.

Objectives: Since the product is in an earlier stage of development, it limits the information you can accurately gather from real-world testing. That's why the primary objectives of alpha testing are focused around quality initiatives like surfacing and resolving bugs or educating your employees about the product. Once these major issues are out of the way and the product looks more like its final form, you're ready for the beta phase to begin. During beta, you'll focus less on fixing bugs and more on evaluating the overall product experience, including customer satisfaction with the feature set.

‍Tester audience: Because alpha testing has a more technical focus, its testers tend to include employees. This is, in part, because employees are a readily available resource to teams in need of real-world testers. Employees also tend to be more patient with sorting out technical issues than mainstream customer testers. That said, many alpha tests see great success by including technically minded external users and early adopters. Like employees, they're often more patient and enthusiastic to work through technical issues than mainstream testers. You'll want to save these mainstream members of your customer base for evaluating the product experience and satisfaction during beta testing.

1. Communicate your expectations and your goals. Being transparent with your testers is one of the best ways to ensure you hit your test objectives. An easy way to do this is by providing resources like a "Welcome to the test" or "Test overview" web page to make  your requirements and objectives crystal clear.
2. Respond to tester feedback. Alpha testers are taking the time to explore your product and submit feedback. You should reciprocate! Testers who feel "heard" are more likely to submit feedback, which ensures you have enough data to take action or deliver to your extended team.
3. Track metrics. Keep track of your submissions, tester engagement, and survey data throughout the project. See how using the Centercode Platform makes collecting test metrics easy.
4. Integrate when possible. Depending on the tools you are using, you may be able to automate some tasks. Integrating your tools saves time and gets feedback flowing to your development team faster.
5. Keep rituals with your stakeholders.  Set up a weekly meeting with your stakeholders to review results and talk about what's happening this and next week. This can ensure proper coverage of testing and maximum impact of the product.‍
6. Include sales leaders, marketing, and support teams. These are great teams to include in your alpha tests. Sales leaders and regional managers can get a headstart on how to position the product. Marketing can align messaging with their hands-on experience using the product. And support teams will have more time to prepare for what they can expect to see at launch.