The results of the 2018 Customer Validation Industry Survey backed up what we already knew: time is a precious commodity that test managers don’t have a lot of. Responses from over 300 companies showed how and where today’s testing professionals are spending their time. In particular, it revealed how the Beta Testing tools they use impact bandwidth and overall program effectiveness.

In performing an in-depth analysis of the data, our research team discovered some interesting parallels between these tools and the time constraints test managers reported. Based on their analysis, we’ll look at today’s most popular Beta Testing Tools – what’s working, what’s causing time sinks, and an alternative option for each one.

Collecting Feedback: Email

Fifty-eight percent of survey respondents use email in test management. It’s far and away the most popular channel for communicating with testers and soliciting feedback.

Why Test Managers Use It

• Accessible. Most everyone has access to email and knows how it works. The tool itself is free, readily available, and doesn’t require any special instruction.
• Familiar to Use. From the test manager’s perspective, it’s easy to tap out an email and fire it off to a group of testers. Filters and flagging in clients like Gmail make organizing emails straight-forward, if somewhat limited.

Where Time Sinks Occur

• Aggregating Feedback. An email is an individual silo of information. Creating a system that can organize the individual pieces of feedback you receive takes time when you’re building a system within a generic email client.
• Tracking and Responding to Conversations. Keeping in close contact with your testers is central to maintaining high participation throughout your test. It’s easy to lose track of every conversation via email. This inherent lack of structure can muddle follow-up attempts on issues between test managers and users, as well as test managers and developers. Locating submissions and responding to each one individually can also create massive time sinks.

Staying GDPR-Compliant While Using Decentralized Tools
Complying with the EU General Data Protection Regulation (GDPR) means being very careful with how and where you store user data. Transferring a user’s personally identifiable information (PII) across multiple systems while maintaining compliance is a labor-intensive endeavor. These time sinks deepen if a user opts out. Scrubbing identifiable data from each system – not to mention emails, spreadsheets, and other siloed tools – is difficult if not impossible to guarantee.

One Alternative: Using Forms

Creating a survey form to receive product feedback allows you to bucket information as it comes in and funnel the results into a spreadsheet for further analysis. For example, you can set up a form with a date of when the issue happened, a product topic, a rating scale as to how serious the issue is to the submitter, and a text box for them to describe the issue.

Right off the bat, a form will give you:

• The submission date, as well as a self-reported ‘when this issue happened’ date to better gauge where in the product cycle this issue occurred
• What aspect of the product this issue applies to in order to sort submitted issues
• An indication of how important the issue may be in relation to other issues
• An idea of what the issue is and potential ways to test it

Each submission would still need to be reviewed, but the secondary details as to who, what, where, and when, have already been taken care of.

Analyzing Feedback: Spreadsheets

Sixty-eight percent of respondents use spreadsheets to analyze feedback. This a no brainer – spreadsheets can be a very effective tool for Customer Validation. But the things that make spreadsheets great can also make them time-consuming.

Why Test Managers Use It

• Accessible. Like email, spreadsheets are accessible, easy to access, and easy to customize.
• Adaptable. With formatting and a little knowledge, spreadsheets can suit a myriad of purposes.
• Organized. Features like pivot tables ease sorting and ordering data.

Where Time Sinks Occur

• Formatting. While the flexibility of spreadsheets makes them a popular and useful tool, creating efficient structures can actually slow you down. It’s generally done through trial and error. The perfect spreadsheet might be built for test A, but there is a good chance it won’t work for test B.
• Making Updates. Like email, spreadsheets are individual silos of information. Test managers spend generous amounts of time manually copying, pasting, and rekeying data. While spreadsheets can link to other sheets, setting up those formulas takes time. In addition, having too many people working in the same file can cause conflicts as to how data should be organized.

One Alternative: Google Data Studio

Google Data Studio and Tableau are powerful business intelligence tools that let you set up customized and detailed reports. Other stakeholders in your project can view these reports without affecting the data the reports are looking at, allowing them to further investigate project details on their own without damaging the source data.

These reports update in real time alongside the source data. This enables you to point to a configured report at that source. You save a significant amount of time by keeping stakeholders up-to-date as the project continues without the need to change focus for further analysis, reporting, and release.

Managing Feedback: Jira

Forty-two percent of respondents use Jira to collect and manage feedback. A Development team standard, Jira is an indispensable issue tracking and prioritization tool.

Why Test Managers Use It

• Used by Developers. Reporting bugs and other issues in Jira gives developers a direct line into real-world product issues.
• Structured. Jira’s Agile structure mirrors many Customer Validation best practices for organizing information. Multi-workflow scheduling helps streamline issues to keep product development moving even as bug reports are coming in.
• Specialized. As an issue-tracking tool, Jira is well-equipped to organize bug submissions and other issues streaming in from Customer Validation projects.

Where Time Sinks Occur

• Managing Permissions. Inviting testers into Jira means setting an appropriate level of access within the program. While it’s possible for testers to work within Jira, they need a separate project workspace and custom permissions to do so. Ensuring they don’t have visibility into everything your Dev team is working on takes care – and time.
• Managing Multiple Tools. Without survey capabilities or a clear way to communicate with testers, test managers need to use multiple Beta Testing tools at once. This causes friction as test managers move the data in and out of different systems.
• Organizing Feedback. Because testers can create their own tickets, how they catalog and submit feedback needs to be monitored (and often corrected) to ensure the project data stays tidy. Issues that come into the project from testers in Jira, as well as the feedback that integrates with Jira via email, will need to be combined manually to prevent cluttering and confusion.

One Alternative: Integrations

Integrations are absolutely worth every moment spent in learning and setup. By syncing together multiple systems, you minimize any discrepancies between otherwise siloed programs.

For example, you can connect Jira to Centercode or another Beta Testing tool that handles issue submissions from testers. If someone submits a bug report in your Beta Testing tool, an integration can format and forward the bug report data to create a new ticket in Jira.

Later, if that same issue is resolved, a Jira integration can send a status update to the original ticket in the Beta Testing tool to mark it ‘Resolved’ and provide an update. This is a version of closing the loop in cases where testing and issue submissions and issue tracking take place in separate tools.

Upgrading Your Beta Testing Tools

Industry professionals leading the Customer Validation industry are looking at ways to centralize their Beta Testing data. This means their eyes are on tools with built-in features and templates that streamline processes and minimize time investments. They’re also looking at integrations as a means for both updating feedback and siphoning it along the proper channels.

The Centercode Platform is currently the only all-in-one platform on the market designed to meet the exact needs of Customer Validation. That said, some customer testing professionals use a wide combination of Beta Testing tools to communicate with testers and analyze feedback.

You can access the full Customer Validation Industry Report 2019 to read more about the tools and processes being utilized by industry leaders!