Motivate testers with the right messages at the right time to boost participation and feedback—download the new playbook with templates now!
There is no shortage of metrics in business. As Peter Drucker famously said, "You can't manage what you can't measure." And as a product manager, you're responsible for managing the vision and communicating the information you glean from metrics into a strategy that various teams within your organization rely on.
Metrics are essential because they help you shape the product as it develops and forecast the success of every release. If you're a seasoned product manager, you're probably already familiar with and/or tracking some of these common KPIs:
These metrics are powerful tools and absolutely essential to understanding and shaping your product. So what's the problem? Most of those metrics are either lagging indicators or reactive.
By the time enough product sales have accumulated and your department is asking for NPS, you're one to three months post launch. And there's very likely a big gap between the score you were aiming for and the one you're getting now. After all, there are a lot of factors that influence customer satisfaction outside of the product itself, like the purchase or support experience — things you'll be held accountable for but wouldn't have been able to account for.
You know that the success of your product and your career hinges on continuously deepening your knowledge of your product, your business, your industry, and your customers. But there's just so much you can't know. If that uncertainty keeps you up at night, you're in good company.
How do you know if people will like your product? How do you close the gap between what you think you know and how it all shakes out after you've launched? What role can metrics play in helping you find the answers to these questions proactively instead of waiting for star ratings, reviews, or support tickets to come through?
The good news is that there really is a simple way to get real-time insight about your product — one that helps you accurately predict your product's success and calls attention to issues you can fix before they impact satisfaction scores after launch.
The great news is that if you're user testing (and most product managers are), then you're already halfway there. The future of product management metrics is deepening the insights you get from user testing.
Let us show you how future-minded product managers are closing gaps in knowledge and enhancing their expertise of their customers, product, and industry through a whole new generation of user testing metrics.
"When the product succeeds, it's because everyone on the team did what they needed to do, but when the product fails, it's the product manager."
This quote explains exactly why being a product manager is so stressful. You have a lot on your shoulders. It's not always fair, but it is part of the job.
User testing outcomes align with product management outcomes, which is why the majority of product managers are performing user tests on a semi-regular basis. It eases some of that pressure by granting peace of mind.
But if you're only treating your user tests as a gut check to surface glaring issues before launch, you're missing a powerful opportunity to unlock deep insight into your product and how successful it's going to be. By exploring your user testing metrics, you uncover some phenomenal benefits, including:
In short, you get the good stuff.
Here's where it gets a little tricky: a lot of product managers recognize that there's utility in user testing metrics, but 70% struggle to track them. This is either because they
It's understandable because…well, these metrics can be really difficult to deliver. First of all, how do you decide which metrics are important for you to measure? Bugs collected? Bugs fixed? Surveys completed? How many active testers there are? How engaged they are? You can waste a lot of time by not knowing which metrics are the most impactful to track.
Then there's the matter of actually collecting these results and data signals from all over the place. If you aren't using an all-in-one platform, you're saddled with the time-consuming task of pulling data from multiple spreadsheets, tester emails, and tools. You might also need to rely on your stakeholders in Engineering, Sales, and Product Success, which means you're beholden to their timelines and bandwidth before you can even start.
All this work and you haven't actually gotten to the part where you're calculating the impact of your efforts. If you're a time-crunched product manager (and let's face it, most are), you probably don't have a spare few hours each week to locate all the data, clean it up, and manually perform the calculations.
I thought you said it was simple!
Stick with us, we're almost there.
For this reason, product managers end up taking two paths when it comes to their user testing metrics:
But there's a third option that blows the other two out of the water:
How are the product managers who perform option #3 getting it done? That's right, it's magic!
Just kidding. These PMs aren't wizards. They're also not more intelligent, more capable, or getting less sleep than any of their peers who are struggling to do it all. Instead, they're using a two-part solution.
The first is that they're focusing on three core user testing metrics. The second is they're reinforcing their processes with automation and relying on smart reporting to stay on the ball as they run user tests alongside their other responsibilities. OK, that second part is a little bit like magic — but it's attainable even if you're still waiting on that letter from Hogwarts.
This approach guarantees you're getting a unique perspective on your product and your customers that's more nuanced and granular than broader, whole-experience-view metrics like NPS or CSAT. By working with the medium-sized data you get during user testing, you're empowered to come up with focused solutions, whittle down the guesswork in building business cases, and easily prioritize fixes and ideas.
Meanwhile, automation makes product management more…manageable. No surprises there; it's likely that you're already using some level of automation to deliver the metrics you're currently tracking. But until recently, there haven't been tools for product managers looking to automate their user tests — let alone automate the metrics that increase their visibility and showcase their impact.
But before we talk about the automation, let's look at the metrics themselves. Here are the three core user testing KPIs that are leading the future of product management metrics — the questions they answer, the benefits of tracking them, and how they build on each other to get you closer to your product management goals.
The Product Success Score is an estimated measure of confidence based on the feedback within the project. As you fix issues, implement ideas, and receive praise, your success score will increase. And the more engagement you have, the more accurately this score forecasts your product's success on the market.
The challenge with measurements that product managers use to try and evaluate product and feature acceptance, like star ratings and NPS, is that they cover the customer experience, rather than the product experience on its own. In short, they aren't designed to forecast the success of your product.
This also makes these scores impossible to calculate accurately before launch. Try and use them to predict your product's success before your product has hit the market and you often end up with a totally different number a few months later.
What makes the Product Success Score vital as a product management metric is that it offers contextual, easy-to-act-on insight about your product and the way customers experience each feature or area. By weighing the impact of positive and negative feedback submitted by target market testers, you have a holistic product success score to work with that enables you to track shifts in satisfaction and product acceptance.
The Product Success Score relies on an impact measurement for each piece of feedback submitted by user testers. This impact is calculated by weighing factors like how important the feature is to the overall product experience (priority), how often issues occur (popularity), and how severe those instances are (severity).
Taking the temperature of how your user test is performing can be difficult. It's usually done through gut feeling or benchmarking against past experiences of running tests. Neither of these are particularly helpful, especially when you have limited bandwidth to deal with problems (like low engagement, for example) when something goes wrong.
The Delta Health Score measures the level of engagement within your user testing project, based on the number of active testers, tester engagement, and feedback quality. It uses industry benchmarks to set a threshold for how much feedback (positive or negative) you need about product areas or features in order to have a good understanding of customer acceptance. It then grades the amount of feedback you've gotten over the course of your test against that threshold.
This score makes it immediately obvious when your project needs your attention, like whether a part of your product needs more tester feedback, whether or not you need to extend a phase of testing, or if testers are being blocked from testing the product.
It is notoriously difficult for product managers to show their value in quantitative terms. It's even harder when you have limited control over the factors that influence user satisfaction, but you're still accountable to the overall success of your product.
The Test Impact Score measures the impact your efforts to address feedback from user testing have had on your product. This metric weighs the Product Success Score from the beginning of your project with what that score is currently to show how the issues you're fixing and ideas you're implementing are impacting overall satisfaction and acceptance over time.
Rather than pointing to how many issues have come in versus how many issues got fixed (which is useful in its own right), the Test Impact Score takes it one step further by showcasing the impact of those implementations over time. Not all changes made to a product have an equal impact on the user's experience, after all.
This score not only highlights how much influence your efforts have on your product but how efficiently you're prioritizing what needs to be done — the hallmark of a great product manager.
Now that you understand the three core user testing KPIs and how they serve you and your product, let's talk about the second part of the future of product management metrics: automation.
Automation matters because, while you could manually calculate the core KPIs you just learned about, it's just not realistic given the time constraints and limited bandwidth of a modern product manager.
Here's how automated user testing metrics are making a difference for future-minded product managers.
And it doesn't end at metrics: product managers are automating the whole of their user testing processes with tools like the Centercode Platform. This enables them to centralize their data and optimize their processes, increasing the efficiency, visibility, and ROI in addition to clearly and accurately measuring these returns.
