Note: This is the second part of our spotlight on the most recent episodes of HBO’s “Silicon Valley”. If you haven’t yet, read part one: HBO’s “Silicon Valley” Shows Us Exactly Why You Need to Run a Decent Beta. If you’re continuing on, spoiler alert!
Sunday night’s episode of “Silicon Valley” had Richard ready to shut down Pied Piper. They had achieved a successful launch with more than 500,000 downloads, but their daily active user count was dismally low. Users were overwhelmed by the app and so they were abandoning it in droves.
This sad state of affairs resulted from the fact that the Pied Piper team didn’t run a true beta test before launching their platform. They had a beta version, but they only gave it to close friends (who were mostly engineers). Their friends loved it, but as it turns out, their customers didn’t.
Now Pied Piper is on incredibly shaky ground; things are primed to go belly up at any moment. But while Pied Piper’s current predicament makes for great television, there are steps the team could have taken during their beta phase that would have avoided this situation entirely. With the hype and momentum that they already had, their launch could have catapulted them into being the next tech unicorn.
In today’s blog post, I want to dive into exactly what Pied Piper could’ve done to run a good beta test and then launch a product that didn’t “totally freak people out.”
Recruit Members of Your Target Market
The Pied Piper team’s cardinal sin was to send the beta version to only a handful of their engineer friends. They did this for two reasons: it was easy and it felt safe. The team thought that this would help prevent pre-release leaks about their product. And their friends were right there, waiting to get a hold of the platform. They’d be the ones who’d “get” Richard’s vision.
While the team believed they would receive “unbiased, honest feedback,” the problem with this approach is that their friends can’t give objective feedback about the real user experience. First, they’re too technical to give any feedback about how the product would feel to real users. Second, they’re biased because they know the Pied Piper team and want them to succeed. We even see this with Monica when she downplays and even dismisses her negative opinions about the platform’s UI and feel.
What the team needed to do was recruit targeted, enthusiastic strangers to test out their platform. They needed people with no connection to the company that reflected the actual target market of the platform. Then they could really see how the platform performed with their customer base. And I have no doubt one of the first things they would have heard was that the product “totally freaked people out.”
Set Beta Test Objectives
The second thing the Pied Piper team neglected to do was set objectives for their beta test. They went in with a vague idea of looking for bugs and seeing if people liked their platform, but that was it. Without well-defined objectives, you can’t collect the kind of feedback that will help you meet your beta test goals.
Instead, what the Pied Piper team should have done was sit down and identify a few objectives they wanted to achieve during beta, and then build a plan around that. In their case, since they were introducing a technologically complex product, they needed to evaluate the onboarding experience and the ease of use of the product. This would’ve made it obvious that there were issues with new user drop-off. They would have identified the biggest problems and fixed them before launch.
Image credit: John P. Johnson/HBO
Use NDAs to Increase Security
While the Pied Piper team seemed concerned about the security of their beta product — they didn’t have their beta testers sign NDAs. They likely didn’t feel it was needed since they were sending the beta to friends they trusted, but midway through the episode, they realized a Hooli employee had gotten a hold of a beta key. This meant a competitor had early access to their product. Taking basic security measures, like qualifying testers and having them sign NDAs, would help prevent these security issues.
Have a Method for Collecting Feedback
The final big miss for the Pied Piper team was not having a way to collect structured feedback. All the “rave” reviews that Pied Piper’s team did actually receive were in the form of emails and short phone conversations, which only pumped up their egos for their technological success rather than address any elements of the platform that could’ve been improved.
When Monica did have some negative feedback to give, her only option was to tell Richard face-to-face, which lead to an awkward confrontation and quickly dismissing her own feedback as irrelevant.
What they needed was a central, secure place for their beta testers to submit a variety of different types of feedback, such as bug reports, feature requests, journals, and surveys. This would allow their beta testers to provide high quality, detailed feedback about their experiences and paint a much more complete picture of the status of the platform team that would be a lot harder to ignore for the Pied Piper.
The Pied Piper Beta Test
Image credit: John P. Johnson/HBO
Given everything discussed above, if I were running a beta test for Pied Piper (with what I currently know), it’d probably look something like this:
Number of Beta Testers: 100 engaged testers
Test Period: 2 weeks
Test Objectives:
- Find unknown bugs
- Evaluate onboarding experience
- Measure ease of use of the platform
- Gauge response to customer messaging
- Improve support materials
- Get customer validation of product experience
Beta Candidate Requirements:
- Own a smartphone
- Keep pictures and videos on multiple devices
- Send pictures/video between devices at least three times a month
- Doesn’t consider themselves to be “highly technical”
Security:
- All testers must be willing to sign an NDA
- Assign a specific beta key to each tester
Feedback Collected:
- Surveys
- Bug reports
- Feature requests
- Journals
- Discussion forums
There’s obviously a lot more detail and consideration that goes into building a complete beta test plan, but even this level of planning would’ve made a huge difference in the results. It would’ve given the Pied Piper team a clear path toward gathering the feedback that would truly improve their product, so at the end of the beta test, they could be confident that they understood the complete user experience and had done everything in their power to perfect it before launch.
I can’t wait to see how the Pied Piper team pulls themselves out of this tough position in the coming episodes. Hopefully, they take the advice from their focus groups to heart and don’t have to rely on click farms forever to keep themselves afloat!
Header image courtesy of HBO.
