The Delta ’20: Virtual Customer Validation Conference saw over 400 Customer Validation professionals from companies like Bose, Roku, Autodesk, TechSmith, and countless others. Over the course of three days, attendees picked up new tips and techniques, workshopped their programs, networked across industries, and brought their experiences to the table in the largest gathering of customer testing professionals yet.
They also asked a lot of eye-opening questions — both about the current state of customer testing and what’s in store for the future. In his live Ask-Me-Anything, Centercode CEO Luke Freiler tackled some of those questions head-on. He discussed Ted, the “tester engagement director” bot that codifies the Customer Validation Framework into a collection of features for prescriptive tester engagement. He also shared insights into our roadmap for 2020.
In case you missed it, here are some of the most revealing questions from the AMA!
Mario Espinosa, DAZN: Hey Luke, Ted looks great! Can’t wait to start using it. Is there anything you currently have in the works with Ted that isn’t available for release yet but is in the works for the future?
Luke: Hi Mario! Absolutely, there are so many opportunities to carry this concept forward. The next two areas I’d like Ted to lend a hand to are tester recruitment (helping find and select testers) and feedback management (encouraging relevant discussion).
Sharon Rylander, Square Panda: I agree, that’s very exciting! What do you anticipate the handoff will look like between Ted and the beta manager, day-to-day — e.g., will there be a code phrase that highly active users can type in an email that would send an alert to the manager to pay attention or take action?
Luke: Hi Sharon! The full answer to this would be huge — but in short, we look at Ted as having two relationships: the PM and the tester. There are separate libraries and facts for both. For example, if the PM has an imminent phase planned, but not yet published, Ted will remind them to do so. The goal is to focus the PM’s relationship with the testers to the feedback or discussion area, which is intended to be a single place to engage and take action.
Alex Larsen, Trimble: Are you concerned about Ted automating away some of the human element that makes the connection between test managers and participants so genuine and fun? How do you balance increased productivity with a personal connection and driving engagement?
Luke: Absolutely not. We ultimately want to achieve two things: (1) give you back many more hours to engage in a more meaningful way by doing the heavy lifting for you, and (2) centralize that activity into a place where you can more easily take action and see their feedback through (so everyone wins).
Michael Flores, Avid: Will Ted’s user impact score system replace the current user score system or co-exist with it? What do you recommend that project managers do if their user score systems are already robust and carefully managed?
Luke: It will co-exist. User Impact Score will be automated and available in all the same areas — but the existing scoring will be fully available and leverageable in Macros and such.
Ellen Murphy, JUUL Labs: Hi Luke, yesterday you mentioned a mobile app coming in Q3/4. Can you give us any more information about its functionality, timeline for live release with customers, etc.?
Luke: Hi Ellen! The short version is that the entire tester experience will be available — including Ted, feedback/discussion, surveys, etc. — as well as Admin dashboards/reports. What it doesn’t include is administrative functionality like survey creation. We’re looking to support as much mobile native functionality as possible, like the camera, badges, and notifications.
We don’t have a release date beyond 2020. It’s a parallel project alongside Ted, driven by different resources (although we’ll be pushing a couple from Ted onto it as soon as it’s out).
Conor Mulhern, Alarm.com: I run a CV program for a B2B company that develops products for both our customers and their customers as well. My current Customer Validation experience exclusively focuses on our customers, but I’d love to be able to accommodate their customers on their behalf in the future. I worry that we wouldn’t be able to create an experience that our customers would feel meets their brand needs. Do you have anything on the roadmap to accommodate that B2B2C experience?
Luke: Very interesting question. We’ve run a lot of tests like that ourselves (for example working with contractors who do installations, and then the users of the product they’re installing) – and they certainly have their own set of challenges – most of which we’ve been able to handle. The branding aspect of the question is something I’ve never considered before – I’d have to think that through.
Chris Mitchell, Autodesk: What are your thoughts about moving more into User Research/Usability, as well as maintaining a focus on validation? They all blur together for us, and we often have “competing” teams doing similar things with different tools.
Luke: Very, very mixed. The problem with leaning more into this space is that it puts us more directly against tools like Qualtrics — which is very different. They focus 100% on surveys and analytics, which I’d roughly estimate is only about 25% of our space. If I’m going to compete directly with that, I need to put a lot of resources into it. It also means spending a LOT of energy “catching up” as opposed to innovating, which is something I think our space has tons of room for.
Sharon Rylander, Square Panda: Do you see Centercode’s business opportunity with larger and more structured companies, or do you think there’s also an opportunity with scrappy startups?
Luke: This is definitely one of those things that keeps me up at night. Here are some numbers: at one point, 60% of our marketing leads were companies under 50 people and they ultimately generated 8% of our revenue. I made the call based on that data to focus on companies above that size — but it was a heavy heart (at the time we were below that threshold, now we’re not). As I said yesterday, our platform is a reflection of the industry it serves — big and fragmented. Ted is a huge attempt to address that, meaning we could provide something more streamlined.
My hope is that Ted will open us to create new editions of our product focused on a much wider audience, which can have a lower cost of sales – and everyone wins. I LOVE startups, my favorite Centercode success stories were all startups (Roku, Nest, etc.). In the meantime, we’re constantly brainstorming what we can offer those audiences at no cost.
Alex Larsen, Trimble: Have you or your team noticed a general increase in the number of companies taking Customer Validation more seriously over time? What are the trends in the marketplace when it comes to customer testing? Are we trending in the right direction when it comes to future job opportunities for people in our line of work?
Luke: We’re definitely seeing growth, but it’s still early. One of the things I didn’t get to speak to yesterday was what we’re looking to do with our certification program. We’re in the process of digitizing it to be self-serve, which we’d like to offer freely. The goal is to market it to anyone displaced by the (hopefully not) imminent recession. In other words, we’re looking to provide a new skill set to people who need it, but it’ll be available to others. Our cert ambitions were never monetary; we just want more people doing this, which will ultimately lead to growth for us.
Michael Rothstein, Roku: I’d love to see a collaborative environment where Centercode project managers (such as those attending this conference) can post questions, solutions, and feature requests, and we can all work together to share ideas. The best platform to run this could be the Centercode platform itself using discussion groups. Your support team is fantastic, but I think providing an additional way for us to share and learn from each other would help all of us. What are your thoughts on this?
Luke: This is something our customer success and product teams have goals set for and are working toward. We actually did this in the past, but I’m hoping this time we’ll see a lot more activity. Please help make that happen. (:
