In Spring of 2021, Centercode's biggest release yet was imminent and it was finally time to gather customer feedback. Coincidentally, the release included an entirely new framework, delta testing, built to capture actionable feedback from testers with minimal effort.

To eat your own dog food means to use your own products. At Centercode, we are firm believers in the value of dogfooding and in this case, we realized we would get to leverage our new delta testing framework to test out the latest version of Centercode. In short, we'd run our own delta test, utilizing our own tool to collect feedback on the machine itself!

Here are a few lessons we learned and wins we celebrated along the way.

Establishing a Testing Process to Eat Our Own Dog Food

Setting up a testing program can be challenging, but when you are asking testers to test your own testing software…it definitely adds some complexity! One of the first challenges we had to tackle was how we were going to give those that were testing the new version of our software something to actually do with it.

It was unrealistic to expect a customer to adjust their current testing routine over to the new version and dive in headfirst, but we needed them to at least get their feet wet with something to be able to experience the extended benefits of delta testing. We decided to split our efforts, allowing us to focus each half on entirely different goals.

The first half was all about getting our testers using the new version of Centercode, purely as a tester. Asking testers to explore an entirely new platform and framework alone was too daunting. We decided to purchase and ship out 'Meta Quest 2' units to each participant, and ran a project called, "Oasis" in which they participated as if they were testing the Oculus (a launched and successful product). During Oasis, testers were asked to provide feedback about the Oculus and complete activities using our new delta testing format.

The second half was the piece that ultimately mattered most to our team. We spun up a separate project, Friar, specifically focused on capturing feedback on the experience of participating in the Oasis test. This gave us an opportunity to ask customers how it felt to be a tester experiencing the new interface, interacting with Ted, and participating in a delta test for the first time.

Dogfooding with Delta Testing vs. Beta Testing

We were able to transition from running beta projects to running two simultaneous delta projects with different focuses incredibly smoothly. Leveraging the new delta features, we were able to plan ahead and create a test plan using Centercode's provided template, then import that test plan into each project to ensure we wasted no time once the new build was delivered.

We ran through a mental checklist (Centercode has recently released a launch dashboard to make this even easier): we imported our plan, have the testing activities scheduled and ready to deploy to testers, and Ted is activated and ready to engage testers - what’s left to do?

With everything set up, all we had to do now was focus on the results. While Ted’s efforts were busy engaging with testers and notifying them of their weekly activities, our entire focal point was on addressing the feedback that we were receiving in real time.

Ted, Centercode’s virtual Tester Engagement Director, is a real hero - take a look at some of the results:

61 pieces of feedback and 420 activities were submitted with no effort on our part beyond letting Ted work his magic - a great sign for the first ever Delta test and Ted's first time running wild.

Beyond Ted’s support , access to the various Delta Dashboards made it easy to review the top project feedback and share valuable results with our stakeholders in just a few clicks.

The Results of Eating Our Own Dogfood

As a company, Centercode has managed successful testing programs for decades. With the launch of the delta testing framework and the most comprehensive update to the Centercode platform ever, we felt a little anxious at the thought of how we were going to effectively test it (and launch from there).

Thankfully, the transition couldn't have been smoother. Our best practices still apply - the only difference is that we have to put much less effort into the parts of managing a testing program that, quite frankly, weren't as fun as the rest.

For instance, we spent no time managing tester engagement (thanks to the help of our friend Ted) or building custom reports (thanks to the automated dashboards). Instead, we got to focus our time on engaging with valuable tester feedback and preparing for informative discussions with our stakeholders.

All in all, we had a great experience and we'd encourage any organization considering a transition from beta to Delta to give it a shot - it’s critical to the success of your product! It's designed to require less effort while producing more value, and Centercode is here to support each step of the way.