Continuous Testing

What is continuous testing?

Continuous testing is the process of automated software testing during the delivery pipeline of software. Teams perform continuous testing to evaluate the functional quality of the software so they can assess the risk of the upcoming release. The scope of continuous testing is commonly aligned with user stories, which are used to automate testing with important user-facing test cases.

Why is continuous testing important?

As organizations become more agile and development teams release more frequently, updating the quality assurance approach has become a bigger priority. With continuous testing, testing teams have been able to accelerate their testing cycles and cover more test cases than their manual testing. 

Here are some more benefits to continuous testing:

• Faster improvements in product quality
• Aligns teams using DevOps or Continuous Integration and Continuous Delivery (CI/CD)
• More predictable timeline for testing in the project schedule
• Reduces business risk to launching with failures
• Integration capabilities with customer and user testing for optimal product experience
  

What is continuous customer testing?

We know not everything can be automated in testing because resources are typically scarce and test cases may not be compatible. That's why many organizations have started to add customers to help scale their testing efforts. Continuous customer testing is a delta program that embeds customers into software testing processes for products that are continuously updated. It allows teams to leverage customers to test updates, fixes, and improvements prior to being released to the general public.

Teams that leverage continuous customer testing see:

• More testing coverage for new features and improvements
• Faster testing cycles
• Fewer defects at release 
• Higher confidence and less risk in new updates
• An overall increase to customer satisfaction

How does continuous customer testing work?

Continuous customer testing engages the customer base in weekly cycles with the goal of collecting actionable feedback on specific features or user experiences. There are five general phases to a continuous customer test: Phase Scoping, Tester Recruitment (recruitment and replenishment), Engagement Management, Feedback Management, and Results Distribution.

The work being performed in the Continuous Customer Testing phases is as follows:

• Phase Scoping Pick the features, experiences, and activities that you'd like your testers to focus on.
• Tester Replenishment Fresh testers are added to address increasing requirements and/or offset any natural attrition.
• Engagement Management Testers are engaged by an automated agent who responds to their individual performance and behavior.
• Feedback Management Issues, Ideas, and Praise are enriched by collaboration. Results are then prioritized by Impact Score.‍
• Results Distribution Smart dashboards automatically summarize all results, broken down by feature and phase.