Whether it’s the rise of AI, CI/CD, or DevOps, the lines between testing and development are blurring. But as testing methodologies evolve and bleed into development, professional testers are questioning how this evolution will affect their role in product development.
In our interview with Jason Arbon, CEO of Test.ai, we dive deeper into the future of testing and how people running tests can adapt to imminent technological changes.
Many people are jumping on the test automation bandwagon. Do you envision AI and automation taking over the majority of testing efforts in the near future?
Jason: Yes, AI-powered test automation will perform the majority of software testing in the near future. I wouldn’t even really consider it ‘envision’ – it is simply going to happen. AI will drive cars and save millions of lives. It will generate movies to entertain us. It will also test apps and improve software quality on a global scale. AI is amazingly well suited to testing software.
Are there specific areas that are best suited for using AI and test automation?
Jason: Basic test automation will be the first to leverage AI. All those apps have the same login screens, shopping carts, product pages, etc. AI enables tests to be written once, then re-run across platforms, and more importantly across applications.
This will allow test developers to focus on the more complex, more fun, and more appreciated aspects of test automation. Given that AI can drive applications and benchmark them against each other, performance testing will be the second to leverage AI.
Surprisingly, qualitative measurements of application quality – things we are used to having a UX or usability panel tell us in a multi-page report – will also leverage AI-based approaches.
AI can ‘see’ more apps than any human, and AI can learn from the aggregate feedback of thousands of experts and users alike. When thinking about how it will impact different areas of testing, the key thing to consider is how AI can scale. If scaling is useful for an area of testing, it is ripe for the picking.
Many companies leverage customers and/or employees for beta testing, in conjunction with test automation efforts. Do you foresee this type of beta testing going away in the near future?
Jason: Beta testing is perhaps the most undervalued and most important aspect of testing. At Google, we had tons of engineers (literally metric tons of them), tens of thousands of automated tests, large manual internal and external testing teams. However, the one thing we couldn’t live without was our beta users. They found the most important bugs faster than most any other process.
The one thing we couldn’t live without was our beta users. Software is designed to serve humans, so the best feedback is from the very people it serves.
Software is designed to serve humans, so the best feedback is from the very people it serves. Too much software testing time and money is spent trying to emulate the user, when it is pretty quick, and provides better data, to just ask users about the quality and usefulness of software via beta programs.
When is the best time to conduct beta testing with real users?
Jason: The best time to conduct beta testing is continuously. The Google Chrome team does this best. They always have a deployed alpha and beta version in production, and they’re constantly monitoring usage and feedback.
Especially as AI works to automate most of the menial work, software teams will have the bandwidth to focus more on deploying beta programs and processing the beta feedback. I’ve seen top software companies spend months on internally focused design and engineering, only to have their beta customers reject it outright (e.g., they didn’t use the beta version or politely told them what was wrong).
Beta feedback should not only be continuous, but it should also start as early as possible – before prototypes. If you aren’t familiar with pretotyping, I’d recommend a quick read. It shows how beta users can give useful feedback even on pretotypes. Bring in the beta user as early and often as possible. It is a cheaper, better, faster way to test software.
The best time to conduct beta testing is continuously. Bring in the beta user as early and often as possible. It is a cheaper, better, faster way to test software.
Lastly, some believe testers are a dying breed. What do you recommend professional testers do to stay relevant?
Jason: DevOps, automation, agile, instrumentation, monitoring, CI/CD, better tooling, better languages, were all supposed to kill software testing. However, the fact is that spending on testing activities is growing 20%+ year over year. Even companies like Google struggle to fill full-time manual testing positions.
The percentage of software project budgets spent on testing is rising. It went from 18% a few years ago to over 23%. It is true that all the above improvements have helped with software testing and quality, but the testing problem is growing faster than our new tools and practices can keep up.
Testers don’t have to do anything and can still grow more relevant in the near term with supply and demand. Even as AI and automation pick up more of the work, the reality is that only a small fraction of the testing that should be happening is happening. There is still a lot more work to do.
There are many ways to stay relevant, but folks should now be very deliberate about their career growth in the near term.
I’d suggest folks do one of three things to stay relevant. One: focus on being a generalist. Imagine if you were the only tester left on your QA team (i.e., they were automated away), what work would be left to do? Focus on that. That’s often test lead/manager type of work.
Two: become an expert in an aspect of testing to keep ahead of the bots. Be the best exploratory tester ever, be the best usability engineer in the world, or be the best edge-case API tester in the field.
Finally, three: embrace and extend the coming wave of automation and especially AI/Data. Ramp up on the basics of machine learning/AI and get creative in ways to leverage AI/Data to solve the testing problems you are facing. There are many ways to stay relevant as a tester, but folks should now be very deliberate about their career growth in the near term.
Always a pleasure speaking with you, Jason. We’ll be keeping tabs on where you take Test.ai this year and beyond.
Stay in touch with news and thought leadership from inside the testing industry by subscribing to the Centercode newsletter.
