Video
While A/B testing is a staple in marketing, experimental design is a powerful way to solve complex digital marketing issues. By testing hundreds of variables at once, marketers can focus their digital engagement strategies more effectively. Keri Larkin, a partner with Bain's Customer Strategy & Marketing practice, discusses how digital marketers can use experimental design to better understand customers and to boost both impact and return on investment.
Read the Bain Brief: Boosting Your Digital Marketing Effectiveness
Read the transcript below.
KERI LARKIN: A/B testing, often known as champion/challenger testing, is a staple in digital marketing and serves a very important role. But it has limitations. You can only test a few variables at a time, and testing combinations of variables can take a very long time and require many tests. There is a more sophisticated, more powerful way of testing, which is experimental design.
Experimental design increases the variability and therefore enables you to test many, even hundreds, of combinations of variables by testing just a few. We've seen incredible results with experimental design. We've increased response rates and conversion rates three-to-five times historical champion campaigns. Experimental design is best suited for very tough strategic marketing issues and questions. It enables you to really understand which variables drive customers to respond and to act, and can often yield very surprising insights.
For example, in one of our tests, we had a customer group that just didn't respond to any incentives. A zero response. And so we would offer richer and richer incentives, but to the customer, it didn't matter. It was giving money away essentially. In another case, we tested interstitials. And the timing of the interstitial actually had incredible impact, and that impact varied based on the customer group and the customer's reason for visiting the site—their actual intent.
Whenever you're testing digitally, it's very important to understand the physical world as well. And that can come in the form of creating a very seamless customer experience. So you, for example, would want call-center reps to be aware of the test, and to have their scripts incorporate the test. So a customer experiencing the test would call in and have a seamless experience.
It also means a lot of collaboration internally. You may include groups from finance or legal or business units to ensure there's coordination on the test and the new offers or new pricing that's incorporated, but also to ensure that the results of the test will be implemented when the test is complete.
With greater use of mobile and digital advertising, that will put increased pressure and importance on driving a return for the investment dollars going into digital. Experimental design enables you to really understand why [and how] customers want to learn, engage and purchase on the site, and therefore [shows you] how to focus resources and investments to drive the most impact.
Read the Bain Brief: Boosting Your Digital Marketing Effectiveness