"Testing is still the best way to find true breakthroughs"
- Bob Stone, veteran marketer and author of Successful Direct Marketing Methods
Testing is essential for the long-term success of marketing programs. It's the only way to prove what works in the marketplace. No amount of high-powered back-end statistical analysis can substitute for the clear, real-world answers you can get from testing.
Split-run techniques—often called A/B splits, test-control, champion- challenger, or one-variable-at-a-time testing—have given marketers some valuable answers, but split-run techniques are like a Model T to today's Ferrari...
THE SCIENCE OF TESTING HAS EVOLVED FAR BEYOND SPLIT-RUN TECHNIQUES
With the right techniques, you can rapidly test dozens of marketing-mix variables, all at once, with about the same sample size as an A/B split, yet with much greater accuracy, plus in-depth results that quantify every variable on its own and in combination with others. Here's just one example of performance before, during, and after one 18–element test:.
"ALMOST ANY QUESTION CAN BE ANSWERED, CHEAPLY, QUICKLY AND FINALLY, BY A TEST CAMPAIGN." – Claude Hopkins, Scientific Advertising (1923)
Since Claude Hopkins wrote Scientific Advertising in 1923, marketers have championed the split-run approach... and overlooked significant innovations. Like an advanced civilization hidden in the jungle, the specialized field of "scientific testing" has flourished for years out of sight of most of the marketing world. Unfortunately, experts developing these new techniques largely ignored the marketing world, while complex statistics—in 700+ page tomes covering subjects like nongeometric designs, Galois fields, confounding schemes, reflection, and effect heredity—made this vast realm of scientific testing seem out of reach.
Fortunately, newly-refined statistics and market-focused strategies now offer an abundance of testing alternatives with little additional effort. Split-run tests are only the simplest of myriad options that offer the freedom to test more variables, more rapidly, and with greater accuracy. Advantages include:
|Benefit||Scientific "Matrix" Testing||Split-run Testing|
|Quantity||2-3 Variables in one test
(many variables can be tested simultaneously)
|1 variable = 1 test|
|Speed and Sample Size||Constant Sample Size
(no matter how many variables are tested- often 1/10th to 1/20th the sample size normally required)
|Increasing sample size (testing 5 variables = 5x sample size)|
|Depth of Insights||Main effects, comparative effects and interactions
(can quantify the impact of each element on its own, plus interacts, where elements have a different impact in certain combinations)
|One main effect
(interactions are impossible to uncover)
|Flexibility||Wide range of test designs
(six major types, plus additional design techniques, chosen based on your objectives, with split-run tests as the simplest option)