How should we test campaigns?
Our answer
A/B test one change at a time on clicks or orders, with at least 500 recipients per variation. Hold out a random 10% from promotional sends to see whether campaigns add sales.
- Confidence
- High
- Strongest proof
- 4 official rules
- 4 practitioners, 3 opinions or claims
- Sources
- 7 sources
- 11 statements
- KSCISMWZDP
New here? Take the 1-minute tour
Why this is the answer
3 reasons and our judgmentJudge tests on clicks or orders, not opens. Klaviyo recommends two variations and one variable at a time, and warns that its open-rate tests count Apple's automatic opens. sendXmail (email marketing agency) and Suped judge on clicks, orders or revenue per recipient instead.
Small lists need bigger changes. Suped (email deliverability software maker) calculates that a 10% lift on a 20% open rate needs about 6,500 recipients per variant. Customer.io's rule of thumb is at least 500.
An A/B test picks the better email, not whether sending pays. Zettler Digital (Klaviyo and paid media agency) holds out a random share of recipients and compares store revenue per person, and Made With Intent advises the same.
Our judgment
Keep two kinds of test apart: A/B tests to choose between versions, and random holdouts to see whether sending adds sales. Size the test before sending, and on a small list test bigger changes with a full 50/50 split and carry the result to later sends. That combines Suped's sample sizes, Customer.io's rule of thumb, sendXmail's click metric and Klaviyo's warning about Apple opens. Repeat a promising or borderline result before changing the program, since Klaviyo labels significance at a 90% win probability while Suped and Customer.io work to 95% confidence.
Our judgment, built from Klaviyo, Zettler Digital, Made With Intent, Suped, Customer.io and sendXmail: our reading of these sources, not a finding from any one of them.
What the sources say
11 statements from 7 sourcesKlaviyo recommends judging subject line tests on opens and content tests on clicks, testing one variable with 2 variations, and warns that open counts include Apple's automatic opens.
“Open-rate A/B tests count recorded opens, including MPP opens, so the winning variation may reflect automatic opens as well as human engagement.”
Klaviyo labels a campaign A/B test result statistically significant when its win probability is above 90% and each variation has more than 50 successful deliveries; promising results should be retested.
“For promising results, repeat the test to confirm the finding.”
Klaviyo's A/B testing advice: write down what you expect, test one variable with no more than 4 variations, check statistical significance, and start a new test if content changes.
“A large sample alone does not establish that 1 variation is better.”
A Klaviyo community manager tells an account whose attributed share fell from 40% to 26% after tighter settings to keep settings fixed, pair attributed revenue with other metrics, and use holdouts for lift.
“treat attributed revenue as a measurement and optimization signal, not as a direct measure of incremental revenue.”
Suped, which sells deliverability tools, says subject line tests need a p-value of 0.05 or lower, and by its own illustration about 70,000 recipients per variant to detect a 3% relative lift.
“The first hour mostly measures people who open email immediately.”
Customer.io says testers usually aim for 95% confidence, suggests at least 500 recipients per variation as a rule of thumb, and warns that small samples may never reach significance.
“A rule of thumb is a sample size of at least 500 recipients per variation, but it’s worth taking the time to calculate sample sizes specific to your needs and confidence levels”
sendXmail, which sells copy testing, says across 80+ brands AI subject lines beat human ones on clicks by 12 to 19% given audience data, but the gap shrank to about 1 to 2 points by paying customers.
“Pick click-through rate as your success metric from the start, not open rate.”
Made With Intent suggests suppressing a random 10% of recipients before a promo email, then comparing revenue per recipient, including a seven-day tail, adjusted for the margin cost of the discount.
“Before your next promotional email, suppress a random 10% of eligible recipients.”
Postscript, which sells the feature, says its AI SMS testing sends AI variants against a control group and usually beats the original message, but gives no counts, method or size of lift.
“As the model learns what converts, it automatically shifts more traffic to the winners.”
Zettler Digital, a Klaviyo agency, says last click overstates email and SMS. It tests one program at a time with a random 90/10 or 80/20 holdout for at least 2 to 4 weeks, measured on Shopify revenue per profile.
“Last-click attribution will almost always overstate the impact of email and SMS, especially for brands with strong brand demand, repeat customers, and short consideration cycles.”
Klaviyo's global holdout groups need at least 400,000 profiles. It recommends holding out about 5% of marketable profiles for 3 months, and the lift counts all conversions, attributed or not.
“We additionally recommend that you run holdout groups for 3 months to gather enough data.”
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it3 figures from cited sources
- The strongest disagreementThe best case against the answer, from a cited source
- How sure we areWhat its confidence rests on, and what would change the answer