How should we test whether email and SMS campaigns add sales?
Our answer
Hold out a random 5 to 20% of the eligible audience from one email or SMS program for at least 2 to 4 weeks. Then compare store revenue per person between the groups, net of discounts.
- Confidence
- Medium
- Strongest proof
- 4 official rules
- 2 practitioners, 6 opinions or claims
- Sources
- 10 sources
- 12 statements
- K
ZDMWALAKCI
New here? Take the 1-minute tour
Why this is the answer
3 reasons and our judgmentAttributed revenue can't show what messages add. Klaviyo staff say that takes a holdout. The version you send always wins opens and clicks, so Customer.io judges its holdouts on a conversion goal.
Holdouts are small and run for weeks. Zettler Digital (Klaviyo and paid media agency) splits 90/10 or 80/20 for at least 2 to 4 weeks. Attribuly and LTV.ai hold out 5 to 10%, and Klaviyo recommends about 5% for 3 months.
A fixed holdout share can be too small. Kixie (sales calling software maker) warns that a 10% holdout on a small list often can't detect a lift worth acting on. It sizes the test from your conversion rate and the smallest lift that matters.
Our judgment
Hold out 5 to 20% of the eligible audience, toward the high end on a small list, and check the share with a sample size calculation. Run at least 2 to 4 weeks, longer for slow purchase cycles, and read the result at an end date set in advance. Judge on store revenue per person net of discounts and sending costs.
Our judgment, built from Zettler Digital, Attribuly, LTV.ai, Made With Intent, Klaviyo, Kixie, Bluecore and Customer.io: our reading of these sources, not a finding from any one of them.
What the sources say
12 statements from 10 sourcesCustomer.io's holdout tests withhold a message from a share of the audience, and it says to judge them on a conversion goal, since the sent version always wins on opens and clicks.
“When you use holdout tests, you should set conversion criteria and use the conversion metric to determine the true utility of your message.”
Klaviyo's flow splits can send a random percentage of profiles down a path; the percentage applies to the profiles reaching that path, not to everyone entering the split.
“If you use a random sample condition on a path, the percentage applies to the profiles evaluated for that specific path; not to all profiles entering the split.”
Causality Engine, a measurement vendor, says email platform revenue is credit, not a test, and adding it to Meta and Google claims can exceed real sales. Below 400,000 profiles, test one busy flow at a time.
“The revenue your email platform reports is credit, not a test.”
Kixie, which sells sales texting software, says to randomize SMS holdouts by customer, size them from baseline conversion and the smallest lift worth acting on, and keep calls and email equal across groups.
“Ten percent is not a rule, it is a habit.”
A Klaviyo community manager tells an account whose attributed share fell from 40% to 26% after tighter settings to keep settings fixed, pair attributed revenue with other metrics, and use holdouts for lift.
“treat attributed revenue as a measurement and optimization signal, not as a direct measure of incremental revenue.”
Eightx, a finance firm, says brands under $3M in revenue almost certainly shouldn't pay for Attentive, and those at $3M to $10M should prove SMS incremental with a holdout before signing an annual deal.
“Do not sign an annual deal here without first running a held-out test on your current tool to prove SMS is incremental enough to justify the upgrade.”
Made With Intent suggests suppressing a random 10% of recipients before a promo email, then comparing revenue per recipient, including a seven-day tail, adjusted for the margin cost of the discount.
“Before your next promotional email, suppress a random 10% of eligible recipients.”
LTV.ai, which sells email software, says to keep 5 to 10% of customers from all email and SMS for 60 to 90 days or more. It claims first holdouts often show 30 to 60% of reported revenue, with no data.
“The holdout test is uncomfortable because it means deliberately not emailing some customers.”
Attribuly, which sells attribution software, says to hold 5 to 10% of cart abandoners out of recovery messages for 14 to 28 days, suppressed across email, SMS, onsite prompts and retargeting.
“Enforce clean suppression across email, SMS, on-site prompts, and paid retargeting to avoid contamination.”
Bluecore's help says email and SMS holdouts need a specific hypothesis set in advance and an end date, reach significance faster on big campaigns, and break if group size, content or audience change mid-test.
“Holdout group tests should have an end date and not run indefinitely.”
Zettler Digital, a Klaviyo agency, says last click overstates email and SMS. It tests one program at a time with a random 90/10 or 80/20 holdout for at least 2 to 4 weeks, measured on Shopify revenue per profile.
“Last-click attribution will almost always overstate the impact of email and SMS, especially for brands with strong brand demand, repeat customers, and short consideration cycles.”
Klaviyo's global holdout groups need at least 400,000 profiles. It recommends holding out about 5% of marketable profiles for 3 months, and the lift counts all conversions, attributed or not.
“We additionally recommend that you run holdout groups for 3 months to gather enough data.”
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it3 figures from cited sources
- Where each source standsA chart placing each source on the question
- The strongest disagreementThe best case against the answer, from a cited source
- How sure we areWhat its confidence rests on, and what would change the answer