How should we decide when to act on an unclear result?
Our answer
First sort an unclear result. Rerun a broken test. On a sound one, hold or take a small reversible step judged on the low end of its range, with 80 to 90% confidence for reversible moves and 95% otherwise.
- Confidence
- Medium
- Strongest proof
- 3 official rules
- 1 dataset, 1 practitioner, 11 opinions or claims
- Sources
- 12 sources
- 16 statements
- G
CC
New here? Take the 1-minute tour
Why this is the answer
3 reasons and our judgmentAn unclear result is not proof of no effect. Sellforte (marketing mix modeling firm) calls an incremental return on ad spend of 4, with a 90% interval of 0 to 8, inconclusive. A narrow range near zero means the effect is small (Atticus Li).
Holding is a decision in its own right. Stella (incrementality testing software maker) uses 90% confidence on reversible monthly moves and 95% on irreversible ones. Google's experiments default to 80%.
Small tests overstate the effect, so judge on the bound nearest zero. Recast saw a small geo test read 11 times where a larger study found about 2.3 times, and it treats that bound as the conservative estimate.
Our judgment
- Sort an unclear result into broken, inconclusive or small before deciding anything. Sellforte's setup checks, Clean Commit's 58/42 split check and Convert's tracking and sample ratio checks catch a broken test. Sellforte and Atticus Li separate a wide interval that spans the bar from a narrow one near zero.
- On an inconclusive result, hold or take a small reversible step judged on the end of the range closest to zero, and move your estimate only partway. This combines Stella's hold rule, Recast's conservative bound and Common Thread Collective's practice of weighting a first result by its confidence. Acting on a finished lift or geo test sizes the step against breakeven.
- Use 80 to 90% confidence for cheap, reversible moves and 95% for moves that are hard to undo. The range draws on Google's 80% default, Clean Commit's 85% or 90% for easy-to-reverse changes and Stella's 90% and 95%.
- Rerun bigger or longer once, with the sample size fixed in advance, and move on after two or three sound flat tests. This follows Atticus Li's rules, Convert's advice to expand the audience when traffic was low and Juan Justo's warning against peeking and slicing.
Our judgment, built from Sellforte, Clean Commit, Convert, Atticus Li, Stella, Recast, CTC, Google and Juan Justo: our reading of these sources, not a finding from any one of them.
What the sources say
16 statements from 12 sourcesGoogle recommends running a Google Ads experiment for 2 to 3 weeks, uses an 80% confidence interval by default, and auto-applies a winning trial to 100% of traffic unless you turn that off.
“We recommend allowing your experiment to run for 2 to 3 weeks to gather data.”
Juan Justo says to calculate a test's sample size and duration before starting, so you know when to stop and don't end it the moment it looks significant.
“please run a pre-test”
Juan Justo says slicing one test's results by new versus returning, landing page or country is bad practice, since the smaller segments give noisy, extreme reads.
“that's a really bad practice just because you don't you don't actually get a clean read”
Google advises running experiments one at a time for at least 4 to 6 weeks, discards the first 7 days, and calls a spend gap under 20% between arms normal. Avoid reactive changes in the first 2 weeks.
“It’s recommended that you run the experiment for at least 4-6 weeks or longer if you have a long conversion delay.”
When traffic is too low for statistical significance, this podcast suggests choosing which landing page to back from engagement signals such as heat maps, as an editorial call.
“we're just getting more engagement on this new variation.”
Stella, which sells testing, says a budget call should state direction, size and a check date, with breakeven at 1 divided by margin (a 40% margin means 2.5x). Its step-size examples are illustrative.
“Cutting a channel because the point estimate looked soft is how brands kill channels that were working.”
Sellforte, which sells incrementality testing, says to judge a test by its setup, uncertainty and fit with similar tests. In its hypothetical, an iROAS of 4 with a 0 to 8 interval is inconclusive, not proof of zero.
“A narrow confidence interval cannot repair a biased comparison.”
Sellforte, which sells testing, says no fixed spend makes a channel too small to test; it is too small when no feasible design can detect an effect that would change the budget call. Then use broader evidence.
“A full pause in selected units produces more signal than a shallow cut everywhere”
Convert, which sells A/B testing software, says an inconclusive test does not prove no effect: check for tracking bugs, re-test if it was underpowered but moved, and move on if a clean test showed nothing.
“If tests are inconclusive due to low traffic, then you need to expand the test audience and re-run.”
Google's GeoX guidance says a geo test's minimum detectable effect depends on geo volatility and matching. Fewer than about 20 geos means low power, and a lower budget makes an inconclusive result more likely.
Haus simulated a year of test-driven budget moves 36 million times: precise tests beat doing nothing 82% of the time vs 62% for noisy ones, and 15 noisy tests a year did no better than 6.
“The precise scenarios beat a frozen-budget baseline 82% of the time. The noisy ones, only 62%.”
Clean Commit, a Shopify agency, says small stores should test offers first, since they move conversion 15 to 40%, and skip A/B tests under about 1,000 sessions or 20 to 30 conversions a month.
“traffic isn't really what powers a test. Conversions are.”
CTC, which sells geo tests, says across hundreds of its tests Facebook acquisition's median incrementality factor is 1.14, with single tests ranging from 0.5x to 2.4x, so one test is not enough.
“single test results are insufficient”
August Stanislaus says lift studies often give inconsistent results, so teams have to read between the lines and accept some uncertainty.
“You'll run the same test three times and get three different answers and then you'll have to try to just read between the lines and get comfortable with a little bit of uncertainty.”
Atticus Li, a testing consultant, says an inconclusive A/B test found no difference, not proof there is none: keep the control, extend only to a fixed sample size, or try a bolder change.
“After two or three well-powered tests in the same area come back inconclusive, it is time to accept that this element is not a significant driver of the metric you are targeting.”
Recast says small, noisy tests that reach significance tend to overstate impact: one GeoLift read 11x ROI where a larger lift study found about 2.3x. It advises using the interval bound nearest zero.
“treat the bound of the confidence interval closest to zero as your conservative estimate”
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it3 figures from cited sources
- Where each source standsA chart placing each source on the question
- The strongest disagreementThe best case against the answer, from a cited source
- How sure we areWhat its confidence rests on, and what would change the answer