All answers

The Prevailing
Answers for
Business model:
Business model, pick up to two

How should we decide when to act on an unclear result?

The PrevailingPublished Oct 29, 2026

Our answer

First sort an unclear result. Rerun a broken test. On a sound one, hold or take a small reversible step judged on the low end of its range, with 80 to 90% confidence for reversible moves and 95% otherwise.

Confidence
Medium
Strongest proof
3 official rules
1 dataset, 1 practitioner, 11 opinions or claims
Sources
12 sources
16 statements

New here? Take the 1-minute tour

Why this is the answer

3 reasons and our judgment

An unclear result is not proof of no effect. Sellforte (marketing mix modeling firm) calls an incremental return on ad spend of 4, with a 90% interval of 0 to 8, inconclusive. A narrow range near zero means the effect is small (Atticus Li).

Holding is a decision in its own right. Stella (incrementality testing software maker) uses 90% confidence on reversible monthly moves and 95% on irreversible ones. Google's experiments default to 80%.

Small tests overstate the effect, so judge on the bound nearest zero. Recast saw a small geo test read 11 times where a larger study found about 2.3 times, and it treats that bound as the conservative estimate.

Our judgment

  • Sort an unclear result into broken, inconclusive or small before deciding anything. Sellforte's setup checks, Clean Commit's 58/42 split check and Convert's tracking and sample ratio checks catch a broken test. Sellforte and Atticus Li separate a wide interval that spans the bar from a narrow one near zero.
  • On an inconclusive result, hold or take a small reversible step judged on the end of the range closest to zero, and move your estimate only partway. This combines Stella's hold rule, Recast's conservative bound and Common Thread Collective's practice of weighting a first result by its confidence. Acting on a finished lift or geo test sizes the step against breakeven.
  • Use 80 to 90% confidence for cheap, reversible moves and 95% for moves that are hard to undo. The range draws on Google's 80% default, Clean Commit's 85% or 90% for easy-to-reverse changes and Stella's 90% and 95%.
  • Rerun bigger or longer once, with the sample size fixed in advance, and move on after two or three sound flat tests. This follows Atticus Li's rules, Convert's advice to expand the audience when traffic was low and Juan Justo's warning against peeking and slicing.

Our judgment, built from Sellforte, Clean Commit, Convert, Atticus Li, Stella, Recast, CTC, Google and Juan Justo: our reading of these sources, not a finding from any one of them.

What the sources say

16 statements from 12 sources
  • GoogleOct 26, 2026Supports

    Google recommends running a Google Ads experiment for 2 to 3 weeks, uses an 80% confidence interval by default, and auto-applies a winning trial to 100% of traffic unless you turn that off.

    “We recommend allowing your experiment to run for 2 to 3 weeks to gather data.”

    Read it on support.google.com

  • Andrew FarisOct 26, 2026Supports

    Juan Justo says to calculate a test's sample size and duration before starting, so you know when to stop and don't end it the moment it looks significant.

    “please run a pre-test”

    Read it on podcasters.spotify.com

  • Andrew FarisOct 26, 2026Caveat

    Juan Justo says slicing one test's results by new versus returning, landing page or country is bad practice, since the smaller segments give noisy, extreme reads.

    “that's a really bad practice just because you don't you don't actually get a clean read”

    Read it on podcasters.spotify.com

  • GoogleOct 26, 2026Caveat

    Google advises running experiments one at a time for at least 4 to 6 weeks, discards the first 7 days, and calls a spend gap under 20% between arms normal. Avoid reactive changes in the first 2 weeks.

    “It’s recommended that you run the experiment for at least 4-6 weeks or longer if you have a long conversion delay.”

    Read it on support.google.com

  • Cody PlofkerSep 29, 2026Caveat

    When traffic is too low for statistical significance, this podcast suggests choosing which landing page to back from engagement signals such as heat maps, as an editorial call.

    “we're just getting more engagement on this new variation.”

    Read it on podcasters.spotify.com

  • StellaSep 23, 2026Supports

    Stella, which sells testing, says a budget call should state direction, size and a check date, with breakeven at 1 divided by margin (a 40% margin means 2.5x). Its step-size examples are illustrative.

    “Cutting a channel because the point estimate looked soft is how brands kill channels that were working.”

    Read it on stellaheystella.com

  • SellforteSep 10, 2026Supports

    Sellforte, which sells incrementality testing, says to judge a test by its setup, uncertainty and fit with similar tests. In its hypothetical, an iROAS of 4 with a 0 to 8 interval is inconclusive, not proof of zero.

    “A narrow confidence interval cannot repair a biased comparison.”

    Read it on sellforte.com

  • SellforteSep 3, 2026Supports

    Sellforte, which sells testing, says no fixed spend makes a channel too small to test; it is too small when no feasible design can detect an effect that would change the budget call. Then use broader evidence.

    “A full pause in selected units produces more signal than a shallow cut everywhere”

    Read it on sellforte.com

  • ConvertSep 1, 2026Supports

    Convert, which sells A/B testing software, says an inconclusive test does not prove no effect: check for tracking bugs, re-test if it was underpowered but moved, and move on if a clean test showed nothing.

    “If tests are inconclusive due to low traffic, then you need to expand the test audience and re-run.”

    Read it on convert.com

  • GoogleSep 1, 2026Supports

    Google's GeoX guidance says a geo test's minimum detectable effect depends on geo volatility and matching. Fewer than about 20 geos means low power, and a lower budget makes an inconclusive result more likely.

    Read it on developers.google.com

  • HausJul 23, 2026Supports

    Haus simulated a year of test-driven budget moves 36 million times: precise tests beat doing nothing 82% of the time vs 62% for noisy ones, and 15 noisy tests a year did no better than 6.

    “The precise scenarios beat a frozen-budget baseline 82% of the time. The noisy ones, only 62%.”

    Read it on haus.io

  • Clean CommitJul 17, 2026Supports

    Clean Commit, a Shopify agency, says small stores should test offers first, since they move conversion 15 to 40%, and skip A/B tests under about 1,000 sessions or 20 to 30 conversions a month.

    “traffic isn't really what powers a test. Conversions are.”

    Read it on cleancommit.io

  • Common Thread CollectiveJul 2, 2026Supports

    CTC, which sells geo tests, says across hundreds of its tests Facebook acquisition's median incrementality factor is 1.14, with single tests ranging from 0.5x to 2.4x, so one test is not enough.

    “single test results are insufficient”

    Read it on commonthreadco.com

  • DTC Newsletter / DTC PodcastJun 8, 2026Disagrees

    August Stanislaus says lift studies often give inconsistent results, so teams have to read between the lines and accept some uncertainty.

    “You'll run the same test three times and get three different answers and then you'll have to try to just read between the lines and get comfortable with a little bit of uncertainty.”

    Read it on podcasters.spotify.com

  • Atticus LiApr 7, 2026Supports

    Atticus Li, a testing consultant, says an inconclusive A/B test found no difference, not proof there is none: keep the control, extend only to a fixed sample size, or try a bolder change.

    “After two or three well-powered tests in the same area come back inconclusive, it is time to accept that this element is not a significant driver of the metric you are targeting.”

    Read it on atticusli.com

  • RecastFeb 24, 2026Supports

    Recast says small, noisy tests that reach significance tend to overstate impact: one GeoLift read 11x ROI where a larger lift study found about 2.3x. It advises using the interval bound nearest zero.

    “treat the bound of the confidence interval closest to zero as your conservative estimate”

    Read it on getrecast.com

In the full answer

For readers who sign in
  • What to do this week3 steps
  • The numbers behind it3 figures from cited sources
  • Where each source standsA chart placing each source on the question
  • The strongest disagreementThe best case against the answer, from a cited source
  • How sure we areWhat its confidence rests on, and what would change the answer

Read next

More in Team and process