All answers

The Prevailing
Answers for
Business model:
Business model, pick up to two

How should a store our size run A/B tests, and how much traffic does a test need?

The PrevailingPublished Oct 28, 2026

Our answer

Test only big changes. Plan for a few hundred orders in each version, run at least 2 full weeks and judge on revenue per visitor. Skip A/B tests below about 1,000 sessions or 20 to 30 conversions a month.

Confidence
Medium
Strongest proof
6 official rules
1 dataset, 7 practitioners, 3 opinions or claims
Sources
10 sources
17 statements

New here? Take the 1-minute tour

Why this is the answer

3 reasons and our judgment

A readable test needs hundreds of orders per version. Intelligems puts it at a couple hundred per variant, about a month for a best-seller at 100 orders a week. Clean Commit (Shopify development agency) plans 350 conversions per group for a 10 to 15% lift.

Small lifts need far more traffic. Catching a 20% lift at a 3% conversion rate takes about 13,900 visitors per version, and a 10% lift about 53,100 (Fudge, Shopify AI theme tool maker).

Tests run for weeks, not days. Intelligems and Convertibles run them 2 to 3 weeks, and Shoplift sets a floor of two to four weeks for Shopify Plus stores. Clean Commit skips A/B tests under about 1,000 sessions or 20 to 30 conversions a month.

Our judgment

  • Plan for a few hundred orders in each version. That is the span between Intelligems' couple hundred orders per variant and Clean Commit's 350 conversions per group.
  • Run each test for at least 2 full weeks. That is the shortest run Intelligems, Convertibles, Shoplift and Clean Commit give, counted in whole weeks as Fudge advises.
  • Judge a test on revenue or profit per visitor, with purchases as a guardrail. This combines Talon.One's rule to decide on incremental contribution, Intelligems' profit per visitor and Clean Commit's purchase-rate guardrail.
  • On Shopify, try Rollouts for theme, checkout, discount and catalog tests before paying for a testing app. Shopify's help pages say Experiments run on the Grow plan or higher, with a 50/50 split by default.

Our judgment, built from Intelligems, Clean Commit, Convertibles, Shoplift, Fudge, Talon.One and Shopify: our reading of these sources, not a finding from any one of them.

What the sources say

17 statements from 10 sources
  • ShopifyUndatedSupports

    Shopify's built-in Experiments need the Grow plan or higher and cannot test headless checkouts or Liquid template changes. During a test, ads and feeds may show a different offer from the store.

    “Product ads, listings, feeds, and third-party apps might not display the same offer.”

    Read it on help.shopify.com

  • ShopifyUndatedSupports

    Shopify Experiments test theme, checkout, catalog or discount changes against a control, by default with a 50/50 traffic split and an end date 90 days after the start.

    “By default, a new Experiment uses a traffic split of 50% control and 50% treatment, and an end date 90 days after the Experiment starts.”

    Read it on help.shopify.com

  • ShopifyUndatedCaveat

    Shopify says editing an Experiment's treatment or control while it runs can affect results. You can run several Experiments on one resource at once; each gets its own exclusive share of traffic.

    “You can run multiple Experiments on the same resource simultaneously.”

    Read it on help.shopify.com

  • FudgeOct 27, 2026Supports

    Fudge, which sells a theme editing tool, works out A/B sample sizes at 95% significance and 80% power: a 3% baseline with a 20% relative lift needs about 13,900 visitors per variant.

    “a '20% relative lift' at 95% significance and 80% power gives '13,900 visitors per variant'”

    Read it on fudge.ai

  • ShopifyOct 23, 2026Supports

    Shopify's Rollouts now handle catalog changes, so stores can schedule product availability and pricing per market and run experiments between catalogs on the online store.

    “Rollouts now supports catalog changes, so you can schedule product availability and pricing per market.”

    Read it on changelog.shopify.com

  • Talon.OneOct 19, 2026Supports

    Talon.One says scale a promotion only when a randomized holdout shows positive incremental contribution after discount cost. In its hypothetical example, a discount adds 50 orders but cuts contribution by $3,750.

    “A promotion should not scale because redemption rises.”

    Read it on talon.one

  • ShopifyOct 2, 2026Supports

    Shopify added discounts to Rollouts in October 2026, so stores can test offers against existing ones and compare conversion before launching widely. Existing eligibility rules still apply.

    “Test discount changes against your existing offers and review conversion results to help decide which offer to keep.”

    Read it on changelog.shopify.com

  • Cody PlofkerSep 29, 2026Disagrees

    When traffic is too low for statistical significance, this podcast suggests choosing which landing page to back from engagement signals such as heat maps, as an editorial call.

    “we're just getting more engagement on this new variation.”

    Read it on podcasters.spotify.com

  • Homestead / Mars MenSep 17, 2026Caveat

    MarsMen runs small page changes as normal on-site A/B tests, but tests a new angle, offer or page format by duplicating the Meta campaign to the new page and checking for added spend at target efficiency.

    “incremental spend that would have not happened otherwise”

    Read it on splitbase.com

  • Blend CommerceAug 28, 2026Caveat

    Blend Commerce, which sells A/B testing, says most tests produce no clear winner, and relays Optimizely's finding that 12% of 127,000+ experiments significantly improved their main metric.

    “Most A/B tests do not produce a conclusive winner.”

    Read it on blendcommerce.com

  • IntelligemsAug 21, 2026Caveat

    Intelligems studied 157 A/B test launches across 69 brands and 170M+ visits and found no systematic effect on paid traffic (paid social median +0%), including 29 redirect and theme tests. It sees no ad spend or bid data.

    “The big-swing tests nudged paid no more than the quiet ones”

    Read it on intelligems.io

  • IntelligemsAug 14, 2026Supports

    Intelligems suggests a couple hundred orders per variant before trusting a result, about a month for a product doing 100 orders a week. With little traffic, make bigger changes and test one variant against a control.

    “A loose guidepost is a couple hundred orders per variant before you trust a result”

    Read it on intelligems.io

  • ShopliftJul 31, 2026Supports

    Shoplift, which sells price testing software, says prices should be retested regularly, winners rolled out to all traffic, and most Shopify Plus stores need at least two to four weeks per test.

    “For most Shopify Plus stores that's a floor of two to four weeks.”

    Read it on shoplift.ai

  • Clean CommitJul 17, 2026Supports

    Clean Commit, a Shopify agency, says small stores should test offers first, since they move conversion 15 to 40%, and skip A/B tests under about 1,000 sessions or 20 to 30 conversions a month.

    “traffic isn't really what powers a test. Conversions are.”

    Read it on cleancommit.io

  • IntelligemsJun 17, 2026Supports

    Intelligems, which sells price testing software, says to test the current price against prices 5 to 10% lower and higher at once for two to three weeks, judging on profit per visitor.

    “Based on data from thousands of price tests, a 5% price change typically moves conversion rate by around 10% in the opposite direction.”

    Read it on intelligems.io

  • ShopifyJun 5, 2026Supports

    Shopify's Rollouts, under Markets > Rollouts in the admin, can schedule, gradually publish and A/B test themes, checkout and customer account setups, including two entirely different themes.

    “between two entirely different themes or checkout setups to find the winner”

    Read it on changelog.shopify.com

  • ConvertiblesDec 28, 2025Disagrees

    Convertibles says to run a popup A/B test until it reaches 95% confidence, with at least 1,000 visitors per variation, which takes 2 to 3 weeks for most Shopify brands.

    “Until you hit statistical significance at 95% confidence. Minimum 1,000 visitors per variation.”

    Read it on convertibles.dev

In the full answer

For readers who sign in
  • What to do this week3 steps
  • The numbers behind it3 figures from cited sources
  • Where each source standsA chart placing each source on the question
  • The strongest disagreementThe best case against the answer, from a cited source
  • How sure we areWhat its confidence rests on, and what would change the answer

Read next

More in Capture, testing and team