How should we test changes during peak season?
Our answer
Run and call your store tests before your peak. Stop theme, app and checkout changes 10 to 14 days before it, and start no new A/B tests during it, because peak results rarely hold afterward.
- Confidence
- Medium
- Strongest proof
- 2 official rules
- 1 dataset, 5 practitioners, 8 opinions or claims
- Sources
- 12 sources
- 16 statements
BFD
New here? Take the 1-minute tour
Why this is the answer
3 reasons and our judgmentPeak tests rarely give results you can keep. Tests started in Black Friday week rarely reach valid results and won't hold in January, says Brillmark (A/B test development agency), which freezes code about 10 days before Black Friday.
WeltPixel and Fudge stop changes even earlier. WeltPixel (Shopify tracking app maker) finishes checkout, theme and app changes two full weeks out. Fudge (Shopify AI theme tool maker) freezes structural storefront changes after September 30 and ships only campaign pages in November.
Promotional periods read differently from normal ones. Lift and holdout tests by Common Thread Collective (ecommerce ad agency) measured promotional periods 30 to 50% away from normal periods. Shopify and Datma both keep price tests out of holidays and sales events.
Our judgment
- Finish and call site tests before the peak and start none during it. Brillmark's warning that peak-week tests don't hold in January and Common Thread Collective's 30 to 50% gap between promotional and normal reads point the same way.
- Stop theme, app and checkout changes 10 to 14 days before the peak. That range spans Brillmark's freeze about 10 days before Black Friday and WeltPixel's two full weeks, and Fudge's September 30 freeze keeps structural rebuilds well before the season.
- Apply this timing to whatever your peak is, not only Black Friday. Sean Frank of Ridge says more than half of Shopify brands don't peak then, and Common Thread Collective advises finding your category's biggest moment.
- Test the offer itself before the peak, on Shopify's Rollouts where your plan allows. Menachem Ani advises testing discounts in October with Rollouts, within Shopify's Rollouts rules.
Our judgment, built from Brillmark, CTC, WeltPixel, Fudge, Sean Frank, Menachem Ani and Shopify: our reading of these sources, not a finding from any one of them.
What the sources say
16 statements from 12 sourcesShopify's built-in Experiments need the Grow plan or higher and cannot test headless checkouts or Liquid template changes. During a test, ads and feeds may show a different offer from the store.
“Product ads, listings, feeds, and third-party apps might not display the same offer.”
Fudge, which sells a page builder, says freeze structural storefront changes after September 30 and ship only campaign pages on top in November, with everything done by November 25.
“The single biggest mistake: rebuilding the storefront in November. Freeze the storefront, ship campaign pages on top.”
A Google Ads consultant says a conversion rate drop in Black Friday week usually comes from traffic quality, site UX or the offer, not the ads, and suggests testing discounts in October. No data.
“When conversion rate drops in BF week, the ads get blamed first.”
CTC says a ROAS target from a Black Friday lift test may not fit January; keep the test dates and promotion attached so a seasonal result does not become a permanent budget rule.
“Keep the test dates and promotional context attached to the result so a seasonal finding doesn't become a permanent budget assumption.”
CTC's Luke Austin says a handful of its lift tests in promotional periods showed incrementality 30 to 50% different from normal periods, in either direction, which should change BFCM ROAS targets.
“it's 30, 40, 50% variation in, in some direction against your normal”
Brillmark, which builds A/B tests, says tests started in Black Friday week rarely give valid results; freeze code about 10 days before and skip new tests if not live by late October.
“Tests started in Black Friday week rarely reach valid results and the data won't hold in January.”
Intelligems, which sells A/B testing software, says pause a test for Black Friday only if restarting within days, otherwise end it and relaunch after BFCM; orders during a pause do not count.
“If a test has to stop for Black Friday, pause it when you will restart it within days, and end it when it would sit idle for weeks.”
Ridge's CEO says more than half of Shopify brands do not peak in Black Friday week, and that brands in categories like swimwear, patio, garden or pet should ignore BFCM advice. The figure has no source.
“If you try to compete in the Meta auction, strong gifting brands like mine will demolish you.”
Charle, which sells Shopify migrations, says migrate only when the platform caps growth that spending can't lift, and puts a Shopify Plus move at $25,000 to $50,000 for a simple store and past $300,000 for complex ones.
“Migrating rarely moves conversion rates on its own.”
WeltPixel, which sells tracking apps, says finish checkout, theme and app changes by about November 13 and leave tracking untouched from about November 16, noting Shopify publishes no freeze date.
“The freeze exists to prevent optional changes, not repairs.”
Shopify's Rollouts, under Markets > Rollouts in the admin, can schedule, gradually publish and A/B test themes, checkout and customer account setups, including two entirely different themes.
“between two entirely different themes or checkout setups to find the winner”
Across 225 geo tests on its platform, Stella, which sells testing, found a median 2.31x incremental ROAS. It suggests 4 to 6 weeks for most brands and skipping tests that can't get a few hundred conversions.
“The confidence interval matters more than the point estimate”
Shopify's price elasticity guide says change price 10% to 20%, run at least 7 to 14 days outside holidays, change nothing else, and keep the price if revenue and profit rose. Its example is hypothetical.
“Adjust your price by a meaningful percentage (typically 10% to 20%) to trigger a measurable reaction from customers.”
Datma, which sells store analytics, says Shopify keeps no price history; it ranks before and after comparisons as weakest and single-product price tests over 2 to 4 weeks as most reliable.
“Shopify doesn't maintain a log of historical product prices.”
A CTC strategist says find your category's biggest moment rather than defaulting to Black Friday; for one client CPMs are cheaper at another time with the same conversion rate, and health brands pick up after Christmas.
“because CPMs are cheaper and they get the same conversion rate”
Shoplift, which sells A/B testing, says Black Friday traffic can spike 5 to 10 times normal and most tools advise pausing tests, but its merchants tested through the weekend. The 5 to 10 times figure has no source.
“BFCM weekend isn't just your highest traffic, it's your most valuable data.”
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it3 figures from cited sources
- Where each source standsA chart placing each source on the question
- The strongest disagreementThe best case against the answer, from a cited source
- How sure we areWhat its confidence rests on, and what would change the answer