How should we prioritize the list of site improvements?
Our answer
Rank every site idea on likely profit impact, evidence, cost and time to result. Fix obvious bugs at once, favor offer changes over cosmetic ones, schedule big changes for quiet months and re-rank each quarter.
- Confidence
- Medium
- Strongest proof
- 1 dataset
- 8 practitioners, 1 opinion or claim
- Sources
- 7 sources
- 10 statements
- BCCC
SNNC
New here? Take the 1-minute tour
Why this is the answer
3 reasons and our judgmentA simple score on the same few factors does the job. Eightx and Blend Commerce both score projects on return, proof or odds, cost and time, and Eightx re-scores every 90 days.
Offer changes belong near the top because bigger effects need less traffic to prove. Clean Commit puts them first: about 2,600 visitors per variation detect a 50% lift, against 15,000 for a 20% lift. Busier pages reach a result sooner too (Convertcart).
Timing matters too, since big changes suit quiet months and light tests suit the weeks before a sale. Shoplift puts navigation and checkout changes in quiet months and light tests before a sale.
Our judgment
- Any simple score for ranking site ideas will do. A score of return, evidence, cost and time works without one named framework, because Eightx and Blend Commerce score much the same factors in different ways.
- Keep offer tests running. Rank offer changes and the busiest pages high, and keep one or two offer tests running beside smaller ones. This follows Clean Commit's tiers and applies to every store the advice Intelligems and Convertcart give stores short of traffic, to test where the most shoppers see the change.
- Let your sales calendar overrule the score. Fit the ranked list to the sales calendar, taking Shoplift's point that structural changes belong in quiet months and light ones before a peak.
- Give each site idea a size and a metric. Add a rough effort estimate and the metric it should move, applying Nielsen Norman Group's advice on getting research onto a product roadmap to a store's improvement list.
- Judge price tests on profit. Judge a price test on profit per visitor, not revenue per visitor, since a lower price can raise revenue while cutting profit. This applies Eightx's scoring of return on margin rather than revenue and matches the answer on testing prices (Measuring a price change).
Our judgment, built from Eightx, Blend Commerce, Clean Commit, Intelligems, Shoplift, Nielsen Norman Group and Convertcart: our reading of these sources, not a finding from any one of them.
What the sources say
10 statements from 7 sourcesBlend Commerce, which sells A/B testing, says most tests produce no clear winner, and relays Optimizely's finding that 12% of 127,000+ experiments significantly improved their main metric.
“Most A/B tests do not produce a conclusive winner.”
Intelligems suggests a couple hundred orders per variant before trusting a result, about a month for a product doing 100 orders a week. With little traffic, make bigger changes and test one variant against a control.
“A loose guidepost is a couple hundred orders per variant before you trust a result”
Blend Commerce, which sells both CRO and redesigns, advises optimizing a Shopify store before redesigning it, unless the theme itself is bloated, outdated or not built mobile-first.
“a full redesign is rarely the first or best solution.”
Blend Commerce, which sells CRO audits, scores each recommendation out of 5 on proof, ease, cost, time and impact, weighted to a total out of 100, and won't run several tests at once in shared setups.
“We won't run multiple tests at once to maintain clean, accurate results”
Convertcart, a CRO firm, says 414 of its 1,103 client experiments (37%) produced a significant win, and blames failures mostly on winners never shipped and tests too small for the traffic.
“Assign a named owner accountable for getting every winner into production”
Clean Commit, which sells CRO work, ranks test ideas by whether they change what the customer pays or gets, and says to start with a free shipping threshold. It relays GoodUI medians: 6.6% single-element, 23% bundled.
“If you've never run a Tier 1 test, start with a free shipping threshold.”
Clean Commit, a Shopify agency, says small stores should test offers first, since they move conversion 15 to 40%, and skip A/B tests under about 1,000 sessions or 20 to 30 conversions a month.
“traffic isn't really what powers a test. Conversions are.”
Shoplift, which sells A/B testing software, says ecommerce test roadmaps should follow the sales calendar: heavy tests in quiet periods, light ones before peaks, with three to four weeks per test.
“Do not redesign checkout in November, do not waste a quiet January, match the test to the moment.”
Nielsen Norman Group tells researchers to turn findings into recommendations tied to what the product manager is measured on, sized by effort and ready before quarterly planning. It gives no data.
“Research findings are not roadmap arguments.”
Eightx, which sells fractional CFO services, ranks DTC projects by scoring six factors and multiplying them, so a zero on any one kills a project. Return is scored on 12-month margin, not revenue.
“The website redesign is last because it eats 120 person-days for a return the team is not actually confident in.”
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it3 figures from cited sources
- Where each source standsA chart placing each source on the question
- The strongest disagreementThe best case against the answer, from a cited source
- How sure we areWhat its confidence rests on, and what would change the answer