How should we judge an email agency's results?
Our answer
Judge an email agency on what it adds to store revenue, against a pre-agency baseline and a holdout. Use attribution settings locked in your own account. Check setup at 90 days and return at 6 months.
- Confidence
- Medium
- Strongest proof
- 3 official rules
- 5 practitioners, 3 opinions or claims
- Sources
- 9 sources
- 11 statements
- K
IEZDSAM
New here? Take the 1-minute tour
Why this is the answer
3 reasons and our judgmentAttributed revenue is easy to inflate. An agency paid on it has every reason to widen the attribution window (Maestra). Klaviyo applies attribution changes to past as well as future conversions.
Agencies should be judged against the baseline before they started. Cut one that can't show incremental revenue against it (Enrich Labs). InboxArmy, an email marketing agency, judges early signals at 90 days and return at 6 months.
A holdout shows what email causes. Zettler Digital and Everboost hold out part of the audience, Zettler Digital comparing store revenue per profile. Attributed revenue is not a direct measure of incremental revenue, says Klaviyo's community manager.
Our judgment
Lock and record attribution settings in an account you own before the agency starts. Klaviyo's rule on past conversions, Maestra's and Enrich Labs' conditions of hiring and Eightx's point that default settings flatter email all lead there. Check at about 90 days that the core flows are live and judge return at about 6 months, combining InboxArmy's review points with Enrich Labs' month 3 scorecard and quarterly decision. Prove what the agency adds with a holdout on one busy flow in the first quarter, not a before and after comparison alone, because some attributed revenue would have happened without the email. Zettler Digital's minimum of 2 to 4 weeks lets the result land before Enrich Labs' month 3 scorecard.
Our judgment, built from Klaviyo, Maestra, Enrich Labs, Eightx, InboxArmy, Zettler Digital, Everboost and Sorted Agency: our reading of these sources, not a finding from any one of them.
What the sources say
11 statements from 9 sourcesGoogle requires senders to keep Postmaster Tools spam rates below 0.3%, and those sending 5,000+ messages a day to Gmail must offer one-click unsubscribe and a visible unsubscribe link.
“Keep spam rates reported in Postmaster Tools below 0.3%.”
Sorted, a Klaviyo agency, puts published retainers at $2,500 to $10,000 a month and says below about $20,000 a month in store revenue, build core flows yourself or hire a freelancer.
“Below about $20,000 a month in store revenue, a $2,500+ retainer is hard to pay back.”
Enrich Labs, which sells an AI marketing tool, says to hire an email agency on outcomes: you own the ESP account, no one can promise inbox placement, and cut them if they cannot show incremental revenue.
“Nobody can promise inbox placement.”
Maestra, an email platform vendor, puts Klaviyo agencies at $2,500 to $10,000 a month and one in-house email manager at $11,200 to $13,200, and warns against paying agencies on revenue share.
“An agency paid on attributed revenue has every reason to widen it”
A Klaviyo community manager tells an account whose attributed share fell from 40% to 26% after tighter settings to keep settings fixed, pair attributed revenue with other metrics, and use holdouts for lift.
“treat attributed revenue as a measurement and optimization signal, not as a direct measure of incremental revenue.”
Everboost, a retention agency, says Klaviyo's attributed revenue counts refunds and renewals and is not cash. It suggests tighter windows, holdouts, and judging email on repeat buying cohort by cohort.
“is a claim about influence, not a ledger of cash”
Eightx says Klaviyo's default 5-day attribution counts opens, including Apple privacy opens, which inflates email revenue. Switching to click-based attribution can cut it by a third or more in its clients.
“Switch to click-based attribution before you trust the dashboard number.”
InboxArmy, an email agency, says judge an agency at 90 days for early signals and 6 months for clear return, with ROI as incremental email revenue minus agency cost. Incremental means before versus after.
“Evaluate performance at 90 days for early signals and 6 months for clear return on investment.”
Zettler Digital, a Klaviyo agency, says last click overstates email and SMS. It tests one program at a time with a random 90/10 or 80/20 holdout for at least 2 to 4 weeks, measured on Shopify revenue per profile.
“Last-click attribution will almost always overstate the impact of email and SMS, especially for brands with strong brand demand, repeat customers, and short consideration cycles.”
Klaviyo credits sales by last touch by default, within 5 days of an email click, email open or text click and 12 hours of a text delivery. Bot clicks are excluded by default, and changes apply to past data too.
“selecting a longer email open window instead of clicks may introduce a bias towards opens, which may not be conclusive indicators of conversions.”
Klaviyo's global holdout groups need at least 400,000 profiles. It recommends holding out about 5% of marketable profiles for 3 months, and the lift counts all conversions, attributed or not.
“We additionally recommend that you run holdout groups for 3 months to gather enough data.”
In the full answer
For readers who sign in- What to do this week3 steps
- The numbers behind it3 figures from cited sources
- Where each source standsA chart placing each source on the question
- The strongest disagreementThe best case against the answer, from a cited source
- How sure we areWhat its confidence rests on, and what would change the answer