Here's a confession that might get my CMO card revoked: I've watched marketing teams burn through testing budgets like they're trying to win a prize for most experiments conducted. Spoiler alert: there's no trophy. Just a lot of inconclusive data and confused executives wondering why the measurement program that was supposed to bring clarity delivered a migraine instead.
The instinct makes sense. Platform attribution is increasingly unreliable, privacy regulations keep tightening, and everyone from the CFO to the board wants proof that marketing dollars actually move the needle. So we test everything. Meta campaigns. Google Shopping. That influencer partnership someone swore would be "huge." The result? A calendar so packed with holdout experiments that we're essentially running a research lab instead of a marketing department.
There's a better way. And it starts with accepting an uncomfortable truth: your testing capacity is finite, and not every question deserves an answer.
The Real Cost of Testing Everything
Every incrementality test requires a holdout group, which means deliberately not showing ads to a portion of your audience. That's revenue you're leaving on the table during the test window. For a mid-sized B2B SaaS company, a four to six week geo-holdout test on a major channel can easily represent six figures in opportunity cost.
Now multiply that by the eight or ten tests some teams try to squeeze into a year.
Tom Leonard's recent piece on MarTech nails the core problem:
Testing capacity is limited, so spend it on questions where uncertainty has meaningful financial consequences.
That's the filter most teams skip. They test because they can, not because the answer will change anything.
I've sat in rooms where teams presented incrementality results for channels that were already validated by their MMM, their observational analysis, and frankly, common sense. The test confirmed what everyone already knew. Meanwhile, the channel that was quietly eating budget while delivering questionable results? Still untested six months later.
Where Uncertainty Actually Lives
The tests worth running share a common trait: they address questions where being wrong costs real money.
Leonard offers a useful example: Meta's Advantage+ Shopping Campaigns reporting attractive marginal CPAs while overall customer acquisition costs climb. That's a signal worth investigating. Either Meta is finding incremental customers efficiently and something else is driving CAC higher, or the platform is claiming credit for conversions that would have happened anyway.
The P&L implications of getting that wrong are significant. If ASC is highly incremental, you scale. If you're out on the diminishing returns curve, you reallocate those dollars. That's a test worth the holdout cost.
Contrast that with testing a channel where your MMM, your sales response data, and your gut all agree it's working. What exactly changes if the incrementality test confirms what you already believe? You've spent testing capacity to feel slightly more confident about a decision you were going to make anyway.
The IDEATE Framework (Yes, It's an Acronym, and I'm Sorry)
I'm generally allergic to cute acronyms, but the IDEATE framework Leonard outlines has enough practical value that I'll forgive the naming. It stands for Insight, Draft Hypothesis, Envision Paths, Arrange the Test, Track Results, and Execute on Findings.
Most of those steps are self-explanatory. The two where teams consistently stumble are Insight and Envision Paths.
Insight is about being honest with yourself before you even design the test. What have you actually seen in your data that makes you question current assumptions? If you can't articulate a specific observation that triggered the question, you're probably testing out of anxiety rather than strategy.
Envision Paths is where the real discipline lives. Before you run the test, map out what you'll actually do with each possible outcome. If the channel tests as highly incremental, what changes? If it tests as marginally incremental? If it tests as not incremental at all?
If your answer to all three scenarios is "we'll probably keep doing what we're doing," you don't need the test. You need to admit you've already made the decision.
Starting Without a Baseline
Roman Petrochenkov, Head of Analytics Growth at Carwow, makes a point that's worth internalizing: your first incrementality test shouldn't try to solve everything. The goal is to have a successful test, meaning one that teaches you how to set up experiments, how to learn from them, and how to communicate results internally.
He recommends starting with a mid-sized campaign that's least reliant on creative variables and funnel complexity. For some businesses, that's branded search. The creative plays a minimal role, most of the journey leads to the homepage, and the risks of a holdout are lower than testing your top-performing prospecting campaign.
Will this first test deliver groundbreaking insights? Probably not. But it builds the organizational muscle for the tests that will.
The Results Have a Shelf Life (Use Them)
Here's where I see even sophisticated teams drop the ball. They run a well-designed test, get statistically significant results, present them in a deck that gets polite nods in a meeting, and then... nothing changes.
Incrementality results should inform decisions long after the test ends. If you discovered that a channel's true incremental ROAS is 40% lower than platform attribution suggested, that finding should reshape how you allocate budget for the next several quarters. It should recalibrate your MMM. It should change how you evaluate that channel's performance in weekly reviews.
Measured's approach of combining incrementality testing with continuously calibrated MMM points toward where this is heading: tests that don't just answer one-time questions but feed into models that guide ongoing optimization.
The Cadence Question
Postie's guidance on test frequency is refreshingly direct: a standard brand with average agility can feel good running a test every six months. More agile organizations might go quarterly, but only if they can actually act on the findings between tests.
The question to ask before scheduling your next test: "When I get the results, what will we do?" If the answer is "we'll figure it out next year," extend the test window and save yourself the operational complexity of running multiple shorter experiments.
The Uncomfortable Math
Running fewer tests feels counterintuitive in an era where "data-driven" is the highest compliment a marketing team can receive. But data-driven doesn't mean data-drowned. It means using the right data to make better decisions.
Three well-designed tests that address genuine uncertainty and lead to meaningful budget reallocation will outperform a dozen tests that confirm what you already knew or answer questions nobody was going to act on anyway.
The goal isn't to test more. It's to know more. And sometimes, the fastest path to knowing more is admitting which questions don't actually need answering.
