That 12-point lead over your competitor in ChatGPT citations? It might be real. It might also be the statistical equivalent of flipping a coin three times and declaring yourself a probability genius.
New research from IQRush, due for full release next week, quantifies what many of us suspected: the AI visibility metrics populating your dashboards are snapshots of a moving target, not fixed competitive intelligence. The paper provides a method for distinguishing genuine signal from noise, and the findings should make every marketing leader reconsider how they're allocating budget to this emerging channel.
I've spent two decades turning marketing data into board-ready forecasts. The pattern here is familiar: a new measurement category emerges, vendors rush to productize it, and executives start making resource decisions before anyone asks the uncomfortable question about statistical validity. We did this with social sentiment scores. We did it with early attribution models. Now we're doing it with AI visibility rankings.
The Variance Problem Nobody Wants to Model
Generative AI systems are designed to introduce randomness into their outputs. This is a feature, not a bug, from the model's perspective. But it creates a measurement nightmare for marketers trying to track brand visibility.
The IQRush research illustrates this with a concrete example: when testing SearchGPT on running gear queries, Tom's Guide appeared in roughly 9.5% of citations while Runner's World showed up in about 6.0%. On a dashboard, that looks like a clear winner. Tom's Guide is outperforming by 3.5 percentage points.
Except the margin of error on those figures overlapped. With a single sample, you couldn't actually say Tom's Guide was outperforming at all. The apparent gap was within the range of statistical noise.
This isn't an edge case. It's the default condition for most AI visibility measurements being sold today.
What "Stable" Actually Requires
The paper establishes two conditions that must both be true before a ranking becomes trustworthy. First, the order has to stop changing as you add more data. Early in the measurement process, rankings fluctuate wildly because no site has established a clear edge. Only after sufficient responses accumulate do the top performers separate from the pack.
Second, the gap between top sites must exceed the margin of error for each. If two brands are separated by 2 points but each has a 4-point confidence interval, you're looking at a statistical tie dressed up as a competitive insight.
In 30 platform-topic tests, the number of responses needed for both conditions to be met ranged from 33 to 94 (counting only responses that included citations). Three out of 30 tests never reached stability even after 125 questions.
Let that sink in. Ten percent of the test cases couldn't produce a reliable ranking at any sample size the researchers tested.
The Vendor Incentive Problem
IQRush sells software that measures AI visibility using the methodology their paper advocates. That's worth noting. But separate research from Rand Fishkin and Gumshoe.ai
reached similar conclusions after testing nearly 3,000 responses across ChatGPT, Google AI, and Claude. Their finding: AI tools almost never return the same list twice, and ranking order changes constantly.Two independent teams arriving at the same conclusion through different methods is the kind of convergent evidence that should shift your priors.
The uncomfortable implication is that much of the AI visibility tracking market is selling precision it cannot deliver. Not because the vendors are dishonest, but because the underlying phenomenon is genuinely unstable. You can't measure a fixed position when the position isn't fixed.
What This Means for Your Budget
If you're currently paying for AI visibility tracking, here's the question to ask your vendor: what sample size are you using per query, and how are you calculating confidence intervals?
If the answer is vague, or if they're reporting rankings based on single-digit response counts, you're making decisions on noise. The IQRush research suggests you need somewhere between 33 and 94 cited responses per query before rankings stabilize, and even then, 10% of cases may never reach statistical significance.
This doesn't mean AI visibility is unmeasurable. It means the measurement requires more rigor than most current tools provide. Visibility percentage (how often your brand appears at all, regardless of position) may be more defensible than positional rankings, because it's less sensitive to the order-shuffling that happens between responses.
A Framework for the Skeptical CMO
Before your next pipeline review, run this diagnostic on any AI visibility data you're using for resource allocation:
First, check the sample methodology. How many responses per query? Is the vendor running repeated measurements or reporting single snapshots? If they can't tell you, the data isn't board-grade.
Second, demand confidence intervals. Any metric reported without error bars is a point estimate pretending to be a fact. A 15% visibility share with a ±8% confidence interval tells a very different story than the same number with ±2%.
Third, test for stability over time. If your "ranking" swings by 20% week over week with no change in your content or competitive activity, you're probably measuring noise. Real competitive shifts don't oscillate that fast.
Fourth, separate visibility from value. Even if you could measure AI citations perfectly, you'd still need to prove they drive pipeline. The attribution chain from "mentioned in ChatGPT" to "closed-won revenue" is long and leaky. Don't let the novelty of the channel exempt it from the same ROI scrutiny you'd apply to any other marketing investment.
The Honest Forecast
AI-driven search is growing. The channel matters. But the measurement infrastructure hasn't caught up with the hype cycle, and that gap is where marketing budgets go to die.
My recommendation: treat AI visibility as an emerging signal worth monitoring, not a KPI worth optimizing against. Invest in the fundamentals that make your brand citable (authoritative content, clear expertise signals, structured data) rather than chasing dashboard positions that may be artifacts of randomness.
When the measurement science matures, you'll be ready. Until then, model the uncertainty explicitly, or don't put it in front of your CFO. Because nothing erodes credibility faster than presenting statistical noise as competitive intelligence.
