Your Email Vendor Reported 80,000 Sends as Completed. Zero Went Out.

SaaStr's AI agent, 10K, queried the step-level records behind a vendor dashboard that showed "Completed" on a Daily newsletter send. Underneath that green status: 80,708 attempts, 80,708 errors, zero emails delivered. The vendor's status page never changed color.

That single finding reframes how marketing ops teams should evaluate email infrastructure. It surfaced because the bake-off ran at production volume (500,000+ emails across three vendors), not the usual few hundred test sends that pass for due diligence.

The Measurement Problem Most Bake-Offs Ignore

A typical vendor evaluation sends a few hundred emails, checks the vendor's own dashboard, and picks a winner based on who had the friendliest rep. The problem: vendor-reported "delivered" doesn't mean inbox placement. It means the receiving server accepted the message. Whether it landed in the inbox, a tab, spam, or vanished entirely is a different question.

Seed-list inbox placement testing catches provider-specific issues that "delivered" metrics miss. The method sends to monitored test mailboxes (Gmail, Microsoft 365, Yahoo) and checks whether each copy landed in inbox, tabs, spam, or wasn't found at all. But seed-list results are directional, not definitive. Seed accounts lack real engagement history, so results can vary between runs and don't guarantee real-world inbox placement.

A blended inbox placement score can hide provider-specific failures entirely. Reading results by provider matters more than any single number. And "missing" is its own category: treating it as spam, or ignoring it, leads to the wrong remediation and the wrong vendor choice.

What the SaaStr Bake-Off Actually Measured

The cleanest comparison was a matched split on SaaStr's AI Day campaign: two groups of about 2,300 people each, identical content, sent the same morning. The challenger got 50.5% unique opens versus the incumbent's 21.0%. Unique clicks: 12.2% versus 4.2%. Same audience, content, and send time. The gap points to how each vendor delivered the email, not what was in it.

Limits were stated upfront: one campaign, about 2,300 per side. Opens are noisy (Apple MPP, pre-fetching). Raw click counts included security scanners. But clicks moved in the same direction as opens, which made the gap harder to dismiss.

Then came the 80,708-send failure. The AI agent ran four diagnostic sends, changing one variable each time: sender identity, tracking, subscription settings, plain-text content. All four failed identically. That ruled out content, configuration, and list quality, putting the failure squarely on the vendor's transport layer.

The Scorecard That Caught the Real Differences

The agent refused to pick a winner on opens alone. Its seven-criteria scorecard: completed delivery confirmed at the step level, inbox placement, spam complaints, unsubscribes, human clicks (scanner traffic filtered out), sponsor reporting quality, and operator effort.

Operator effort turned out to be one of the biggest differentiators. How much human time does each vendor consume in consent reconciliation, audience prep, schedule checks, and reporting? Most bake-off spreadsheets never include it. That's an oversight when you're a three-person team (as SaaStr is) or when your ops team is already stretched across RevOps, CRM, and attribution work.

The migration plan reflected real deliverability discipline: ramp volume from 20K to 40K to full list, stepping up only if bounces, complaints, and unsubscribes hold. Confirm unsubscribe sync in both directions before scaling. Map every workflow that depends on the incumbent before canceling anything.

Why Provider-Level Monitoring Is Now Table Stakes

Gmail and Yahoo's bulk-sender requirements (SPF, DKIM, DMARC with at least p=none, one-click unsubscribe, spam rates below 0.3%) are gating factors for deliverability. But compliance alone doesn't guarantee inbox placement. Engagement, complaints, and sending reputation still drive filtering outcomes.

Broad benchmark data gives context: Validity's benchmark report shows Gmail inbox placement at about 89.8%, Microsoft at 77.4%, and Yahoo at 87.3%. Microsoft's spam placement rate sits around 15.1%, more than double Gmail's 6.4%. Those numbers aren't B2B SaaS-specific, though. They're directional context from broad datasets, not performance targets for your list. And Validity has flagged that Gmail inbox placement can shift quickly, reinforcing the need for continuous monitoring over one-time audits.

What This Changes for Your Next Vendor Evaluation

The hypothesis is straightforward: if you run vendor comparisons at production volume with provider-level inbox placement data (not blended delivery rates), you'll catch failures that small test sends and vendor dashboards hide. The SaaStr case showed a 2.4x open rate gap and an 80,000-send silent failure that would have gone undetected in a standard trial.

Whether an AI agent runs the comparison or your ops team does, the scorecard matters more than the automation. Score on step-level delivery confirmation, provider-specific placement, human engagement with scanner traffic stripped, and operator effort. Pair seed-list results with Gmail Postmaster Tools spam-rate monitoring to reduce false conclusions. Repeat the test; a single run is a signal, not a verdict.

SaaStr's team trusted the result because the agent queried the records underneath the dashboard labels. A green status page and a "Completed" tag are vendor claims. Step-level data is evidence. The distance between those two things was 80,708 emails that never left the building.