The Framework That Makes Your Documentation Board-Ready for AI
Nine days ago, Addy Osmani, Director of Engineering at Google Cloud AI, published something that should have landed in every CMO's pipeline review: a framework called Agentic Engine Optimization. The SEO community noticed. The developer relations teams noticed. But the executive teams who actually own the budget for documentation, content operations, and digital infrastructure? Most of them missed it entirely.
That's a problem, because AEO isn't a technical curiosity. It's a revenue signal.
Here's the math that should get your CFO's attention: Adobe Digital Insights' April 2026 data shows AI-referred retail traffic now converts 42% better than non-AI traffic – an 80-point swing from a year earlier. If your documentation, product pages, and technical content can't be consumed by AI coding agents, you're not just invisible to a new channel. You're invisible to the channel that converts best.
What Osmani Actually Published (And What X Got Wrong)
The framework addresses a specific problem: AI coding agents – Claude Code, Cursor, Cline, Aider, Codex CLI, Gemini CLI – consume documentation fundamentally differently from humans. They issue a single HTTP request, strip HTML, count tokens, and either use your content as context or silently discard it. Your analytics recorded nothing useful. Scroll depth was zero. Time-on-page was 400 milliseconds.
But the agent was absolutely there. And depending on how your docs were structured, it either completed the task successfully or hallucinated a solution because the content was too token-heavy, poorly structured, or blocked by a misconfigured robots.txt.
Popular X posts reduced Osmani's token guidance to "keep docs under 15k tokens." That's the quick-start number only. The real framework is tiered by page purpose: 15k for quick-starts, 20k for conceptual guides, 25k for API reference per endpoint, and a 30k hard ceiling for single pages. The llms.txt discovery file itself should stay under 5k tokens.
Why does this matter when frontier models now offer 1M+ context windows? Three reasons that hit your P&L directly. First, cost economics: GPT-5.4 charges a 2× input premium above its 272k standard window. A single oversized page can flip an agent workflow from cheap to expensive inference. Second, RAG-chunk efficiency: most production agents don't dump whole pages into context – they chunk, retrieve, and compose. Third, agent branching: a Claude Code session might consume documentation from 5–10 different services in parallel. Even at 1M capacity, ten 30k-token pages burn 30% of the window before the agent has written a single line of code.
The Five Signals That Determine Whether Agents Use Your Content
Osmani's framework organizes everything around five auditable signals:
Discoverability. Can agents find your documentation without rendering JavaScript? This means server-side rendered HTML, robots.txt that doesn't block AI crawlers, and an llms.txt file at the domain root listing content in a structured way.
Parsability. Is the content machine-readable without visual layout interpretation? Semantic HTML, heading hierarchy that matches logical structure, fenced code blocks, actual tables instead of visual approximations.
Token Efficiency. Does the content fit within agent context windows without truncation? Per-page token counts tracked as documentation metadata, budgets enforced by page type, markdown-first to minimize tokenization overhead.
Capability Signaling. Does the documentation tell agents what your API does, not just the syntax? This is where skill.md files come in – declaring what a service does, its inputs, constraints, and safe-usage boundaries.
Access Control. Does your robots.txt actually let AI traffic through? Not a wall, but a known surface. Block agents from checkout and account pages; allow them on public documentation and marketing.
Search Engine Land covered the framework four days after publication, noting that Osmani released an open-source audit tool called agentic-seo that scores a 100-point audit against these criteria and outputs JSON for CI/CD.
The Adoption Landscape (And the Mueller Counter-Position)
The llms.txt standard, proposed by Jeremy Howard at Answer.AI in September 2024, now has real adoption. Verified publishers include Vercel, Stripe, Shopify, GitHub, Anthropic, and OpenAI. Cloudflare publishes per-product but returns 404 at the root apex. HubSpot hasn't published yet.
Here's the governance wrinkle: Google's John Mueller has stated publicly that Google does not use llms.txt for ranking and has called the per-page markdown-twin practice "a stupid idea." This is the official current position from Google Search.
But that's the wrong frame. AEO isn't about Google Search rankings. It's about whether AI agents – including Google's own Gemini CLI – can successfully use your content to complete tasks on behalf of users. Google added Google-Agent to its official list of user-triggered fetchers on March 20, 2026, formalizing an identity for AI-powered systems that browse the web on behalf of users. This isn't another Googlebot variant. It appears in HTTP requests when an AI agent visits a site to complete a user-initiated task.
The distinction matters for your board deck: crawlers read your site; agents use it on behalf of humans.
The Pilot Plan: What to Ship in the Next 30 Days
If I were presenting this to a CFO, here's the 2–3 week pilot I'd propose:
Week 1: Audit and baseline. Run the agentic-seo CLI against your top 20 documentation pages. Filter server logs for AI agent user-agents (axios/1.8.4, curl/8.4.0, got, colly). Establish baseline token counts per page type.
Week 2: Quick wins. Publish llms.txt at domain root with descriptions that tell agents what they'll find, not just page titles. Audit robots.txt for unintended blocks on AI crawlers. Front-load answers within the first 500 tokens of your highest-traffic pages.
Week 3: Measurement framework. Segment AI agent traffic in analytics. Track token counts as documentation metadata. Establish correlation between AEO audit scores and AI-referred conversion rates.
Risks and mitigations. The Mueller counter-position creates internal confusion – mitigate by framing AEO as agent-readiness, not SEO. Token budgets require content rewrites – mitigate by starting with quick-start pages only. llms.txt maintenance adds operational overhead – mitigate by auto-generating from existing sitemap with descriptions.
The Revenue Equation
The World Economic Forum's January 2026 analysis framed this shift clearly: "Search engine optimization is no longer the name of the game – it's now agentic engine optimization." The brands that win this shift will be those whose information is easy for AI models to find, trust, and recommend.
The 42% conversion lift from AI-referred traffic isn't theoretical. It's measurable in your attribution data today – if you're tracking it. The question for your next pipeline review isn't whether to invest in AEO. It's whether you can afford to let your documentation remain invisible to the channel that converts best.
Model or it didn't happen. The framework is published. The audit tool is open-source. The revenue signal is clear. What's your 30-day pilot?
