Your Pages Look Great to Buyers and Invisible to AI Crawlers: A Three-Layer Fix

OpenAI runs at least three distinct crawlers, and blocking one doesn't block the others. GPTBot handles training data collection. OAI-SearchBot powers ChatGPT search results. ChatGPT-User fetches pages on demand when a user asks a question. A single robots.txt rule aimed at GPTBot leaves the other two untouched, and most B2B SaaS sites treat all three as one thing. That confusion is where crawl-readiness problems start.

Crawl Eligibility Is Not One Setting

In an opt-in sample of 47 WordPress sites covering 12.8 million bot and human visits between November 2025 and May 2026, GPTBot accounted for 40.7% of AI-crawler visits, with Bytespider at 22.1% and ClaudeBot at 13.7%. A separate dataset from Trakkr, covering 84 brands and 575,788 AI-crawler visits from June 2025 through February 2026, put GPTBot at 57.2%. The numbers don't agree because the samples, time windows, and bot definitions differ. That's the point: your logs are the only source that matters for your site.

Robots.txt is a published directive, not a technical barrier. A compliant crawler reads it and obeys; a non-compliant one ignores it. If you have content that genuinely needs protection (pricing models, proprietary data, gated research), it belongs behind authentication or WAF rules, not a polite request in a text file.

The operational move: build a per-user-agent policy. Decide separately whether GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Bytespider, and PerplexityBot should access each URL path. Validate compliance in your server or CDN logs over a two-week window. Cloudflare maintains an AI Crawl Control bot reference that reflects growing standardization at the CDN layer. If your CDN offers these controls, use them. If it doesn't, that's a procurement conversation worth having.

Chunk Extractability: Self-Contained or Useless

"Chunkability" isn't about hitting a magic word count. A paragraph that says "as shown in the table above" is useless to a retrieval system that never saw the table. A claim that says "revenue grew 34%" without a date, denominator, or source link is a liability once it leaves the page.

The practical heuristic, drawn from Hygraph's content-structure guidance: direct answer first, then key conditions and evidence, then explanation and background. Each section should be self-contained and extractable. Keep qualifiers, dates, units, and source links inside the same logical block as the claim they support. When a retrieval system grabs a chunk, everything needed to evaluate accuracy travels with it.

For CMS templates: give answer blocks stable HTML selectors. Use meaningful heading hierarchy so sections have clear boundaries. Render the answer in server-side or static HTML rather than requiring JavaScript execution. A crawler that fetches your page and gets a nav shell, a loading spinner, and no content has technically "crawled" you and found nothing worth extracting.

Citation Readiness Is the Part Everyone Skips

A fetched page isn't an indexed page. An indexed page isn't a retrieved page. A retrieved page isn't a cited page. Four distinct stages, and most teams collapse them into one dashboard metric that tells them almost nothing.

Google's AI-features guidance emphasizes following standard search fundamentals and indicates that special AI markup or machine-readable files are not required for Google's AI features. Useful context, but it applies to Google. Other AI systems have their own retrieval logic, and none of them publish a guaranteed citation algorithm.

What you can control: make your claims verifiable. Attach the source, the date, and the denominator to every factual statement. Link to the canonical URL you want cited. No checklist can guarantee an AI system will cite you. The defensible baseline is accessible pages, useful original content, clear structure, and verifiable claims.

Run It This Week

Setup: Pick five money pages (pricing, product, key landing pages). Pull server or CDN logs for the past 14 days filtered to known AI crawler user-agents.

Crawl audit: For each page, record which AI crawlers hit it, what response code they got, and whether the robots.txt rule matches your intended policy. Flag any page where the policy says "allow" but logs show zero fetches, or where the policy says "block" but fetches appear anyway.

Chunk test: Fetch each page with a non-rendering HTTP client (curl, wget). Compare what you get to what the browser shows. If the answer block requires JavaScript, deliver the essential answer in server-rendered HTML with a stable selector.

Citation check: For each page's primary claim, verify that the date, source, and denominator sit in the same HTML section. If a qualification lives in a tooltip, an accordion, or a different page, move it adjacent to the claim.

Ownership: Assign each failure to a named person. Crawl-access fixes go to the technical SEO lead. Extraction fixes go to content engineering. Any failure without an owner and a fix ticket blocks the "AI-ready" label on that URL.

The hypothesis: if we fix server-rendered answer blocks on five money pages and verify crawler access per user-agent, then the share of AI-crawler fetches returning complete answer content will increase from baseline (measured in logs) because the retrieval system currently receives incomplete or empty HTML. Success = 100% of target pages returning answer content to non-rendering fetches. Guardrail = no increase in blocked-crawler errors for user-agents we intend to allow. Stop-loss = if fixing rendering breaks existing Googlebot indexing, revert and diagnose before proceeding.

You won't know whether citations increase until you build separate tracking for that stage. But a page that can't be fetched and extracted can't be cited, period. Fix the retrieval contract first. Measure citation impact after.