How AI engines decide
what to cite.

There is no single "AI crawler," and blocking one is not blocking another

OpenAI runs four separate crawlers and they do different jobs. GPTBot gathers content for model training. OAI-SearchBot is the one that decides whether you can surface in ChatGPT's search results. ChatGPT-User fetches a page because a person asked, which means robots.txt may not govern it at all.

Google publishes its own list, and Bing publishes theirs.

The common mistake: blocking GPTBot to opt out of training, then wondering why you never appear in ChatGPT. Those are different bots. Read your own robots.txt and check which one you actually disallowed.

Named and cited are separate results, and the split is the finding

We baselined a real estate development firm in August. Ten buyer questions, one engine, every answer stored verbatim, the whole run repeated four times across two endpoints so a single lucky reply couldn't move the number.

Named in 2 of 10 answers. Cited as a source in 1 of 10. The split matters more than either number. Both branded questions hit. All five category questions missed. All three pricing questions missed.

Findable if you already know the name. Invisible if you don't. That is a different problem from "we need more content," and it needs a different fix.

What this means for you: a report that gives you one visibility score has averaged away the only useful distinction. Ask for the branded and unbranded numbers separately.

Structured data helps a machine parse you, and Google says what it does not do

Schema markup is worth adding. It tells a parser which text is a question, which is an answer, which is a price. Google's own introduction is clear about what it is for and what it is not.

It is a parsing aid. It is not a lever that makes a thin page authoritative. If the underlying page doesn't answer the question, marking it up as an answer changes nothing.

It has now been measured. Ahrefs compared 1,885 pages that added schema against 4,000 matched pages that did not (May 2026). Citations moved +2.2% on ChatGPT and +2.4% on AI Mode, both indistinguishable from zero. Google's AI optimization guide says structured data isn't required for its AI features.

The tell: anyone who sells schema markup as the AEO fix, without touching what the page says, is selling the cheap half.

Google publishes what governs its AI features, and it is mostly ordinary

Google's page on AI features is the closest thing to an official answer that exists. Read it before you buy anything described as a proprietary AEO methodology.

Most of what it asks for is unglamorous and already familiar. Be crawlable. Be useful. Be clear about who wrote it.

Worth noticing: there is no published ranking formula for AI answers. Anyone quoting one is quoting themselves.

Part of citation is decided inside the model, not on your website

Anthropic documents a citations feature where the model returns the specific source passages its answer rests on. That decision happens at answer time, over whatever documents were supplied.

You cannot optimize your way into a mechanism running on someone else's servers. What you can do is be the source that is easy to quote: a clear claim, in one place, attributable to a named author.

The useful consequence: write passages that survive being lifted out of context, because that is exactly what happens to them.

One ask proves nothing, because an answer is a distribution

Ask the same question twice and you can get two different answers. Any measurement built on a single query is noise reported as a finding.

Ask each question several times. Record every answer verbatim. Treat a change as real only when it survives repetition. Our baseline above was run four times across two endpoints before a single number went in a document.

We measured how much it moves: across 720 answers, 1 in 3 companies named showed up in half the answers or fewer, and a single check missed those companies 3 times out of 4.

Ask your vendor: how many times did you run each query, and can I see the raw answers? If the answer is once, and no, the number is decoration.

Only Google's crawler is documented to run JavaScript

Google says it can process content inside JavaScript as long as it isn't blocked. The others are a different story. Vercel and MERJ tracked the AI crawlers and found none of OpenAI's, Anthropic's or Perplexity's run JavaScript. Their own crawler pages don't say otherwise.

So text that only appears after a script runs, like reviews, prices or FAQs loaded by a widget, can be invisible to ChatGPT, Claude and Perplexity while Google sees it fine.

The test: view your page source, not the rendered page. If the answer a buyer needs is not in that raw HTML, three of the four big engines cannot quote it.

Google says it ignores llms.txt

The llms.txt file gets sold as the AI version of a sitemap. Google's July 2026 guide says it is fine to have one, and that it "will neither harm nor help" visibility, "as Google Search ignores them." OpenAI, Anthropic and Perplexity don't document using it either.

The tell: an llms.txt file as a paid line item. It costs nothing to add and nothing documented reads it.

Bing and Google now report your AI citations directly

Until this year every AI visibility number came from someone asking the engines and counting. Now two engines report it themselves. Bing Webmaster Tools added AI Performance in February 2026: how often Copilot cited your site, which pages, and a sample of the questions behind it. Google added a generative AI report to Search Console in June 2026, as reported by PPC Land, with impressions for AI Overviews and AI Mode combined, rolling out to some sites first.

Both show impressions and citations, not clicks. Neither covers ChatGPT. But they are the first numbers that come from the engines rather than a tracker.

Ask your vendor: are you pulling Bing's AI Performance report and Google's generative AI report every month? Both are free.