Optimising Your Content for AI Discovery

For the past decade, being found online meant ranking in Google. That is no longer exclusively true. A growing share of information queries now go to AI assistants - ChatGPT, Claude, Perplexity, Gemini - and these systems surface information differently from search engines. They retrieve, synthesise, and cite. The rules that determine what gets cited are different from the rules that determine what ranks in search.

This guide covers what is currently known about AI-native content optimisation, what tools are available, and how to measure whether it is working.

How AI systems retrieve and cite content

AI language models do not browse the web in real time (unless using a web search tool). They have been trained on large text corpora, and their "knowledge" is baked into model weights. However, AI search products - Perplexity, ChatGPT with Search, Claude with web search enabled - do retrieve and synthesise current web content in response to queries.

The retrieval step works similarly to a search engine: crawlers index pages, and a retrieval system selects relevant content. But the synthesis step is different: the AI reads the retrieved content and generates a response that may or may not attribute the source. What gets cited depends on:

llms.txt - the emerging standard

llms.txt is a plain-text file placed at the root of a domain (e.g., yoursite.com/llms.txt) that provides a structured description of the site for AI crawlers and language models. It is analogous to robots.txt but oriented toward AI understanding rather than crawl access.

A well-formed llms.txt file includes:

robots.txt for AI crawlers

Major AI crawlers respect robots.txt directives. If you want AI systems to index your content - and you should - your robots.txt must explicitly permit the relevant user agents:

Structured data (Schema.org)

Schema.org structured data in JSON-LD format helps AI systems and search engines understand the type of content on a page. For a research and standards organisation, the most useful schemas are:

Writing for AI retrieval: answer-first content

AI systems prefer content that answers questions directly and immediately. Traditional SEO writing - which builds to a conclusion - is less effective for AI retrieval than answer-first writing. The structure that works best:

Measuring AI discoverability

Traditional SEO has Google Search Console. AI discoverability measurement is less mature but workable: