Structuring /llms.txt for AI search ingestion.
A developer guide to creating clean, token-efficient Markdown indexes so ChatGPT, Claude, and Perplexity can parse your platform without HTML overhead.
What is the /llms.txt standard?
Traditional search crawlers parse complex HTML DOM trees, JavaScript bundles, and CSS stylesheets. When conversational AI models (like GPTBot, ClaudeBot, and PerplexityBot) search the web in real time, full HTML pages consume unnecessary context window tokens and introduce parsing noise.
The /llms.txt standard is a proposed machine-readable publishing format (hosted at yourdomain.com/llms.txt) that provides AI assistants with a structured, curated markdown map of your core product documentation, APIs, and use cases.
Standard File Structure (llmstxt.org Spec)
A valid /llms.txt file is served as plain text (text/markdown; charset=utf-8) at your root domain:
# [Your Product Name] > A concise 2-3 sentence overview of what your SaaS does, its target audience, and primary workflow benefits. ## Core Documentation - [Quickstart Guide](/docs/quickstart.md): Essential onboarding, API key generation, and basic setup. - [API Reference](/docs/api-reference.md): Endpoint parameters, request/response JSON schemas, and error codes. - [Features & Architecture](/features.md): Overview of product modules, capabilities, and integrations. ## Secondary Resources - [FAQ & Pricing](/pricing.md): Plan tiers, limits, and common technical questions. - [Full Documentation Export](/llms-full.txt): Single-file combined Markdown archive for deep RAG vectorization.
Three Technical Implementation Rules
Direct 200 OK (No Redirects)
Serve the file directly from your root domain with a 200 HTTP status code. Avoid 301/302 redirect loops.
Clean Markdown Files
Ensure every link in your llms.txt points to raw, clean Markdown files stripped of navigation bars, headers, and footers.
robots.txt Alignment
Ensure your robots.txt explicitly allows GPTBot, ClaudeBot, and PerplexityBot to read your /llms.txt directory.
Verify Your /llms.txt & AI Crawler Readiness
Run a free diagnostic scan to check your /llms.txt status, robots.txt permissions, and live brand citations across ChatGPT, Perplexity, and Claude.