Why Was llms.txt Proposed?
When a software developer uses an AI agent like Cursor to query an API library, or asks Perplexity to summarize a product, the AI crawler faces three major engineering obstacles:
- Token Waste: Standard web pages are packed with navigation bars, cookie banners, tracking scripts, and stylesheets that consume thousands of tokens before reaching the actual article text.
- Dynamic Rendering Failures: Many AI scrapers do not execute full headless browser JavaScript, causing Single Page Applications (SPAs) to return blank HTML.
- Information Fragmentation: Important technical definitions are frequently split across dozens of nested subpages, exhausting the agent's context window during multi-hop retrieval.
Proposed by Jeremy Howard (founder of fast.ai), llms.txt solves this by providing a single, standardized, token-efficient Markdown file pointing directly to clean text resources.
How llms.txt Compares to robots.txt and sitemap.xml
It is critical not to confuse llms.txt with existing search infrastructure. Each serves a distinct layer of the web ecosystem:
| Protocol | Target Consumer | Primary Purpose | Standard Status |
|---|---|---|---|
| robots.txt | All automated web crawlers | Access control & crawl permissions (Allow / Disallow) | Official IETF RFC 9309 |
| sitemap.xml | Search engine bots (Googlebot, Bing) | Exhaustive inventory of canonical URLs & revision dates | Universal Search Standard |
| llms.txt | LLMs, coding assistants, AI search agents | Token-compact Markdown index of core knowledge & documentation | Community Proposal (Draft) |
The Official Syntax of an llms.txt File
An llms.txt file is written in standard GitHub-flavored Markdown. It consists of four structural sections:
# Project or Website Name
> A concise blockquote summary (under 40 words) describing the primary purpose of the platform.
## Core Documentation
- https://example.com/docs/api : Complete REST API endpoint reference and authentication.
- https://example.com/docs/quickstart : 5-minute onboarding guide for developers.
## Optional / Advanced Sections
- https://example.com/docs/architecture : Deep dive into system internals and benchmarks.Syntax Rules to Follow:
- H1 Title: Exactly one top-level
#heading defining the project or organization. - Blockquote Description: A single
>paragraph summarizing the site for an LLM system prompt. - H2 Sections: Logical groupings (e.g.
## Top Pages,## API Guides). - Bulleted Links: List items in the format
- URL : Description. The description explains the context so the AI agent decides whether fetching the sub-document is necessary.
Real-World Example: Inspecting dontbeac.app/llms.txt
On this publication, we maintain a live llms.txt at dontbeac.app/llms.txt. Here is the exact production file our Next.js edge server delivers to AI crawlers:
# dontbeac.app — AI Tools, Guides & Tech
> Practical, authoritative guides on frontier AI models (Gemini 4 Argon, Claude Opus 5.5, GPT-6 Astra), Google Antigravity, and startup credit programs. Updated weekly.
## Top Pages
- https://dontbeac.app/blogs/what-is-gemini-4-argon : Google DeepMind's Gemini 4 Argon explained — 2M context, Antigravity integration, benchmarks.
- https://dontbeac.app/blogs/how-to-use-gemini-4-argon-in-antigravity : Guide to configuring Gemini 4 Argon inside Google Antigravity IDE.
- https://dontbeac.app/blogs/claude-startups-program-eligibility-application-credits : Claude for Startups — how to get $1,000 to $100,000 in free API credits.
- https://dontbeac.app/about : About the publication, founder identity, and technical scope.Separating Facts from SEO Hype: What llms.txt Does NOT Do
With the rise of Generative Engine Optimization (GEO), several marketing agencies have made exaggerated claims about llms.txt. Let us review the empirical facts:
Reality: Googlebot does not use llms.txt to rank web pages. Google AI Overviews extract citations directly from standard indexed HTML pages that satisfy user intent and demonstrate high information gain.
Reality: Tools like Cursor, Windsurf, and Claude Code actively fetch llms.txt when developers prompt @docs https://your-site.com. Providing an llms.txt ensures the agent retrieves exact syntax rather than broken HTML scraps.
How to Implement llms.txt in Next.js (App Router)
In modern Next.js applications, adding llms.txt requires zero backend configuration:
- Navigate to your project's
public/folder. - Create a new plain text file named
llms.txt. - Populate it with your curated markdown URLs and descriptions.
- Deploy to Vercel or your hosting platform. Next.js automatically serves it at
https://yourdomain.com/llms.txtwithContent-Type: text/plain.
Summary & Strategic Recommendation
Implementing an llms.txt file costs virtually nothing (under 15 minutes of work) and genuinely assists agentic coding tools in parsing your documentation. However, it should never distract engineering teams from foundational technical SEO: fast server-rendered HTML, valid canonical tags, descriptive titles, and authoritative, original research.