llms.txt

What is llms.txt?

A plain-text Markdown file placed at the root of your website (https://yoursite.com/llms.txt). It gives AI tools and agents that choose to read it a clear, curated overview of your site: who you are, what you offer, and which pages matter most. Where robots.txt tells crawlers what not to access, llms.txt tells agents what is available and where to start.

Why AI agents need it

Agents parsing raw HTML waste context on navigation, cookie banners, JavaScript, sidebars, and footer links. A curated llms.txt gives them a clean starting point, a machine-readable summary designed for exactly this use. Every other file in the agent-readiness package builds on the assumption that an agent can find an answer key first.

The business value for you

When an agent, a customer-support bot, a coding assistant, or a research agent working for your customer, lands on your website, clean Markdown can be the difference between a useful answer about your business and wasted context. For example:

• In Ramp’s production experiment across roughly 50 marketing pages, Markdown was the only tested format that reliably surfaced in LLM responses. Read the walkthrough: Ramp’s agent-marketing case study.
• SerpApi found Markdown helped AI agents understand their documentation faster and more accurately once they arrive, with far less wasted context than the same pages as HTML. Read the walkthrough: SerpApi’s llms.txt case study.

Sources: Ramp Builders, Marketing to AI agents • SerpApi, Why We Added llms.txt to SerpApi (Despite the Controversy)

How it came to be

Jeremy Howard of Answer.AI proposed llms.txt in September 2024, with the format documented at llmstxt.org. His premise: "Site authors know best, and can provide a list of content that an LLM should use." In May 2026 the convention gained browser-vendor recognition: Google Chrome's Lighthouse added an llms.txt audit under a new experimental "Agentic Browsing" category. Absence is currently treated as not applicable (it does not fail the audit); the audit still shows Google's developer tooling recognizing the file for agents.

Sources: Answer.AI, The /llms.txt file proposal (September 2024) • Google Chrome for Developers, Lighthouse llms.txt audit (May 2026)

Adoption

Adoption is early but accelerating fast. Thunderbit's May 2026 crawl found 5.86% of the Tranco Top 10,000 websites publish a valid llms.txt, up from roughly 0.3% of top sites measured a year earlier. Originality.ai's year-long tracking study counted 36,120 sites with the file by May 2026, an 8.8x increase in twelve months. The same tracking wave also reported that most published files still see little or no AI-bot traffic, another reason we do not promise citations or referrals from llms.txt alone. For context on how standards mature: a ProGEO.ai study found 7.4% of Fortune 500 companies had llms.txt as of March 2026, while robots.txt, which took nearly three decades to reach RFC status, is at 92.8%. Early movers still get the readiness layer in place before the category is crowded; that is infrastructure timing, not a traffic guarantee.

Sources: Thunderbit 2026 • PPC Land (July 2026) covering Originality.ai adoption counts, Ahrefs May 2026 request-log findings (including the widely cited 97% zero-request rate), and ProGEO.ai's Fortune 500 study

The technical reality

Most hand-rolled files fail the strict format the Howard proposal defines: a single # heading with the site name, a > blockquote summary, named ## link-list sections, and an ## Optional section at the end. Getting there requires analyzing your site, extracting accurate business language, writing clear natural-language descriptions, and keeping the file synchronized as your website changes.

A useful file is a curated index, not a sitemap dump. Prefer product or service pages, pillar guides, case studies, docs, and pricing pages that would still explain your business if an agent read only those URLs. Skip tag archives, login and checkout flows, thin stubs, and near-duplicates. Link labels should describe the destination, vague “click here” or “read more” text gives an agent nothing to navigate on.

Four patterns that consistently produce weak files: (1) listing every URL with no descriptions, (2) vague link labels, (3) letting the file go stale so it points at moved or deleted pages, and (4) shipping an oversized community llms-full.txt that crowds the context window instead of helping. Our generator emits the strict Howard format with a self-declared standards-conformance section, and the managed subscription refreshes it monthly so the index stays aligned with your site. The free Pre-Check tells you whether your existing file passes. We ship Howard-spec llms.txt and deliberately skip community variants like llms-full.txt, which are outside the formal llmstxt.org proposal.

Further reading: DerivateX, LLMs.txt Guide: What It Does and Doesn’t Do (2026)

What happens if you skip it

Agents that look for an answer key don't find one, so they reconstruct your business from raw HTML, incompletely, or not at all. Google Chrome's Lighthouse Agentic Browsing category includes an audit for llms.txt; today a missing file is treated as not applicable rather than a failing check, and the convention still appears in Google's own developer tooling. One honest boundary: Google Search has said llms.txt is not a ranking signal and is not required for AI Overviews or AI Mode. We sell this file for the agentic-browsing surface. We leave Google Search placement and guaranteed AI traffic unpromised.