Why software company SerpApi decided to use llms.txt, and their results
SerpApi, a search-API company, published a clear argument we share: llms.txt helps AI agents understand a website faster and more accurately once they arrive. They treat it as agent infrastructure, separate from SEO ranking hacks.
That is the value we want every AgentReadyWebsite customer to grasp. llms.txt is one of typically 13 infrastructure files we install and refresh monthly for you.
Source: SerpApi, Why We Added llms.txt to SerpApi (Despite the Controversy)
Our value in one sentence
When an AI agent visits your website, agent-readable files make that agent more productive: it understands what you offer faster, with less wasted context, and with a higher chance of getting the facts right, so the agent gives accurate information about your business to potential customers.
Agent productivity on arrival
SerpApi’s framing starts with a developer scenario: when someone tells a coding agent “use SerpApi to search Google,” the agent must figure out what SerpApi is, find the right endpoint, learn the parameters, and ideally see a working example. That is a documentation problem.
The same pattern applies to almost any business an agent is asked to evaluate. When a customer tells their AI agent, “find me a company that sells X,” the agent needs to figure out what your website offers, find the right solution or service page, understand the offering, and ideally see proof (pricing, case studies, how it works). That is a business-comprehension problem.
SerpApi measured whether llms.txt helps agents understand their documentation once they arrive. Their answer was yes. Citation counts in Google AI Overviews sit on a different surface.
How agents actually read a website
Humans scan a page visually. Agents usually do something closer to a fetch: they request a URL, receive a response, and try to extract meaning inside a limited context window.
SerpApi quotes a blunt description from DerivateX’s guide: when an AI tool retrieves a webpage, it does not read it the way a human does. It receives raw HTML filled with navigation menus, cookie banners, JavaScript bundles, advertising scripts, and footer links. For a system working inside a fixed context window, that structural noise competes directly with the content that matters.
DerivateX cites roughly a 10× token reduction when an agent reads Markdown instead of parsing HTML. SerpApi then measured their own documentation pages, and found the gap can be much larger.
As cited by SerpApi: DerivateX’s LLMs.txt Guide: What It Does and Doesn’t Do (2026)
Bouncer vs table of contents
robots.txt tells crawlers what not to access. llms.txt tells agents what is available and where to find it. One is a bouncer; the other is a table of contents.
Jeremy Howard’s proposal (Answer.AI / llmstxt.org) exists because context windows are too small to swallow most websites whole. A curated Markdown index at /llms.txt gives agents a starting map instead of forcing them to reconstruct your business from noisy HTML.
SerpApi’s measured results
SerpApi compared the raw size of their documentation pages that agents can read as Markdown versus the same pages as standard HTML. These are their numbers for their docs surface, not a promise that every site will see the same multiples. They are still the clearest public measurement we have of why agent-readable formats matter.
| Page | Markdown | HTML | Reduction |
|---|---|---|---|
| Google Search API | 19.6 KB | 354.9 KB | 18× |
| Google AI Mode API | 7.7 KB | 358.0 KB | 46× |
| Google AI Overview API | 3.2 KB | 224.1 KB | 70× |
| Integrations | 2.3 KB | 63.3 KB | 28× |
| Pricing | 8.4 KB | 143.8 KB | 17× |
| Google Shopping API | 8.3 KB | 225.5 KB | 27× |
| Amazon Search API | 24.9 KB | 275.6 KB | 11× |
| Google Maps API | 12.9 KB | 240.8 KB | 19× |
Across their markdown-enabled pages, SerpApi reports an average of about 32× smaller in Markdown. Even their worst case in the sample (Amazon Search API) was still 11×.
In token terms, they translate one example like this: an agent reading their Google AI Mode HTML page burns roughly 79,000 tokens on content that is not relevant, while the Markdown version lands around 1,800 tokens of usable content.
Their llms.txt index itself is under 16 KB, a map of 60+ endpoints, guides, and pricing smaller than many hero images.
Table and token figures: SerpApi, Why We Added llms.txt (Despite the Controversy)
What that looks like for an agent
Without a clean agent entry point, the agent often fetches a marketing or docs HTML page and inherits everything humans tolerate in a browser: a heavy <head>, scripts, font preloads, a sidebar with hundreds of links, interactive widgets, and a footer. Somewhere in the middle sits the actual explanation of what you sell. By the time the agent has extracted the useful paragraphs, a large share of its context window may already be full, and the examples, pricing details, or caveats at the bottom can get truncated.
With llms.txt (and other agent-readable files), the same agent can start from a curated table of contents: who you are, what you offer, and which pages carry the signal. It chooses the right next URL instead of guessing from noisy HTML. Less noise means more room for accurate parameters, offers, and constraints, which is what “higher accuracy” means in practice.
AgentReadyWebsite focuses on the discovery and comprehension layer for your existing site: a standards-aligned package that starts with foundational llms.txt and the companion files agents need to interpret your business. We do not ship SerpApi-style per-page .md documentation endpoints today.
Low cost, low risk, real upside
SerpApi’s conclusion matches our view of agent-readiness: they are not pretending llms.txt boosts AI Overview rankings. The cost of publishing a well-formed file is low. It does not interfere with ordinary SEO crawling or indexing. The upside is real when an agent tasked with understanding a product lands on the domain and can map the offer quickly.
They also note they are not alone: Yoast added llms.txt generation with the argument that the file is for AI assistants trying to give accurate answers based on your content, and Drupal’s LLM Support recipe provides full spec coverage. The ecosystem is building tooling because this solves a content-delivery problem for AI systems, not because it is an SEO ranking silver bullet.
Plugin-generated files can be a start; many are still thin URL lists that miss the strict Howard format agents (and Google Chrome’s Lighthouse Agentic Browsing audit) expect. That is why AgentReadyWebsite treats llms.txt as foundational and installs it inside a fuller package of agent protocol files with monthly refreshes. Learn more on our llms.txt explainer.
What this means for AgentReadyWebsite customers
SerpApi’s llms.txt case study is unique, although your website is not like their API documentation website, the agent problem is the same: noise, limited context, and the need for a curated answer key.
- Faster comprehension, agents get a table of contents for your business instead of reverse-engineering it from menus and scripts.
- Higher accuracy, more of the context window stays on services, pricing, and proof instead of cookie banners.
- Managed, not one-and-done, we generate a standards-aligned package (typically 13 files), deliver it ready to install, and refresh it monthly as your site and the standards change.
That is the productivity outcome: agents that visit your site waste less effort and leave with a clearer understanding of what you sell.
See how agent-ready your site is today
Run the free Pre-Check for your Agent Readiness Score, then read SerpApi’s full write-up if you want every measurement detail.