robots.txt

What is it?

The standard crawler-rules file at your website root (https://yoursite.com/robots.txt), governed by IETF RFC 9309. We enhance it two ways: explicit rules for modern AI user-agents (the crawlers behind AI training, AI search, and AI assistants), and the optional Cloudflare Content Signals directive that declares how your content may be used, for search, for AI input, or for AI training.

Why AI agents need it

robots.txt is where agents look first. It defines crawl rules and points to your sitemap, and it is the natural home for AI-era access preferences. A file written in 2015 for Googlebot says nothing about the dozens of AI crawlers that exist today, leaving them to guess, or to be blocked (or admitted) by accident.

The business value for you

You keep traditional crawler guidance and AI-era signals aligned in one familiar, standards-backed place. Content Signals let you be specific rather than binary: you can stay visible to AI search and assistants (ai-input) while declaring a preference against your content being used for model training (ai-train), or whatever combination fits your business.

How it came to be

The robots exclusion protocol dates to 1994 and became IETF RFC 9309 in 2022, a rare example of a community convention graduating to a full standard. Cloudflare introduced the Content Signals Policy in September 2025 as an addition to robots.txt, released under a CC0 license so any site on any host can use it.

Sources: IETF RFC 9309, Robots Exclusion Protocol • Cloudflare, Giving users choice with the Content Signals Policy (September 2025)

Adoption

robots.txt itself is nearly universal, Cloudflare's 2026 agent-readiness analysis found 78% of sites have one, but notes "the vast majority are written for traditional search engine crawlers, not AI agents." Content Signals adoption stands at 4% of sites and growing; Cloudflare reported serving managed robots.txt with Content Signals for over 3.8 million domains at launch. The gap between the two numbers is the opportunity: almost everyone has the file, almost no one has updated it for agents.

Sources: Cloudflare, Introducing the Agent Readiness score (2026) • Cloudflare, Content Signals Policy announcement

The technical reality

The risk in editing robots.txt is breaking what already works. You must combine classic directives with AI-specific user-agent rules, keep existing sitemap references and crawl settings intact, and avoid syntax conflicts, one overly broad Disallow can hide your site from crawlers you want. That is why we upsert: your existing rules, user-agent groups, and sitemap lines are preserved, and the AI-era additions are merged around them. We never overwrite your file.

One honesty note: Content Signals are advisory preference declarations, not technical enforcement, and Google has said its crawlers do not currently use the content-signal directive. We include Content Signals because they are the clearest published vocabulary for expressing AI-usage preferences at scale, not because every crawler obeys them today.

What happens if you skip it

Your content's AI-usage terms stay undeclared: AI crawlers apply their own defaults, which may mean training on content you'd rather protect, or skipping content you'd rather have represented. And agents reading a stale robots.txt get no signal that your site has thought about them at all.