Cloudflare's 2026 AI bot rules, explained for data teams

Cloudflare sits in front of a large share of the web (in our own scan, 24.5% of responding top-1,000 homepages were served through it), so its defaults decide whether your crawler, scraper or browser agent gets a page, a 403, a challenge or a bill. In sixteen months it has changed those defaults twice and added three new mechanisms. This guide puts them in order, explains what each one does at the HTTP level, and shows what we measured across the Tranco top 1,000 domains on 6 November 2026.

It describes technical and operational behavior. It is not legal advice.

The timeline

July 2025: the default block and Pay Per Crawl

On 1 July 2025 Cloudflare said it had become the first infrastructure provider to block AI crawlers “accessing content without permission or compensation, by default.” Every new domain is asked whether it wants to allow AI crawlers and starts from “the default of control.” Existing domains kept their settings.

Pay Per Crawl launched the same day in private beta. A site sets one domain-wide price and chooses, per crawler, to allow, charge or block. On a paid URL:

1. A crawler that asks without payment intent gets HTTP 402 Payment Required and a crawler-price header.
2. To accept, the crawler retries with a crawler-exact-price header.
3. Crawlers must register with Cloudflare and sign every request with Web Bot Auth (signature-agent, signature-input and signature headers).

Some trade press reports that Pay Per Crawl evolved into a “pay per use” model in July 2026, according to ppc.land. Cloudflare’s own July 2026 post doesn’t describe that change, so treat it as unconfirmed.

Web Bot Auth: signed requests

Web Bot Auth builds on HTTP Message Signatures (RFC 9421). The bot holds a key pair and publishes its public keys in a directory. Each request carries Signature and Signature-Input headers, plus a Signature-Agent header that tells the site where to find the keys. A site can then verify that a request really comes from the operator it claims, rather than trusting a user-agent string anyone can copy. Browserbase was an early partner, according to CXO Digital Pulse.

The practical point: identity is moving from “what you say you are” to “what you can prove you are.” Paid access already requires a signature.

Content Signals: preferences in robots.txt

The Content Signals Policy (24 September 2025) adds three yes/no preferences to robots.txt:

• search: building a search index and returning links and short excerpts (not AI-generated summaries)
• ai-input: feeding content into AI models at answer time, for example retrieval
• ai-train: training or fine-tuning models

A typical line reads Content-Signal: search=yes, ai-train=no. A signal that’s left out expresses no preference. Cloudflare says its managed robots.txt feature is enabled on more than 3.8 million domains. Signals are preferences, not blocks. Enforcement comes from bot rules.

July 2026: Search, Agent and Training

The second “Content Independence Day” post (1 July 2026) sorts automated traffic by behavior:

Three details matter for data teams:

• New defaults from 15 September 2026. For new domains, Training and Agent bots are blocked by default on pages that display ads. The reasoning is that ads mean the page exists for human attention. Search stays allowed.
• Multi-purpose crawlers are judged by all their behaviors. The post says Googlebot, Applebot and BingBot would be blocked for customers who block Training.
• Verified no longer means allowed. Verified bots used to pass by default. Now Verified status only makes a bot allowable within its category. A bot that abuses content-use signals loses it, and bots that reproduce content in full can’t be Verified.

Cloudflare is also testing an optional use field for robots.txt: use=immediate (interact, store nothing), use=reference (index, excerpt, link back; the default) and use=full (summarize and reproduce). Managed robots.txt adds use=reference.

What this means operationally

Your browser agent is a bot, by definition. If you point ChatGPT agent, Claude in Chrome or a Browser Use script at a new, ad-supported Cloudflare site, the default is now to block it. Expect jobs that worked in August to start returning challenges.

Plan for several access lanes, not one. For any source, the realistic outcomes are now open, blocked, paid (402) or signed-only. A pipeline needs to know which lane each source is in and record it.

Identify honestly. A declared user agent, a contact URL and, where it applies, signed requests are how you stay in the “allowable” bucket. Spoofing a browser to dodge a block gets operators de-listed, as Perplexity was in August 2025, and it makes your pipeline fragile, because it breaks whenever detection improves.

Read the signals before you build. Check robots.txt per bot, Content Signals and any 402 pricing during feasibility, not after launch. Some sources will be off the table, and you want to know that before you write the extractor.

Monitor access, not just data. A rising rate of 403s, cf-mitigated: challenge headers or 402s is an early warning. Track it next to row counts and freshness.

Budget for paid access. Where a site charges per crawl, that’s a real per-page cost in your unit economics.

What we measured: the AI Crawler Access Index

On 6 November 2026 we fetched robots.txt and made one homepage request for each of the Tranco top 1,000 domains. 510 domains served a valid robots.txt and 689 homepages responded. Full method and downloadable rows are in the AI Crawler Access Index. “Blocked” below means robots.txt disallows the root path for that bot, through a named rule or a wildcard. Percentages are of the 510 domains with a valid robots.txt.

What stands out:

• Training is blocked most. 151 domains (29.6%) block at least one training bot at the root, against 89 (17.5%) for search bots and 84 (16.5%) for user-triggered agents.
• Sites separate training from search. 51 domains block GPTBot but leave OAI-SearchBot unblocked.
• Big sites block more. Among the top 100, 50 served a valid robots.txt, and 19 of those block at least one training bot.
• Content Signals are still rare. Only 17 domains (3.3%) publish them.
• Bot defenses show up before robots.txt does. Of the 689 homepages that responded to our plainly identified research client, 99 (14.4%) returned 403. 169 (24.5%) were served through Cloudflare, and 37 answered with a Cloudflare challenge.

For a data team, the takeaway is that robots.txt is only half the picture. About one in seven responding top sites turned away an honest, non-browser client at the front door, whatever its robots.txt said.

Getting data from an access-lane web

These rules don’t stop you from getting web data. They make a source’s access status something you have to check, document and watch for changes. Our managed data feeds do that work for each source: we check feasibility before we build, monitor access alongside freshness and row counts, and put a source URL and timestamp on every value. If you want to know where a specific source stands, ask us. A free 100-row sample takes two business days, or a $950 feasibility check (credited against setup) gives you a written answer on harder cases.