Research · November 6, 2026 · CC BY 4.0

AI Crawler Access Index

How the top 1,000 sites on the web treat AI crawlers and agents: who blocks training bots, who still lets AI search in, and how often a plain automated client gets turned away.

Look up a site CSV JSON

29.6%

of sites with a robots.txt fully block at least one AI training crawler

151 of 510 sites

21.2%

block OpenAI's GPTBot, versus 11.2% for its search crawler

108 vs 57 sites

14.4%

of homepages refused a plain, clearly labelled HTTP client with a 403

99 of 689 sites

3.3%

publish Cloudflare-style Content Signals in robots.txt

17 of 510 sites

Who gets blocked: training bots, far more often

Share of the 510 domains with a valid robots.txt that block each AI bot at the root. Many sites opt out of training while staying visible in AI answers.

Training crawlers · Collect content to train models

Bytespider 24.9%
CCBot 23.9%
GPTBot 21.2%
ClaudeBot 21.0%
Meta-ExternalAgent 19.6%
Google-Extended 18.2%
Applebot-Extended 18.0%

AI search crawlers · Index pages for AI search answers

PerplexityBot 16.7%
Claude-SearchBot 13.1%
OAI-SearchBot 11.2%

User-triggered fetchers · Fetch a page when a person asks the assistant

Perplexity-User 14.3%
Claude-User 13.1%
ChatGPT-User 12.7%

Blocked = the robots.txt group that applies to the bot disallows "/". Source: Scraping Agency crawl of the Tranco top 1,000 (list K9YYW), November 6, 2026. Axis 0–25%.

Opting out of training, staying in AI search

Training is where the line is drawn. 151 sites (29.6%) block at least one training crawler at the root, against 89 (17.5%) for AI search crawlers and 84 (16.5%) for user-triggered fetchers.
Sites split OpenAI's bots. 51 sites block GPTBot but leave OAI-SearchBot alone: no training, but still eligible to be cited in ChatGPT search.
The top of the list is stricter. Among the top 100 domains that serve a robots.txt, 19 of 50 block at least one training crawler.
Content Signals are still rare. Only 17 sites (3.3%) publish the Content-Signal directive that Cloudflare introduced in 2025.
robots.txt is the polite layer; bot management is the enforced one. 14.4% of responding homepages answered our clearly labelled research client with a 403, 24.5% were served through Cloudflare, and 37 returned a Cloudflare challenge.

For anyone relying on agents to collect web data, the practical takeaway is that access is no longer a given. Plan the access route per source, and expect it to change. That is a large part of what we do in a managed feed.

Look up a site

Search the 1,000 domains in this edition.

Rank	Domain	Training bots blocked	AI search blocked	User fetchers blocked	Homepage

Not in this edition's top 1,000, or no website responded.
Loading data…

Showing the first matches. Full rows, with the verdict for every bot, are in the CSV.

Every bot in the index

User agent	Operator	Type	Blocked	Restricted	Open	Named
GPTBot	OpenAI	training	108 (21.2%)	313	89	134
ClaudeBot	Anthropic	training	107 (21.0%)	314	89	127
Google-Extended	Google	training	93 (18.2%)	327	90	112
Applebot-Extended	Apple	training	92 (18.0%)	334	84	90
Meta-ExternalAgent	Meta	training	100 (19.6%)	328	82	90
CCBot	Common Crawl	training	122 (23.9%)	304	84	117
Bytespider	ByteDance	training	127 (24.9%)	300	83	111
OAI-SearchBot	OpenAI	search	57 (11.2%)	360	93	88
Claude-SearchBot	Anthropic	search	67 (13.1%)	356	87	73
PerplexityBot	Perplexity	search	85 (16.7%)	333	92	118
ChatGPT-User	OpenAI	user	65 (12.7%)	354	91	95
Claude-User	Anthropic	user	67 (13.1%)	357	86	70
Perplexity-User	Perplexity	user	73 (14.3%)	348	89	63

How we measured it

We took the top 1,000 domains of the Tranco list K9YYW, a research-grade ranking that averages several popularity sources.
On November 6, 2026 we requested https://<domain>/robots.txt and the homepage once each, with the user agent ScrapingAgencyResearch/1.0 (+https://scrapingagency.com/research/ai-crawler-access-index/), 16 requests in parallel and a 12-second timeout.
For each of 13 AI user agents we applied RFC 9309 group selection (the most specific matching group, otherwise *) and longest-match rules to decide whether the root path is blocked, partly restricted, or open.
For homepages we recorded the HTTP status, whether the response came through Cloudflare, and whether Cloudflare returned a challenge.

Limitations. One snapshot from one network location. robots.txt states preferences and does not prove enforcement. A 403 to our client does not mean a browser would be blocked. Many Tranco domains are infrastructure without a website. The full per-domain results are in the downloads, and the collection script is available on request.

Cite as: Scraping Agency, AI Crawler Access Index, 2026-11 edition, https://scrapingagency.com/research/ai-crawler-access-index/ (CC BY 4.0).

Questions we get

Something else? Ask us directly.

Does a robots.txt block actually stop AI crawlers?

robots.txt is a request, not an enforcement mechanism. The major AI operators say their declared crawlers honor it, but enforcement comes from bot-management layers such as Cloudflare, which is why we also measured how homepages respond to a plain client.

Why only 510 of 1,000 domains?

The Tranco top list includes many infrastructure domains (CDNs, APIs, ad and tracking hosts) that serve no website or robots.txt. 510 domains returned a valid robots.txt file and 689 returned a homepage response.

What does "blocked" mean here?

The robots.txt group that applies to that user agent (the most specific matching group, else the * group, per RFC 9309) disallows the site root. "Restricted" means some paths are disallowed but the root is allowed.

Can I use the data?

Yes. The dataset is published under CC BY 4.0. Please cite "Scraping Agency, AI Crawler Access Index" with a link to this page.

Need data from sites that turn bots away?

We plan a reliable access route for every source and keep it working. Get a free 100-row sample within 2 business days.

Get a free sample See pricing