26% of top websites block AI crawlers

AI crawler blocking is a robots.txt policy that forbids specific AI bots from crawling a website. On 26 August 2026 we scanned the robots.txt files of the 800 most-linked domains on the web (Majestic Million ranking). The result: 26.6% of these sites block at least one AI crawler completely, and the split between training bots and search bots is where it gets interesting.

How many websites block AI crawlers

Of the 800 domains, all served a parseable robots.txt. A crawler counts as blocked when its own User-Agent group contains Disallow: /.

AI crawler	Operator	Fully blocked
CCBot	Common Crawl	20.8%
GPTBot	OpenAI (training)	19.2%
ClaudeBot	Anthropic	18.5%
Bytespider	ByteDance	18.2%
meta-externalagent	Meta AI	15.9%
PerplexityBot	Perplexity	15.0%
Applebot-Extended	Apple	14.6%
anthropic-ai	Anthropic	14.1%
Amazonbot	Amazon	13.5%
claude-web	Anthropic (older)	12.4%
facebookbot	Meta	11.0%
chatgpt-user	OpenAI (Browse)	10.9%
Google-Extended	Google (training)	16.6%
OAI-SearchBot	OpenAI (search)	8.0%

One third of sites (33.2%) address AI bots in any form. Two thirds (66.8%) carry no AI-specific rule at all, which means their content is available to every trainer by default.

Which big websites block GPTBot

154 of the 800 domains fully block GPTBot. The head of that list reads like a content-ownership map:

• Instagram (#4)
• TikTok (#22)
• Vimeo (#27)
• WhatsApp (#28)
• Amazon (#30)
• Yahoo (#52)
• Spotify (#60)
• New York Times (#67)
• Medium (#71)
• SoundCloud (#74)
• Forbes (#75)
• BBC (#94)
• CNN (#97)

Subscription and data businesses dominate. These are companies whose content has a price tag, so the training question is a revenue question for them.

Training bots get blocked, search bots stay welcome

The clearest signal in the data is the asymmetry inside one company’s bot family. OpenAI runs a training crawler (GPTBot, blocked by 19.2%) and a search crawler (OAI-SearchBot, blocked by 8.0%). Sites block the trainer at more than twice the rate of the searcher.

That split is deliberate and rational. Blocking the training bot keeps your content out of future model weights while leaving ChatGPT Search free to cite and send traffic today. Google-Extended (16.6%) works the same way for Gemini training without touching Google Search indexing.

If you write one robots.txt rule this year, make it a decision between these two categories instead of a blanket ban.

llms.txt exists on 13% of top sites

The proposed llms.txt standard shows up on 12.9% of the scanned domains. For comparison, sitemap.xml declarations in robots.txt are near-universal on professional sites. Treat llms.txt as cheap optionality, not as a lever: Google has stated it does nothing for LLMs today, and our numbers show adoption is still niche even among large publishers.

How we measured

• Sample: top 800 domains from the Majestic Million (backlink-based global ranking), retrieved on 26 August 2026.
• Each domain’s /robots.txt was fetched and parsed into User-Agent groups. A bot is “fully blocked” when its group contains Disallow: / and “partially restricted” when its group lists other Disallow paths. Partial restrictions are excluded from the percentages above.
• /llms.txt was probed separately and counted when it returned non-HTML content.
• Limitations: X-Robots-Tag headers, firewall rules, and JavaScript gates are invisible to this method. Real blocking rates are likely somewhat higher. 287 further domains were skipped because they did not serve a parseable robots.txt within the timeout.
• Raw data: ai-crawler-blocking-2026.csv, one row per domain, one column per bot.

Should your site block AI crawlers

Decide the two categories separately. If model training on your content is acceptable, leave GPTBot and Google-Extended alone and keep OAI-SearchBot open so ChatGPT Search can cite you. If your content is paid, block the trainers and think hard about the searcher. Either way, measure what you are losing before you flip the switch: run your domain through the AI Visibility Check to see whether AI engines cite you today.

llms.txt exists on 13% of top sites

The proposed llms.txt standard shows up on 12.9% of the scanned domains. For comparison, sitemap.xml declarations in robots.txt are near-universal on professional sites. Treat llms.txt as cheap optionality, not as a lever: Google’s John Mueller has said the file cannot help LLMs today (Search Engine Journal), and our numbers show adoption is still niche even among large publishers.

Frequently asked questions

What percentage of websites block GPTBot?

In our scan of the 800 most-linked domains, 19.2% fully block GPTBot in robots.txt. Another share restricts it to specific paths.

Does blocking GPTBot remove my site from ChatGPT search?

No. ChatGPT Search uses OAI-SearchBot, which is a separate crawler. Blocking GPTBot stops training use, blocking OAI-SearchBot stops search citations. They are independent switches.

Is llms.txt worth adding?

It costs minutes and breaks nothing. Google has said it does not help LLMs today, and only 12.9% of top sites ship one. Set expectations accordingly.

How do I check if my site blocks AI crawlers?

Open https://your-domain.com/robots.txt and look for your name in any User-agent group. Or run the AI Visibility Check and read the crawler-access section of the result.

---

Want the same view for your own site, including consent and security findings beyond robots.txt? Talk to us.