26% of top websites block AI crawlers
By Philipp Kant 4 min read
AI crawler blocking is a robots.txt policy that forbids specific AI bots from crawling a website.
On 26 August 2026 we scanned the robots.txt files of the 800 most-linked domains on the web (Majestic Million ranking). The result: 26.6% of these sites block at least one AI crawler completely, and the split between training bots and search bots is where it gets interesting.
How many websites block AI crawlers
Of the 800 domains, all served a parseable robots.txt. A crawler counts
as blocked when its own User-Agent group contains Disallow: /.
| AI crawler | Operator | Fully blocked |
|---|---|---|
| CCBot | Common Crawl | 20.8% |
| GPTBot | OpenAI (training) | 19.2% |
| ClaudeBot | Anthropic | 18.5% |
| Bytespider | ByteDance | 18.2% |
| Google-Extended | Google (training) | 16.6% |
| meta-externalagent | Meta AI | 15.9% |
| PerplexityBot | Perplexity | 15.0% |
| Applebot-Extended | Apple | 14.6% |
| anthropic-ai | Anthropic | 14.1% |
| Amazonbot | Amazon | 13.5% |
| claude-web | Anthropic (older) | 12.4% |
| facebookbot | Meta | 11.0% |
| chatgpt-user | OpenAI (Browse) | 10.9% |
| OAI-SearchBot | OpenAI (search) | 8.0% |
One third of sites (33.2%) address AI bots in any form. Two thirds (66.8%) carry no AI-specific rule at all, which means their content is available to every trainer by default.
Which big websites block GPTBot
154 of the 800 domains fully block GPTBot. The head of that list reads like a content-ownership map:
- Instagram (#4)
- TikTok (#22)
- Vimeo (#27)
- WhatsApp (#28)
- Amazon (#30)
- Yahoo (#52)
- Spotify (#60)
- New York Times (#67)
- Medium (#71)
- SoundCloud (#74)
- Forbes (#75)
- BBC (#94)
- CNN (#97)
Subscription and data businesses dominate. These are companies whose content has a price tag, so the training question is a revenue question for them.
Training bots get blocked, search bots stay welcome
The clearest signal in the data is the asymmetry inside one company’s bot family. OpenAI runs a training crawler (GPTBot, blocked by 19.2%) and a search crawler (OAI-SearchBot, blocked by 8.0%). Sites block the trainer at more than twice the rate of the searcher.
That split is deliberate and rational. Blocking the training bot keeps your content out of future model weights while leaving ChatGPT Search free to cite and send traffic today. Google-Extended (16.6%) works the same way for Gemini training without touching Google Search indexing.
If you write one robots.txt rule this year, make it a decision between these two categories instead of a blanket ban.
How we measured
- Sample: top 800 domains from the Majestic Million (backlink-based global ranking), retrieved on 26 August 2026.
- Each domain’s
/robots.txtwas fetched and parsed into User-Agent groups. A bot is “fully blocked” when its group containsDisallow: /and “partially restricted” when its group lists other Disallow paths. Partial restrictions are excluded from the percentages above. /llms.txtwas probed separately and counted when it returned non-HTML content.- Limitations: X-Robots-Tag headers, firewall rules, and JavaScript gates are invisible to this method. Real blocking rates are likely somewhat higher. 287 further domains were skipped because they did not serve a parseable robots.txt within the timeout.
- Raw data: ai-crawler-blocking-2026.csv, one row per domain, one column per bot.
Should your site block AI crawlers
Decide the two categories separately. If model training on your content is acceptable, leave GPTBot and Google-Extended alone and keep OAI-SearchBot open so ChatGPT Search can cite you. If your content is paid, block the trainers and think hard about the searcher. Either way, measure what you are losing before you flip the switch: run your domain through the AI Visibility Check to see whether AI engines cite you today.
llms.txt exists on 13% of top sites
The proposed llms.txt standard shows up on 12.9% of the scanned
domains. For comparison, sitemap.xml declarations in robots.txt are
near-universal on professional sites. Treat llms.txt as cheap
optionality, not as a lever: Google’s John Mueller has said the file
cannot help LLMs today (Search Engine Journal),
and our numbers show adoption is still niche even among large
publishers.
FAQ
What percentage of websites block GPTBot?
In our scan of the 800 most-linked domains, 19.2% fully block GPTBot in robots.txt. Another share restricts it to specific paths.
Does blocking GPTBot remove my site from ChatGPT search?
No. ChatGPT Search uses OAI-SearchBot, which is a separate crawler. Blocking GPTBot stops training use, blocking OAI-SearchBot stops search citations. They are independent switches.
Is llms.txt worth adding?
It costs minutes and breaks nothing. Google has said it does not help LLMs today, and only 12.9% of top sites ship one. Set expectations accordingly.
How do I check if my site blocks AI crawlers?
Open https://your-domain.com/robots.txt and look for the crawler’s
name (GPTBot, ClaudeBot, Google-Extended, OAI-SearchBot) in any
User-agent group. Or run the
AI Visibility Check and read the
crawler-access section of the result.
Want the same view for your own site, including consent and security findings beyond robots.txt? Talk to us.
