Which AI crawlers can read your site?
Enter a URL. We read your robots.txt for 18 AI bots, test what really gets through your firewall, and show whether ChatGPT, Claude, and Perplexity can find and cite you.
Reads robots.txt and sends four homepage requests. No signup, nothing stored. The numbers behind it: 26% of top sites block AI crawlers.
What this checks
18 AI bots, one by one
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more, each read against your robots.txt the way the bot itself reads it.
Training vs. search
Blocking training crawlers keeps you out of future models. Blocking search crawlers keeps you out of AI answers today. The check tells the two apart.
What actually gets through
We request your homepage as a browser and as three AI bots. A CDN or firewall that blocks bots shows up here, even when robots.txt says yes.
noai, noindex, llms.txt
Opt-out directives in meta tags and headers, plus whether an llms.txt file exists.
Visible where your buyers ask.
We make sites readable and citable for AI search: crawler policy, structure, schema, and the content AI assistants quote.
Questions about the AI Crawler Check
Which AI crawlers does the check cover?
18 user agents from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon, DuckDuckGo, Meta, Common Crawl, ByteDance, and Cohere. Each one is tagged by purpose: training, search, or fetching a page for a user.
What is the difference between training and search crawlers?
Training crawlers such as GPTBot and ClaudeBot collect pages for model training. Search crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot index pages so an assistant can cite and link them in answers. Blocking a search crawler takes the site out of those answers.
Why test the firewall when robots.txt allows a bot?
robots.txt is a request; CDNs and firewalls block at the network level. The check fetches the homepage as a regular browser and as GPTBot, ClaudeBot, and PerplexityBot. An error or challenge page for the bots alone means they are blocked, whatever robots.txt says.
Does llms.txt matter?
llms.txt is a proposed file that points AI tools to a site's key content. Use by the large AI providers is unconfirmed, so the check reports it for information and does not score it.
How many websites block AI crawlers?
In our scan of the robots.txt files of the 800 most-linked websites in August 2026, 26.6 percent blocked at least one AI crawler completely. GPTBot was blocked by 19.2 percent, OpenAI's search crawler by only 8 percent.
From the blog
- The best free robots.txt testers, ranked
Google retired its robots.txt Tester in 2023. The free tools that replace it, which parser each uses, and how to test GPTBot and other AI bots.
- 26% of top websites block AI crawlers
Robots.txt scan of the 800 most-linked websites: 26.6% block at least one AI crawler, GPTBot 19.2%, OAI-SearchBot only 8%. Full per-bot numbers.
