The best free robots.txt testers, ranked
By Philipp Kant 6 min read
A robots.txt tester is a tool that checks whether a given crawler may fetch a given URL under a site’s robots.txt rules, and which rule decides it. Google retired the robots.txt Tester in Search Console in late 2023 and replaced it with a read-only robots.txt report, so the free options now sit in other tools.
The choice depends on the question. “Is this URL blocked for Googlebot?” needs Google’s parser. “Can ChatGPT and Claude read my site?” needs the AI bots’ user agents, and a look past robots.txt at the firewall.
| Tool | Parser | AI bots | Edit rules before publishing | Account |
|---|---|---|---|---|
| Tame the Bots robots.txt checker | Google’s open-source parser | GPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended, Applebot, custom | Yes | None |
| Search Console robots.txt report | Google’s own | No | No | Verified domain property |
| Bing Webmaster Tools robots.txt tester | Bing’s own | No | Yes | Verified site |
| Screaming Frog SEO Spider (free) | Screaming Frog’s own | Not in the free version | Paid version only | None, 500 URLs per crawl |
| Google’s robotstxt library | Google’s open-source parser | Any | Yes, it is code | None |
| kant.dev AI Crawler Check | kant.dev’s own | 18 AI bots, plus live firewall probes | No | None |
1. Tame the Bots, Google’s parser in a browser
Tame the Bots runs its robots.txt checker on Google’s open-source “Robots.txt Parser and Matcher Library”, the same code Googlebot uses, and mimics Google’s fallbacks (Googlebot-Image falling back to Googlebot’s rules, for example). Paste a list of URLs and it shows, per URL, whether it is allowed and which rule decides.
What sets it apart: a long list of user agents, including GPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended, Applebot, and Bingbot, and a custom mode that tests rules before they go live. Free, funded by donations, no account.
2. Search Console robots.txt report, Google’s own view
The report that replaced the Tester shows which robots.txt files Google found for the top 20 hosts of a domain property, when it last tried to fetch each one, the fetch status, and parsing errors and warnings. After an urgent fix you can ask Google to recrawl the file from the same screen.
It does not test URLs. For that, Google points to the URL Inspection tool. Both need a verified domain property, so they only work on your own sites.
3. Bing Webmaster Tools, a tester with an editor
Bing added a robots.txt tester to Webmaster Tools in 2020, and it does what Google’s old one did: enter a URL, see whether it is blocked, for which user agent, and by which line. It tests as Bingbot or AdIdxBot (the Bing Ads crawler), and its editor lets you change the rules and retest before publishing. Needs a verified site.
Worth checking for one more reason: Bing’s index feeds Copilot and parts of ChatGPT search.
4. Screaming Frog SEO Spider, the crawl-wide view
The free version of the desktop crawler handles up to 500 URLs per crawl and lists every URL it found but was not allowed to fetch, under the “Blocked by Robots.txt” filter. That answers a different question than the other tools: not “is this URL blocked?” but “which of my pages are?”
Editing and testing a custom robots.txt is part of the paid licence, as is most crawl configuration.
5. Google’s robotstxt library, for developers
Google open-sourced the C++ parser and matcher its crawlers use. Anyone building their own checks gets Google’s exact interpretation, including the edge cases that hand-written regexes get wrong. Tame the Bots (above) is built on it.
6. kant.dev AI Crawler Check, for AI bots and firewalls
Our free AI Crawler Check reads the live
robots.txt and reports the verdict for 18 AI crawlers, split into
training bots and the ones that feed AI search answers. Then it sends real
requests with those bots’ user agents, because a firewall or CDN often
blocks a bot that robots.txt allows. It also reports noai tags and
llms.txt.
It does not test single URLs against draft rules; use Tame the Bots or Bing for that.
What every robots.txt tester misses
robots.txt is a request, not a lock. Three things decide access beyond it:
- Firewalls and CDNs. Bot protection at Cloudflare or a hosting provider can block GPTBot while robots.txt says yes, or let a crawler through that robots.txt forbids. Only a live request shows which.
- User-triggered fetchers. Google documents that its user-triggered fetchers, which run when a person asks a product to open a page, “generally ignore robots.txt rules”. The same logic applies to the AI agents that browse for a user.
- JavaScript. A bot that is allowed in still only reads the HTML response. Content that needs JavaScript stays invisible to GPTBot and ClaudeBot; see can AI crawlers read JavaScript.
How Google reads robots.txt
The rules a tester should apply, from Google’s documentation:
| Situation | What Google does |
|---|---|
| Allow and Disallow conflict | The more specific (longer) rule wins; on a tie, the least restrictive |
| Several user-agent groups match | Only the most specific group counts; order in the file does not matter |
| File larger than 500 KiB | Content past the limit is ignored |
| robots.txt returns 4xx (not 429) | Treated as if no robots.txt exists: everything may be crawled |
| robots.txt returns 5xx | Crawling pauses, then Google uses the last good copy for up to 30 days |
| A change is published | Google caches robots.txt for up to 24 hours |
The 4xx row surprises people most: a missing robots.txt is no block at all. The 5xx row is the dangerous one: a server error on robots.txt stops crawling of the whole site for a while.
How many large sites block which AI bot, from our own scan of 800 domains, is in 26% of top websites block AI crawlers.
FAQ
What replaced Google’s robots.txt Tester? The robots.txt report in Search Console (Settings), launched in November 2023, plus the URL Inspection tool for single URLs. Neither lets you edit and test draft rules; Tame the Bots and Bing Webmaster Tools do.
How do I check whether GPTBot is blocked? Test the URL with GPTBot as the user agent in Tame the Bots, or run the AI Crawler Check, which checks GPTBot and 17 other AI crawlers at once and also sends a live request to catch firewall blocks.
Does blocking GPTBot remove my site from ChatGPT? No. GPTBot collects training data. ChatGPT’s search answers use OAI-SearchBot, and visits a user triggers use ChatGPT-User. Each has its own robots.txt entry.
Is Google-Extended a crawler? No. It is a robots.txt token. Disallowing it tells Google not to use the content for Gemini training and grounding; Googlebot still crawls the pages for Search.
How long until a robots.txt change takes effect? Google caches the file for up to 24 hours. In Search Console you can request a recrawl of robots.txt to speed it up.
Check which AI bots can read your site, and whether your firewall agrees with your robots.txt, with the free AI Crawler Check.
Ask an AI about this post
