How it works
- Enter one page URL, for example
https://example.com/pricing. - The tool fetches your live
robots.txtand evaluates it for every bot token the way RFC 9309 describes: the most specific user-agent group applies, the longest matching rule wins, Allow wins a tie, and*and$wildcards are honoured. - For bots that are real crawlers, it requests the page with that bot's User-Agent string, five at a time, and compares each response with a normal browser request.
- You get a score card, a bot-by-bot table and three ready-to-copy
robots.txttemplates.
What it checks
- Robots.txt rule per bot — allowed or blocked, with the reason and the exact line number of the rule that decides it. A missing file (404) means everything is allowed; a server error (5xx) means everything is treated as blocked.
- Server response per bot — the HTTP status your server returns to each crawler's User-Agent, next to the status a browser gets.
- WAF and bot challenges — detects challenge pages such as a
cf-mitigatedheader, "Just a moment", "Attention Required" or captcha pages served instead of your content. - OpenAI — GPTBot (training), OAI-SearchBot (search) and ChatGPT-User (user-fetch).
- Anthropic — ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user-fetch).
- Perplexity — PerplexityBot (search) and Perplexity-User (user-fetch).
- Control tokens — Google-Extended and Applebot-Extended are not crawlers, so only their robots.txt rule is checked.
- Other AI bots — CCBot, Bytespider, Amazonbot, Meta-ExternalAgent, DuckAssistBot, MistralAI-User and cohere-ai.
- noai signals —
noaiandnoimageaivalues in the meta robots tag or theX-Robots-Tagheader.
Why it matters
AI assistants fetch pages for different reasons. Training bots collect text for future models; search bots build the index an assistant answers from; user-fetch bots open a page because someone asked about it right now. Blocking a search or user-fetch bot can keep your pages out of those answers, while blocking training bots is a legitimate choice many publishers make. The two layers can disagree: robots.txt may allow a bot while a firewall rule or bot-protection setting quietly serves it a 403 or a challenge page.
The score reflects this split. It is the share of search and user-fetch bots that pass both layers, multiplied by 100. Training bots are listed but not scored. This is our own criterion for AI search reachability, not a rule published by OpenAI, Anthropic, Perplexity or Google, and a high score does not mean any assistant will cite you.
How to fix common problems
Blocked in robots.txt by accident. Use the line number in the table to find the rule. A broad User-agent: * with Disallow: / catches every AI bot that has no group of its own. Add a specific group for the bots you want, or start from the "Appear in AI search, opt out of training" template, which allows search and user-fetch bots and blocks training bots. The "Allow all" and "Block all" templates cover the other two choices. Test the edited file with the Robots.txt Tester; to build one from scratch, use the Robots.txt Generator.
Robots allows, server blocks. If the server column shows 403, 429 or a challenge page while the browser request returns 200, check your CDN or firewall bot settings and allow the AI crawlers you want to reach you.
Content not readable. Most AI crawlers do not run JavaScript, so confirm your text is in the raw HTML with the JavaScript Rendering Checker and point them to your key pages with an llms.txt checked by the llms.txt Validator.
Frequently asked questions
Can this tool detect IP-based blocking?
No. Our requests come from our own server's IP address with each bot's User-Agent, not from the bots' published IP ranges. A firewall that blocks only those IP ranges would let our test through while still blocking the real bot.
Why are training bots not part of the score?
Blocking training bots such as GPTBot or ClaudeBot is a legitimate choice that does not have to affect whether search and user-fetch bots can reach you. The table still shows their status, so you can confirm your opt-out works.
What is the difference between search and user-fetch bots?
Search bots such as OAI-SearchBot crawl pages ahead of time for an assistant's search index. User-fetch bots such as ChatGPT-User or Claude-User open a page on demand when a person asks the assistant about it. Each vendor documents its bots, for example in OpenAI's crawler overview.
Why are Google-Extended and Applebot-Extended not fetched?
They are control tokens, not crawlers. Google and Apple read the robots.txt rule for these tokens to decide how content crawled by their normal bots may be used for AI features. There is no separate bot to request your page, so only the rule is checked.
Does a 404 robots.txt block AI crawlers?
No. Under RFC 9309 a missing robots.txt (404) means crawlers may fetch everything. A server error (5xx) is the opposite: crawlers should treat the whole site as disallowed until the file loads again, so an unstable robots.txt can block bots you meant to allow.
Is the AI Crawler Access Checker free?
Yes, it is free to use, and the report includes the bot table and the copyable templates. For a wider view of how AI-ready a page is, try the AI Visibility Checker.