No Signup Unlimited Usage Real Data 100% Free Privacy First Fast Results Export Reports Mobile Friendly

AI Crawler Access Checker

Check whether GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and 12 other AI bots can reach a page: the robots.txt rule (with line number) and a live request with each bot’s real User-Agent, WAF challenge detection and copy-ready robots.txt templates.

Fetching live data…

Quick answer

The AI Crawler Access Checker tests whether the bots behind ChatGPT, Claude, Perplexity and other AI services can reach a page. For each bot it checks two layers: what your live robots.txt says, and how your server actually answers a request sent with that bot's real User-Agent. You see exactly which rule or response blocks which bot, and why.

How it works

  1. Enter one page URL, for example https://example.com/pricing.
  2. The tool fetches your live robots.txt and evaluates it for every bot token the way RFC 9309 describes: the most specific user-agent group applies, the longest matching rule wins, Allow wins a tie, and * and $ wildcards are honoured.
  3. For bots that are real crawlers, it requests the page with that bot's User-Agent string, five at a time, and compares each response with a normal browser request.
  4. You get a score card, a bot-by-bot table and three ready-to-copy robots.txt templates.

What it checks

  • Robots.txt rule per bot — allowed or blocked, with the reason and the exact line number of the rule that decides it. A missing file (404) means everything is allowed; a server error (5xx) means everything is treated as blocked.
  • Server response per bot — the HTTP status your server returns to each crawler's User-Agent, next to the status a browser gets.
  • WAF and bot challenges — detects challenge pages such as a cf-mitigated header, "Just a moment", "Attention Required" or captcha pages served instead of your content.
  • OpenAI — GPTBot (training), OAI-SearchBot (search) and ChatGPT-User (user-fetch).
  • Anthropic — ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user-fetch).
  • Perplexity — PerplexityBot (search) and Perplexity-User (user-fetch).
  • Control tokens — Google-Extended and Applebot-Extended are not crawlers, so only their robots.txt rule is checked.
  • Other AI bots — CCBot, Bytespider, Amazonbot, Meta-ExternalAgent, DuckAssistBot, MistralAI-User and cohere-ai.
  • noai signals — noai and noimageai values in the meta robots tag or the X-Robots-Tag header.

Why it matters

AI assistants fetch pages for different reasons. Training bots collect text for future models; search bots build the index an assistant answers from; user-fetch bots open a page because someone asked about it right now. Blocking a search or user-fetch bot can keep your pages out of those answers, while blocking training bots is a legitimate choice many publishers make. The two layers can disagree: robots.txt may allow a bot while a firewall rule or bot-protection setting quietly serves it a 403 or a challenge page.

The score reflects this split. It is the share of search and user-fetch bots that pass both layers, multiplied by 100. Training bots are listed but not scored. This is our own criterion for AI search reachability, not a rule published by OpenAI, Anthropic, Perplexity or Google, and a high score does not mean any assistant will cite you.

How to fix common problems

Blocked in robots.txt by accident. Use the line number in the table to find the rule. A broad User-agent: * with Disallow: / catches every AI bot that has no group of its own. Add a specific group for the bots you want, or start from the "Appear in AI search, opt out of training" template, which allows search and user-fetch bots and blocks training bots. The "Allow all" and "Block all" templates cover the other two choices. Test the edited file with the Robots.txt Tester; to build one from scratch, use the Robots.txt Generator.

Robots allows, server blocks. If the server column shows 403, 429 or a challenge page while the browser request returns 200, check your CDN or firewall bot settings and allow the AI crawlers you want to reach you.

Content not readable. Most AI crawlers do not run JavaScript, so confirm your text is in the raw HTML with the JavaScript Rendering Checker and point them to your key pages with an llms.txt checked by the llms.txt Validator.

Frequently asked questions

Can this tool detect IP-based blocking?

No. Our requests come from our own server's IP address with each bot's User-Agent, not from the bots' published IP ranges. A firewall that blocks only those IP ranges would let our test through while still blocking the real bot.

Why are training bots not part of the score?

Blocking training bots such as GPTBot or ClaudeBot is a legitimate choice that does not have to affect whether search and user-fetch bots can reach you. The table still shows their status, so you can confirm your opt-out works.

What is the difference between search and user-fetch bots?

Search bots such as OAI-SearchBot crawl pages ahead of time for an assistant's search index. User-fetch bots such as ChatGPT-User or Claude-User open a page on demand when a person asks the assistant about it. Each vendor documents its bots, for example in OpenAI's crawler overview.

Why are Google-Extended and Applebot-Extended not fetched?

They are control tokens, not crawlers. Google and Apple read the robots.txt rule for these tokens to decide how content crawled by their normal bots may be used for AI features. There is no separate bot to request your page, so only the rule is checked.

Does a 404 robots.txt block AI crawlers?

No. Under RFC 9309 a missing robots.txt (404) means crawlers may fetch everything. A server error (5xx) is the opposite: crawlers should treat the whole site as disallowed until the file loads again, so an unstable robots.txt can block bots you meant to allow.

Is the AI Crawler Access Checker free?

Yes, it is free to use, and the report includes the bot table and the copyable templates. For a wider view of how AI-ready a page is, try the AI Visibility Checker.

Who uses the AI Crawler Access Checker

Related tools

JavaScript Rendering Checker

See what AI crawlers that do not run JavaScript get from a page: raw HTML vs a headless-Chromium render, field by field (title, canonical, robots, headings, words, links, JSON-LD, hreflang), JS dependency %, render-only text highlighted and SSR/prerender fixes.

Open tool →

AI Prompt-Friendly Content Checker

Check how well your content answers the prompts real people type into AI chatbots. Scores prompt-alignment — direct answers, self-contained context, definitions, steps and comparisons — so ChatGPT and Perplexity can lift a complete answer from your page.

Open tool →

Topical Authority Map

Map how completely your site covers a topic — the way Google and AI engines judge topical authority. Crawls your internal links, titles and headings, compares them against the real subtopics people search for, and shows your covered subtopics plus the content gaps to build for a stronger topic cluster.

Open tool →

AI FAQ Optimizer

Turn your content into AI-ready FAQs. Detects existing questions, suggests high-intent questions you should answer, grades your answers for snippet-readiness and generates copy-paste FAQPage JSON-LD schema. Paste text or enter a URL.

Open tool →

Entity Extractor

Extract the named entities (people, organizations, places, concepts) a page contains — the way AI and Google’s Knowledge Graph read it. Lists every entity with mention counts and a type guess, scores entity richness and diversity, and flags where schema markup would define your entities more clearly. Paste text or enter a URL.

Open tool →

AI Content Score

A single 0–100 quality score for how well content performs for AI-driven search. Blends readability, structure, answer-friendliness, depth, originality signals and E-E-A-T into one grade with a prioritized improvement list. Paste text or enter a URL.

Open tool →