Robots.txt Tester

How it works

1. Enter a domain or full URL (for example https://example.com).
2. The tool requests /robots.txt from that host exactly as a crawler would and reads the raw response.
3. It parses the file line by line, grouping directives under each User-agent block.
4. You get a clean, structured breakdown of every rule plus the HTTP status and any sitemap URLs found.

What it shows

• HTTP status — whether the file exists (200), is missing (404) or errors.
• User-agent blocks — rules grouped per crawler, including the * wildcard.
• Disallow rules — the exact paths each crawler is told not to fetch.
• Allow rules — explicit exceptions that override a broader Disallow.
• Crawl-delay — any throttling directive a crawler is asked to respect.
• Sitemap declarations — every Sitemap: line and its URL.
• Raw file — the unmodified source so you can spot stray characters.

Common use cases

• Catching accidental blocks — confirm a Disallow: / isn't hiding your whole site.
• Pre-launch checks — verify a staging block was removed before going live.
• Sitemap discovery — find the sitemap URLs a site advertises to crawlers.
• Auditing competitors — see which sections a rival keeps out of the index.

Example

Run the tool on a store and you might see a User-agent: * block that disallows /cart/ and /checkout/, an explicit Allow: /cart/info exception, and a Sitemap: https://example.com/sitemap.xml line — confirming the right pages are blocked while products stay crawlable.

Frequently asked questions

Is the Robots.txt Tester free?

Yes — completely free, with no signup and no limits. Test as many domains as you like.

Does robots.txt stop a page from being indexed?

Not on its own. A Disallow only stops crawling; a blocked URL can still appear in results if other pages link to it. To keep a page out of the index use a noindex tag — check tags with the Meta Tag Analyzer — and read our robots.txt guide.

How is this different from generating a robots.txt?

This tool reads and validates a live file. To build a new one from scratch use the Robots.txt Generator and then confirm the sitemap it references with the Sitemap Finder & Validator.