Robots.txt Generator

How it works

1. Configure your rules — pick the user-agent and add the paths to allow or disallow.
2. Optionally set a crawl-delay and add your sitemap URL.
3. Generate the file and copy or download the output.
4. Upload it as robots.txt in your site's root directory.

What it generates

• User-agent blocks — target all crawlers with * or name a specific bot.
• Disallow rules — block paths like /admin/ or /cart/ from being crawled.
• Allow rules — re-open specific paths inside a blocked folder.
• Crawl-delay — an optional pause between requests for bots that honour it.
• Sitemap reference — a full Sitemap: line pointing to your XML sitemap.
• Clean formatting — correct ordering and one directive per line, ready to deploy.

Common use cases

• New site launch — create a sensible robots.txt before search engines start crawling.
• Blocking private areas — keep admin, checkout or staging paths out of the index.
• Saving crawl budget — stop bots wasting time on faceted or duplicate URLs.
• Adding a sitemap — make sure crawlers can discover your sitemap automatically.

Example

To block your admin area but allow everything else and declare a sitemap, the tool outputs: User-agent: *
Disallow: /admin/
Allow: /
Sitemap: https://example.com/sitemap.xml — copy it straight into your root file.

Frequently asked questions

Is the Robots.txt Generator free?

Yes — completely free, with no signup and no limits. It runs in your browser and never stores your input.

Where do I put the file?

It must live at the root of your domain, at https://example.com/robots.txt. After uploading, confirm the rules behave as intended with the Robots.txt Tester and make sure your sitemap resolves with the Sitemap Finder & Validator.

Can a wrong rule hurt my SEO?

Yes — a stray Disallow: / can block your whole site from being crawled. Always double-check before deploying. Our robots.txt guide explains each directive and common pitfalls in detail.