Robots.txt Tester for Ghost Users

Why Ghost Users need this

Your SEO tool says the tag and author archives are thin duplicates, and Ghost gives you no bulk noindex switch to deal with them. The advice on the forum is to edit the templates or ship a custom robots.txt in the theme, so you add a few Disallow lines and upload the theme. Then comes the part nobody checks: whether the file that is live on your domain blocks only what you meant. One stray rule such as Disallow: /t can also catch real posts whose slugs begin with those letters.

The Robots.txt Tester fetches the live file from the exact host you enter, groups the rules per user-agent, and lets you paste up to 20 paths to see whether Googlebot, Googlebot-Image and Bingbot may crawl each one, with the deciding line number. Run it after every theme upload, before you wonder why a post vanished. It reads robots.txt only, so it says nothing about noindex tags or whether a page is indexed.

Ghost Users specific guidance

What to enter first

Start with your domain as readers reach it, then use the paths box for the URLs that matter: a recent post such as /how-we-price-our-newsletter/, a tag archive like /tag/news/, an author page like /author/jane/, and /rss/. Ghost themes can ship their own robots.txt that overrides the default, so the live file is the only version worth trusting; a copy sitting in your theme folder proves nothing about what the server returns. The host matters, because your custom domain, its www or non-www variant and a .ghost.io address are separate runs. The whole check takes about two minutes after you upload a theme. A clean pass here shows only that robots.txt does not stop the crawl, not that the pages are indexable.

How to read the URL test

For each path, look at the group used and the deciding rule with its line number. If a post shows Blocked, the printed rule tells you whether a broad line such as Disallow: /t caught it by prefix. Tag and author paths showing Blocked mean the theme file does what you wanted for those archives, but Blocked only means not crawled: an archive already linked from your posts can stay in the index, and Google can no longer read a noindex tag on it. Allowed on a post is the result you want, yet it says nothing about the post's meta robots tag. Also compare the Sitemaps declared line with the sitemap your Ghost site actually serves at /sitemap.xml.

What to fix, and in which order

First, repair any post, page or /rss/ URL that shows Blocked: edit the robots.txt in your theme, upload the theme again and rerun the test. Second, decide per archive whether to block crawling or to add a noindex meta tag in the tag or author template; the Ghost forum thread describes both routes, and mixing them on one URL defeats the noindex. Third, if you migrated from another platform, test the old URL patterns you redirect in redirects.yaml, because a crawler that is told not to request an old URL never sees its 301. Finally, run the Meta Tag Analyzer for noindex and use Search Console for index status, since this report can see neither.

Example scenarios

Common mistakes

• Using a prefix without a trailing slash. Disallow: /t or Disallow: /a matches every URL beginning with those letters, including posts. Write Disallow: /tag/ and /author/ so only the archives are matched.
• Testing the theme file instead of the live file. The robots.txt in your theme folder is not proof of what is served until the theme is uploaded and active. Always test the domain readers use.
• Blocking archives and expecting noindex to work. A crawler that cannot fetch a tag page never reads a noindex tag placed in its template. Choose crawl blocking or noindex for each archive, not both.
• Forgetting the old URLs after a migration. Rules copied from the previous platform can disallow the very paths your redirects.yaml rules start from. Test a few old URLs before and after the move.
• Reading Allowed as indexed. Allowed only says robots.txt permits the request. Noindex tags, redirects and index status need the Meta Tag Analyzer and Search Console.

Frequently asked questions

Where is the robots.txt for a Ghost site and can I change it?

Ghost serves a default robots.txt at yourdomain.com/robots.txt, and a theme can include its own robots.txt that overrides it, as Ghost's theme structure page explains. To change the rules, edit that file in your theme and upload the theme again. Then run the live domain through the tester to confirm the new lines are what the server returns, because an old theme that is still active will keep serving the earlier rules. A correct file still says nothing about noindex tags.

How do I noindex tag pages in Ghost?

Ghost has no bulk noindex setting for tag and author archives, which is why forum answers point to editing the tag and author templates so they output a noindex meta tag. A custom robots.txt is the other route, but Disallow only stops crawling, and a crawler that cannot fetch a page never sees its noindex tag. Use one method per archive, run the tester on a path like /tag/news/ to see what the file blocks, then check the page's robots tag separately.

Will a Disallow in my Ghost theme robots.txt remove tag pages from Google?

Not by itself. Disallow stops crawlers from requesting the URL, but a tag page that is already indexed or linked from your posts can stay in results, usually with little description text. The tester reports Blocked or Allowed per path and which line decided it, and it states that Blocked does not remove a URL from the index. Treat Blocked as a crawl instruction only. Removal needs a noindex tag Google can crawl, or a removal request in Search Console.

Why does the tester say my post is Allowed when it is not in Google?

Allowed only means robots.txt does not stop Googlebot from requesting that URL. The post can still carry a noindex tag, sit behind a redirect, be too new, or simply not be indexed yet, and none of that appears in a robots.txt report. Check the post's robots meta tag with the Meta Tag Analyzer and its index status in Search Console, using the URL Inspection tool, before you touch the theme file again. Fix the right layer first.

Can I see whether GPTBot or ClaudeBot is blocked on my Ghost blog?

Yes, for what the file asks. The News & AI crawlers section covers 11 tokens, including GPTBot, ClaudeBot, PerplexityBot and CCBot, and shows for your tested paths whether each is Allowed, Partly blocked or Blocked, plus the group it follows. It reads the file only, so it cannot say whether a bot obeys it, and a block at your CDN or firewall is invisible to it. Compare the rows against what you intend before publishing the theme.

Reference: Ghost's theme structure documentation confirms that a theme can ship its own robots.txt that overrides Ghost's default file.