Why WordPress Site Owners need this
You launched the new site weeks ago, the posts are live, and Google still shows nothing. For many WordPress site owners the cause is one tick box left on from development: Settings > Reading > Search engine visibility, "Discourage search engines from indexing this site". A leftover plugin rule or a copied staging file can do the same job quietly. The opposite problem is just as common: tag, date and author archives and internal search pages are open to crawlers, filling the index with thin near-duplicates that sit beside your real posts. Before you touch a plugin or reinstall anything, read what your server actually hands to crawlers. The Robots.txt Tester fetches the live robots.txt for the domain you enter, groups the rules by user-agent, lists every Disallow and Allow line, and shows any Sitemap declarations. You can also paste up to 20 paths, such as a post, a /tag/ archive and a /?s= search URL, and see whether Googlebot, Googlebot-Image and Bingbot may crawl each one. It reads robots.txt only, so it cannot tell you whether a page is noindex or indexed; treat it as the first check, not the verdict.
WordPress Site Owners specific guidance
Check the setting, then the file
Open Settings > Reading in the dashboard and look at Search engine visibility. If the box is ticked, untick it and save. Then run the tester on your bare domain, and run the www or non-www version separately if you are unsure which one is canonical, because each host is its own run. WordPress serves a virtual robots.txt unless a physical file sits in the site root, so what you see may come from core, from the file editor of an SEO plugin such as Yoast SEO under Tools, or from a file someone uploaded by hand. Read the HTTP status first: 200 means a file exists, while a 404 means crawlers are allowed by default. A clean file does not prove the site is indexable, because the tester never sees the robots tag inside your pages.
Test the URLs that matter
Paste a recent post such as /2026/10/launch-checklist/, a /tag/news/ archive, an /author/jane/ page, a /?s=test search URL and /wp-admin/admin-ajax.php into the paths box. For each crawler you get Allowed or Blocked, the group it obeyed and the deciding line number. Posts should read Allowed. If wp-admin is blocked with an Allow exception for admin-ajax.php, the longer pattern wins, so the exception overrides the broader Disallow. Remember that Blocked means not crawled, not removed from the index: a URL that was indexed earlier can linger. The result also says nothing about whether an archive is thin or worth keeping.
Fix in this order
Deal with the visibility setting first, then any Disallow: / line, then stray rules left by plugins or old developers. For thin tag or author archives, prefer a noindex setting in your SEO plugin over a robots.txt Disallow, because a crawler that is blocked cannot read the noindex tag. Confirm the tag with the Meta Tag Analyzer. Once the sitemap you want is the live one, whether the core /wp-sitemap.xml or your plugin sitemap, check that a Sitemap line points to it. Re-run the tester after every change, and use Search Console for index status, which this tool cannot show.
Example scenarios
1. Site still invisible after launch
- Situation
- Illustrative example: an owner moves a 42-post blog from a staging copy to the live domain. Three weeks later no post appears in Google, and the Discourage search engines box from the build phase is still ticked.
- Input
https://example-blog.comwith the path/2026/10/launch-checklist/- Reading the result
- On a current WordPress install that box adds a noindex tag, not a robots.txt rule, so you would typically see only the default /wp-admin/ rule and Allowed for Googlebot, Googlebot-Image and Bingbot. Untick the box in Settings > Reading and confirm the robots tag with the Meta Tag Analyzer. Allowed only proves crawling is permitted, not that Google will index anything.
2. Archives left wide open
- Situation
- Illustrative example: a food blog has 120 posts but 340 tag pages, most holding one or two posts, plus author and search URLs. The owner suspects they compete with the recipes.
- Input
https://example-recipes.comwith paths/tag/quick-dinner/,/author/sam/,/?s=pasta,/category/soups/- Reading the result
- If no rule matches, all four read Allowed, which means robots.txt is not what keeps them out. Decide in your SEO plugin which archives get noindex, then verify the tag with the Meta Tag Analyzer. The tester cannot see noindex, so Allowed here is not a warning in itself.
3. Images blocked by a tidy-up rule
- Situation
- Illustrative example: a developer adds Disallow: /wp-content/ to hide plugin folders. Over the next weeks the 90 product photos in the uploads folder stop showing in image results.
- Input
https://example-shop.comwith the path/wp-content/uploads/2026/09/hero.webp- Reading the result
- Googlebot-Image and Googlebot would both show Blocked, with the Disallow: /wp-content/ line as the decider. Narrow the rule so uploads stay crawlable, then re-test. The result does not show how long recovery takes or whether the images will return to results.
Common mistakes
- Trusting the dashboard checkbox alone. Unticking the visibility setting does not remove a Disallow: / that a plugin or a physical robots.txt file still outputs. Always read the live file afterwards.
- Blocking archives to keep them out of Google. A Disallow on /tag/ stops crawling but not indexing, and it hides any noindex tag from the crawler. Use noindex for archives you want out of the index.
- Testing only the homepage. A rule can block /category/ or /wp-content/uploads/ while the homepage reads Allowed. Paste real post, archive and image URLs into the paths box.
- Editing a file that is not served. A physical robots.txt in the site root overrides the virtual one, so edits in a plugin editor may change nothing. Re-run the tester after saving to see what crawlers receive.
- Reading Allowed as indexed. Allowed only means robots.txt does not forbid the fetch. The page can still be noindex, redirected or ignored, so confirm in Search Console.
Frequently asked questions
Why is my WordPress site not indexed by Google?
Start with Settings > Reading and check that Discourage search engines from indexing this site is unticked. Then run the tester on your domain and look for a Disallow: / line under User-agent: *, which shuts out every crawler. If both are clean, the cause lies elsewhere: a noindex tag on the pages, a very new domain, or no sitemap submitted. The tester covers only robots.txt, so check the page's robots tag and the Pages report in Search Console for the actual reason.
Should I block tag pages in robots.txt or noindex them?
Usually noindex. A robots.txt Disallow stops crawling, but Google cannot read a noindex tag on a page it is not allowed to fetch, and a blocked URL can still show up if other pages link to it. In Yoast SEO or Rank Math you can set tag archives to noindex in the taxonomy settings. Use the tester afterwards to make sure you did not also Disallow /tag/, which would hide the noindex from crawlers.
Does WordPress have a real robots.txt file?
Not unless you made one. When no robots.txt sits in the site root, WordPress generates the response on the fly, which is why you cannot find it over FTP. The tester requests it over HTTP the way a crawler does, so it shows the generated version or your uploaded file, whichever the server returns. If the content surprises you, look for a physical file and for the SEO plugin's file editor before blaming the theme.
Yoast vs Rank Math: does the plugin change my robots.txt result?
Both can edit robots.txt from the dashboard, and either one sits on top of the same WordPress default, so the plugin choice matters less than what the live file says. Whichever you use, run the tester after saving, because a physical file on the server or a caching layer can mean the edit you made is not what crawlers receive. Keep only one plugin writing rules, otherwise you end up debugging two sets.
Does an Allowed result mean Google will index my page?
No. Allowed means robots.txt does not forbid crawling that path for Googlebot, Googlebot-Image or Bingbot. The page can still carry a noindex tag, redirect elsewhere, or be judged too thin to keep. A staging copy may show Allowed and still be noindex, and a Blocked page can remain indexed if it was indexed before or is linked. Check the robots tag with the Meta Tag Analyzer and index status in Search Console.
Reference: The Ahrefs guide to WordPress SEO tips calls the search engine visibility setting an SEO death sentence and tells owners to check it first.