Why SEO Beginners need this
You typed site:yourdomain.com into Google and got nothing back. After weeks of reading seo basics for beginners, the obvious guess is that your content is weak, so you start rewriting titles and adding keywords. Hold off. Google can't index pages it can't reach, and one leftover line in robots.txt can tell Googlebot to stay out of the whole site. Moz's beginner material lists crawler directives among the reasons a new site may not show up at all, next to not being crawled yet and having no external links. Also remember that changes can take anywhere from hours to several months to show, so an empty search a few days after launch proves little. The Robots.txt Tester fetches the live file from the host you enter and shows, for Googlebot, Googlebot-Image and Bingbot, whether each path you list is allowed or blocked and which line decided it. Ten minutes with it tells you whether crawling is the problem or whether you should look elsewhere before touching a single page.
SEO Beginners specific guidance
Run it on the exact host with real paths
Enter your domain the way visitors reach it, for example https://www.yourshop.com, then paste up to 20 paths in the extra box, one per line: /blog/my-first-post/, /category/shoes/, /products/red-sneaker. Include the page you most want found, because a rule aimed at one folder behaves differently from a rule aimed at the whole site. The tool reads the robots.txt of exactly the host you typed, so the version with www and the version without are two separate runs. If both open in a browser, run both. Do this before rewriting a single title. A clean result here does not prove Google has indexed anything; it only says the file does not forbid crawling those paths.
Reading the result in order
Look at the HTTP status in the Summary first. A 404 gives the one-line report that no robots.txt was found and all crawlers are allowed by default, so the file is not your problem. A 5xx is worse: Google treats it as crawl nothing until the file is reachable again, and the report shows only the Summary. On a 200, go to the URL test. Each path shows Allowed or Blocked for Googlebot, Googlebot-Image and Bingbot, with the deciding rule and its line number. The longest matching pattern wins and Allow wins a tie, so a narrow Allow can override a broader Disallow. A Disallow: / under User-agent: * is the classic culprit. Blocked means not crawled; it does not by itself remove a URL from the index.
What to do after Blocked or Allowed
If a path is Blocked, open the robots.txt Content section, find the line number the report names, then edit that line in the robots.txt file at the root of your site, or in whatever setting your host or CMS offers for it. Rerun the test to confirm the change is live, then wait a few weeks before judging; changes can take hours to several months. If everything is Allowed, stop blaming robots.txt. Check that a sitemap appears under Sitemaps declared, look at the page's robots tag with the Meta Tag Analyzer, and read the index status in Search Console. Allowed does not prove the page is good, linked or indexed.
What this tool does not tell you
- It reads robots.txt only, so a noindex in the page's robots tag or in an X-Robots-Tag header is invisible to it; use the Meta Tag Analyzer for the page tag.
- It cannot tell whether a URL is indexed or why it is missing from results; Search Console shows index status.
- The News and AI crawlers section shows what the file asks, not whether a bot obeys it, and it cannot see firewall or CDN blocking.
Example scenarios
1. A new recipe blog that never appears
- Situation
- Illustrative example: a beginner builds a 15-post recipe blog on a staging copy, switches everything off for crawlers while testing, then moves it to the live domain with the file unchanged. Three weeks later site: shows nothing.
- Input
https://www.example-recipes.comwith paths /,/recipes/lentil-soup/and/blog/- Reading the result
- The Summary shows HTTP 200 and one User-agent block. The URL test marks all three paths Blocked for Googlebot, Googlebot-Image and Bingbot, with Disallow: / under User-agent: * as the deciding rule. Sitemaps declared reads 0. The leftover staging block is the first thing to fix. Blocked does not tell you whether Google indexed anything earlier.
2. Only the tag pages are blocked
- Situation
- Illustrative example: a blogger with 120 posts installed a plugin that added a Disallow for /tag/ and wonders why tag archives are missing while posts are fine.
- Input
https://www.example-blog.comwith paths/blog/first-post/,/tag/seo-tips/and/category/guides/- Reading the result
- The post and category paths show Allowed under the * group. The tag path shows Blocked with Disallow: /tag/ and its line number. That is probably intentional, so the missing tag archives are not a mistake to fix. If the post itself is missing from Google, robots.txt is not the cause.
3. No robots.txt at all
- Situation
- Illustrative example: a small portfolio site of 8 pages was built by hand and never had a robots.txt file.
- Input
https://www.example-portfolio.com- Reading the result
- The report is one line: No robots.txt found (HTTP 404). All crawlers are allowed by default. Nothing blocks crawling, so look at the sitemap, internal links and the page's robots tag next.
Common mistakes
- Blaming content before checking access. Rewriting titles and body text while a Disallow: / sits in the file wastes weeks, because Googlebot never gets to read the new copy.
- Testing only the homepage. A rule can block /blog/ or /products/ while / is Allowed, so list real post, category and product paths in the paths box.
- Treating Allowed as indexed. Allowed only means robots.txt does not forbid crawling; whether Google has indexed the URL is answered in Search Console.
- Mixing up www and non-www. Each host has its own robots.txt, so a clean result for one version says nothing about the other.
Frequently asked questions
Why is my website not showing up on Google?
There are several common reasons: the site is new and not crawled yet, nothing links to it, or a crawler directive blocks it. Robots.txt is the quickest to rule out. Run the Robots.txt Tester on your domain and a few real page paths; if Googlebot shows Blocked, a rule is keeping it out. If everything is Allowed, move on to your sitemap, page-level noindex tags and Search Console, and give a new site a few weeks before judging.
Does robots.txt stop my page from being indexed?
Not on its own. A Disallow rule only stops crawling, so a blocked URL can still appear in results if other pages link to it or if it was indexed earlier. The reverse also holds: an Allowed result does not mean the page is indexed, only that robots.txt does not forbid crawling it. To keep a page out of the index you need a noindex tag, which the Meta Tag Analyzer can check, and Search Console shows real index status.
How long until SEO changes show up in Google?
Google says changes can show up within hours or take several months, so wait a few weeks before judging any change. If you remove a Disallow line from robots.txt, rerun the Robots.txt Tester to confirm the fix is live, then be patient. Searching site: again the same afternoon will not speed anything up, and fixing the file only removes an obstacle; it does not promise that Google will index or rank the pages.
Should I pay someone who promises to submit my site to Google?
Be careful. Google states that no one can guarantee a number one ranking, and offers of priority submit or a special relationship with Google are red flags. You also stay responsible for what a hired company does with your site. Before paying anyone, spend ten minutes on the free basics yourself: check whether robots.txt blocks Googlebot, whether a sitemap is declared, and whether Search Console recognises your site.
Reference: Moz's guide to how search engines work explains why a new site may not appear at all, including when crawler directives block it.