Why News Publishers need this
Being first on a story rarely pays off the way editors expect. Google tends to favour more recent articles, so a scoop can be outranked by rewrites within hours, and publishers report lower click-through when an AI Overview appears, even on page-one results. When every visit counts that much, an avoidable crawling mistake hurts twice. A Disallow line left over from a redesign, a rule that catches /live/ or /amp/ paths, or a wildcard written too broadly can keep crawlers away from exactly the breaking-news sections you most need fetched quickly. Eligibility for news surfaces also rests on pages showing clear bylines and dates, and none of that matters if the crawler is told to stay out.
The Robots.txt Tester reads the live file of the host you enter and checks the paths you list, such as an article URL, an archive page and a live-blog URL. It reports whether Googlebot, Googlebot-Image and Bingbot may crawl each one, and what the file asks of Googlebot-News and ten other tokens, so a blocked section shows up in minutes rather than after a traffic dip.
News Publishers specific guidance
What to enter first
Open the tester, type the host readers actually see, with www if that is your public address, and paste five to ten real paths into the box: a recent article, a section front, a tag page, a live-blog URL, an AMP or print variant if you still publish one, and an old archive URL. The box takes up to 20, so cover every template your CMS produces. A video or m. subdomain is a separate run. Save the list in a note so the same set can run after each deploy. A clean result proves only that the file allows those paths; it says nothing about whether they are indexed or appear in Top Stories.
How to read the crawler rows
In the URL test, look at the group used and the deciding rule with its line number. The longest matching pattern wins and Allow beats Disallow on a tie, so a shorter Allow never overrides a longer matching Disallow. An old rule near the bottom of a long file is the usual culprit. Then read News & AI crawlers. Googlebot-News follows the Googlebot group when it has none of its own, while every other token uses its own group or the * group. Each token shows Allowed, Partly blocked with an N/M paths count, or Blocked. Google-Extended and Applebot-Extended are control tokens, not crawlers. The report shows what the file asks; it cannot tell whether a bot obeys, and a firewall or CDN bot block is invisible to it.
What to fix, and in which order
Fix anything that blocks Googlebot or Googlebot-News from articles first, since that is the loss you feel immediately. Next, decide your AI policy on purpose: edit robots.txt at the web root or in your CMS so each token you want to refuse has its own User-agent group with Disallow: /, then run the check again and confirm it reads Blocked. Re-test after each edit, because adding a group for one bot means that bot stops following the * group. Last, confirm the Sitemap: lines point at your current news sitemap. Blocked means not crawled; it does not remove a URL from the index, and the tool does not see meta robots or X-Robots-Tag, so use the Meta Tag Analyzer and Search Console for those.
Example scenarios
1. A paging rule that catches live blogs
- Situation
- Illustrative example: During a redesign a developer adds Disallow: /*?* to cut parameter noise, forgetting that the live-blog template pages through updates with ?page=2 style URLs.
- Input
https://www.example-news.comwith paths/politics/2026/11/council-budget-vote,/live/election-night?page=2and/archive/2019/- Reading the result
- You would typically see the article and archive paths Allowed, while the live-blog URL shows Blocked for Googlebot with the line number of the wildcard rule. Googlebot-News, which uses the same group as Googlebot when it has none of its own, would read Partly blocked, 2/3 paths allowed, with the live-blog path listed. The fix is narrowing the rule; the result does not show whether those live pages were ever indexed.
2. An AI policy with a gap
- Situation
- Illustrative example: A publisher decides to refuse AI training crawlers but stay in search. Whoever edits the file adds groups for GPTBot and CCBot with Disallow: / and forgets ClaudeBot and PerplexityBot.
- Input
https://www.example-news.comwith three paths:/business/2026/11/rate-decision,/opinion/columnistsand/video/storm-footage- Reading the result
- GPTBot and CCBot would show Blocked with 0/3 paths allowed, while ClaudeBot and PerplexityBot fall to the * group and show Allowed, 3/3. That exposes the gap between intent and file. It does not prove GPTBot actually stays away, since the tool reads the file only.
3. The file errors during a traffic spike
- Situation
- Illustrative example: A breaking story sends traffic up sharply and the origin server starts returning 503 for /robots.txt for several minutes while article pages are still cached at the edge.
- Input
https://www.example-news.comwith one path:/world/2026/11/ferry-accident- Reading the result
- Expect a warning at the top that Google treats this as crawl nothing until the file is reachable again, and HTTP status 503 in the Summary. The rows below it are worked out from whatever the server sent with the error, usually a page with no rules, so they read Allowed; ignore them until the file returns 200. The fix is on the server or CDN side, not in the file. The result is a snapshot of that moment and says nothing about earlier fetches.
Common mistakes
- Reading Blocked as de-indexed. A Disallow stops crawling only. A blocked story can still show in results if other pages link to it.
- Putting a bot rule under the * group. Anything under User-agent: * binds every crawler that has no group of its own, including Googlebot. Aim AI restrictions at named tokens instead.
- Testing only the homepage. Rules usually bite on patterns such as /live/, /amp/, tag archives or parameters. Paste a real URL for each template you publish.
- Trusting a Blocked AI row as enforcement. The row shows what your file asks. Bots that ignore it, or a CDN rule that blocks them, are outside what this check can see.
- Running the wrong host. A staging domain, a www host and a video subdomain each serve their own robots.txt. Check the host you actually publish on.
Frequently asked questions
Does blocking GPTBot in robots.txt affect my Google News SEO?
Not by itself. GPTBot and Googlebot-News are separate tokens, each matched against its own group in the file, so a Disallow under User-agent: GPTBot does not change what Googlebot or Googlebot-News may fetch. The mistake to avoid is a rule under the * group, which every crawler without its own group follows. Run your article paths through the tester and compare the Googlebot row with the GPTBot row; they should differ if you intended them to.
Does the tester check my news sitemap?
It lists every Sitemap: line found in robots.txt, so you can confirm the news sitemap URL is declared next to your main one. It does not open the sitemap, validate its entries or count fresh story URLs. Use the sitemap finder for that step. If the line is missing, add a Sitemap: line with the full https address to the file and run the check again to see it appear in the Summary.
Will a Disallow remove a story from Google News or search?
No. A Disallow stops crawling only, and a blocked URL can still appear in results if other pages link to it. The tester says as much beside its URL test: Blocked means not crawled, not removed from the index. To take a story out of search use a noindex tag or the removal tools in Search Console, and remember noindex only works when the page is crawlable, so do not block it in robots.txt at the same time.
Can the tester tell if AI Overviews use my articles?
No. AI Overviews are a Google search feature and the tester only reads robots.txt. It can show whether Googlebot may crawl an article and whether Google-Extended, which is a control token rather than a crawler, is allowed or blocked. Those rows describe what your file asks for, not what appears in answers or how many clicks you get. To track how answer engines surface your brand, the AI visibility checker is the better fit.
How often should a news site test its robots.txt?
Run it after every deploy that touches templates, redirects or the CDN, and after any CMS or plugin update that can rewrite the file. A newsroom also benefits from a monthly pass over the AI rows, because bot lists and house policy change. Keep the same 8 to 10 paths in a note so results stay comparable. The tool runs on demand only; it does not monitor the file or alert you when someone edits it.
Reference: Eligibility for Google's news surfaces depends on visible dates, author credits and clearly disclosed sponsored content, as set out in the Google News policies in Publisher Center Help.