Why Webflow Designers need this
The staging address that Webflow gives every project can end up in Google and sit beside the live site as a duplicate. Many designers respond by blocking it and assume the matter is closed, but robots.txt controls crawling, not indexing, so pages already in the index stay there. Meanwhile the live domain may carry a rule nobody remembers adding, and on a CMS with thousands of items you have limited access to the underlying code to notice it. The Robots.txt Tester reads the file from the host you enter, groups the rules by user-agent, lists any Sitemap lines, and tells you whether Googlebot, Googlebot-Image and Bingbot may crawl the paths you paste in. Run it once on yoursite.webflow.io and once on your custom domain, because they are separate runs and you do the comparing. What it cannot tell you is whether a page is noindex or already indexed; that needs the page's robots tag and Search Console. Treat the result as one half of a launch check: it answers who is allowed to fetch what, nothing more.
Webflow Designers specific guidance
What to check first
Open the tool and enter yoursite.webflow.io, then paste three paths into the optional box: /, a real CMS item such as /blog/first-post, and its collection list page /blog. Read the HTTP status before anything else. A 404 means no file exists, so every crawler is allowed by default. Then repeat with your custom domain, for example www.yoursite.com, since the two hosts are read separately. Write down the deciding rule and its line number for each path. What this does not prove: an Allowed result on the staging host says nothing about a noindex tag, so a staging site can be open to crawlers and still be marked noindex, or the other way round.
How to read the result
Look at the URL test table first. For each path it names the group used, Allowed or Blocked, and the line that decided. The longest matching pattern wins and Allow wins a tie, so a short Disallow: /blog does not beat Allow: /blog/ on a post. If the live domain shows Blocked for a CMS collection path, find that rule under Rules by user-agent and compare it with the raw file for stray characters. A Disallow: / under User-agent: * on the live domain is the pre-launch leftover to remove. A 5xx status deserves attention too, since Google treats an unreachable file as crawl nothing until it returns. Blocked never means removed from the index, and a blocked staging URL that was indexed earlier or is linked can still appear in results.
What to fix and in which order
Fix the live domain first, because a blocked section there costs more than a visible staging copy. Edit the robots.txt rules in your project's SEO settings, publish, and re-run the tester on the custom domain with the same paths. Change one thing at a time so the line number of each rule stays traceable. For the webflow.io address, open the tags on a few pages with the Meta Tag Analyzer and look in Search Console for pages already indexed, because a clean robots.txt result says nothing about either. Last, confirm the Sitemap lines point to the live host, not the staging one, and put the final results in your handoff notes.
Example scenarios
1. Staging address left wide open
- Situation
- Illustrative example: A designer duplicates a client project for a 60-page redesign, and the staging link gets pasted into a public forum thread where crawlers can follow it.
- Input
- yoursite.webflow.io with paths /,
/servicesand/blog/first-post - Reading the result
- If the file has no rule covering these paths, the URL test would show Allowed for Googlebot, Googlebot-Image and Bingbot. That only means crawling is possible. Next, read each page's robots tag with the Meta Tag Analyzer and search Search Console for staging URLs, because the tester cannot see noindex or index status.
2. Leftover block on the live domain
- Situation
- Illustrative example: A freelancer launches a 40-page brochure site, but a Disallow: / rule written for the review phase is still in the robots.txt rules of the published project.
- Input
- www.yoursite.com with paths /,
/blogand/blog/first-post - Reading the result
- The Summary would show HTTP status 200 and at least one Disallow rule, and the URL test would show Blocked for all three crawlers with the deciding line number, for instance line 2. Remove the rule, publish, and run the same three paths again until they read Allowed.
3. Large CMS and lookalike paths
- Situation
- Illustrative example: A directory built on 3,000 CMS items blocks /blog-drafts/ but must keep /blog/ open, and the designer is unsure which rule wins on a post.
- Input
- www.yoursite.com with 20 paths, one per template, such as
/blog/first-post,/blog-drafts/first-postand/categories/seo - Reading the result
- The URL test names the longest matching pattern for each path, and Allow wins an exact tie. Pick one path per template rather than all 3,000 items. The Summary also lists the Sitemaps declared, so you can check that line against the host you intend.
Common mistakes
- Testing only the custom domain. The webflow.io host is read on its own, so a clean result on www.yoursite.com tells you nothing about the staging copy. Run both hosts every time.
- Treating Blocked as removed. A Disallow stops crawling only, and pages already in Google or linked from elsewhere can stay in results. Use the page's robots tag and Search Console to deal with indexed URLs.
- Trusting Allowed as proof of indexability. Allowed means a crawler may fetch the path, not that the page is indexable. A noindex tag can still be present, and the tester does not read it.
- Testing only the homepage. A rule can hit a collection path like /blog while / stays open. Paste one real URL per CMS template into the paths box.
- Skipping the re-run after publishing. Edits in the SEO settings only matter once the published file changes. Run the tester again after every publish and compare the line numbers.
Frequently asked questions
How do I disable webflow.io subdomain indexing?
The tester does not change anything; it only shows what the webflow.io host currently tells crawlers. Enter yoursite.webflow.io with a few paths, and see whether each is Allowed or Blocked and which line decides. Keep in mind that robots.txt controls crawling, not indexing, so a block does not remove pages already in Google. Check the page's robots tag with the Meta Tag Analyzer and the index status in Search Console before you assume the staging copy is gone.
Is Webflow good for SEO when robots.txt is this easy to get wrong?
Webflow can work well for SEO, but it is hands-on: schema markup, for instance, is usually added as custom JSON-LD or through an app. Robots rules are the same kind of manual check. The tester covers only the crawl side: it shows the status, the rules per user-agent and the Sitemap lines. It says nothing about rankings, speed or content quality, so use it as one step of a checklist, not as a verdict on the platform. Run it on both hosts before each handoff.
What belongs on a Webflow SEO checklist for robots.txt before launch?
Run the live domain and read the HTTP status, confirm no Disallow: / sits under User-agent: *, paste your main CMS paths such as /blog/first-post and /blog and confirm Allowed for Googlebot and Bingbot, and note the Sitemap lines. Then run the webflow.io host separately. Afterwards check each page's robots tag with the Meta Tag Analyzer, since the tester cannot see noindex. Keep the line numbers of the deciding rules in your handoff notes. Do this after the final publish, not before.
Does blocking webflow.io in robots.txt remove it from Google?
No. A Disallow only stops crawling, and a URL that was already indexed or is linked from elsewhere can still appear in results. Google also cannot see a noindex tag on a page it is not allowed to fetch, so blocking can get in the way of cleanup. Decide the order deliberately, check the current tag with the Meta Tag Analyzer, and track removal in Search Console, which the tester does not read. Treat the tester result as the crawl picture only.
Why do webflow.io and my own domain show different results?
Because each host serves its own robots.txt and the tool reads exactly the host you enter. yoursite.webflow.io and www.yoursite.com are two separate runs, and the comparison is yours to make. A difference is not an error by itself; it is a reason to ask which one you intended. If one returns 404 and the other has rules, the 404 host allows all crawlers by default. Compare the raw file text of both for stray characters.
Reference: Branded Agency's Webflow SEO guide notes that the default webflow.io staging domain can cause duplicate content if indexed and that robots.txt controls crawling, not indexing.