Robots.txt Generator for Webflow Designers

Why Webflow Designers need this

You launch a client site, and a week later the webflow.io staging address shows up in search results, a near-copy of the live pages. Many Webflow designers hit this sooner or later, and the first instinct is to block it with robots.txt. That only controls crawling, not indexing, so pages Google already holds will not disappear because of a Disallow line. Large CMS builds add their own strain: thousands of collection items, limited access to the underlying code, and no native schema feature, which leaves a lot of hand-managed work. Robots.txt is one of the few crawl controls you can edit directly in Site settings, under the SEO tab, so it is worth getting right before publishing. The Robots.txt Generator builds the text for that box: you enter a sitemap URL, list the paths to disallow one per line, choose a default policy, and copy the output. It writes one User-agent: * group and nothing fancier. It does not check your paths, publish anything or test the result, so treat its output as a draft to paste, publish and then verify.

Webflow Designers specific guidance

Decide what needs blocking before you generate

Open Site settings, then the SEO tab, and look at what the Robots.txt field already contains before generating anything. Next, write down the paths that should stay out of crawling: the /search results page, a members-only folder, a folder of thank-you pages you do not want found directly. Type those into the Disallow paths box, one per line, each starting with a slash, and leave the default policy on Allow all robots. For the webflow.io address, use the Disable Webflow subdomain indexing option in the same SEO tab instead of leaning on robots.txt. What this does not prove: a Disallow line does not make a page private, and it does not remove anything Google has already stored.

Read the generated block line by line

The output always opens with User-agent: * and then one Disallow line per path, or an empty Disallow: when you listed none. A Sitemap line follows if you filled in the field, so enter the full address, for example https://yourdomain.com/sitemap.xml, using the same domain you set as default under the Publishing tab of Site settings. Scan the first Disallow line with your own eyes: if it reads Disallow: / on a live site, you picked Block all robots and every path line was ignored. If you rely on Webflow's auto-generated sitemap, turn that on in the same SEO tab first. The tool does not fetch the sitemap URL, so a typo in it will not be flagged.

Paste, publish, then test the live file

Copy the output, paste it into the Robots.txt field, save, and publish the site to your custom domain, because nothing changes on the live site until you publish. Then open yourdomain.com/robots.txt in a new tab and compare it with what you pasted. Run one blocked URL such as /search and one CMS item URL such as /blog/post-slug through the Robots.txt Tester to confirm that one is stopped and the other is not. Allow lines, crawl-delay or a second user-agent group must be typed in by hand, and nothing checks them for you. A passing test shows crawlers may fetch a URL, not that Google will index it.

Example scenarios

Common mistakes

• Using robots.txt to hide the webflow.io staging copy. Blocking the staging address stops crawling but does not remove URLs already in Google. Use the Disable Webflow subdomain indexing option in the SEO tab as well.
• Leaving Block all robots selected after launch. That setting outputs Disallow: / and ignores every path you typed. Fine for a private build, harmful on a live domain.
• Forgetting to publish after pasting. The Robots.txt field is saved in settings, but the live file only changes when you publish the site. Always open yourdomain.com/robots.txt afterwards.
• Blocking a path and expecting it to disappear from Google. A Disallow rule controls crawling, not indexing, so a linked URL can still show up in results. Removal needs a different approach than robots.txt.
• Trusting hand-added lines. Allow lines, crawl-delay or a second user-agent group typed into the field are not validated by the generator. A typo can block more than intended, so test the published file.

Frequently asked questions

How do I disable webflow.io subdomain indexing?

Open Site settings, go to the SEO tab and switch on the option to disable Webflow subdomain indexing, then publish the site. That setting is Webflow's own control for the webflow.io address, so use it rather than writing staging rules into your live robots.txt; keep in mind that robots.txt rules control crawling rather than indexing. Pages Google has already stored will not vanish because of a Disallow line, so check the results with a site: search a few days later. The Robots.txt Generator is meant for the live domain's rules, not for removing old staging URLs.

Where do I add a robots.txt file in Webflow?

In Webflow, open Site settings, choose the SEO tab and find the Robots.txt field. Paste the generated text there, save, and publish the site to your custom domain. The file then appears at yourdomain.com/robots.txt, which you should open in a browser to confirm it matches what you pasted. Editor changes do nothing until you publish, so a rule that seems ignored is often just unpublished. The generator itself cannot install or publish anything.

Can I hide a CMS collection from Google with a Disallow rule?

You can stop crawlers fetching a folder such as /team-members/ by entering that path in Disallow paths, but this controls crawling, not indexing. A URL can still appear in results if other pages link to it. Keeping items out of the index usually calls for a noindex tag, and crawlers can only read that tag if they are allowed to fetch the page. So avoid combining a Disallow and a noindex on the same URLs.

Is Webflow good for SEO if I only change robots.txt?

Robots.txt is a small part of the picture. Webflow gives you control over titles, meta descriptions and CMS bindings, yet it has no built-in schema feature, so JSON-LD goes in as custom code, and very large CMS sites with thousands of items are harder to manage. A tidy robots.txt keeps crawlers away from search and private pages, but it says nothing about content quality, rankings or whether your templates output the right tags.

Can the generator add crawl-delay or Allow lines for my Webflow site?

No. It writes a single User-agent: * group with Disallow lines and an optional Sitemap line. Allow lines, crawl-delay, a second user-agent group or a second Sitemap line have to be typed into the Webflow Robots.txt field by hand, and the tool will not validate them. After adding anything manually, publish and run the live file through the Robots.txt Tester so a typo does not block more than you meant.

Reference: Branded Agency's Webflow SEO guide notes that the default webflow.io staging domain can cause duplicate content if indexed, and that robots.txt controls crawling, not indexing.