No Signup Unlimited Usage Real Data 100% Free Privacy First Fast Results Export Reports Mobile Friendly

XML Sitemap Generator

Crawl a website (up to 500 URLs) and build a clean XML sitemap that lists only indexable pages: URLs blocked by robots.txt, non-200 responses, noindex pages, pages canonicalised elsewhere and non-HTML files are left out and shown with the reason. lastmod comes from the Last-Modified header.

Fetching live data…

Quick answer

The XML Sitemap Generator crawls your website from the home page and builds a ready-to-upload sitemap.xml that lists only indexable pages. URLs blocked by robots.txt, redirects, error pages, noindex pages, pages that point their canonical elsewhere and non-HTML files are left out, and each one is shown with the reason, so the file you submit sends a clean signal.

How it works

  1. Enter your website URL and choose how many URLs to crawl (100, 250 or 500) and how deep to go (2, 3 or 5 clicks from the home page).
  2. The tool fetches the start page, follows any redirect to the final address and reads robots.txt. The www and non-www versions count as one site, but every URL in the file uses the host your site actually answers on.
  3. It follows internal <a href> links level by level, eight pages at a time, within a time budget of about 45 seconds. URLs that robots.txt disallows for Googlebot are never requested, and links to images, PDFs, scripts or feeds are recognised by their extension and skipped.
  4. Every fetched page goes through the same filter: status 200, an HTML Content-Type, no noindex, and a canonical tag that points to itself. Pages that pass go into the sitemap; the rest go into the excluded list.
  5. Finally it compares the result with the sitemap you already have, scores the crawl and gives you the XML with Copy and Download buttons.

What you get

  • A valid sitemap.xml — the sitemaps.org 0.9 format in UTF-8, with absolute, XML-escaped URLs, the home page first and the rest ordered by depth.
  • Honest lastmod dates — <lastmod> is taken from each page's Last-Modified response header. If the server sends no such header, the tag is left out instead of being filled with today's date. If almost every page reports "right now", the dates are dropped as well and the report tells you why.
  • No priority or changefreq — Google says it ignores both, so the file stays short.
  • Excluded URLs with reasons — robots.txt rule and line number, redirect target, HTTP status, noindex source (meta robots, googlebot meta or X-Robots-Tag), canonical target or Content-Type, plus the page where each link was found.
  • A comparison with your current sitemap — the tool reads the sitemap listed in robots.txt (or /sitemap.xml), including up to five child files of a sitemap index, and shows which indexable pages it misses and which non-indexable URLs it lists.
  • A crawl health score — based on how many crawled URLs are indexable, broken and redirecting internal links, links to noindex pages, and whether robots.txt answers and points to a sitemap.

Why it matters

A sitemap tells search engines which URLs you consider important and canonical. Google describes it as a way to help crawlers find your pages, especially on new, large or poorly linked sites. A sitemap full of redirects, 404s and noindex pages sends mixed signals: you ask for a page to be indexed while the page itself says the opposite.

Because this generator builds the list from what a crawler actually finds and keeps only URLs that pass every indexability check, the file matches what your site really serves. The excluded list is often just as useful as the sitemap itself, because it shows broken internal links, redirect chains, stray noindex tags and duplicate URLs in one place.

How to publish your sitemap

  1. Click Download sitemap.xml and upload the file to the root of your site, so it opens at https://yoursite.com/sitemap.xml.
  2. Add one line to robots.txt: Sitemap: https://yoursite.com/sitemap.xml. The Robots.txt Generator can write the whole file for you, and the Robots.txt Tester checks that the rules do what you expect.
  3. Submit the sitemap in Google Search Console under Sitemaps, as described in Build and submit a sitemap.
  4. Fix what the excluded list shows: update internal links that point to redirects or 404s (the Broken Link Checker lists every source), and remove noindex from pages that should rank.
  5. Regenerate the file when you add or remove pages, and use the Sitemap Finder & Validator later to confirm search engines can find it. For one page at a time, the Indexability Checker explains exactly why a URL can or cannot be indexed.

Frequently asked questions

Why are some of my pages missing from the generated sitemap?

Either they did not pass a check or the crawler did not reach them. Look them up in the excluded table for the reason. If a page is not listed at all, no internal link pointed to it within the chosen depth, or the crawl stopped at the URL or time limit. The report says clearly when that happens and how many discovered URLs were not fetched.

How many URLs can it crawl?

Up to 500 URLs per run, within about 45 seconds. The sitemap protocol itself allows 50,000 URLs and 50 MB per file, so a larger site needs a sitemap index made of several files, which is usually produced by your CMS. For a full technical crawl of a bigger site, run the Technical Site Audit (Crawler).

Why does the tool not add priority and changefreq?

Google states that it ignores both values. A reliable lastmod is the field it can use, and only when the date reflects a real content change. That is why lastmod comes from your server's Last-Modified header and is left out when that header is missing.

Does it follow nofollow links and pages with query strings?

Nofollow links are followed so that their targets can be checked, and the report counts them. Query-string URLs are crawled too, but only the first 50 are queued, which protects the crawl from endless filter, session or calendar URLs. Tracking parameters such as utm_source, gclid and fbclid are removed before URLs are compared.

Learn & Resources

Guides, tips and tutorials hand-picked for this tool — go from running a check to actually fixing it.