XML sitemaps: what to include and what to leave out
An XML sitemap is a hint list of URLs you consider important and eligible for indexing—not a dump of every path your CMS can generate. Bloated sitemaps waste crawl attention and dilute trust in your lastmod dates. Too-thin sitemaps hide new sections. This article gives a clear include/exclude policy, maintenance habits, and worked examples you can adapt without inventing vanity metrics.
1. What a sitemap is for (and what it is not)
Search engines discover URLs through links first. Sitemaps supplement discovery and help crawlers learn about new or updated URLs faster, especially on large sites, sites with sparse internal linking, or sites that publish many URLs at once. A sitemap does not force indexing, guarantee rankings, or override stronger signals such as noindex, canonical conflicts, or soft 404 behavior.
Think of the sitemap as your public inventory of “please consider these canonical, indexable URLs.” Every URL in that inventory should return a successful response, represent content you want in search, and match the URL you prefer elsewhere on the site. If that standard feels strict, good—strictness is the point.
For foundational crawl/index concepts, see the practical SEO guide. For building and checking files, use the XML sitemap generator and XML sitemap validator on Free SEO Hub, then apply the policy rules below rather than exporting every database row.
2. What to include by default
Include a URL when all of the following are true:
- It returns HTTP 200 (or a deliberate soft success you stand behind—not an error page dressed as 200).
- It is indexable: no
noindexmeta/header, not blocked from crawlers in a way that prevents indexing intent. - It is the canonical URL for that content (self-canonical or the target of consolidations).
- It offers unique, intentional value: a real article, product, category, documentation page, or landing page you would proudly share.
- It is meant for search discovery, not only for logged-in users or transactional completion.
Strong include candidates
- Home page and primary section hubs.
- Published blog posts, guides, and evergreen resources.
- Product detail pages that are in stock or intentionally kept as archival product pages with useful content.
- Category and collection pages that are curated, not infinite filter combinations.
- Help center articles and public documentation.
- Location or service pages that are substantive and unique.
If you maintain hreflang, each language/region canonical can appear in the appropriate sitemap strategy your stack supports—just keep language clusters consistent with on-page annotations. The hreflang generator helps with markup; the sitemap should still list only URLs that meet the include rules.
3. What to leave out
Exclude aggressively. A shorter, cleaner sitemap is usually more useful than a complete dump of your router table.
Always leave out
- URLs with
noindex(thank-you pages, many filtered views, internal utilities). - Non-canonical duplicates: parameter variants, print views, session IDs, alternate sort orders.
- Redirecting URLs (3xx). List the destination instead.
- Error responses (4xx/5xx) and soft 404s that show “no results” with a 200 status.
- Login, cart, checkout, account, and wishlist flows.
- Internal search result pages.
- Staging, preview, and draft URLs.
- Paginated admin-like archives with thin or near-duplicate listings if they are not strategic hubs.
- Facet combinations that explode combinatorially (
color×size×brand×sort). - Tracking-parameterized copies of otherwise good pages.
Usually leave out (unless you have a deliberate strategy)
- Tag pages that only reorganize existing posts without unique commentary.
- Author pages with almost no bio or unique links.
- Out-of-stock products that 200 with empty shells—either enrich them, redirect, or exclude until fixed.
- Auto-generated city pages that repeat the same boilerplate.
Robots.txt and sitemaps should agree in spirit. If you Disallow a pattern to reduce crawl waste, do not list matching URLs in the sitemap. If you noindex a template, remove it from the sitemap even if crawlers can still fetch it. For the broader directive map, read robots.txt vs noindex vs canonical.
4. Honest lastmod, priority, and changefreq
lastmod is useful when it reflects a real content or meaningful template change. It becomes noise when every URL is stamped with today’s date on each sitemap regeneration. Search systems can learn to ignore unreliable dates. Update lastmod when the visible content changes, when a product’s critical attributes change, or when a guide is substantively revised—not when a sidebar timestamp or ad unit rotates.
priority and changefreq are widely treated as weak or ignored signals. Do not spend editorial energy fine-tuning them. If your generator requires values, use simple defaults and invest effort in URL selection and accurate lastmod instead.
A workable editorial rule: bump lastmod only when a human would notice the page changed, or when structured data / primary media that search cares about changed. Cosmetic CSS refreshes do not need a sitewide lastmod wave.
5. Index files, splits, and size limits
Large sites should split sitemaps by type—products, categories, articles, videos—then reference them from a sitemap index file. Splits make debugging easier: if product URLs start failing validation, you can regenerate one file without touching the blog inventory.
Respect format limits (URL count and uncompressed file size caps defined by the sitemap protocol). More important than maxing those limits is keeping each file thematically coherent. Reference the sitemap index from robots.txt with a Sitemap: line, and submit the index in Search Console.
Image and video sitemap extensions are worthwhile when media is central to discovery (recipes with photos, product galleries, course trailers). Do not add image entries for decorative icons. The extension should describe media that is genuinely part of the page’s purpose.
6. Worked examples
Example A: 200-URL marketing site with a blog
Include: home, service pages, about, contact (if indexable and useful), each published article, one resources hub.
Exclude: thank-you page after contact form, preview URLs for drafts, tag archives with one post each, PDF mirror URLs that duplicate HTML guides, UTM copies.
Process: Generate from CMS “published + indexable” status only. Validate sample URLs with the sitemap validator and spot-check canonical tags with the canonical URL generator patterns your templates use.
Example B: Mid-size ecommerce catalog
A store has 8,000 products, 120 categories, and millions of potential filter URLs.
Include: in-stock and strategically kept product URLs, curated categories, a handful of campaign landing pages with unique copy, size guides and care guides.
Exclude: all multi-filter combinations, internal search, cart/checkout, wishlist, sort-order variants, out-of-stock shells that show empty templates, old promo query URLs.
Architecture: sitemap-products-1.xml … sitemap-products-n.xml, sitemap-categories.xml, sitemap-content.xml, wrapped by sitemap-index.xml. Regenerate product files when inventory or PDP content changes; do not rewrite lastmod on untouched products.
Example C: SaaS documentation
Include: each public doc article, API reference pages that are public, changelog entries that are substantive.
Exclude: logged-in-only workspace URLs, search-within-docs result pages, outdated redirects still linked in the sidebar (fix links first, then ensure destinations are listed), versioned duplicates if you canonicalize to “latest.”
Tip: When you deprecate a feature, 301 the doc to the successor and remove the old URL from the sitemap in the same release whenever possible.
Example D: News site publishing daily
Include: article URLs as they publish; section fronts if they are curated.
Exclude: minute-by-minute wire dumps that are thin duplicates, printer views, “live blog” pagination fragments that are not meant to stand alone.
lastmod: update when the article text updates for corrections or new developments. A ticker widget changing every minute should not rewrite lastmod.
Example E: Cleaning a polluted sitemap
Search Console shows thousands of “Discovered – currently not indexed” URLs that match filter patterns listed in the sitemap.
Actions: 1) Export sitemap URLs. 2) Filter out any URL with more than one facet parameter. 3) Drop noindexed templates. 4) Replace redirects with destinations. 5) Resubmit the cleaned index. 6) Use the weekly routine in Search Console checklist to confirm the junk backlog stops growing.
7. Quality assurance before you submit
- Parse the XML; ensure it is well-formed.
- Sample 20–50 URLs across types: fetch status codes, canonical targets, robots meta.
- Confirm every sampled URL matches the host you verified in Search Console (www vs non-www, http vs https).
- Confirm the sitemap is reachable and referenced from robots.txt.
- Compare count trends: a sudden jump often means a generator bug included filters or previews.
- After major template changes, re-run the SEO audit checker on a few listed URLs to catch accidental noindex or canonical drift.
If you use multiple tools environments, keep a single source of truth in the CMS or build pipeline. Manually edited “shadow” sitemaps diverge quickly and create conflicting inventories.
8. Maintenance cadence
On every publish: add new indexable URLs automatically; set accurate lastmod.
On every unpublish or consolidate: remove old URLs; prefer redirects for retired content that has replacements.
Weekly: glance at sitemap processing in Search Console and spot-check a random sample—especially after releases.
After migrations: rebuild from the new canonical host only; keep old host sitemaps from advertising obsolete URLs.
Sitemaps are part of technical hygiene alongside robots rules and canonical consistency. Browse related posts in the blog, and use sitemap tools when you need generators and validators in one place.