XML Sitemaps and robots.txt, Explained Without the Jargon

Two small text files control how search engines see your entire website. One is a map, the other is a bouncer — and most site owners have never opened either. Here’s what they do, what they don’t do, and exactly how to set them up correctly.

Updated August 2026 · 15 min read

The most expensive SEO mistake I’ve ever seen wasn’t a bad link profile or a thin content strategy. It was a single line in a text file. A small e-commerce site had rebuilt their store, and the developer left a staging-site rule behind: Disallow: /. Two words and a slash. That line told every search engine crawler on the internet to stay away from the entire site. Traffic fell off a cliff over six weeks, and nobody noticed until a customer mentioned they couldn’t find the store on Google anymore. The fix took thirty seconds. The recovery took months.

That file was robots.txt. Together with the XML sitemap — a file most people have heard of but couldn’t explain at gunpoint — it forms the front door of your site as far as search engines are concerned. Before Google reads a single word of your content, it asks robots.txt where it’s allowed to go, and checks your sitemap for a list of what’s worth visiting.

The good news is that these two files are genuinely simple. Not “simple once you understand the eight prerequisites” simple — actually simple. By the end of this guide you’ll know what each file does, what the common traps are, and how to set both up correctly in under half an hour, whether you’re on WordPress or something else entirely.

What an XML sitemap actually is (and isn’t)

An XML sitemap is a list. That’s it. It’s a file — usually at yoursite.com/sitemap.xml — that lists the URLs on your site you’d like search engines to crawl and index. Each entry can include a little extra information: when the page was last modified, and optionally how important you consider it relative to your other pages.

Think of it as handing Google a table of contents. Google is perfectly capable of finding your pages by following links, and on a small, well-linked site it will find most of them eventually. The sitemap just makes the job faster and more reliable — especially for pages that are new, buried deep in your site structure, or not linked from many other pages yet.

What a sitemap is not is a ranking factor. Putting a page in your sitemap doesn’t make it rank. It doesn’t even guarantee Google will index it — Google treats sitemaps as a hint, not an instruction. If a page is thin, duplicated, or low quality, listing it in a sitemap changes nothing. The sitemap’s job is discovery, not promotion.

XML sitemap vs. HTML sitemap

Worth clearing up because the names trip people: an HTML sitemap is a page on your site, written for humans, that links to your main sections — like the index at the back of a book. An XML sitemap is a machine-readable file written for crawlers. This article is about the XML one. HTML sitemaps are a nice-to-have for large sites and a legitimate internal linking tool (see my internal linking strategy guide for more on that), but they’re not what Google asks for in Search Console.

What robots.txt actually does

robots.txt is a plain text file that lives at the root of your domain — yoursite.com/robots.txt. It’s the first thing a well-behaved crawler requests when it visits your site. The file contains rules, and each rule says: this group of crawlers may (or may not) access these paths.

A minimal, perfectly valid robots.txt looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Line by line: User-agent: * means “the following rules apply to every crawler.” Disallow: /wp-admin/ means “don’t crawl anything under the /wp-admin/ path.” The Allow line carves out one exception, because WordPress uses that specific file to load content on public pages.

Now the part that surprises almost everyone: robots.txt does not hide anything. It’s a request, not a lock. Well-behaved crawlers (Google, Bing, reputable tools) honor it. Scrapers, spammers, and bad bots ignore it completely. And more subtly — a page blocked by robots.txt can still appear in Google search results. If other sites link to a blocked page, Google knows it exists and may list it with no description, because it was never allowed to read the content. If you truly want a page out of Google’s index, robots.txt is the wrong tool; you want a noindex tag or password protection.

A sitemap tells Google where to look. robots.txt tells it where not to bother. Neither one makes a page rank — they just make crawling efficient.

How the two files work together

These files are complementary, and the cleanest way to keep them straight is to think about their opposite purposes:

One detail ties them together neatly: it’s good practice to put a line at the bottom of robots.txt pointing to your sitemap, like Sitemap: https://yoursite.com/sitemap.xml. Crawlers that read robots.txt will then find your sitemap even if you never submit it anywhere. It costs one line and there’s no downside.

There’s also a coordination rule that matters: never block a page in robots.txt that you want indexed. If Google can’t crawl the page, it can’t see the content, the canonical tags, or the noindex tag — so you lose control of how it’s handled. Pages you want indexed should be crawlable and listed in the sitemap. Paths you don’t care about (admin screens, internal search results, cart pages) can be blocked in robots.txt and left out of the sitemap.

Setting up your XML sitemap

On WordPress (the easy mode)

You almost certainly already have a sitemap and don’t know it. WordPress has generated a basic one at /wp-sitemap.xml by default since version 5.5. Visit your domain with that path appended and you’ll see it. It works, but it’s bare-bones — no images, limited control over what’s included.

Most sites use an SEO plugin instead, and this is the route I’d take. Yoast SEO and Rank Math (both free) generate a proper sitemap at /sitemap_index.xml, which is an index of smaller sitemaps — one for posts, one for pages, one for categories, and so on. The plugin version is better in three ways: it updates instantly when you publish, it lets you exclude specific post types or taxonomies from a settings screen, and it pings search engines when things change. If a plugin sitemap is active, it takes over from the core one, so you don’t have to worry about duplicates.

Outside WordPress

Shopify, Squarespace, Wix, and Webflow all generate sitemaps automatically — usually at /sitemap.xml — with no action needed from you beyond submitting it. If you’re running a custom or static site, you have options: generators like Screaming Frog (free up to 500 URLs) will crawl your site and export a valid sitemap, or your static site generator almost certainly has a sitemap plugin (Hugo, Jekyll, Next.js, and Astro all do). Writing one by hand is fine for tiny sites — the format is just a <urlset> wrapper around <url> entries with <loc> and <lastmod> tags — but anything generated is less error-prone.

Whatever generates it, submit it to Google Search Console: open the Sitemaps report in the left sidebar, paste in the sitemap URL, and hit Submit. Google will show you when it was last read, how many URLs it found, and any errors it hit. If you’re new to that whole dashboard, my Google Search Console guide walks through setup from scratch.

What belongs in your sitemap — and what doesn’t

This is where people quietly sabotage themselves. A sitemap isn’t a dump of every URL your site can produce. It’s a curated list of pages you’d be proud to have indexed. A bloated sitemap full of junk URLs wastes Google’s crawl time on your site and can dilute how much attention your real pages get.

Run your sitemap against this checklist:

• Include only canonical, indexable pages. If a page has a noindex tag, redirects somewhere else, or canonicalizes to another URL, it should not be in the sitemap. Mixed signals confuse the crawler.
• Include only 200-status URLs. No 404s, no redirect chains. A sitemap that points to dead ends is telling Google your housekeeping is sloppy.
• Include only absolute URLs with the correct protocol. All HTTPS, all matching your preferred domain version (www or non-www — whichever you actually use). A sitemap listing http:// URLs on an https:// site is a classic migration leftover.
• Exclude internal search results, filters, and tag archives. WordPress tag pages, date archives, and ?filter=color parameter URLs create near-duplicate pages by the thousand. Keep them out.
• Keep individual sitemap files under the limits. 50,000 URLs or 50 MB uncompressed per file. Big sites split into multiple sitemaps referenced by an index file — which is exactly what the SEO plugins do automatically.
• Keep lastmod dates honest. Updating the modified date on every URL every day to look “fresh” is an old trick Google explicitly ignores. Set it when content genuinely changes.

Writing a correct robots.txt

Here’s a sensible default for a WordPress site — the same bones I use on nearly every project:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://yoursite.com/sitemap_index.xml

What each piece does: the wp-admin block keeps crawlers out of your dashboard (which they can’t usefully index anyway), with the ajax exception so public page features still work. The two search lines block your internal search result pages — an endless pit of thin, duplicate URLs on most sites. The Sitemap line points crawlers at your map. Everything else on the site is crawlable by default, which is what you want.

A few rules of the road that save real pain:

• One file, one location. It must live at the domain root: yoursite.com/robots.txt. A robots.txt in a subfolder is ignored entirely.
• User-agent blocks come first. Rules are grouped: a User-agent line starts a group, and the Disallow/Allow lines that follow apply to it. Most sites only need the * group. Add specific blocks (like User-agent: GPTBot with Disallow: /) only when you have a reason.
• Paths are prefixes. Disallow: /blog blocks /blog, /blogging, and /blog/anything. Add the trailing slash if you mean just the folder: Disallow: /blog/.
• Disallow with no value means allow everything. An empty Disallow: is not a block. The dangerous line is Disallow: / — that slash means the whole site. This is the staging-site trap from the intro.
• Never use robots.txt for security. Blocking /secret-page/ in robots.txt is like announcing “there’s something at /secret-page/ I’d rather you didn’t look at” — publicly, in a file anyone can read. Use authentication or noindex for genuinely private content.

The mistakes I see over and over

After auditing a lot of sites, the same handful of errors come up constantly. Check your own setup against this list:

1. Blocking pages you want indexed. Usually a leftover from a redesign or a staging migration. Symptom: Search Console shows pages as “Blocked by robots.txt” that should be live in search. Fix: remove the offending Disallow line, then request indexing on the affected URLs.

2. Blocking CSS and JavaScript. An old-school practice of disallowing /wp-includes/ or /assets/ to “save crawl budget.” Bad idea — Google renders your pages like a browser does, and if it can’t load your stylesheets and scripts, it sees a broken page and may judge it as not mobile-friendly. Let it crawl your assets.

3. Sitemap full of non-canonical URLs. The sitemap lists http versions, www versions, trailing-slash variants, or parameter URLs while the site has moved on. Every entry should resolve to a 200-status canonical page. If you’re seeing lots of “Discovered – currently not indexed” or redirect reports, my guide to fixing crawl errors and indexing problems covers the diagnosis workflow.

4. Assuming robots.txt removes pages from Google. It doesn’t — it prevents crawling, not indexing. To deindex a page, allow crawling and add a noindex meta tag, so Google can actually read the instruction.

5. Five SEO plugins fighting over the sitemap. I’ve seen sites with two plugins both generating sitemaps at different URLs, listing different pages. Pick one tool, disable the other’s sitemap feature, and submit the single canonical sitemap to Search Console.

Testing both files before you call it done

Setup isn’t finished until you’ve verified. This takes ten minutes and catches everything on the mistakes list.

For robots.txt: open yoursite.com/robots.txt in your browser and read every line slowly, asking “does this block anything public?” Then use the URL Inspection tool in Search Console on a few important pages — it will tell you flat out whether crawling is allowed for that URL and which rule is blocking it if not. (The old standalone robots.txt Tester was retired; URL Inspection is the current way.)

For the sitemap: load the sitemap URL itself and confirm it renders a list of real URLs. Then check the Sitemaps report in Search Console for a green “Success” status and a discovered-URL count that roughly matches your real page count. If you have 40 pages and the sitemap reports 3,000 URLs, you’ve got archive and parameter junk in there. Finally, cross-check with the Pages indexing report: the pages you care about should show as indexed, and the exclusions should be pages you deliberately excluded.

Google’s own sitemap documentation and robots.txt guide are the authoritative references if you want the full spec — both are refreshingly readable as technical docs go.

What neither file can do for you

Setting these files up correctly is real SEO work, but keep it in proportion. A clean sitemap and a correct robots.txt remove friction between your content and the crawler. They don’t create demand, authority, or relevance.

If your pages aren’t ranking, the cause is almost never the sitemap. It’s the content quality, the search intent match, the internal linking, or the lack of backlinks. The technical files are the foundation — a house needs one, but the foundation isn’t why anyone buys the house. Spend your thirty minutes here, get it right, verify it, and then put your energy back into content and links, where rankings actually come from.

The one ongoing habit worth keeping: after any site migration, redesign, CMS change, or domain move, re-check both files first. Every horror story in this article — including the store that vanished from Google for six weeks — traces back to a migration where nobody looked at robots.txt on launch day. Make it the first item on your launch checklist and you’ll never have one of those stories of your own.

Setup verified against Google Search Central documentation as of August 2026. Interface details in Search Console shift occasionally, but the underlying file formats and rules have been stable for years. Related reading on this site: Google Search Console guide, fixing crawl errors and indexing problems, and the one-hour SEO audit checklist.