These two files are the front door to how search engines crawl your site. Neither is complicated, but a small mistake in either can quietly cost you traffic. Here's what each does and how to keep them correct.
robots.txt
Served at /robots.txt, this file tells crawlers where they may and may not go. A minimal, correct file looks like:
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xmlKey point: Disallow controls crawling, not indexing. A disallowed page can still appear in results (without a description) if other sites link to it. To keep a page out of the index, let it be crawled and use a noindex meta tag instead.
The classic mistake
Never block your CSS and JavaScript. Google renders pages like a browser; if you Disallow: /assets/, it can't see your layout and may judge the page as broken or non-mobile-friendly. Only block genuinely private or duplicate areas (admin, cart, faceted-search URLs), and never accidentally ship Disallow: / from a staging config to production — it deindexes the entire site.
More sites are hurt by an over-eager robots.txt than helped by a clever one.
sitemap.xml
A sitemap lists the URLs you want indexed, with optional metadata:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/</loc>
<lastmod>2026-06-24</lastmod>
</url>
</urlset>Only include canonical, indexable URLs that return 200 OK. Don't list redirects, 404s, or noindex pages — a sitemap full of errors erodes trust in the whole file. Reference it from robots.txt and submit it in Google Search Console.
Keeping them healthy
Sitemaps rot. Pages get deleted, URLs change, redirects pile up — and your sitemap keeps pointing at them. Periodically crawl your own sitemap and check every URL's status, so redirects (3xx) and errors (4xx/5xx) don't sit there unnoticed. SEO Inspector includes a sitemap validator that fetches your sitemap (and any sub-sitemaps) and lists exactly which URLs return a redirect or an error.
