Not every page should be in Google. Thank-you pages, internal search results, filtered listings, staging URLs — indexing them dilutes your site and wastes crawl budget. The meta robots tag is how you keep them out, and it's routinely confused with robots.txt, which does something completely different.
The meta robots tag
It's a single line in the <head> that instructs crawlers how to treat the page:
<meta name="robots" content="noindex, follow" />name="robots" targets all crawlers; you can target one with name="googlebot". The two-part value combines an index directive and a follow directive.
noindex vs robots.txt
This is the distinction that trips everyone up:
- robots.txt Disallow controls crawling — whether Google fetches the page at all.
- meta noindex controls indexing — whether the page appears in results.
Here's the trap: if you Disallow a page in robots.txt, Google can't crawl it — which means it can't see your noindex tag. A blocked page can still get indexed (URL only, no description) from external links. So to reliably keep a page out of the index, allow crawling and use noindex. Don't do both. See robots.txt basics for the crawling side.
The directives that matter
The values you'll actually use:
index/noindex— allow or block this page from search results.follow/nofollow— follow the links on this page, or not.noarchive— don't show a cached copy.nosnippet,max-snippet,max-image-preview— control the SERP snippet.
The default (if you specify nothing) is index, follow, so you only need the tag when you want to change that. noindex, follow is the common combination for a page you want kept out of search but whose links should still be crawled.
X-Robots-Tag for non-HTML
You can't put a meta tag in a PDF or an image. For those, send the same directives as an HTTP response header:
X-Robots-Tag: noindex, nofollowThis works for any file type and can be applied server-wide by pattern — handy for keeping generated PDFs or export files out of the index.
Common mistakes
The dangerous one: a stray site-wide noindex shipped from a staging environment to production, which deindexes everything. It's the meta-tag equivalent of Disallow: /. Others: blocking a page in robots.txt and adding noindex (Google never sees the noindex); or noindexing a page you actually want to rank because a template applied it too broadly.
Checking your directives
A rogue noindex is invisible on the rendered page and can silently tank a page's traffic. SEO Inspector shows the meta robots directives for the current page front and centre — right next to the canonical — so a page that's accidentally set to noindex is impossible to miss.
