Robots Checker PRO
Why isn't this page indexed? You get one verdict plus every signal behind it: the robots.txt rule that matched, meta robots, X-Robots-Tag, canonical and HTTP status. Evaluated with Google's real matching algorithm, across 27 crawlers including 16 AI bots.
What it does
Indexable, Noindex, Blocked by robots.txt or Canonicalised elsewhere, with every signal that produced it listed underneath.
The file parsed into groups, your crawler's group highlighted, and the winning line marked in the source. Click the verdict to jump to it.
16 bots from OpenAI, Anthropic, Google, Perplexity, Meta, Apple and Common Crawl, split into training, AI search and live fetch.
Meta robots and X-Robots-Tag broken into tokens, each marked applied or ignored. Canonical read from HTML and the Link header.
Every link tested against robots.txt, with nofollow, sponsored and ugc flagged, plus optional highlighting right on the page.
Sitemaps from robots.txt checked for status and URL count. Optionally paint a green tick or red cross on the toolbar icon.
Why it's different
Most robots checkers do a rough substring match. This one implements Google's actual matching rules from the Robots Exclusion Protocol (RFC 9309).
- Specificity, not order. The most specific user-agent group wins, rather than the first one that happens to match.
- Longest path wins. On an equal-length tie,
AllowbeatsDisallow. - Real wildcards.
*and the$end-anchor are handled properly, matched against path plus query. - Merged groups. Records sharing a user-agent are combined, the way Google combines them.
So when Googlebot has its own group with Allow: /, you'll see that it
overrides the site-wide Disallow, instead of a false alarm.
Screenshots
Changelog
Every version, newest first.
- Tabs: Status, robots.txt, Directives, AI Bots, Links, Sitemaps, Tools.
- robots.txt matcher built to Google's spec: group specificity, longest-match,
Allowwins ties,*and$wildcards, merged groups. - Crawler picker with 27 user agents; every tab recomputes for the one you choose.
- AI Bots matrix with 16 crawlers across 7 companies, split by training / AI search / live fetch, showing whether each is blocked by its own rule or inherits
User-agent: *. - Canonical read from both the HTML tag and the
LinkHTTP header, with conflicts flagged. - URL tester, JSON export, on-page link highlighting, and an optional toolbar badge that follows your selected crawler.
- Light and dark themes, both verified to meet WCAG AA contrast; full keyboard navigation.
Get Robots Checker PRO
Free on the Chrome Web Store. Private by design — everything runs locally in your browser, no account needed.
Add to Chrome → Browse other tools →