Free robots.txt Tester & Validator
Paste your robots.txt and test any URL against it · per user-agent, including AI crawlers · using the same longest-match rules Google applies.
100% client-side · nothing you enter is ever sent to a server or stored.
Preview · sign in with a free account to run it
How to use it
- Paste the robots.txt.
- Pick a user agent (Googlebot, GPTBot, ClaudeBot, PerplexityBot and others) or type a custom one.
- Enter a path and see whether it's allowed or blocked, and which line decided it.
Example
Input
User-agent: * Disallow: /admin/ Allow: /admin/help User-agent: GPTBot Disallow: /
Result
Googlebot /admin/help allowed (line 3: Allow: /admin/help) Googlebot /admin/settings blocked (line 2: Disallow: /admin/) GPTBot /blog/post blocked (line 6: Disallow: /)
How rules are applied
- Most specific group
- A crawler follows the group naming it, such as User-agent: GPTBot, and ignores the * group completely. That's why GPTBot is blocked above even though * allows /blog.
- Longest match wins
- Within a group, the rule with the longest matching path decides (RFC 9309). On a tie, Allow wins.
- No matching rule
- Crawling is allowed by default.
- robots.txt isn't security
- It's a request well-behaved crawlers honour. It hides nothing, and listing /admin/ tells everyone where to look.
Go deeper
Page updated
Frequently asked questions
How are Allow and Disallow conflicts resolved?
Per RFC 9309, the most specific (longest) matching rule wins, and on a tie Allow beats Disallow. Wildcards (*) and end-anchors ($) are supported · which is exactly where hand-written files usually go wrong.
Does robots.txt block AI crawlers?
Compliant ones, yes · GPTBot, ClaudeBot, Google-Extended and PerplexityBot honor it. But robots.txt is a request, not access control: it hides nothing from someone who ignores it. Blocking AI crawlers also removes you from AI answers · check what that trade-off means with our AI Visibility tool.
Does Disallow remove a page from Google?
No. It stops crawling, but a blocked page can still be indexed from links. Use a noindex tag on a crawlable page to keep it out of results.
Want the full picture?
This tool covers one narrow check. A full SimplyScan runs 60 checks across 14 categories · exposed keys, RLS, XSS, auth, and speed · in 30 seconds.
Run a full security scanRelated tools
User-Agent Parser
Paste a User-Agent string · from logs or analytics · and identify the browser, OS, device class, and whether it's a known bot or AI crawler.
Secret Scanner
Paste any code, config, or bundle excerpt and get flagged secrets · AWS keys, Stripe live keys, OpenAI keys, GitHub tokens, JWTs, private key blocks and more.
.env File Linter
Paste your .env and get instant findings · secrets that will ship to the browser, duplicate keys, quoting problems · plus a generated .env.example.