Free robots.txt Tester & Validator
Paste your robots.txt and test any URL against it · per user-agent, including AI crawlers · using the same longest-match rules Google applies.
Frequently asked
How are Allow and Disallow conflicts resolved?
Per RFC 9309, the most specific (longest) matching rule wins, and on a tie Allow beats Disallow. Wildcards (*) and end-anchors ($) are supported · which is exactly where hand-written files usually go wrong.
Does robots.txt block AI crawlers?
Compliant ones, yes · GPTBot, ClaudeBot, Google-Extended and PerplexityBot honor it. But robots.txt is a request, not access control: it hides nothing from someone who ignores it. Blocking AI crawlers also removes you from AI answers · check what that trade-off means with our AI Visibility tool.
Related tools
- User-Agent Parser · Paste a User-Agent string · from logs or analytics · and identify the browser, OS, device class, and whether it's a known bot or AI crawler.
- Secret Scanner · Paste any code, config, or bundle excerpt and get flagged secrets · AWS keys, Stripe live keys, OpenAI keys, GitHub tokens, JWTs, private key blocks and more.
- .env File Linter · Paste your .env and get instant findings · secrets that will ship to the browser, duplicate keys, quoting problems · plus a generated .env.example.