Free robots.txt Tester & Validator
Paste your robots.txt and test any URL against it · per user-agent, including AI crawlers · using the same longest-match rules Google applies.
Frequently asked
How are Allow and Disallow conflicts resolved?
Per RFC 9309, the most specific (longest) matching rule wins, and on a tie Allow beats Disallow. Wildcards (*) and end-anchors ($) are supported · which is exactly where hand-written files usually go wrong.
Does robots.txt block AI crawlers?
Compliant ones, yes · GPTBot, ClaudeBot, Google-Extended and PerplexityBot honor it. But robots.txt is a request, not access control: it hides nothing from someone who ignores it. Blocking AI crawlers also removes you from AI answers · check what that trade-off means with our AI Visibility tool.
Run a full security scan →