SimplyScan logoSimplyScan
What We ScanPricingFree ToolsBlogCreate AccountLog In
  1. Home
  2. Free Tools
  3. robots.txt Tester
Free Tool · Runs in your browser

Free robots.txt Tester & Validator

Paste your robots.txt and test any URL against it · per user-agent, including AI crawlers · using the same longest-match rules Google applies.

100% client-side · nothing you enter is ever sent to a server or stored.

Preview · sign in with a free account to run it

How to use it

  1. Paste the robots.txt.
  2. Pick a user agent (Googlebot, GPTBot, ClaudeBot, PerplexityBot and others) or type a custom one.
  3. Enter a path and see whether it's allowed or blocked, and which line decided it.

Example

Input

User-agent: *
Disallow: /admin/
Allow: /admin/help

User-agent: GPTBot
Disallow: /

Result

Googlebot  /admin/help      allowed  (line 3: Allow: /admin/help)
Googlebot  /admin/settings  blocked  (line 2: Disallow: /admin/)
GPTBot     /blog/post       blocked  (line 6: Disallow: /)

How rules are applied

Most specific group
A crawler follows the group naming it, such as User-agent: GPTBot, and ignores the * group completely. That's why GPTBot is blocked above even though * allows /blog.
Longest match wins
Within a group, the rule with the longest matching path decides (RFC 9309). On a tie, Allow wins.
No matching rule
Crawling is allowed by default.
robots.txt isn't security
It's a request well-behaved crawlers honour. It hides nothing, and listing /admin/ tells everyone where to look.

Go deeper

  • Can ChatGPT and Claude Find Your App? A Guide to AEO

Page updated 21 November 2026

Frequently asked questions

How are Allow and Disallow conflicts resolved?

Per RFC 9309, the most specific (longest) matching rule wins, and on a tie Allow beats Disallow. Wildcards (*) and end-anchors ($) are supported · which is exactly where hand-written files usually go wrong.

Does robots.txt block AI crawlers?

Compliant ones, yes · GPTBot, ClaudeBot, Google-Extended and PerplexityBot honor it. But robots.txt is a request, not access control: it hides nothing from someone who ignores it. Blocking AI crawlers also removes you from AI answers · check what that trade-off means with our AI Visibility tool.

Does Disallow remove a page from Google?

No. It stops crawling, but a blocked page can still be indexed from links. Use a noindex tag on a crawlable page to keep it out of results.

Want the full picture?

This tool covers one narrow check. A full SimplyScan runs 60 checks across 14 categories · exposed keys, RLS, XSS, auth, and speed · in 30 seconds.

Run a full security scan

Related tools

User-Agent Parser

Paste a User-Agent string · from logs or analytics · and identify the browser, OS, device class, and whether it's a known bot or AI crawler.

Secret Scanner

Paste any code, config, or bundle excerpt and get flagged secrets · AWS keys, Stripe live keys, OpenAI keys, GitHub tokens, JWTs, private key blocks and more.

.env File Linter

Paste your .env and get instant findings · secrets that will ship to the browser, duplicate keys, quoting problems · plus a generated .env.example.

SimplyScan logoSimplyScan

Security Scanner for Vibe-Coded Apps.

Product

See What We ScanHow It WorksView Pricing

Resources

Security BlogSecurity ChecklistTrust BadgeUptime MonitoringIntegrationsMCP Server (Claude, Cursor & Windsurf)API DocumentationBackend Audit GuideThreat Detection GuideFAQTerms of ServicePrivacy Policy

Platforms

All platforms →Lovable ScannerBolt.new ScannerCursor ScannerReplit Scannerv0.dev ScannerWindsurf ScannerBubble Scanner

Free Tools

All free tools →SSL CheckerSecurity HeadersDNS Security CheckerCORS TesterJWT DebuggerCSP EvaluatorHash GeneratorSEO Checker

Account

Create AccountLog In

© 2026 SimplyScan. All rights reserved.