# robots.txt ยท simplyscan.io # # GROUP SEMANTICS (RFC 9309) โ€” READ BEFORE EDITING: # A crawler obeys exactly ONE group: its most specific User-agent match. It does # NOT also read the `User-agent: *` group. So a named group containing only # `Allow: /` means that bot ignores every Disallow in this file. # # This file used to list Googlebot, Bingbot, GPTBot, ClaudeBot etc. each in their # own `Allow: /`-only group while the Disallow rules lived solely under `*`. Net # effect: the admin path and every app route were disallowed for every crawler # EXCEPT the ones the rules were written for. # # The named groups exist on purpose (an explicit welcome to AI/answer engines โ€” # that is the AEO play), so the fix is not to delete them: every named group now # repeats the shared Disallow block verbatim. # # >>> If you add a path to one group, add it to ALL of them. <<< # # Why disallow app routes when they already render `noindex`? # The noindex is applied by JavaScript at render time, and Lovable hosting serves # the prerendered HOMEPAGE (HTTP 200, canonical="/") for every unmatched path. # Non-rendering crawlers (Bing's fallback pass, GPTBot, ClaudeBot, PerplexityBot, # CCBot) never execute that JS, so they would index a byte-identical homepage # clone at /login, /dashboard, /scan/*. robots.txt is the only directive they # honour before fetching. User-agent: Googlebot Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success User-agent: Bingbot Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success User-agent: Twitterbot Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success User-agent: facebookexternalhit Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success User-agent: GPTBot Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success User-agent: OAI-SearchBot Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success User-agent: PerplexityBot Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success User-agent: Perplexity-User Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success User-agent: Google-Extended Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success User-agent: ClaudeBot Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success User-agent: Claude-Web Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success User-agent: anthropic-ai Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success User-agent: Applebot-Extended Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success User-agent: meta-externalagent Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success User-agent: Bytespider Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success User-agent: CCBot Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success User-agent: * Allow: / Disallow: /gabriel-25 Disallow: /gabriel-25/login Disallow: /dashboard Disallow: /scan/ Disallow: /verify/ Disallow: /newsletter/ Disallow: /login Disallow: /signup Disallow: /pro-success # Two sitemap sources, deliberately: # - the live one, composed from the database per request (a post published # at 09:03 is listed at 09:03 โ€” no build needed). Plain-text format: the # serving gateway rewrites XML content types, and the txt sitemap format # is correct under text/plain. # - the baked XML snapshot (with lastmod), regenerated on every site rebuild Sitemap: https://wpyoohiwuephrhgeoctb.supabase.co/functions/v1/seo-sitemap?format=txt Sitemap: https://simplyscan.io/sitemap.xml