How to Check AI Agent Website Visibility and AEO Signals

Quick answer: Review your site robots.txt directives, root markdown resources, and server response headers across every public domain. Record all crawler user agent rules, file retrieval HTTP status codes, and payload formatting. Decide whether to permit search crawlers or training bots based on your explicit organisational visibility requirements.

By Gabriel CA · Kraftwire Software

Inspect your current robots.txt file, root directory markdown files, and server response headers across all deployment environments. Record the observed crawler directive lines, response status codes, and header values against your written visibility criteria. Decide whether your public web endpoints should allow search indexing, foundation model extraction, or direct agent retrieval.

How Should You Inspect Search Crawler Robot Directives?

Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links. developers.openai.com

To help ensure your site appears in search results, we recommend allowing OAI-SearchBot in your site’s robots.txt file and allowing requests from our published IP ranges below. developers.openai.com

Inspect your root robots.txt file for specific User-agent declarations matching your visibility requirements. Record the exact User-agent strings, Allow directives, and Disallow paths present in the live file. Decide whether your operational crawler rules match your visibility targets for interactive answer engines.

Compare your firewall access lists against the published IP address ranges specified by search crawler operators. Record any blocked IP addresses, request timestamps, and the specific firewall rule identifiers triggered during testing. Decide whether network firewall configurations require adjustment to admit verified crawler traffic from published subnets.

Review your web server access logs for requests originating from automated search agent user strings. Record the observed user agent string, requesting IP address, path accessed, and resulting HTTP status code. Decide whether to separate search indexation permissions from generative model scraping rules across your site properties.

How Should You Review Foundation Model Training Boundaries?

Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models. developers.openai.com

Inspect your robots.txt configuration for directives that specifically govern foundation model training crawlers. Record the presence or absence of the GPTBot user agent token along with its associated paths. Decide whether to alter root directives if your organisational policy prohibits foundational dataset collection.

Compare path-level disallow rules across public marketing directories and private API documentation folders on your domain. Record the matching paths, the assigned crawler tokens, and the expected boundary restrictions from your project brief. Decide whether separate path exclusions are warranted for proprietary data directories versus public documentation areas.

Test crawler simulation requests against your staging environment using an assumed user agent string GPTBot/1.0. Record the HTTP status code, response headers, and whether content was served or access was denied. Decide whether the observed crawler header responses align with your agreed content syndication requirements.

How Should You Test Root Markdown Context Endpoints?

We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content. github.com

The file can be placed at the site root, or at any path within it, covering the pages under that path. github.com

Test whether a request to /llms.txt at your site root returns a valid markdown document. Record the observed HTTP status code, the declared Content-Type header, and the total character length of the body. Decide whether adding a dedicated markdown descriptor file supports your discovery requirements for autonomous software agents.

Inspect any secondary paths such as /docs/llms.txt or /api/llms.txt to verify localized scoping rules. Record the exact paths checked, their response status codes, and whether the file contents reflect local directory context. Decide whether to create sub-path markdown descriptors for separate product sections based on your site architecture.

Compare the URLs linked within your root /llms.txt file against your current public sitemap document. Record any discrepancies between the links listed in the markdown file and your canonical URL catalogue. Decide whether root placement satisfies your site-wide navigation needs or if supplementary descriptors are needed.

How Should You Inspect Markdown Documentation Link Structures?

This file offers brief background information, guidance, and links to detailed markdown files. github.com

Inspect the structure of your /llms.txt document to verify that it provides concise guidance and links to detailed markdown resources. Record the number of linked markdown files, their section headings, and the byte size of each referenced file. Decide whether linked markdown summaries provide sufficient context without exceeding your documentation scope limits.

Review each markdown URL referenced in your summary file to verify that every destination endpoint resolves successfully. Record the HTTP response status code for each link, noting whether each destination returns a 200 response. Decide whether referenced markdown files require revision based on your agreed factual accuracy standards.

Test whether the content inside linked markdown files can be read without executing client-side scripts. Record whether dynamic rendering scripts are required or if static text is delivered directly in the initial response. Decide whether document formatting conforms to clean plain text standards suitable for programmatic processing.

How Should You Test Endpoint Delivery Latency and Headers?

Test endpoint latency across all agent discovery paths using an automated command line HTTP client. Record the connection duration, time to first byte, and total transfer duration in milliseconds for each test run. Decide whether response latencies fall within acceptable limits according to your architectural performance guidelines.

Review server response headers on markdown files to verify the presence of appropriate caching directives. Record the values of Cache-Control, ETag, and Last-Modified headers observed across your test requests. Decide whether caching rules need adjustment to prevent stale metadata delivery to external consuming agents.

Compare the transfer payload byte size of your /llms.txt file against an assumed hypothetical limit of 50000 bytes. Record the actual measured byte count, noting whether it sits above or below your agreed ceiling. Decide whether payload sizes exceed your internal limits for static markdown documentation delivery.

How Should You Document Visibility Configuration Auditing Routines?

Prepare an automated verification task to check the integrity of your robots.txt and markdown discovery files periodically. Record the scheduled check frequency, alerting destinations, and baseline hash values for each monitored file. Decide whether automated monitoring alerts should be configured for changes to crawler access files.

Compare the live production robots.txt file with the version maintained in your version control repository. Record any diff lines, the commit identifier representing the intended state, and the author of the last deployment. Decide whether observed drift between staging and production rules requires an immediate deployment hold.

Document the combined findings from your crawler directives, root markdown files, and latency checks in a summary report. Record the operational status of each tested signal, the audit date, and the names of the reviewing engineers. Decide whether the overall agent discovery posture conforms to your written digital distribution policy.