Configuring robots.txt for AI search means adding per-agent Allow and Disallow rules that separate "search/citation" crawlers (which power Google AI Overviews, ChatGPT, and Perplexity answers) from "model-training" crawlers, rather than blocking or allowing AI wholesale. Robots.txt is a voluntary text file hosted at the domain root that signals which user-agents may access which paths. It does not enforce access, and compliant crawlers are merely expected to respect it.
That separation is the core decision for AI visibility. You want pages discoverable and citable by answer engines that retrieve, summarize, and link back, while keeping bulk ingestion for foundation-model training disallowed unless you deliberately opt in. In practice that means writing per-agent rules: allow the retrieval-focused agents that power citations, disallow the training-focused agents that feed model weights, instead of using a single User-agent: * rule for all AI.
Because the Robots Exclusion Protocol relies on voluntary compliance, a disallow is a public request, not a firewall. For high-value or sensitive content, treat robots.txt as the first layer and plan to verify behavior and apply stronger controls at the CDN or server edge where needed. The next section names exactly which bots fall into which category today.
The split matters because purpose determines value. Training bots ingest pages offline to build future models with no live citation or referral back to you. Search and citation bots crawl to answer a query right now (powering Google AI Overviews, ChatGPT search, Claude search, and Perplexity answers), which is where discoverability and referral traffic actually come from.
| AI Company | Search/Citation Bot (typically allow) | Training Bot (typically block) | Robots.txt Compliance Notes |
|---|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot | Respects robots.txt tags per official docs |
| Anthropic | Claude-SearchBot, Claude-User | ClaudeBot | Respects robots.txt and Crawl-delay; IP blocking not recommended |
| ByteDance | Bytespider | — | Frequently flagged for low robots.txt adherence - verify separately |
For OpenAI, the vendor publishes OAI-SearchBot and GPTBot as distinct robots.txt tags to let you manage access. Publishers who allow OAI-SearchBot can track referral traffic while disallowing GPTBot to exclude pages from potential training. In practice, that maps to OAI-SearchBot and ChatGPT-User as search/citation and GPTBot as training.
Anthropic documents its framework explicitly: ClaudeBot helps enhance utility and safety by collecting web content that could potentially contribute to training, Claude-User supports Claude AI users by accessing websites when users ask questions, and Claude-SearchBot navigates the web to improve search result quality. All three support the non-standard Crawl-delay extension, and Anthropic notes that blocking IP addresses may not work correctly or persistently because bots need to read robots.txt first.
Use the same pattern for other vendors: verify the exact user-agent string in the vendor's own docs before adding it. Perplexity publishes PerplexityBot / Perplexity-User, Google publishes Google-Extended and Googlebot, Apple publishes Applebot and Applebot-Extended, and Common Crawl publishes CCBot. ByteDance's Bytespider is the classic edge case operators flag for low robots.txt adherence, so treat it as "verify separately" and plan enforcement below rather than relying on robots.txt alone.
Knowing which bots to target is only useful once you know where those rules actually get pasted for your specific platform.
WordPress generates a virtual robots.txt if no file exists at the domain root. You have two paths: upload a static robots.txt file via SFTP or your host's file manager to overwrite the virtual one, or edit it from inside an SEO plugin. Yoast SEO and Rank Math both expose a robots.txt editor under their Tools / File Editor screens. If a physical file exists on disk, WordPress will serve that instead of the virtual version, so check for leftovers after switching methods.
Webflow sites don't allow arbitrary file uploads to /. You manage the file in Site settings > SEO > Indexing, where a text area lets you add custom rules and publish them with the site Set robots.txt rules. Webflow automatically appends your sitemap reference unless you toggle Remove sitemap.xml from robots.txt on, and it also offers a separate Content-Signal header control in the same Indexing panel. The editor is plain text with no per-bot validation, so keep syntax tight.
robots.txt.liquid overrideShopify serves a managed default file that is good for most stores. To customize, you must create a robots.txt.liquid template in the theme. From your admin go to Online Store > Themes >... > Edit code, then Add new template > robots and create it Editing robots.txt.liquid. Use Liquid to extend the default output rather than fully replacing it with static text, which Shopify notes is not recommended because it stops future SEO safeguards from applying. Shopify also notes that rules are advisory and not guaranteed to be followed by every crawler.
On self-hosted stacks, edit the actual file at the document root, e.g. /var/www/html/robots.txt, so yourdomain.com/robots.txt returns it. If your CMS locks the file, use your CDN layer (a Cloudflare Transform Rule, Worker, or similar edge rule) to serve custom content at that path.
Ready-to-adapt pattern referencing the directory above:
# Block training-use crawlers, allow search/citation crawlers
User-agent: [TRAINING-BOT-NAME]
Disallow: /
User-agent: [SEARCH-BOT-NAME]
Allow: /
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
Replace the bracketed placeholders with the exact user-agents from the directory above and repeat the block as needed.
Warning: not every CMS lets you fully control robots.txt — check what your platform actually exposes and whether it merges defaults before you assume your rules are live.
Publishing the rule is only half the job. Rules are only useful if they're actually followed, and robots.txt remains a voluntary convention: well-behaved crawlers check it before fetching, but nothing in the protocol forces compliance and there is no legal weight behind a Disallow line.
Inspect server and CDN logs. After you publish a new Disallow rule, look for two events in chronological order: a fetch of /robots.txt, then a fetch of the path you blocked. If you see the blocked path fetched after robots.txt without respecting the rule, with a familiar user-agent pattern in the log, that's a compliance failure. Filter by user-agent and host IP, not just UA alone, because UA strings are trivial to spoof.
Verify the caller's identity. For major crawlers, use infrastructure checks, not UA text. Google documents the canonical method: run a reverse DNS lookup on the accessing IP, verify that the domain name is either googlebot.com, google.com, or googleusercontent.com, then run a forward lookup on that hostname and confirm it resolves back to the original IP. OpenAI, Anthropic, and Common Crawl publish their IP ranges or reverse-DNS patterns in their docs, so you can apply the same two-step check to GPTBot, ClaudeBot, CCBot and similar operators. Automatically matching IPs against the vendor-published JSON lists scales this beyond one-off lookups.
Use bot-aware analytics. If you are behind Cloudflare, Fastly, Akamai or similar, enable bot analytics to group requests by verified vs unverified bot score, ASN, and request rate. Exporting a short window of logs around the robots.txt change makes it obvious whether a specific AI crawler stopped or kept crawling.
Some crawlers, notably Bytespider and generic scraping toolkits that rotate UAs, have a documented history of ignoring robots.txt. For those, escalate beyond the text file:
/blog/* or /docs/*.Robots.txt has zero legal or technical enforcement, so treat it as a signal and pair it with CDN/WAF blocking for non-compliant crawlers.
Once access is correctly gated and verified, the final layer is making sure the content those bots can reach is actually worth citing. robots.txt is the gatekeeper, and llms.txt is the guide. Robots.txt tells crawlers what they may fetch; llms.txt gives models a clean, curated map of what actually matters on your site: a plain Markdown file at the domain root that summarizes your expertise and points to your best pages.
Beyond just keywords, AI engines prioritize well-structured content with clear metadata, FAQs, and schema markup. HarperFlow automates these optimizations, ensuring your content is highly discoverable and relevant, reducing manual work while boosting answer engine performance.
The proposal on llmstxt.org is deliberately small. It lives at /llms.txt, uses Markdown, starts with an H1 containing the project or site name as the only required section, adds a blockquote summary, then uses H2 headers to group file lists where each entry is a Markdown link with optional notes. There is even an Optional section for secondary resources that can be skipped when context is short. The spec is built to coexist with current web standards and complement robots.txt by providing context for allowed content, while robots.txt remains the file that tells automated tools what access is considered acceptable for search indexing bots.
That split explains why opening access alone rarely earns citations. Search and answer engines can retrieve a page but still skip it if the answer is buried in navigation, JavaScript, or vague prose. Models quote pages where the answer is immediate, self-contained, and traceable.
Use this close-the-loop checklist today:
Once the doors and map are in place, focus turns to structure at scale. HarperFlow is built for that next layer, turning approved content into traceable, answer-oriented pages that answer engines can parse and cite without rework.
No, you should keep them separate. OAI-SearchBot powers referral traffic from ChatGPT search results, while GPTBot is the tag you disallow to exclude pages from potential training. Blocking the search bot removes citations, not training risk.
Yes. Use per-agent blocks with a specific path like Disallow: /private/ under the training bot's User-agent. The file must still live at the top-level directory of a site to be found, and you can mix Allow and Disallow per bot. Check server logs after to confirm the rule is honored.
Check identity with reverse DNS and vendor-published ranges, not just the User-agent string. Google documents the method as reverse lookup the IP, verify the domain, then forward lookup back to the same IP, and Anthropic publishes similar guidance. Anthropic also notes that blocking IP address(es) may not work correctly or persistently because crawlers need to fetch robots.txt first.
That means it does not follow voluntary compliance. Treat robots.txt as a public request, not a firewall, and enforce at the edge with a WAF rule, rate limit, or challenge. On Cloudflare, the Block AI Bots feature blocks verified bots classified as crawling for purpose of AI training, as well as a number of unverified bots that behave similarly.
Not if you configure it by behavior. Cloudflare separates AI bot behavior into Search, Agent, and Training with options to block on all pages, block on pages with ads, or allow. You can block Training while keeping Search allowed to preserve citations and referral traffic.
No, they serve different roles. Robots.txt is the file that tells automated tools what access is considered acceptable for search indexing bots, while llms.txt is designed to coexist and complement it by providing context for allowed content. One gates access, the other maps what is worth citing.
Shopify recommends extending the default via a robots.txt.liquid file in your theme's templates folder instead of replacing it with static text. That method preserves future SEO safeguards, and you can still reference your sitemap. Shopify notes rules are directional and advisory, not guaranteed to be followed by every crawler.
Yes, for bots that honor it. Anthropic supports the non-standard Crawl-delay extension, so you can add Crawl-delay under its User-agent. It reduces fetch rate without removing you from search or training pools.