robots.txt generator for AI crawlers

What is a robots.txt file?

A robots.txt file is a plain-text file at the root of your site that tells automated crawlers which parts of it they may and may not fetch. It follows the long-standing Robots Exclusion Protocol: a crawler that respects the standard reads robots.txt before it starts, matches its own user-agent name against the rules you have written, and obeys the Allow and Disallow lines in the most specific group that applies to it. It is the front door of your site for bots — the first thing a well-behaved crawler checks.

Until recently that mostly meant search-engine crawlers like Googlebot and Bingbot. Now a new class of visitor has arrived: AI crawlers that gather pages to train large language models and to answer questions inside assistants like ChatGPT, Claude, Perplexity and Google’s AI Overviews. They identify themselves with their own user-agent names, which means you can set rules for them individually — and that is exactly what this generator helps you do.

Should you allow or block AI crawlers?

There is no single right answer; it depends on what you want from AI search. Allowing AI crawlers is how your content becomes eligible to be read, summarised and cited inside AI answers. If being mentioned when someone asks ChatGPT or Perplexity about your space is valuable to you — and for most brands building visibility, it is — then blocking those crawlers removes you from the very surface you are trying to win. You cannot be the answer if the model was never allowed to read you.

Blocking makes sense in narrower cases: paywalled or subscription content you do not want ingested for training, private or member-only areas, staging sites, or work you would rather models did not reproduce. Many teams take a middle path — allowing the crawlers that power live, cited answers (so they still appear in AI results) while blocking the ones that only scrape for model training. The important thing is that it is your call, made deliberately, rather than a default you never set. One caveat worth knowing: robots.txt is a request, not a wall. Reputable operators honour it, but it does not physically stop a crawler that ignores the standard, so treat it as a clear published preference, not access control.

Which AI crawlers this generator covers

The tool lists the AI user-agents worth setting a rule for today, each with a short note on who runs it:

• GPTBot — OpenAI’s crawler for training data, and OAI-SearchBot plus ChatGPT-User for ChatGPT’s search index and live browsing.
• ClaudeBot — Anthropic’s crawler that gathers content for Claude.
• PerplexityBot — indexes pages so Perplexity can cite them in answers.
• Google-Extended — the toggle for Gemini and Vertex AI training, separate from normal Google Search indexing.
• Applebot-Extended, CCBot (Common Crawl, which feeds many models), Bytespider, Amazonbot and Meta-ExternalAgent round out the major operators.

A subtle but important point: Google-Extended and Applebot-Extended control AI training only. Blocking them does not remove you from Google Search or hurt normal rankings — it just opts you out of that company’s AI model training. That separation is deliberate, and it is why the generator lists these tokens on their own.

How to use this robots.txt generator

Set each crawler to Allow or Block — or use the presets to flip them all at once and then adjust the exceptions. Add any paths you want kept private under Disallow paths (one per line, like /admin/ or /cart/); those apply to the general User-agent: * group. If you have an XML sitemap, paste its URL so crawlers can find it. The file on the right updates as you go.

When it looks right, hit Copy or Download, then upload the file to the root of your domain so it lives at https://yourdomain.com/robots.txt. If you already have a robots.txt, merge these AI-crawler groups into it rather than overwriting your existing rules. Everything runs locally in your browser, so nothing you enter is sent anywhere. One thing to know about how the file works: a crawler follows only the single most specific group that names it, so a bot with its own Allow: / group will not also read the shared User-agent: * disallow rules — which is why the generator writes an explicit group per bot.

After you publish, open https://yourdomain.com/robots.txt in a browser to confirm it loads as plain text and reads the way you expect. It is also worth revisiting whenever your strategy shifts: the AI-crawler landscape moves quickly, new user-agents appear, and a rule that made sense a year ago may be costing you visibility today. Because a block silently keeps you out of a model rather than throwing an error, an out-of-date robots.txt is easy to forget — so treat it as a living file and re-check it whenever you review your wider AI search strategy.

robots.txt is one lever in AI visibility

Letting the right crawlers in is the necessary first step, but being read is not the same as being cited. Whether an AI assistant actually quotes you depends on clear, well-structured content, sound technical health, and knowing how models describe you today. That is where Suparank comes in: it measures your brand’s visibility across ChatGPT, Perplexity, Gemini and Claude, runs GEO and technical audits to find what is holding you back, and helps you write content AI actually quotes — all from one chat-first workspace.