📈 SEO & Marketing ✔ Free Forever

Robots.txt Generator | Create Robots.txt Files Free

Generate a valid robots.txt file with crawl rules for search bots in seconds. Choose from presets for WordPress, e-commerce, or AI bot blocking. Add custom user-agent rules, allow and disallow paths, crawl delays, and sitemap URLs. This robots.txt generator is free and needs no signup.

✔ 10 quick presets ✔ Multiple user-agent rules ✔ Block AI training bots ✔ Sitemap directive ✔ Download robots.txt ✔ Inline validation
Quick Preset
User-agent Rules
Sitemap URL (optional)
Extra paths to block (all bots, one per line)
Generated robots.txt
# robots.txt will appear here
How to use Place robots.txt at the root of your domain: https://yourdomain.com/robots.txt. Search engines check this file before crawling.

AI crawler presets: training bots, AI search bots and assistants

AI bots are not one group, and treating them as one is how sites vanish from AI answers by accident. The presets separate them by what they do.

  • Training crawlers collect text to train models. Examples are GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent. Blocking them keeps your content out of future training sets.
  • AI search crawlers build the index that answers cite from. Examples are OAI-SearchBot, Claude-SearchBot, PerplexityBot and DuckAssistBot. Blocking them removes you as a source in ChatGPT search, Claude and Perplexity.
  • Assistant fetchers open a single page because a person asked about it. Examples are ChatGPT-User, Claude-User and Perplexity-User.

Which preset to pick

Most publishers want the middle road: block training, allow AI search and assistants. That preset writes a Disallow for every training crawler and an explicit Allow for the rest. There is also a preset that blocks every AI bot, one that allows them all, and a WordPress preset with training crawlers blocked. Some assistant fetchers do not follow robots.txt for links a user pastes in, so a firewall rule is the only hard stop.

About This Tool

How this robots.txt generator builds valid directive blocks, rule by rule

This robots.txt generator runs entirely in your browser. It writes plain text and makes no network requests. Each user agent rule you add becomes an object holding an agent name, a list of disallow paths, a list of allow paths, and an optional crawl delay. On every change, the tool walks the full rule list and reassembles the entire file from scratch, which is why the output box updates instantly as you type.

The directive syntax it follows

Robots.txt has one of the smallest grammars on the web, formalized in 2022 as RFC 9309, the Robots Exclusion Protocol. This robots.txt generator sticks to the directives that protocol and the major search engines actually support.

User-agent Names the crawler a rule block applies to. The tool ships a preset list including Googlebot, Bingbot, DuckDuckBot, Baiduspider, YandexBot, several SEO crawlers like AhrefsBot and SemrushBot, and AI crawlers including GPTBot, ClaudeBot, Google-Extended and PerplexityBot, plus a free text field for any custom agent name and the wildcard *.
Disallow One path per line, written exactly as typed into the output. An empty Disallow value means allow everything for that agent, which is what the Allow All preset produces.
Allow Optional, used to carve out an exception inside a broader Disallow, for example allowing a single AJAX endpoint inside an otherwise blocked admin folder.
Crawl-delay and Sitemap Crawl-delay is written per rule block if you set it, though Google ignores this directive entirely, it is honored by Bing and some others. Sitemap lines are written at the end of the file, outside every user agent block. Add one URL per line if you have several sitemaps.
// generate(), simplified from source lines.push('User-agent: ' + (rule.agent||'*')); rule.disallow.forEach(function(p){ lines.push('Disallow: ' + p); }); rule.allow.forEach(function(p){ lines.push('Allow: ' + p); }); if(rule.crawlDelay){ lines.push('Crawl-delay: ' + rule.crawlDelay); } // sample output for the WordPress preset User-agent: * Allow: /wp-admin/admin-ajax.php Disallow: /wp-admin/ Disallow: /wp-login.php

The six built in presets

PresetWhat it sets up
BlankA single rule for User-agent * with an empty Disallow, ready for you to fill in.
Allow allUser-agent * with an empty Disallow, permitting the entire site.
Block allUser-agent * with Disallow: /, blocking the entire site.
WordPressBlocks /wp-admin/, the login page, internal search results, xmlrpc.php, trackback and cgi-bin, while allowing the admin-ajax.php endpoint. Theme and plugin files stay crawlable so Google can render your pages.
E-commerceBlocks cart, checkout, account, order, admin, tmp and search paths plus add to cart, sorting and filter URLs, the pages that add no unique indexable value.
AI: block training botsKeeps all search engines allowed, then adds a Disallow: / block for the crawlers that collect training data: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, cohere-ai, Diffbot, AI2Bot and Omgilibot. AI search bots such as OAI-SearchBot and PerplexityBot stay allowed.
Robots.txt is a request, not a lock. The protocol is voluntary. Well behaved crawlers honor it, but nothing stops a scraper from ignoring the file outright, and it does not prevent a URL from being indexed if other pages link to it, it only stops crawling. If you need a page to genuinely stay out of search results, pair this with a noindex meta tag or an authentication wall, disallowing a page in robots.txt while it is also linked elsewhere can even backfire, since Google may still index the URL with no description because it was blocked from reading the page.

Validation and export

Live validation

The robots.txt generator checks for a User-agent rule, paths that do not start with a slash, full URLs pasted into Disallow, spaces, a misplaced $, duplicate user agents, Disallow: / for every bot, bad Crawl-delay values, relative sitemap URLs and the 500 KiB size limit. Errors and warnings appear right above the output. Below it, a tester tells you whether any URL is allowed or blocked for a given bot, and which rule decided it.

Copy and download

A copy button uses the Clipboard API with a manual textarea fallback for older browsers, and a download button saves the exact output as robots.txt via a Blob and object URL, ready to drop at your site root.

Runs in your browser RFC 9309 directive syntax One file per site root, always

Robots protocol references

  • RFC 9309 is the formal, current specification for the Robots Exclusion Protocol.
  • Google's robots.txt introduction explains how Googlebot specifically interprets each directive, including its stance on crawl delay.
  • robotstxt.org is the original informal home of the standard, predating the formal RFC by two decades.

Crawl control in practice

Setting up a fresh WordPress install's crawl rules in under a minute, blocking checkout and account paths on a new storefront before launch, deciding whether to let AI training crawlers like GPTBot access your content, and generating a quick file that blocks everything for a staging subdomain that should never appear in search results.

Robots.txt rules people search for most

To block one folder for every crawler, write User-agent: * on one line and Disallow: /folder/ on the next. Disallow: / blocks the whole site, and an empty Disallow: allows everything.

Robots.txt stops crawling, not indexing. A blocked page can still appear in search if other sites link to it. Use a noindex tag to keep a page out of results.

Common Questions

FAQ: Robots.txt Generator

A robots.txt file is a plain text file placed at the root of your website that instructs search engine crawlers which pages or sections to visit and which to skip. Without it, bots crawl everything. A well configured robots.txt protects admin areas, prevents duplicate content indexing, and helps search engines focus their crawl budget on your most important pages.

Upload the file to the root directory of your domain so it is accessible at https://yourdomain.com/robots.txt. It must be at the root. A robots.txt in a subdirectory like /blog/robots.txt has no effect. In WordPress, the file goes in the same folder as wp-config.php, which is usually your public_html or www folder.

Add a user-agent rule with the bot's name and set Disallow to /. For example, to block GPTBot: User-agent: GPTBot followed by Disallow: /. This robots.txt generator has a dropdown of common bots and an AI bot blocking preset that fills in all major AI training crawlers at once.

Yes, significantly. A correct robots.txt prevents bots from wasting crawl budget on thin or duplicate pages like /tag/, /search/, or /cart/, leaving more budget for your core content. Accidentally blocking important pages with a broad Disallow: / rule is one of the most common SEO disasters, which is why using a robots.txt generator with validation reduces the risk.

Crawl-delay tells a bot to wait a certain number of seconds between requests. It is useful for protecting server performance on high-traffic sites or preventing overly aggressive crawlers from slowing your site down. Note that Googlebot ignores Crawl-delay and has its own crawl rate setting in Google Search Console instead.

Yes. This robots.txt generator is completely free to use with no account required. Build, copy, or download your robots.txt file as many times as you need.

Add Disallow: /directory-name/ under the User-agent line. The trailing slash limits the rule to that folder and everything inside it.

Add a group for each bot, such as User-agent: GPTBot followed by Disallow: /. Repeat for ClaudeBot, CCBot and others you want to stop.

No. It only stops crawling. To remove a page, allow crawling and add a noindex meta tag, or use the removal tool in Search Console.

It depends on the bot. Blocking training crawlers such as GPTBot keeps your text out of model training. Blocking AI search crawlers such as OAI-SearchBot also removes your pages from AI answers, which most sites do not want.

Both belong to OpenAI. GPTBot collects content for training models. OAI-SearchBot indexes pages so ChatGPT search can cite them. You can block one and allow the other.

From the blog

Search, without the guesswork

What actually moves rankings and what is folklore.