Robots.txt Generator | Create Robots.txt Files Free
Generate a valid robots.txt file with crawl rules for search bots in seconds. Choose from presets for WordPress, e-commerce, or AI bot blocking. Add custom user-agent rules, allow and disallow paths, crawl delays, and sitemap URLs. This robots.txt generator is free and needs no signup.
# robots.txt will appear here
https://yourdomain.com/robots.txt. Search engines check this file before crawling. AI crawler presets: training bots, AI search bots and assistants
AI bots are not one group, and treating them as one is how sites vanish from AI answers by accident. The presets separate them by what they do.
- Training crawlers collect text to train models. Examples are GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent. Blocking them keeps your content out of future training sets.
- AI search crawlers build the index that answers cite from. Examples are OAI-SearchBot, Claude-SearchBot, PerplexityBot and DuckAssistBot. Blocking them removes you as a source in ChatGPT search, Claude and Perplexity.
- Assistant fetchers open a single page because a person asked about it. Examples are ChatGPT-User, Claude-User and Perplexity-User.
Which preset to pick
Most publishers want the middle road: block training, allow AI search and assistants. That preset writes a Disallow for every training crawler and an explicit Allow for the rest. There is also a preset that blocks every AI bot, one that allows them all, and a WordPress preset with training crawlers blocked. Some assistant fetchers do not follow robots.txt for links a user pastes in, so a firewall rule is the only hard stop.
How this robots.txt generator builds valid directive blocks, rule by rule
This robots.txt generator runs entirely in your browser. It writes plain text and makes no network requests. Each user agent rule you add becomes an object holding an agent name, a list of disallow paths, a list of allow paths, and an optional crawl delay. On every change, the tool walks the full rule list and reassembles the entire file from scratch, which is why the output box updates instantly as you type.
The directive syntax it follows
Robots.txt has one of the smallest grammars on the web, formalized in 2022 as RFC 9309, the Robots Exclusion Protocol. This robots.txt generator sticks to the directives that protocol and the major search engines actually support.
The six built in presets
| Preset | What it sets up |
|---|---|
| Blank | A single rule for User-agent * with an empty Disallow, ready for you to fill in. |
| Allow all | User-agent * with an empty Disallow, permitting the entire site. |
| Block all | User-agent * with Disallow: /, blocking the entire site. |
| WordPress | Blocks /wp-admin/, the login page, internal search results, xmlrpc.php, trackback and cgi-bin, while allowing the admin-ajax.php endpoint. Theme and plugin files stay crawlable so Google can render your pages. |
| E-commerce | Blocks cart, checkout, account, order, admin, tmp and search paths plus add to cart, sorting and filter URLs, the pages that add no unique indexable value. |
| AI: block training bots | Keeps all search engines allowed, then adds a Disallow: / block for the crawlers that collect training data: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, cohere-ai, Diffbot, AI2Bot and Omgilibot. AI search bots such as OAI-SearchBot and PerplexityBot stay allowed. |
Validation and export
Live validation
The robots.txt generator checks for a User-agent rule, paths that do not start with a slash, full URLs pasted into Disallow, spaces, a misplaced $, duplicate user agents, Disallow: / for every bot, bad Crawl-delay values, relative sitemap URLs and the 500 KiB size limit. Errors and warnings appear right above the output. Below it, a tester tells you whether any URL is allowed or blocked for a given bot, and which rule decided it.
Copy and download
A copy button uses the Clipboard API with a manual textarea fallback for older browsers, and a download button saves the exact output as robots.txt via a Blob and object URL, ready to drop at your site root.
Robots protocol references
- RFC 9309 is the formal, current specification for the Robots Exclusion Protocol.
- Google's robots.txt introduction explains how Googlebot specifically interprets each directive, including its stance on crawl delay.
- robotstxt.org is the original informal home of the standard, predating the formal RFC by two decades.
Crawl control in practice
Setting up a fresh WordPress install's crawl rules in under a minute, blocking checkout and account paths on a new storefront before launch, deciding whether to let AI training crawlers like GPTBot access your content, and generating a quick file that blocks everything for a staging subdomain that should never appear in search results.
Robots.txt rules people search for most
To block one folder for every crawler, write User-agent: * on one line and Disallow: /folder/ on the next. Disallow: / blocks the whole site, and an empty Disallow: allows everything.
Robots.txt stops crawling, not indexing. A blocked page can still appear in search if other sites link to it. Use a noindex tag to keep a page out of results.
FAQ: Robots.txt Generator
A robots.txt file is a plain text file placed at the root of your website that instructs search engine crawlers which pages or sections to visit and which to skip. Without it, bots crawl everything. A well configured robots.txt protects admin areas, prevents duplicate content indexing, and helps search engines focus their crawl budget on your most important pages.
Upload the file to the root directory of your domain so it is accessible at https://yourdomain.com/robots.txt. It must be at the root. A robots.txt in a subdirectory like /blog/robots.txt has no effect. In WordPress, the file goes in the same folder as wp-config.php, which is usually your public_html or www folder.
Add a user-agent rule with the bot's name and set Disallow to /. For example, to block GPTBot: User-agent: GPTBot followed by Disallow: /. This robots.txt generator has a dropdown of common bots and an AI bot blocking preset that fills in all major AI training crawlers at once.
Yes, significantly. A correct robots.txt prevents bots from wasting crawl budget on thin or duplicate pages like /tag/, /search/, or /cart/, leaving more budget for your core content. Accidentally blocking important pages with a broad Disallow: / rule is one of the most common SEO disasters, which is why using a robots.txt generator with validation reduces the risk.
Crawl-delay tells a bot to wait a certain number of seconds between requests. It is useful for protecting server performance on high-traffic sites or preventing overly aggressive crawlers from slowing your site down. Note that Googlebot ignores Crawl-delay and has its own crawl rate setting in Google Search Console instead.
Yes. This robots.txt generator is completely free to use with no account required. Build, copy, or download your robots.txt file as many times as you need.
Add Disallow: /directory-name/ under the User-agent line. The trailing slash limits the rule to that folder and everything inside it.
Add a group for each bot, such as User-agent: GPTBot followed by Disallow: /. Repeat for ClaudeBot, CCBot and others you want to stop.
No. It only stops crawling. To remove a page, allow crawling and add a noindex meta tag, or use the removal tool in Search Console.
It depends on the bot. Blocking training crawlers such as GPTBot keeps your text out of model training. Blocking AI search crawlers such as OAI-SearchBot also removes your pages from AI answers, which most sites do not want.
Both belong to OpenAI. GPTBot collects content for training models. OAI-SearchBot indexes pages so ChatGPT search can cite them. You can block one and allow the other.
From the blog
Search, without the guesswork
What actually moves rankings and what is folklore.


