Robots.txt Generator | Create Robots.txt Files Free
Generate a valid robots.txt file with crawl rules for search bots in seconds. Choose from presets for WordPress, e-commerce, or AI bot blocking. Add custom user-agent rules, allow and disallow paths, crawl delays, and sitemap URLs.
# robots.txt will appear here
https://yourdomain.com/robots.txt. Search engines check this file before crawling.
How this robots.txt generator builds valid directive blocks, rule by rule
This tool is pure client side, it writes plain text and never touches a network request. Each user agent rule you add becomes an object holding an agent name, a list of disallow paths, a list of allow paths, and an optional crawl delay. On every change, the tool walks the full rule list and reassembles the entire file from scratch, which is why the output box updates instantly as you type.
The directive syntax it follows
Robots.txt has one of the smallest grammars on the web, formalized in 2022 as RFC 9309, the Robots Exclusion Protocol. This generator sticks to the directives that protocol and the major search engines actually support.
The six built-in presets
| Preset | What it sets up |
|---|---|
| Blank | A single empty rule for User-agent * with nothing disallowed. |
| Allow all | User-agent * with an empty Disallow, permitting the entire site. |
| Block all | User-agent * with Disallow: /, blocking the entire site. |
| WordPress | Blocks /wp-admin/, /wp-includes/, plugin and cache folders, cgi-bin, trackback and xmlrpc.php, while allowing the admin-ajax.php endpoint. |
| E-commerce | Blocks cart, checkout, account, order, admin, tmp, search and tag paths, the pages that add no unique indexable value. |
| Block AI crawlers | Adds a separate Disallow: / block for GPTBot, CCBot, anthropic-ai, Google-Extended, FacebookBot and Omgilibot, each as its own named user agent rule. |
Validation and export
Live validation
The tool checks that at least one User-agent line exists and that every Disallow or Allow path begins with a forward slash, flagging an error or warning banner immediately below the output box.
Copy and download
A copy button uses the Clipboard API with a manual textarea fallback for older browsers, and a download button saves the exact output as robots.txt via a Blob and object URL, ready to drop at your site root.
Robots protocol references
- RFC 9309 is the formal, current specification for the Robots Exclusion Protocol.
- Google’s robots.txt introduction explains how Googlebot specifically interprets each directive, including its stance on crawl delay.
- robotstxt.org is the original informal home of the standard, predating the formal RFC by two decades.
Crawl control in practice
Setting up a fresh WordPress install’s crawl rules in under a minute, blocking checkout and account paths on a new storefront before launch, deciding whether to let AI training crawlers like GPTBot access your content, and generating a quick block-everything file for a staging subdomain that should never appear in search results.
FAQ: Robots.txt Generator
A robots.txt file is a plain text file placed at the root of your website that instructs search engine crawlers which pages or sections to visit and which to skip. Without it, bots crawl everything. A well-configured robots.txt protects admin areas, prevents duplicate content indexing, and helps search engines focus their crawl budget on your most important pages.
Upload the file to the root directory of your domain so it is accessible at https://yourdomain.com/robots.txt. It must be at the root — a robots.txt in a subdirectory like /blog/robots.txt has no effect. In WordPress, the file goes in the same folder as wp-config.php, which is usually your public_html or www folder.
Add a user-agent rule with the bot’s name and set Disallow to /. For example, to block GPTBot: User-agent: GPTBot followed by Disallow: /. This robots.txt generator has a dropdown of common bots and an AI bot blocking preset that fills in all major AI training crawlers at once.
Yes, significantly. A correct robots.txt prevents bots from wasting crawl budget on thin or duplicate pages like /tag/, /search/, or /cart/, leaving more budget for your core content. Accidentally blocking important pages with a broad Disallow: / rule is one of the most common SEO disasters — which is why using a robots.txt generator with validation reduces the risk.
Crawl-delay tells a bot to wait a certain number of seconds between requests. It is useful for protecting server performance on high-traffic sites or preventing overly aggressive crawlers from slowing your site down. Note that Googlebot ignores Crawl-delay and has its own crawl rate setting in Google Search Console instead.
Yes. This robots.txt generator is completely free to use with no account required. Build, copy, or download your robots.txt file as many times as you need.
From the blog
Search, without the guesswork
What actually moves rankings and what is folklore.
