A Beginner’s Guide to robots.txt Allow Disallow Sitemap

What a robots.txt file actually controls

A robots.txt file lives at the root of your domain, at a path like example.com/robots.txt, and it gives instructions to well behaved crawlers about which parts of the site they should or should not request. It does not remove pages from search results by itself, and it does not stop a determined bot from reading a page, since it works on an honor system that major search engines choose to follow. What it does very well is save crawl budget by telling search engines not to waste time on folders like admin panels, internal search results, or duplicate parameter driven pages, and it gives you a single spot to declare your sitemap location so crawlers can discover new content quickly.

Building the file line by line

Start with a User agent line, which tells the rule set which crawler it applies to. Using an asterisk applies the block to every crawler that respects the standard. Below that you list Disallow lines for anything you want left alone, and Allow lines for any exception inside a disallowed folder. The order within a group generally does not matter to major engines, since they apply the most specific matching rule rather than the first one they read, but grouping related lines together keeps the file readable for the next person who edits it.

A complete robots.txt allow disallow sitemap example

Here is a realistic file for a small business site. It blocks the admin area entirely, allows one public facing script inside that folder that the site needs to render correctly, leaves every other page open, and points to the sitemap.

Written out as an actual file, it reads like this:

User-agent: *
Disallow: /admin/
Allow: /admin/public-widget.js
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

Notice the blank line separating the user agent group from the sitemap declaration. The sitemap line is not tied to any single crawler group, so it can sit anywhere in the file, though most people place it at the end for readability. Each rule should also sit on its own line, since combining directives on one line is invalid syntax that some crawlers will simply skip.

Common mistakes that break a robots.txt allow disallow sitemap setup

The most frequent error beginners make is disallowing the entire site by accident with a single line reading Disallow: /, usually left over from a staging environment that was never updated before launch. That one line tells every well behaved crawler to skip the whole domain, and it will quietly tank your rankings without producing any obvious error message. Another common mistake is assuming that Disallow also removes a page from the search index, when in fact a blocked page can still show up in results with just its URL if other sites link to it, because the crawler never got a chance to see a noindex instruction on the page itself.

A third mistake is forgetting that subdomains need their own robots.txt file. A file at example.com/robots.txt has no effect on shop.example.com, which needs its own file at its own root if you want different crawling rules there. Finally, many people forget to update the sitemap line after migrating to a new domain or switching from http to https, which leaves crawlers reading an outdated or broken sitemap reference long after the actual site has moved.

A short checklist before you publish

• Confirm the file sits at the domain root, not inside a subfolder.
• Use a trailing slash on folder paths so you do not accidentally block a similarly named page.
• Double check that pages you want ranked are not caught by a broad Disallow rule higher up.
• List every sitemap you maintain, including separate ones for images or news if you have them.
• Test the file in a crawler simulator before pushing it live.