Robots.txt Guide: Syntax, Examples, and Classic Mistakes

Robots.txt is a plain-text file with outsized power: one line in it can remove your site from search, and websites do this to themselves every year — usually by shipping a staging config to production. It's also widely misunderstood: it controls crawling, not indexing, and using it for the wrong job creates exactly the problems it seems to solve. Here's the complete guide — syntax, examples, the classic mistakes, and what robots.txt is genuinely for.

What robots.txt is (and the one thing it isn't)

The file lives at exactly yourdomain.com/robots.txt and tells crawlers which paths they may fetch, per the crawl stage of the pipeline. Compliant bots (all major search engines) obey it; scrapers ignore it — it's a convention, not security. And the critical distinction: Disallow prevents crawling, not indexing. A disallowed URL that other pages link to can still be indexed — as a bare URL with no description ("indexed, though blocked by robots.txt"). To keep a page out of the index, the tool is a noindex meta tag on a crawlable page; robots-blocking it actually prevents Google from ever seeing that noindex. Getting this backwards is the field's most common confusion.

The syntax, complete

• User-agent: * — which crawler the following rules address (* = all; specific bots like Googlebot can get their own blocks).
• Disallow: /admin/ — don't fetch paths starting with this. Disallow: with nothing = allow everything; Disallow: / = block the entire site (the fatal line).
• Allow: /admin/public-tool — carve an exception inside a disallowed path (Google honours the more specific rule).
• Sitemap: https://yourdomain.com/sitemap.xml — declare your sitemap(s); absolute URL, anywhere in the file.
• Wildcards: * matches any string, $ anchors the end — Disallow: /*?sort= blocks sort parameters; Disallow: /*.pdf$ blocks PDFs.
• Comments with #; rules are case-sensitive; one directive per line.

A sane default file

User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /search Disallow: /*?filter= Allow: /

Sitemap: https://example.com/sitemap.xml

Block the genuinely un-crawlworthy — admin areas, carts and checkouts, internal search results, infinite parameter spaces — and nothing else. Most small sites need even less; an empty-ish robots.txt is a perfectly good robots.txt.

What robots.txt is genuinely for

1. Crawl-waste control: keeping bots out of infinite spaces (faceted filters, calendars, session URLs) — the front line of crawl budget management on large sites.
2. Utility-page hygiene: carts, login flows, internal search — pages with no search value that would otherwise soak crawl attention.
3. Non-HTML assets you'd rather not serve to search: internal PDFs and the like (with the indexing caveat above).

And what it's not for: hiding sensitive content (the file is public — listing /secret-admin/ advertises it), de-indexing (noindex's job), duplicate management (canonicals do that — blocking duplicates prevents Google from seeing the canonical), and blocking CSS/JS (a legacy habit that now breaks rendering — Google must fetch your assets to see your pages).

The classic disasters, and the audit

The hall of shame: Disallow: / shipped from staging (check robots.txt in every deployment checklist); blocking a path and expecting its noindex to work; blocking assets until pages render blank to Google; case mismatches silently not matching. The audit is two minutes quarterly, per the checklist: read the live file, test key URLs with Search Console's robots.txt report, and diff against last quarter — robots.txt changes should be as deliberate as URL changes, because their blast radius is comparable.

Frequently asked questions

Do I even need a robots.txt?

Without one, everything is crawlable — a fine default for small sites. Have one anyway for the Sitemap line and for deliberate hygiene; an intentional file beats an accidental absence.

Can I block AI crawlers in robots.txt?

The same mechanism serves — many AI crawlers declare user-agents you can address individually. Compliance is voluntary as ever, and policy here is a business decision beyond this guide's scope.

How fast do robots.txt changes take effect?

Google refetches the file frequently (typically within a day); un-blocking then waits on recrawl of the affected URLs — hours to weeks by importance. One more reason the fatal Disallow hurts: recovery isn't instant even after the fix. Keep the file boring, keep the audits scheduled — and spend the drama budget where it pays, on the links (hello).