The SEO best practices for robots.txt are: keep it at your root domain, let crawlers fetch your CSS and JavaScript, block only low-value paths, list every sitemap with the Sitemap: directive, and never use it to hide pages you actually want removed from search. The single most important thing to understand is that robots.txt controls crawling, not indexing — it tells compliant bots which URLs they may fetch, but a blocked URL can still appear in search results if other sites link to it. To keep a page out of the index you use a noindex meta tag on a page that stays crawlable, not a Disallow rule.
That is the summary an AI overview will give you. What it can't give you is the reasoning behind each rule, the exact failure modes that silently deindex sites, and a copy-paste reference you can validate against. This guide is that version. Test any file you write against our free robots.txt Analyzer before you deploy it.
The best-practice reference table
Start here. Each row is a rule, why it matters for SEO, and the failure mode it prevents. The rest of the article expands the ones that trip people up.
| Best practice | Why it matters | What goes wrong without it |
|---|---|---|
Place the file at domain.com/robots.txt | Crawlers only read robots.txt at the host root | A file in a subfolder is silently ignored — no rules apply |
| Understand it controls crawling, not indexing | Prevents the "I blocked it but it still ranks" surprise | Blocked URLs get indexed with no snippet; you can't deindex them |
| Let crawlers fetch CSS and JavaScript | Googlebot renders pages before indexing | Broken render → lower rankings or a cloaking flag |
| Block only low-value paths (admin, checkout, params) | Focuses crawl budget on pages that can rank | Wasted crawl budget or, worse, blocking real content |
Add every sitemap with Sitemap: | Speeds up discovery of your URLs | Slower, less complete crawling of new pages |
| Never hide sensitive pages here | The file is public and doesn't enforce access | You publish a map of the private URLs you meant to hide |
Use noindex (not Disallow) to remove a page | noindex is the only directive that deindexes | A blocked page never shows Google the noindex — stays indexed |
Skip Crawl-delay for Google | Google ignores it; Bing/Yandex honor it | False confidence that you slowed Googlebot |
| Test before every deploy | One stray Disallow: / deindexes the whole site | A typo silently removes you from search for days |
robots.txt Basics for SEO
File Location and Syntax
The robots.txt file must be:
- Located at the root of your domain: example.com/robots.txt
- Plain text (never HTML or XML)
- Following consistent formatting
- Accessible without authentication
Basic Structure
User-agent: [bot name or *]
Disallow: [path to block]
Allow: [path to allow]
Crawl-delay: [seconds between requests]
Order Matters
More specific user-agents listed first:
User-agent: Googlebot # Most specific
Disallow: /admin/
User-agent: * # Least specific (default)
Disallow: /private/
SEO Best Practices
Practice 1: Allow Search Engine Crawlers
Why: You want Google, Bing, Yahoo to index your content.
# Allow Google
User-agent: Googlebot
Allow: /
# Allow Bing
User-agent: Bingbot
Allow: /
# Default - allow all good bots
User-agent: *
Allow: /
Best Practice: Don't block legitimate search engines unless you have specific reasons.
Practice 2: Block Only What Needs Blocking
Why: Blocking too much wastes your crawl budget and confuses search engines.
Common Things to Block:
# Admin areas (not meant for public)
Disallow: /admin/
Disallow: /wp-admin/
# Private user areas
Disallow: /account/
Disallow: /profile/
Disallow: /settings/
# Duplicate content filters
Disallow: /*?
Disallow: /search?
Disallow: /filter?
# Temporary pages
Disallow: /temp/
Disallow: /draft/
# Session IDs and parameters
Disallow: /*session=
Disallow: /*utm_
Don't Block:
- Your main content pages
- Categories and taxonomies
- Blog posts and articles
- Product pages
- Contact pages
Practice 3: Allow Search-Related Pages
Important pages should be explicitly allowed:
User-agent: Googlebot
Disallow: /private/
Allow: / # Allow main site
Allow: /public/* # Allow specific sections
Practice 4: Disallow Duplicate Content Intelligently
Parameter-Based Duplicates:
# Block URL parameters that create duplicates
Disallow: /*?
Disallow: /*&
Session Variables:
Disallow: /*session=
Disallow: /*sessionid=
Disallow: /*s=
Tracking Parameters (use Google Search Console instead):
# Block UTM parameters
Disallow: /*utm_
Practice 5: Know That Crawl-Delay Does Not Work for Google
What It Does: Asks a crawler to wait a set number of seconds between requests.
The catch: Google has never supported Crawl-delay and ignores it entirely. Bing and Yandex do honor it, so the directive is only useful for those bots:
# Only Bing and Yandex obey this — Googlebot ignores it
User-agent: Bingbot
Crawl-delay: 5 # Wait 5 seconds between requests
Slowing Googlebot instead: Google retired the manual crawl-rate limiter tool in Search Console in 2024. Googlebot now sets its own crawl rate automatically based on how fast your server responds and how many errors it returns. If Google is genuinely overloading your server, return HTTP 500, 503, or 429 responses and Googlebot backs off within minutes. Do not reach for Crawl-delay expecting it to throttle Google — it won't.
Practice 6: Use Sitemap Directive
What It Does: Tells search engines where to find your sitemap.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-news.xml
Best Practice:
- Include both sitemap.xml (regular pages)
- Include sitemap-news.xml (news/blog content)
- Include sitemap-images.xml (image-heavy sites)
Practice 7: Be Consistent with Canonical Tags
Relationship:
- robots.txt controls crawling
- Canonical tags control indexing
- Both should point to same "main" version
robots.txt: Disallow: /duplicate-page
HTML: <link rel="canonical" href="https://example.com/main-page">
When they conflict, use canonical tags (more powerful).
Practice 8: Test Your robots.txt
Validate every change with our robots.txt Analyzer before deploying, or use one of the options below.
Google Search Console:
- Go to Search Console
- Settings
- Test robots.txt
- Enter a URL to see if it's blocked
Online Testers:
- seotesting.com
- robotstxt.org
- regex.org
Practice 9: Monitor robots.txt Performance
Google Search Console Reports:
- Coverage report shows blocked pages
- Monitoring shows crawl errors
- Inspect URL shows if blocked
Check Regularly:
- After changes, monitor crawl rate
- Watch for unexpected increases/decreases
- Verify blocked pages aren't important
Practice 10: Include All Versions and Subdomains
Subdomains:
# example.com/robots.txt
User-agent: *
Disallow: /private/
# subdomain.example.com/robots.txt
User-agent: *
Allow: /
Each subdomain needs its own robots.txt.
HTTPS and WWW:
# Both should be available
https://example.com/robots.txt
https://www.example.com/robots.txt
http://example.com/robots.txt # Redirect to HTTPS
http://www.example.com/robots.txt # Redirect to HTTPS
Common Robots.txt Examples
Blog
User-agent: *
Allow: / # Allow all
Disallow: /admin/ # Block admin
Disallow: /draft/ # Block drafts
Disallow: /*? # Block pages with query parameters
Allow: /?s= # But allow search
Allow: /?page= # But allow pagination
Disallow: /wp-admin/ # Block WordPress admin
Sitemap: https://example.com/sitemap.xml
E-commerce
User-agent: *
Allow: / # Allow all products
Disallow: /admin/ # Block admin
Disallow: /checkout/ # Block checkout process
Disallow: /*? # Block parameter-based duplicates
Allow: /?sort= # Allow sorting (intentional variants)
Allow: /?filter= # Allow filtering
Disallow: /account/ # Block user accounts
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-products.xml
Large Site with Many Subfolders
User-agent: *
# Allow main content
Allow: /blog/
Allow: /articles/
Allow: /products/
# Block administrative areas
Disallow: /admin/
Disallow: /internal/
Disallow: /private/
# Block duplicate versions
Disallow: /old-version/
Disallow: /staging/
Disallow: /test/
# Block parameters that create duplicates
Disallow: /*?
Allow: /?page=
Allow: /?sort=
Sitemap: https://example.com/sitemap.xml
Common Robots.txt Mistakes
Mistake 1: Blocking Everything
WRONG:
User-agent: *
Disallow: /
# Site won't appear in search results!
Mistake 2: Blocking Important Content
WRONG:
Disallow: /blog/
Disallow: /articles/
# Hides your main content from search engines
Mistake 3: Inconsistent with HTML
robots.txt says: Disallow: /private/
HTML has: <meta name="robots" content="index, follow">
# Conflicting signals confuse search engines
Fix: Use HTML meta tags for specific control, robots.txt for broad rules.
Mistake 4: Overly Complex Blocking
WRONG:
Disallow: /products/*?*?
Disallow: /search?*&*=
# Too complex, may not work as intended
Better: Use simple, clear rules.
Mistake 5: No Sitemap
WRONG:
User-agent: *
Disallow: /private/
# Missing sitemap helps search engines
Better: Include sitemap directive.
Advanced robots.txt Patterns
Disallow by File Type
User-agent: *
Disallow: /*.pdf$
Disallow: /*.zip$
Disallow: /*.exe$
Disallow Session IDs
User-agent: *
Disallow: /*?sid=
Disallow: /*?sessionid=
Block Bad Bots
User-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: *
Allow: /
Specific Googlebot Rules
User-agent: Googlebot
Disallow: /private/ # Googlebot: obeys Disallow, ignores Crawl-delay
User-agent: *
Disallow: /private/
Crawl-delay: 2 # Other bots (Bing/Yandex): slower crawl
Give Googlebot its own group only when you need different Disallow rules — not to set a crawl delay, which it ignores.
robots.txt and SEO Strategy
Preserve Crawl Budget
Search engines allocate crawl budget—limit it to important pages:
# Don't waste budget on parameters
Disallow: /*?
# But allow intentional sorting/filtering
Allow: /?sort=price
Allow: /?filter=category
Prevent Content Duplication
Canonical tags handle this, but robots.txt can assist:
# Block obvious duplicates
Disallow: /duplicate/
Disallow: /old/
# Let search engines find canonical versions
Sitemap: https://example.com/sitemap.xml
Focus Crawling on Essential Pages
Steering crawlers away from thin or duplicate paths concentrates crawl budget on pages that can rank:
# Prioritize essential sections for crawling
User-agent: *
Allow: /products/
Allow: /blog/
Disallow: /*
Note the wording: this controls crawling, not index size. Disallow does not delete anything already in Google's index, and blocked URLs that are linked from elsewhere can still appear as bare, snippet-less results. To actually shrink your indexed footprint, keep those pages crawlable and serve a noindex tag — the next section explains why.
Crawling Is Not Indexing (the mistake that undoes most robots.txt files)
This is the single most misunderstood point, so it earns its own section. Disallow stops a compliant crawler from fetching a URL. It does not stop that URL from being indexed. If another page — anywhere on the web — links to a URL you blocked, Google can add it to the index knowing only the URL and its anchor text. You'll see it in Search Console flagged as "Indexed, though blocked by robots.txt," usually shown to searchers as a bare link with the note "No information is available for this page."
The fix depends on your goal:
| Goal | Wrong tool | Right tool |
|---|---|---|
| Keep a page out of Google | Disallow in robots.txt | noindex meta tag / X-Robots-Tag header on a crawlable page |
| Stop wasting crawl budget on junk | noindex alone | Disallow in robots.txt |
| Remove a page fast | either alone | noindex + Search Console Removals tool |
The trap: noindex only works if Google can crawl the page and read the tag. If you both Disallow and noindex a page, Google never fetches it, never sees the noindex, and the URL can stay indexed indefinitely. Never block a page in robots.txt that you also want deindexed — pick one mechanism, and for deindexing it must be noindex on a crawlable page.
Never Use robots.txt to Hide Sensitive Pages
robots.txt is a public file at a predictable URL — anyone can open example.com/robots.txt and read every path you listed. Disallow: /admin/secret-export/ doesn't hide that directory; it advertises it. And because Disallow is only a request that well-behaved bots honor, it stops nothing that ignores the rules.
For anything genuinely private, use real controls: authentication, server-side authorization, IP restrictions, or a noindex header on a page that already requires login. Treat robots.txt purely as crawl-efficiency guidance for cooperative search engines — never as a security or privacy boundary.
Monitoring and Maintenance
Quarterly Review
- Check robots.txt in Search Console
- Verify Sitemap URLs still valid
- Review blocked pages (are they still needed?)
- Check crawl stats (increased or decreased?)
After Site Changes
- Update robots.txt if URLs change
- Update sitemap if structure changes
- Test in Search Console
- Monitor for crawl errors
Conclusion
A well-configured robots.txt improves SEO by helping search engines crawl your most important pages efficiently while protecting admin areas and reducing duplicate content. Follow these best practices: allow search engine bots, block only necessary content, optimize crawl delay, include sitemaps, and test regularly. Monitor your robots.txt effectiveness through Google Search Console and adjust as your site evolves. Remember that robots.txt is advisory only—it guides good bots but doesn't guarantee search engine behavior. Combine it with canonical tags, noindex directives, and proper site structure for comprehensive crawlability control.