How do I block AI scrapers and LLM training bots?

Blocking AI Scrapers and LLM Training Bots

To block AI scrapers and LLM training bots, add a robots.txt file at your site root that lists each AI crawler's user-agent token with a Disallow: / rule — GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google's AI), CCBot (Common Crawl), PerplexityBot, Bytespider (ByteDance), Amazonbot, and Applebot-Extended (Apple) are the major ones. The critical caveat is that robots.txt is voluntary and advisory — well-behaved bots read it and obey, but robots.txt cannot actually stop anything. For crawlers that ignore the file or spoof a browser user-agent, real enforcement lives at the network edge: a WAF, firewall user-agent rules, or a bot-management service that returns a 403 instead of politely asking.

That is the summary an AI overview gives you. The part it can't give you is the nuance that decides whether this actually works: which of these bots you should keep allowed, why blocking Google-Extended does not touch your search rankings, the difference between a training crawler and a real-time fetch bot, and where the robots.txt request ends and firewall enforcement has to begin. Here is the canonical list of AI crawlers, who runs them, and what each one does with your content.

The rapid growth of large language models created a new problem for site owners: preventing training bots from ingesting content without permission. Unlike traditional web crawlers such as Googlebot and Bingbot — which are beneficial because they index you for search — AI bots like OpenAI's GPTBot and Google-Extended collect content to train machine-learning models. Owners have legitimate concerns about their work training competing services, copyright, and the diminished value of proprietary content.

While robots.txt provides a first line of defense, it's important to understand that it's advisory only — compliant bots respect it, but malicious scrapers ignore it entirely. A comprehensive defense strategy combines robots.txt with edge enforcement (WAF/firewall) and, where warranted, legal and contractual protections.

The AI crawler cheat sheet: bot, token, and operator

This is the reference table to keep. Each row is a distinct crawler with its own user-agent token — the exact string you list after User-agent: in robots.txt. Tokens are case-insensitive but must be spelled exactly; a typo (for example the common mistake GoogleExtended instead of Google-Extended) silently matches nothing and blocks nobody.

AI bot	User-agent token	Operator	What it does
GPTBot	`GPTBot`	OpenAI	Crawls content to train future models
ChatGPT-User	`ChatGPT-User`	OpenAI	Real-time fetch when a user asks ChatGPT to browse a link
OAI-SearchBot	`OAI-SearchBot`	OpenAI	Indexes pages for ChatGPT search results
ClaudeBot	`ClaudeBot` (also legacy `anthropic-ai`, `Claude-Web`)	Anthropic	Crawls content for Claude / model training
Google-Extended	`Google-Extended`	Google	Opt-out for Gemini & Vertex AI training — not Search
CCBot	`CCBot`	Common Crawl	Builds the open dataset many LLMs train on
PerplexityBot	`PerplexityBot`	Perplexity	Indexes pages to cite in Perplexity answers
Bytespider	`Bytespider`	ByteDance (TikTok)	Aggressive crawler feeding ByteDance AI
Amazonbot	`Amazonbot`	Amazon	Crawls for Alexa and Amazon AI products
Applebot-Extended	`Applebot-Extended`	Apple	Opt-out for Apple Intelligence training — not Siri/Spotlight search

How to read this table before you block everything. The tokens split into three jobs. Training crawlers (GPTBot, Google-Extended, CCBot, ClaudeBot, Applebot-Extended, Bytespider, Amazonbot) ingest your content to train models — blocking these is the low-risk default. AI-search bots (OAI-SearchBot, PerplexityBot) index you so an AI answer engine can cite you and send referral traffic — blocking these costs you visibility. Real-time fetchers (ChatGPT-User) act on a specific user request. And two tokens are deliberate opt-out siblings: Google-Extended and Applebot-Extended exist only to remove you from AI training while your normal search crawler (Googlebot, Applebot) keeps indexing you — so blocking them never touches search rankings.

The Challenge with robots.txt

Why robots.txt Alone Isn't Enough

robots.txt is a simple text file that tells crawlers what they should and shouldn't access. However:

It's Voluntary: Following robots.txt rules is entirely optional. Well-behaved bots follow it, but poorly-designed or malicious scrapers often ignore it.

It's Not Secure: The file is world-readable, making your site structure publicly visible.

It's Not Legal: It has no legal enforcement. You can't sue someone for ignoring your robots.txt.

It's Easy to Circumvent: Scrapers can simply ignore the file or access pages through alternative methods.

However, robots.txt still serves a purpose as a first defense against compliant bots and reduces unnecessary server load from casual scrapers.

Adding AI Scrapers to robots.txt

Major AI Bots to Block

OpenAI's GPTBot:

User-agent: GPTBot
Disallow: /

Anthropic's Claude Bot:

User-agent: ClaudeBot
Disallow: /

Google-Extended (opt out of Gemini/Vertex AI training — does not affect Google Search):

User-agent: Google-Extended
Disallow: /

Perplexity Bot:

User-agent: PerplexityBot
Disallow: /

Common Crawl:

User-agent: CCBot
Disallow: /

Microsoft Copilot:

User-agent: Copilot
Disallow: /

Other AI Bots:

User-agent: Bytespider
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Baiduspider-render
Disallow: /

Comprehensive robots.txt Blocking Example

# Block AI training + scraping bots
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

# Allow legitimate search engines
User-agent: Googlebot
Disallow:

User-agent: Bingbot
Disallow:

User-agent: Slurp
Disallow:

# Block all other bots (default)
User-agent: *
Crawl-delay: 10

Before you deploy, paste your file into the analyzer below to catch a stray Disallow: / that would deindex your whole site, confirm each AI token is spelled correctly, and verify your search-engine allow rules survived:

Open the Robots Txt Analyzer toolFree, in your browser on inventivehq.com →

Loading interactive tool...

JavaScript Required

This interactive tool requires JavaScript to function. Please enable JavaScript in your browser to use the full features.

The tool description and documentation above provide information about this tool's capabilities. For the best experience, please enable JavaScript and refresh the page.

Remember that everything above is still a request. The next sections cover the layers that turn a request into enforcement — starting with the firewall, because that is where you actually stop a bot that ignores robots.txt.

Advanced Technical Defenses

Meta Robots Tag

Add to your HTML <head>:

<!-- Prevent archiving/caching by AI services -->
<meta name="robots" content="noarchive,nocache">

<!-- Prevent indexing by specific services -->
<meta name="robots" content="noimageindex">

X-Robots-Tag Header

For APIs or multiple resources, use HTTP headers:

X-Robots-Tag: noarchive, nocache
X-Robots-Tag: noimageindex

Robots.txt with Rate Limiting

User-agent: GPTBot
Disallow: /

# Slow down other bots
User-agent: *
Crawl-delay: 10
Request-rate: 1/10s

Content Security Policy (CSP)

Content-Security-Policy: script-src 'self'; style-src 'self' fonts.googleapis.com;

Prevents scrapers from loading resources that might be rate-limited.

Firewall and Server-Level Blocking

CloudFlare Rules

If using CloudFlare:

Go to Firewall Rules
Create rule to block known bot User-Agents
Example rule:

(cf.bot_management.score < 30) or (http.user_agent contains "GPTBot")

Server-Level Blocking (.htaccess)

# Apache .htaccess
RewriteEngine On

# Block GPTBot
RewriteCond %{HTTP_USER_AGENT} "GPTBot" [NC]
RewriteRule .* - [F]

# Block Claude bot
RewriteCond %{HTTP_USER_AGENT} "Claude-Web" [NC]
RewriteRule .* - [F]

# Block Perplexity
RewriteCond %{HTTP_USER_AGENT} "PerplexityBot" [NC]
RewriteRule .* - [F]

Nginx Blocking

# nginx configuration
if ($http_user_agent ~* (GPTBot|Claude-Web|PerplexityBot)) {
    return 403;
}

Monitoring and Detection

Check for Known Scrapers

Install monitoring to see what bots are visiting:

CloudFlare Analytics:

Bot Management
See traffic by bot type
Block specific bots

Server Logs Analysis:

grep "GPTBot" /var/log/apache2/access.log
grep "CCBot" /var/log/apache2/access.log

Identify Unknown Bots

Look for:

Unusual User-Agent strings
Rapid page requests
Unusual bot patterns
Requests to feed endpoints (indicating content scraping)

Reverse IP Lookup:

Identify bot owners
Add to block list if malicious

Legal and Policy Approaches

Terms of Service Update

Add to your site's ToS:

Automated Access Restrictions:
- Automated access, scraping, or data mining
  of our website is prohibited
- This includes training machine learning models
- Violators will be pursued legally
- Use our API if available for approved access

Copyright and Attribution

Add copyright notices:

<!-- HTML head -->
<meta name="copyright" content="© 2024 Company Name">
<meta name="rights-holder" content="Company Name">

Licensing Statements

<!-- Creative Commons license declaration -->
<link rel="license" href="https://creativecommons.org/licenses/by-nc/4.0/">
<meta name="license" content="CC BY-NC 4.0">

DMCA Takedown Notices

If you catch scraping:

Document the scraping
Send DMCA takedown notice
Report to hosting provider

Content-Level Protections

Hide Content from Bots

<!-- Content hidden via a CSS class (define .bot-trap { display:none } in your stylesheet) -->
<div class="bot-trap" aria-hidden="true">
  protected content
</div>

Note this is a weak measure: modern crawlers execute or parse hidden markup, and hiding real content from bots while showing it to users can be treated as cloaking by search engines. Use it only for genuine honeypot/trap content, never for pages you also want ranked.

JavaScript-Rendered Content

Render important content with JavaScript (harder for bots to scrape):

// Load sensitive content after page load
document.addEventListener('DOMContentLoaded', function() {
    fetch('/api/protected-content')
        .then(r => r.json())
        .then(data => {
            // Inject content
        });
});

Rate Limiting

Limit requests per IP/bot:

// Server-side rate limiting
const rateLimit = require('express-rate-limit');

const limiter = rateLimit({
    windowMs: 15 * 60 * 1000,
    max: 100,
    message: 'Too many requests'
});

app.use('/api/', limiter);

Legitimate Bot Allowlisting

Whitelist Search Engines

# Whitelist good bots
User-agent: Googlebot
Disallow:

User-agent: Bingbot
Disallow:

User-agent: Slurp
Disallow:

User-agent: DuckDuckBot
Disallow:

Tools to Monitor

Dark Visitors: darkvisitors.com
Tracks AI crawlers and their policies
Monitor what's visiting your site
Get updated lists of AI bots

Important Limitations to Understand

Bots Can Ignore Rules

Compliant bots: Follow robots.txt (~70% of bots) Partially compliant: Sometimes follow (~20% of bots) Non-compliant: Ignore entirely (~10% of bots)

User-Agent Spoofing

Scrapers can fake User-Agent headers:

Original: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
Spoofed: Mozilla/5.0 (looks like browser)

Defense requires deeper technical analysis (request patterns, IP reputation, etc.)

Legal Gray Areas

Scraping public content (legal in some jurisdictions)
robots.txt violations (not legally binding)
Terms of Service violations (only if enforced)
Copyright infringement (valid legal claim)

Comprehensive Strategy

For maximum protection:

robots.txt: Block known AI bots (minimal impact, better than nothing)
Firewall Rules: Use CloudFlare or server rules to block User-Agents
Rate Limiting: Limit requests per IP address
Monitoring: Track what's accessing your site
Terms of Service: Explicitly prohibit scraping
Copyright Notices: Claim copyright on content
Legal Action: DMCA/cease-and-desist for persistent violators
Server Config: Block at Apache/.htaccess or Nginx level
Licensing: Consider CC licenses with restrictions
Content Protection: JavaScript rendering, access controls for sensitive data

Important Note

Even with all protections, determined actors can circumvent defenses. The most important protections are:

Legal frameworks (ToS, copyright)
Technical hurdles (making scraping difficult)
Monitoring (catching violators)
Enforcement (legal action when needed)

robots.txt and User-Agent blocking are helpful but incomplete solutions.

Conclusion

Blocking AI scraper bots requires a multi-layered approach. Start with robots.txt to discourage compliant bots and reduce unnecessary load. Add firewall rules to actively block known AI bots. Implement rate limiting to prevent rapid scraping. Update your terms of service to explicitly prohibit automated access. For serious violations, pursue legal remedies. Understanding that robots.txt is advisory only ensures you don't rely exclusively on it but rather use it as part of a comprehensive protection strategy. Regular monitoring via tools like Dark Visitors helps you stay informed about new bots and adapt your defenses accordingly.

Frequently Asked Questions

Can robots.txt actually stop AI bots from scraping my site?

Only the ones that choose to obey it. robots.txt is a voluntary, advisory standard — it is a request, not a wall. Well-behaved crawlers like OpenAI's GPTBot, Google-Extended, and ClaudeBot publish their user-agent tokens and honor a Disallow rule. Bad actors and anonymized scrapers simply ignore the file, and because robots.txt is world-readable it never blocks access — it only asks. For crawlers that ignore the rules, you need enforcement at the network edge: a WAF, firewall user-agent blocks, or bot-management service that returns a 403 instead of politely asking.

What is the correct user-agent to block OpenAI in robots.txt?

OpenAI operates three distinct crawlers, each with its own token. GPTBot collects content to train models — this is the one most people want to block. ChatGPT-User fetches a page in real time when a ChatGPT user clicks a link or asks the assistant to browse. OAI-SearchBot indexes pages for ChatGPT search results. Blocking GPTBot stops training use; blocking OAI-SearchBot removes you from ChatGPT's search citations. List each on its own User-agent line with Disallow: / to cover all three.

Does blocking Google-Extended hurt my Google search rankings?

No. Google-Extended only controls whether your content is used to train and ground Google's generative-AI products (Gemini and Vertex AI). It is completely separate from Googlebot, which handles crawling and ranking for Google Search. You can Disallow Google-Extended and stay fully indexed and ranked in normal search results. The one is not a proxy for the other, and Google has confirmed Google-Extended has no effect on how a page is crawled, indexed, or ranked in Search.

Should I use anthropic-ai or ClaudeBot to block Anthropic?

ClaudeBot is Anthropic's current, active crawler user-agent — that is the one that matters today. The older anthropic-ai and Claude-Web tokens are legacy identifiers that Anthropic has largely retired, but there is no harm in listing all three, since an unused token in your robots.txt costs nothing. The safe pattern is to Disallow ClaudeBot, anthropic-ai, and Claude-Web together so you are covered regardless of which identifier a given request uses.

Will blocking AI crawlers hurt my SEO?

Blocking AI training crawlers (GPTBot, Google-Extended, ClaudeBot, CCBot) has no effect on traditional SEO, because Googlebot and Bingbot are separate crawlers with separate tokens that you leave allowed. The real trade-off is AI-search visibility: if you also block OAI-SearchBot and PerplexityBot, your pages can no longer be cited in ChatGPT search or Perplexity answers. Decide deliberately — many sites block training bots but allow AI-search bots to preserve referral traffic from AI answer engines.

What do I do about AI bots that ignore robots.txt?

Move from a request to enforcement. robots.txt cannot block anything by itself, so add a layer that actually returns a 403: a Cloudflare (or other CDN) bot-management rule, a WAF user-agent block, or server rules in .htaccess or nginx. For scrapers that spoof a browser user-agent, you need behavioral defenses — rate limiting per IP, IP-reputation blocking, and CAPTCHA challenges — because there is no user-agent string to match on. Cloudflare and Fastly also offer one-click 'block AI bots' managed rules that maintain the token list for you.

How do I block every AI crawler at once?

There is no single wildcard that targets 'AI bots' specifically — User-agent: * would also block Google and Bing. Instead, maintain an explicit block list of known AI user-agent tokens (GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, ClaudeBot, anthropic-ai, CCBot, PerplexityBot, Bytespider, Amazonbot, Applebot-Extended, Meta-ExternalAgent) each with Disallow: /, while explicitly allowing Googlebot and Bingbot. Keep the list current using a service like Dark Visitors, which tracks new AI crawlers and their published tokens.

Does blocking CCBot stop AI models from training on my content?

Partly, and only going forward. CCBot is Common Crawl's crawler, and Common Crawl's public dataset is a major training source for many LLMs, so a Disallow on CCBot removes your content from future Common Crawl snapshots. But it does nothing about data already collected in past crawls, and it does not stop AI companies that crawl you directly rather than via Common Crawl. Block CCBot as one line of defense, not as a complete solution.

What is Applebot-Extended and how is it different from Applebot?

Applebot is Apple's crawler for Siri and Spotlight search suggestions — you generally want to allow it. Applebot-Extended is a separate opt-out token that specifically controls whether Apple may use your already-crawled content to train its Apple Intelligence and foundation models. Disallowing Applebot-Extended keeps you in Apple's search features while removing your content from its AI training, exactly like the Applebot / Applebot-Extended split mirrors Googlebot / Google-Extended.

robots.txtAI scrapersweb scrapingcontent protection

How do I block AI scrapers and LLM training bots?

Frequently Asked Questions

Free tools you can use right now

Related articles

What are the SEO best practices for robots.txt?

How Do I Use robots.txt for Different Environments (Staging vs Production)?

How Does Robots.txt Relate to Security and Privacy?