How to block AI scrapers from your website (2026 guide)

If you own a website with anything original on it — articles, transcripts, recipes, tutorials, research — every major AI lab is treating it as free training data unless you tell them not to. This guide is the short, practical answer to how you actually stop them in 2026, ordered from the easiest thing you can do tonight to the actual enforcement layers your hosting provider gives you for free.

The short answer

Three layers, applied together:

1. robots.txt — declares your intent. Compliant crawlers (OpenAI's GPTBot, Anthropic's ClaudeBot, Google's Google-Extended, Perplexity's PerplexityBot, and others) read it before they crawl. Not enforceable, but the easiest, fastest first step.
2. Server-side blocking — Nginx deny, Apache .htaccess, or Cloudflare's one-click "Block AI Scrapers and Crawlers." This actually refuses the request at the edge, so even non-compliant bots can't get the page.
3. Authenticated paywalls or login walls — the only real lock. Content behind credentials cannot be scraped at scale without breaking the law.

Do step 1 in five minutes. Do step 2 the same day. Reserve step 3 for the work that's worth most to you.

Why robots.txt alone is not enough

For ten years, robots.txt was a polite request and Google honoured it because Google had a brand to protect. AI labs are newer, more under-resourced on legal exposure, and there are more of them. As of June 2026 there are at least 24 named AI training crawlers in active circulation, and a non-trivial number of scrapers that send fake user-agents to pretend they're a browser.

robots.txt is still worth doing — most of the named, compliant bots from the big labs do honour it, and once you have a clear Disallow: / on a bot, you can credibly say in any future legal proceeding that you withdrew permission. But treat it as a public statement of intent, not a wall.

Step 1 — the robots.txt that actually covers the field

Drop this at https://yoursite.com/robots.txt. It's the working 2026 list:

# AI training crawlers — disallowed
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: Omgili
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Meta-ExternalFetcher
Disallow: /

User-agent: DuckAssistBot
Disallow: /

User-agent: YouBot
Disallow: /

User-agent: TimpiBot
Disallow: /

User-agent: Kangaroo
Disallow: /

# Allow normal search engines to keep indexing
User-agent: Googlebot
Disallow:

User-agent: Bingbot
Disallow:

A few notes on the distinctions, because they catch people out:

• Googlebot vs Google-Extended — Googlebot is the regular search crawler. Google-Extended is the separate AI-training user-agent. Blocking Google-Extended does not affect your Google Search ranking. It only opts you out of Bard/Gemini training.
• Applebot vs Applebot-Extended — same pattern. Applebot-Extended is the AI-training variant. The base Applebot is still used for Siri Suggestions and Spotlight.
• GPTBot vs ChatGPT-User vs OAI-SearchBot — three different OpenAI agents. GPTBot is for model training, ChatGPT-User is live browsing inside a ChatGPT session, OAI-SearchBot is for ChatGPT Search. They have different policies. Block all three if you want full coverage.

Step 2 — actual enforcement at the edge

robots.txt is voluntary. The crawler reads it, decides whether to honour it, and you have no recourse. To actually refuse connections, you block at the server or CDN layer.

If you use Cloudflare

The fastest path. Cloudflare ships a one-click toggle: Security → Bots → AI Scrapers and Crawlers → Block. It uses Cloudflare's bot fingerprinting (which is much harder to fake than a user-agent string) and refuses the request before it ever reaches your origin. Free plan included. As of mid-2026 it covers all the same agents listed above plus a long tail Cloudflare has identified through traffic analysis. If you only do one thing on this list, do this.

If you use Nginx

In your server block:

if ($http_user_agent ~* (GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-Web|anthropic-ai|Google-Extended|PerplexityBot|Perplexity-User|CCBot|Bytespider|Amazonbot|Applebot-Extended|Diffbot|cohere-ai|FacebookBot|Meta-ExternalAgent|Meta-ExternalFetcher|DuckAssistBot|YouBot)) {
    return 403;
}

If you use Apache (.htaccess)

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-Web|anthropic-ai|Google-Extended|PerplexityBot|Perplexity-User|CCBot|Bytespider|Amazonbot|Applebot-Extended|Diffbot|cohere-ai|FacebookBot|Meta-ExternalAgent|Meta-ExternalFetcher|DuckAssistBot|YouBot) [NC]
RewriteRule .* - [F,L]

Both return HTTP 403 to the bot. Both rely on the user-agent string, so a scraper that lies about who it is will get through. That's why Cloudflare's fingerprinting is materially better than user-agent matching — but matching is still useful because most named crawlers do identify themselves honestly. They want to be in the conversation about future deals.

Step 3 — paywalls and login walls

This is the only category that prevents scraping rather than discourages it. If the content sits behind a credential check, it's not freely accessible, and any scraper that wants it has to either steal credentials or break the terms of service. Both raise the legal stakes high enough that the major labs simply don't bother.

This is why the New York Times put its archive behind a metered paywall before suing OpenAI: scraped public-web content is harder to litigate than scraped paywalled content. The wall is the proof of intent.

A few practical patterns:

• Free article allowance (the metered paywall): the NYT, Substack, Bloomberg model. First three articles free, then a wall. AI crawlers see exactly what an anonymous visitor sees, which is the first three.
• Archive paywall: latest articles free, anything older than 30 days behind a login. Common at trade publications. Locks the training corpus (back catalogue) while keeping the promotional surface (new posts) indexable.
• Members-only podcast feeds: a private RSS URL given to paid members. The public feed is the trailer.

The trade-off is real: anything behind a wall is also invisible to Google and to potential readers. Most independent creators won't paywall their whole site. But for the back catalogue you'd want to license to AI buyers — the work that's actually valuable as training data — putting it behind a wall is what makes it scarce, and scarcity is what makes it priced.

Has it already been taken?

This is the question to answer before you do any of the above, because the answer changes your strategy.

If your work is already inside Common Crawl (a free, public dataset that most large language models have trained on), every major AI lab already has it. You can still block future crawls, but the back catalogue is already in their training data. Your leverage going forward is over new work, not the old.

If your work is not in Common Crawl — usually because it was paywalled, behind a login, or simply too obscure to be crawled comprehensively — then you have rare training data and can charge accordingly.

There's a free check that tells you which side of this line your site sits on, and how many of the named AI bots have visited you in the past 90 days, in about 30 seconds. We built it as part of ArchiveBay: the scrape check. Paste your domain, get an answer.

What about ai.txt?

ai.txt was a 2023 proposal for an AI-specific equivalent of robots.txt. It briefly looked like it would be adopted. As of mid-2026, the standards bodies haven't ratified it, the major labs don't read it, and the recommendation is: don't bother. Stick with robots.txt for declaration and Cloudflare/Nginx/Apache for enforcement.

If ai.txt ever lands as a real standard, the same Disallow: semantics will likely apply and your existing robots.txt will roughly translate. We'll update this post if that changes.

What about lawsuits?

The honest answer in mid-2026: the case law is unsettled. The New York Times v. OpenAI litigation is still grinding through discovery. Getty Images v. Stability AI has produced partial rulings on the input side but nothing dispositive on outputs. Several class actions are open. Until those resolve, the practical effect of "the AI labs might lose in court eventually" on a scraper's decision today is zero.

What is producing actual revenue for creators right now is licensing — voluntary, paid agreements between AI buyers and rights holders. The same companies that are scraping for free will pay for cleaner, structured, attributable data when it's offered in a form their lawyers don't have to argue about. That's what ArchiveBay does for independent creators, the same way Shutterstock did it for the big publishers.

The minimum viable defence

If you have ten minutes:

1. Drop the robots.txt block above onto your site.
2. If you're on Cloudflare, flip the Block AI Scrapers toggle.
3. Run the scrape check to find out what's already happened.

If you have an hour, do the Nginx or Apache rule too. If you have a week, think about which parts of your archive are worth paywalling and which are worth keeping public as promotional surface.

The strategy is not "keep AI from ever seeing your work." That ship has sailed for most public content. The strategy is to make sure you control your future work, you know what's already been taken, and the work that's most valuable to AI buyers stays scarce enough to actually price. That's what makes your archive an asset rather than a thing somebody else trained on for free.

---

Once you've locked the door, the next question is whether what's behind it is worth something to AI buyers. Run the free valuation tool — it gives you a price range in about 30 seconds based on the size, age, and content type of your archive.