# How do you protect your website content from AI crawlers?

Date: 2026-07-16 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/protect-website-content-ai-crawlers/

**Answer:** Start with a robots.txt file that names each AI crawler and says which parts of your site it may visit, choosing separately between training crawlers and search crawlers. Then back it up with server or network rules, because robots.txt is a polite request that well-behaved bots follow, not a lock.

A **crawler** is a program that visits web pages automatically and saves what it finds. Search engines have used crawlers for decades. AI companies now run their own, some to collect text for training models and some to fetch pages that an assistant quotes in its answers. If you publish articles, recipes, photos or course material, you can decide how much of it those bots receive.

The main tool is old and simple: a text file called **robots.txt** at the root of your site, such as `example.com/robots.txt`. It lists rules for named bots. The Internet Engineering Task Force, the body that writes many internet standards, formalized this format in 2022 as RFC 9309. That standard is clear about one limit that shapes everything below: the rules are a request that crawlers are expected to honor, and in its words, "These rules are not a form of access authorization."

## Training bots and search bots are different decisions

The most useful step is to stop thinking of "AI bots" as one group. OpenAI, for example, documents three agents with different jobs:

| Name in robots.txt | What it does | If you block it |
|---|---|---|
| GPTBot | Collects pages that may be used to train OpenAI's foundation models | Your content is marked as not to be used for that training |
| OAI-SearchBot | Finds pages to show in ChatGPT's search answers | Your pages stop appearing in those answers, though a plain link to your site may still show |
| ChatGPT-User | Visits a page when a person asks ChatGPT or a custom GPT to look at it | OpenAI says robots.txt rules may not apply, because a user started the visit |

OpenAI states that each setting is independent. So a news site can refuse training while still being found and cited in ChatGPT search, which many publishers see as a source of readers. Other AI companies publish their own crawler names and rules; check each provider's documentation for the exact token, the name a bot uses to find its section of your file.

## A worked robots.txt

Here is a file for a small publisher who wants to stay visible in AI search but keep articles out of training, and who keeps a members' area off limits to every bot.

```text
# Keep content out of OpenAI model training
User-agent: GPTBot
Disallow: /

# Allow OpenAI's search crawler, except the members' area
User-agent: OAI-SearchBot
Allow: /
Disallow: /members/

# Every other crawler: stay out of the members' area
User-agent: *
Disallow: /members/
```

How a crawler reads it: it looks for a group whose `User-agent` line matches its own name, ignoring capital letters. If nothing matches, it follows the `*` group. Inside a group, `Disallow: /` means "nothing on this site," and a path like `/members/` covers everything that starts with it.

Three practical details from the standard and OpenAI's documentation:

- Crawlers may keep a cached copy of your file, but the standard says not for more than 24 hours. OpenAI says its search systems can take about a day to reflect a change.
- Every path you list is public. Writing `Disallow: /drafts/` tells the whole world that `/drafts/` exists.
- A crawler can claim any name. OpenAI publishes the IP address ranges its bots use, so your host or firewall can check that a visitor calling itself GPTBot really comes from OpenAI.

> **Try it:** Open `yourdomain.com/robots.txt` in a browser. Write down which AI crawlers it mentions, then decide, for each one, whether you want it for training, for search, both or neither, and draft the groups you would add.

## What blocking can and cannot do

Robots.txt **can** tell cooperative crawlers what you want, and companies such as OpenAI document how their crawlers read it. It is free, takes minutes, and can be changed whenever your view changes.

It **cannot** stop a bot that ignores it, remove content that was collected before you added the rule, or control copies of your work that others have reposted elsewhere. It does not protect private pages either: the standard advises using real security, such as a login, for anything that must stay closed.

For actual enforcement, you need rules at the server or network level. Hosting companies and content delivery networks (services that sit in front of your site and serve it quickly worldwide) can refuse requests from specific bots, check that a bot is who it says it is, and show you how often each one visits. Keep the robots.txt file too: it documents your intent, and that record can matter if you later negotiate a licence or raise a complaint.

## Can you charge them for access?

Yes, through some services, though it is still a new and small market. In July 2025 Cloudflare, a content delivery network, announced **pay per crawl** as a private beta. It builds on an HTTP status code that web servers had rarely used: **402 Payment Required**.

The idea works like a toll booth. The site owner sets one price per request for the whole domain, then picks, crawler by crawler, one of three choices: allow for free, charge, or block. A crawler that wants a paid page either states in its request how much it is willing to pay or receives a 402 reply that names the price. If it agrees, it gets the page and Cloudflare records a charge, collects the money from the crawler and passes it to the publisher. To stop impostors from running up someone else's bill, crawlers must prove their identity by signing each request with a cryptographic key they registered in advance.

Two limits are worth knowing before you count on income. Only crawlers that sign up can pay; for any other bot, "charge" behaves like a block, though it signals that a deal is possible. And the launch version offered one flat price per site, with Cloudflare itself saying that richer options, such as different prices for different sections or licences for training versus search, are ideas for later. If you run a site with valuable archives, the practical path today is to set clear robots.txt rules, enforce them at the network level, and ask your host whether it supports paid crawling or direct licensing.

## Sources

- Overview of OpenAI Crawlers: https://platform.openai.com/docs/bots
- RFC 9309: Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309
- Introducing pay per crawl: Enabling content owners to charge AI crawlers for access: https://blog.cloudflare.com/introducing-pay-per-crawl/