How do you stop AI companies from training on your website?

If you publish recipes, lessons, photos or articles, some of that work has probably been read by a crawler, a program that visits pages automatically and stores what it finds. Several AI companies now separate their crawlers by purpose: one collects pages that may be used to train future models, another fetches pages so an assistant can cite them in search answers. That split is what makes a sensible opt-out possible. You can say "no" to training and "yes" to being found.

The main tool is still robots.txt, a plain text file at the root of your site (for example example.ca/robots.txt). Its format is standardized as RFC 9309, published by the Internet Engineering Task Force (IETF) in 2022. Each crawler identifies itself by a name, called a product token, and looks for the group of rules addressed to that name.

Know which crawler does what

OpenAI's crawler documentation is a clear example of the split. As of October 2026 it lists three agents:

The two robots.txt settings are independent, so you can refuse one and accept the other. Other companies publish their own names; Google, for instance, uses a separate token called Google-Extended for its generative AI models. Check each company's documentation for the current list before you write your file, because names and versions change.

A worked robots.txt for "no training, yes search"

Here is a small file for a site that wants to stay out of OpenAI's training data but remain visible in ChatGPT search, and that keeps a private folder away from every crawler:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
Disallow: /members/

User-agent: *
Disallow: /members/

Three details matter. First, a crawler follows only the group that matches its name; if a named group exists, the * group no longer applies to it, which is why /members/ appears twice. Second, under RFC 9309 crawlers should not rely on a cached copy of your file for more than 24 hours, and OpenAI says its search systems take roughly a day to adjust after a change. Third, every path you list becomes public: robots.txt tells the world where your private folder is, so it never replaces a password.

If you are unsure about the syntax, an assistant can draft and explain the file for you:

Write a robots.txt file for my website.
Goals: refuse crawlers that collect content for AI model training,
allow AI search crawlers so my pages can be cited, and keep /members/
closed to all crawlers. Use only crawler names that the companies
document publicly, and add a comment above each group explaining it.
Then list anything robots.txt cannot do for me.

Do robots.txt and opt-out tags really work?

Partly. robots.txt works well with crawlers whose owners choose to respect it, and the large AI labs say theirs do. But the standard itself says plainly that these rules do not authorize or deny access: nothing technical stops a crawler that ignores them, or one that pretends to be another bot. OpenAI publishes the IP address ranges its crawlers use, so a site can check that a visitor calling itself GPTBot really is one, and server or firewall rules can then block what robots.txt only asks.

There are also gaps no file can close. A refusal applies from the day crawlers read it; it does not pull your pages out of models already trained or datasets already copied. Visits triggered by a user, like ChatGPT-User, may not follow robots.txt at all. And your content can travel through third parties, such as a site that republishes your articles, where your file has no say.

Opt-out tags are the next step, and they are not finished. The IETF's AI Preferences working group is writing a shared vocabulary for statements such as "you may use this for search but not for training," along with ways to attach those statements to content through robots.txt and HTTP response headers. As of October 2026 both documents are still Internet-Drafts, revised in August and September 2026, not published standards. Crawlers may not read them yet, so treat any draft syntax as an experiment, not a guarantee.

Where you are matters too. In the European Union, a machine-readable reservation of rights has legal force: the AI Act's Article 53 requires providers of general-purpose models placed on the EU market to identify and respect such reservations made under the 2019 copyright directive. Canada's Copyright Act has no comparable opt-out rule, so a Canadian site relies on robots.txt, its terms of use and, if it comes to that, the courts. If your content is a business asset worth defending, a lawyer can tell you which of those levers fits your case.