How do you keep your work from being used to train AI?

Large language models learn from enormous amounts of text gathered from the public web, and image models learn from pictures gathered the same way. If you publish writing, photos, art or code online, some of it may already be in a training set. Many creators now want a say in whether their future work joins those sets, and since 2023 a few practical controls have appeared.

They fall into two groups: controls for things you publish on a website you run, and controls for things you type into AI tools. Each works differently, and each has limits worth knowing before you rely on it.

How robots.txt works

A web crawler is a program that visits pages automatically and saves what it finds. Search engines use crawlers to build their indexes, and AI companies use them to collect training text. Robots.txt is a plain text file at the root of a website (for example example.com/robots.txt) that tells crawlers which parts of the site they may visit. Each crawler identifies itself with a name, called a user agent, and the file gives rules per name.

Two AI-specific names matter most today:

• GPTBot is OpenAI's crawler. Since August 2023, OpenAI has said site owners can block it in robots.txt or by blocking its IP addresses. OpenAI says pages it collects may be used to improve future models, after filtering out paywalled sources, sites known to gather personal information, and text that breaks its policies.
• Google-Extended was announced by Google in September 2023. It is a name you can use in robots.txt to say whether content Google collects may help improve its Bard chatbot and its Vertex AI generative services, including future model generations.

A worked robots.txt

Here is a file for a small artist's portfolio site that wants to stay in normal search results but opt out of these two AI training uses:

# Allow ordinary search engines and other visitors
User-agent: *
Allow: /

# Opt out of OpenAI model training
User-agent: GPTBot
Disallow: /

# Opt out of Google's generative AI training (Bard, Vertex AI)
User-agent: Google-Extended
Disallow: /

Disallow: / means "nothing on this site." If you only want to protect one section, such as a gallery, you can write Disallow: /gallery/ instead and leave the rest open.

To install it, save the text as robots.txt and upload it to the top folder of your site, so that it opens at yoursite.com/robots.txt. Many website builders and blogging platforms have a robots.txt or "crawlers" setting instead of file access; look in the SEO or privacy section of your dashboard.

Controls for what you type into chatbots

Your prompts and uploaded files are a second route into training data. In ChatGPT, OpenAI says what you type may be used to improve its models, and you can switch that off in Settings, whether or not you have an account. Other chatbots publish their own policies, and they differ from one tool to the next, so read the data or privacy page of each tool you use. If you use AI for client or company work, also check whether your plan's terms exclude your data from training.

Limits to keep in mind

• Nothing is removed retroactively. Blocking GPTBot today does not take your pages out of data collected before you added the rule.
• Robots.txt is voluntary. Reputable companies say they follow it, but it is an instruction, not a lock. A crawler that ignores it can still read public pages.
• Each company uses its own name. A rule for GPTBot says nothing to other crawlers. New AI crawler names appear regularly, so revisit your file from time to time.
• Copies elsewhere are outside your control. If your work has been reposted on another site, that site's robots.txt applies, not yours.

What opt-outs actually exist?

As of June 2024, the dependable ones are few and specific. For websites, there are crawler rules in robots.txt: GPTBot for OpenAI and Google-Extended for Google's generative models, with similar published names from some other companies. You can also block crawlers at the server level, by user agent or by IP address, which is firmer than a robots.txt request but takes more technical work.

For chatbots, there are account settings: the ChatGPT setting that stops your conversations from improving its models, and the equivalent controls or policies in other tools. Where a business or developer plan promises not to train on your data, that promise is another form of opt-out, built into the contract rather than a switch.

What does not exist yet is a single, universal "do not train" signal that every AI company recognizes, or a way to pull your work out of models that have already been trained. Google itself said, when it announced Google-Extended, that it wants to explore more machine-readable ways for publishers to state their choices. Until something broader appears, the practical approach is layered: set your robots.txt, adjust the settings of the tools you use, keep originals and publication dates of your work, and watch for new controls as companies add them.