Refuse AI training in the form each AI company honors. Stay in search and in the AI answers that cite you. Put a contract and a price on everything else.
Updated
Each AI company that trains on the web names its own crawler or robots.txt token for it. Refuse them all by name, and AI training stops without touching your search traffic.
Googlebot, Bingbot and Applebot bring your search traffic and also feed AI Overviews, Copilot and Siri. Keep them, and refuse their AI uses with each vendor's own setting.
Bots that load your page to answer a question meet FORA on the paths you sell: a 403 that tells them where to buy. A search-only contract covers whoever hides.
The AI companies treat their uses as allowed unless a site refuses them, and there is no single signal they all honor. So a refusal has to be said in each vendor's own form, in up to four places.
The same company often runs three bots: one that trains, one that builds its AI search, and one that loads a page when a person asks. Each gets a different answer.
Refused in robots.txt across the site, and a 403 on the paths you sell.
| Name | Vendor | What it's for | Follows robots.txt |
|---|---|---|---|
GPTBot | OpenAI | Model training | Yes |
ClaudeBot | Anthropic | Model training | Yes |
Meta-ExternalAgent | Meta | Model training and indexing | Yes |
Amazonbot | Amazon | Model training and Amazon products | Yes |
MistralAI-Training | Mistral | Model training | Yes |
CCBot | Common Crawl | An open web corpus that AI labs train on | Yes |
Bytespider | ByteDance | Crawling for ByteDance | Not stated |
AI2Bot | Allen Institute for AI | Training open language models | Not stated |
Diffbot | Diffbot | A web knowledge graph and datasets | Not stated |
Omgilibot | Webz.io | Web datasets | Not stated |
A 403 on the paths you sell, where they can buy. Many don't follow robots.txt.
| Name | Vendor | What it's for | Follows robots.txt | Refusing it costs |
|---|---|---|---|---|
ChatGPT-User | OpenAI | Loads a page when a ChatGPT user asks | Not always | — |
Claude-User | Anthropic | Loads a page when a Claude user asks | Yes | — |
Perplexity-User | Perplexity | Loads a page when a Perplexity user asks | Not always | — |
Google-Agent | Agents on Google infrastructure that browse when a user asks | Not always | — | |
Meta-ExternalFetcher | Meta | Loads a page when a user asks | Not always | — |
Amzn-User | Amazon | Loads a page when a user asks | Not always | — |
MistralAI-User | Mistral | Loads a page when a user asks | Not stated | — |
Your choice in robots.txt: refusing one takes you out of that vendor's AI search. On the paths you sell, FORA's edge function refuses them unless you let them through.
| Name | Vendor | What it's for | Follows robots.txt | Refusing it costs |
|---|---|---|---|---|
OAI-SearchBot | OpenAI | The index for ChatGPT search | Yes | Not shown in ChatGPT search answers |
Claude-SearchBot | Anthropic | The index for Claude's search | Yes | Less visible in Claude's search results |
PerplexityBot | Perplexity | The index for Perplexity's results; not for training | Yes | Out of Perplexity's results |
Meta-WebIndexer | Meta | Citations and links in Meta AI | Not stated | Fewer citations in Meta AI |
Amzn-SearchBot | Amazon | Alexa search; not for training | Yes | Out of Alexa search |
MistralAI-Index | Mistral | Mistral's search index; not for training | Yes | Out of Mistral's search |
DuckAssistBot | DuckDuckGo | DuckAssist answers; not for training | Yes | Out of DuckAssist; search unaffected |
YouBot | You.com | You.com search | Yes | — |
Never refused: they bring your search traffic. Their AI uses are refused with the tokens and page settings below.
| Name | Vendor | What it's for | Follows robots.txt | Refusing it costs |
|---|---|---|---|---|
Googlebot | Google Search, including AI Overviews and AI Mode | Yes | Out of Google Search | |
Bingbot | Microsoft | Bing and Copilot | Yes | Out of Bing and Copilot |
Applebot | Apple | Spotlight, Siri and Safari, and context for Apple's AI answers | Yes | Out of Spotlight, Siri and Safari |
No crawler sends these names. They exist only to refuse a use in robots.txt.
| Name | Vendor | What it's for | Follows robots.txt |
|---|---|---|---|
Google-Extended | Gemini training, and grounding in Gemini Apps and Vertex AI; Google Search unaffected | — | |
Applebot-Extended | Apple | Apple's model training; search unaffected | — |
__usage_ai-training | Shared (IETF draft) | Training, for every crawler that honors it: Anthropic now, OpenAI announced, Google early 2027, Microsoft from 1 Feb 2027 | — |
From each vendor's own documentation, checked 11 November 2026. The list is also published as JSON.
Choose what you want, and copy what goes into your robots.txt, your page headers and your site's terms.
Add it to the file at the root of your site. A crawler combines every group that names it, so this works next to the groups you already have.
# AI uses this site does not license without an offer User-agent: GPTBot User-agent: ClaudeBot User-agent: Meta-ExternalAgent User-agent: Amazonbot User-agent: MistralAI-Training User-agent: CCBot User-agent: Bytespider User-agent: AI2Bot User-agent: Diffbot User-agent: Omgilibot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: __usage_ai-training Disallow: / # License https://m4ow.uk/socw/3.txt User-agent: * Content-Signal: search=yes, ai-train=no tdl: https://m4ow.uk/socw/3.txt Allow: /
Set them on every page, at your origin or on your CDN. Or use the HTML tags below instead.
X-Robots-Tag: Bingbot: ai-training:disallow tdm-reservation: 1 Link: <https://m4ow.uk/socw/3.txt>; rel="terms-of-service" Link: <https://publisher.example/terms>; rel="terms-of-service"
The same signals, in the <head> of every page.
<meta name="Bingbot" content="ai-training:disallow" /> <meta name="tdm-reservation" content="1" /> <link rel="terms-of-service" href="https://m4ow.uk/socw/3.txt" /> <link rel="terms-of-service" href="https://publisher.example/terms" />
The Movement for an Open Web's Search Only Terms Contract binds whoever accesses your site, however they identify. Search stays free; every other access owes the fee you set.
tdl line in the User-agent: * group tells every crawler where the contract is.No. Google says Google-Extended "does not impact a site's inclusion in Google Search". Apple says the same of Applebot-Extended.
Yes, with the "Search generative AI" setting in Google Search Console. No robots.txt line does it without taking you out of Google Search too, because AI Overviews draw on Googlebot, the crawler that brings your search traffic.
Perplexity says PerplexityBot builds its search index and is not used to train models, so refusing it takes you out of Perplexity's results. On the paths you sell, FORA's edge function refuses it by default, and you can let it through. Perplexity-User loads pages when someone asks and generally ignores robots.txt: on the paths you sell, FORA gives it a 403.
It is a request, which the AI companies document that they follow. The search-only contract turns the same lines into terms of access, and FORA turns the access into a sale with a signed receipt.
No. The contract is one fixed document at the Movement for an Open Web's address, and you link to it. You set your fee in your own site's terms with the guide's wording, and the contract takes the fee "in the amount set out in the Website Terms and Conditions". A crawler follows robots.txt to the contract, and the contract to your terms. The wording to paste.
None yet. The shared __usage_ai-training token comes closest: Anthropic recognizes it now, OpenAI has announced it, Google plans early 2027 and Microsoft starts on 1 February 2027. The generator includes it next to each vendor's own name.
Every bot is checked against its vendor's documentation, and the date is under the table. The same list is published as JSON for tools that generate from it.
See what an AI agent gets for one of your pages.