Resource · For publishers

What you can do about
AI crawlers. Today.

Refuse AI training in the form each AI company honors. Stay in search and in the AI answers that cite you. Put a contract and a price on everything else.

Updated

The short answer

Refuse AI training. Stay in search. Sell the rest.

Refuse AI training

Each AI company that trains on the web names its own crawler or robots.txt token for it. Refuse them all by name, and AI training stops without touching your search traffic.

Stay in search

Googlebot, Bingbot and Applebot bring your search traffic and also feed AI Overviews, Copilot and Siri. Keep them, and refuse their AI uses with each vendor's own setting.

Sell the rest

Bots that load your page to answer a question meet FORA on the paths you sell: a 403 that tells them where to buy. A search-only contract covers whoever hides.

Why one line is not enough

Each AI company reads its own refusal

The AI companies treat their uses as allowed unless a site refuses them, and there is no single signal they all honor. So a refusal has to be said in each vendor's own form, in up to four places.

Every bot

Who is crawling, and what for

The same company often runs three bots: one that trains, one that builds its AI search, and one that loads a page when a person asks. Each gets a different answer.

Training and datasets

Refused in robots.txt across the site, and a 403 on the paths you sell.

NameVendorWhat it's forFollows robots.txt
GPTBot OpenAI Model training Yes
ClaudeBot Anthropic Model training Yes
Meta-ExternalAgent Meta Model training and indexing Yes
Amazonbot Amazon Model training and Amazon products Yes
MistralAI-Training Mistral Model training Yes
CCBot Common Crawl An open web corpus that AI labs train on Yes
Bytespider ByteDance Crawling for ByteDance Not stated
AI2Bot Allen Institute for AI Training open language models Not stated
Diffbot Diffbot A web knowledge graph and datasets Not stated
Omgilibot Webz.io Web datasets Not stated

Live fetch

A 403 on the paths you sell, where they can buy. Many don't follow robots.txt.

NameVendorWhat it's forFollows robots.txtRefusing it costs
ChatGPT-User OpenAI Loads a page when a ChatGPT user asks Not always —
Claude-User Anthropic Loads a page when a Claude user asks Yes —
Perplexity-User Perplexity Loads a page when a Perplexity user asks Not always —
Google-Agent Google Agents on Google infrastructure that browse when a user asks Not always —
Meta-ExternalFetcher Meta Loads a page when a user asks Not always —
Amzn-User Amazon Loads a page when a user asks Not always —
MistralAI-User Mistral Loads a page when a user asks Not stated —

AI search

Your choice in robots.txt: refusing one takes you out of that vendor's AI search. On the paths you sell, FORA's edge function refuses them unless you let them through.

NameVendorWhat it's forFollows robots.txtRefusing it costs
OAI-SearchBot OpenAI The index for ChatGPT search Yes Not shown in ChatGPT search answers
Claude-SearchBot Anthropic The index for Claude's search Yes Less visible in Claude's search results
PerplexityBot Perplexity The index for Perplexity's results; not for training Yes Out of Perplexity's results
Meta-WebIndexer Meta Citations and links in Meta AI Not stated Fewer citations in Meta AI
Amzn-SearchBot Amazon Alexa search; not for training Yes Out of Alexa search
MistralAI-Index Mistral Mistral's search index; not for training Yes Out of Mistral's search
DuckAssistBot DuckDuckGo DuckAssist answers; not for training Yes Out of DuckAssist; search unaffected
YouBot You.com You.com search Yes —

Search crawlers that also feed AI

Never refused: they bring your search traffic. Their AI uses are refused with the tokens and page settings below.

NameVendorWhat it's forFollows robots.txtRefusing it costs
Googlebot Google Google Search, including AI Overviews and AI Mode Yes Out of Google Search
Bingbot Microsoft Bing and Copilot Yes Out of Bing and Copilot
Applebot Apple Spotlight, Siri and Safari, and context for Apple's AI answers Yes Out of Spotlight, Siri and Safari

robots.txt tokens

No crawler sends these names. They exist only to refuse a use in robots.txt.

NameVendorWhat it's forFollows robots.txt
Google-Extended Google Gemini training, and grounding in Gemini Apps and Vertex AI; Google Search unaffected —
Applebot-Extended Apple Apple's model training; search unaffected —
__usage_ai-training Shared (IETF draft) Training, for every crawler that honors it: Anthropic now, OpenAI announced, Google early 2027, Microsoft from 1 Feb 2027 —

From each vendor's own documentation, checked 11 November 2026. The list is also published as JSON.

Generator

Generate your rules

Choose what you want, and copy what goes into your robots.txt, your page headers and your site's terms.

Your choices

robots.txt

Add it to the file at the root of your site. A crawler combines every group that names it, so this works next to the groups you already have.

# AI uses this site does not license without an offer
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Bytespider
User-agent: AI2Bot
User-agent: Diffbot
User-agent: Omgilibot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: __usage_ai-training
Disallow: /

# License https://m4ow.uk/socw/3.txt
User-agent: *
Content-Signal: search=yes, ai-train=no
tdl: https://m4ow.uk/socw/3.txt
Allow: /

Response headers

Set them on every page, at your origin or on your CDN. Or use the HTML tags below instead.

X-Robots-Tag: Bingbot: ai-training:disallow
tdm-reservation: 1
Link: <https://m4ow.uk/socw/3.txt>; rel="terms-of-service"
Link: <https://publisher.example/terms>; rel="terms-of-service"

HTML tags

The same signals, in the <head> of every page.

<meta name="Bingbot" content="ai-training:disallow" />
<meta name="tdm-reservation" content="1" />
<link rel="terms-of-service" href="https://m4ow.uk/socw/3.txt" />
<link rel="terms-of-service" href="https://publisher.example/terms" />

By hand

  • Set your access fee in your terms page: paste the guide's wording (A), which incorporates the contract, and (B), which states your fee, and fill in the brackets. The contract itself stays unchanged: the wording to paste.
  • Archive your robots.txt with the Internet Archive after each change, as dated evidence.
  • Put FORA's edge function on the paths you sell. Live fetchers that ignore robots.txt get a 403 that tells them where to buy. By default it refuses the AI search crawlers there too. To keep those paths in AI search, let the crawlers through.
The limits

What robots.txt can't stop

Put it in writing

Set your price for everything beyond search

The Movement for an Open Web's Search Only Terms Contract binds whoever accesses your site, however they identify. Search stays free; every other access owes the fee you set.

  1. Set your access fee. The contract sets no default. Choose a fee per item: one price for everything, or one each for written works, images, video, audio and other items. Set it on your own, without agreeing it with other publishers.
  2. Add the contract to your site's terms. Paste two pieces of wording from MOW's guide into your terms page and fill in the brackets: (A) incorporates the contract by its fixed URL, and (B) states your fee. The contract itself stays unchanged; it points to your terms for the fee. The wording.
  3. Add it to robots.txt. A tdl line in the User-agent: * group tells every crawler where the contract is.
  4. Keep the evidence. Screenshot each change, and archive your robots.txt with the Internet Archive after you update it.
robots.txtsearch-only contract
User-agent:
*
tdl:
https://m4ow.uk/socw/3.txt
Allow:
/
Questions

What publishers ask

Will refusing AI training hurt my Google ranking?

No. Google says Google-Extended "does not impact a site's inclusion in Google Search". Apple says the same of Applebot-Extended.

Can I keep my pages out of AI Overviews?

Yes, with the "Search generative AI" setting in Google Search Console. No robots.txt line does it without taking you out of Google Search too, because AI Overviews draw on Googlebot, the crawler that brings your search traffic.

What about Perplexity?

Perplexity says PerplexityBot builds its search index and is not used to train models, so refusing it takes you out of Perplexity's results. On the paths you sell, FORA's edge function refuses it by default, and you can let it through. Perplexity-User loads pages when someone asks and generally ignores robots.txt: on the paths you sell, FORA gives it a 403.

Does robots.txt bind anyone?

It is a request, which the AI companies document that they follow. The search-only contract turns the same lines into terms of access, and FORA turns the access into a sale with a signed receipt.

Do I edit the search-only contract to set my price?

No. The contract is one fixed document at the Movement for an Open Web's address, and you link to it. You set your fee in your own site's terms with the guide's wording, and the contract takes the fee "in the amount set out in the Website Terms and Conditions". A crawler follows robots.txt to the contract, and the contract to your terms. The wording to paste.

Which line refuses every AI company at once?

None yet. The shared __usage_ai-training token comes closest: Anthropic recognizes it now, OpenAI has announced it, Google plans early 2027 and Microsoft starts on 1 February 2027. The generator includes it next to each vendor's own name.

How current is the list?

Every bot is checked against its vendor's documentation, and the date is under the table. The same list is published as JSON for tools that generate from it.

Every AI agent is a customer. FORA is your storefront.

See what an AI agent gets for one of your pages.

Try your website FORA for publishers