Resources · Content marketplaces · Cloudflare

Cloudflare charges AI crawlers on its network. With FORA, you sell a license per resource.

If your site runs through Cloudflare, Cloudflare can charge an AI crawler (a program that fetches pages in bulk) for each page it takes, and pay you. FORA lets you set terms and a price for each resource, sold through the Exchanges you choose. Each sale leaves a signed record of what was bought, and on what terms.

Updated

At a glance

Cloudflare and FORA, side by side

The facts a revenue team and an engineering lead act on. Cloudflare's cells describe Pay Per Crawl, and Pay Per Use where it differs.

CloudflareFORA
Money
Who pays you, and when Cloudflare, as merchant of record, monthly, to a new Stripe Connect account, after settlement periods and payout thresholds [1, 2]. Your Exchange operator, under its own terms. See About FORA.
Fees Not public. Cloudflare's docs, posts and terms state no fee. No Exchange fee at FORA's publisher console, beyond the payment system's. See About FORA.
What you see Which crawlers visit and how often they're charged. Your earnings balance isn't in the dashboard [3]. Each sale with its usage-report status, and daily income with a CSV download, in your operator's publisher console [33].
Contract Pay Per Crawl's program terms: not public. Pay Per Use: each program's terms say what the buyer may do [4]. No dispute process documented. Your agreement with your Exchange operator. Your terms go into each offer the Exchange signs [34]. Disputes: see About FORA.
Status Pay Per Crawl Closed beta, charging listed under Enterprise plans with Bot Management [1, 5]. Pay Per Use Beta [4]. No AI company named as paying through Pay Per Crawl. Live See About FORA.
Your stack
DNS Cloudflare's nameservers, with DNSSEC off first, or your own DNS provider on Business or Enterprise [6, 7, 8]. Every web hostname routed through Cloudflare [9]. No record to add [33].
CDN Cloudflare only [6]. A drop-in edge function. The rest depends on your own setup; FORA adds to what you run rather than replacing it.
Firewall and bot shield Your rules run first, so a crawler they block is never offered a price [10]. Turn off Block AI Bots to join [11]. Your WAF and bot product run first. Let signed requests and your two files through them; FORA's guide gives two setups [35].
Your site and CMS No change for one price. A price per page needs a response header from your origin or a Worker [12]. One file, fora.json, naming your Exchanges, and a sitemap the operator imports your pages from [33].
Search traffic Only crawlers you set to Charge pay. Cloudflare warns that charging a search crawler may hurt your SEO [13]. Googlebot, Bingbot and Applebot always pass [35]. See Search traffic, below.
Upkeep Cloudflare keeps the list of crawlers you can charge [14]. Logpush can't fill in the past, and logs are lost while a job fails [15]. Change your Exchanges by editing your discovery file, with nothing to redeploy [33].
Leaving The docs describe no step to turn Pay Per Crawl off, close its Stripe account or take a final payout. Delete the edge function from your CDN.
The short answer

Two ways to be paid for AI access

Cloudflare charges at the door.

Pay Per Crawl charges a crawler one price per page, and Cloudflare collects and pays out. In Pay Per Use, an AI company proposes a price for a use, and you accept or decline. Both are in beta. Pay Per Crawl needs your site's traffic to run through Cloudflare.

FORA sells the license.

You set the uses, buyers, caps and price for each resource, and can offer one article on several terms. An AI agent (software that acts for a user, one task at a time) buys under an offer your Exchange signs. Pay Per Crawl needs no Exchange of yours, because Cloudflare is one.

Closed beta
Pay Per Crawl is by admission only.

Cloudflare's plan table lists charging under Enterprise plans with Bot Management. You need a new Stripe account, and your balance isn't in the dashboard. Cloudflare names no AI company paying through it.

  1. Discover

    The agent finds the Exchange that sells your content: from your site, or through a Broker's search across catalogs.

  2. Query

    It asks for the content and gets signed offers: the price, the permitted uses, the obligations.

  3. Transact

    It commits to one offer, pays through the Exchange and gets a signed receipt.

  4. Deliver

    It fetches the file through a short-lived link bound to its identity.

  5. Report

    It reports what it used, when the offer asks for a report.

  6. Dispute if needed

    If the file is wrong or never arrives, it opens a dispute backed by signed evidence.

How you're paid

From a crawler's request to your monthly payout

In both programs, Cloudflare charges the buyer and pays you monthly. They differ in who sets the price.

In Pay Per Crawl (closed beta) you set one price for every crawler set to Charge, Cloudflare quotes it with HTTP 402, charges each successful fetch and pays you monthly. Pay Per Use (beta) reverses who sets the price: the AI company sets a price for a use it defines, you accept or decline, the buyer reports its own uses, and Cloudflare charges the buyer and pays you monthly.
Two programs, two price setters.
Text version

In Pay Per Crawl (closed beta) you set one price for every crawler set to Charge, Cloudflare quotes it with HTTP 402, charges each successful fetch and pays you monthly. Pay Per Use (beta) reverses who sets the price: the AI company sets a price for a use it defines, you accept or decline, the buyer reports its own uses, and Cloudflare charges the buyer and pays you monthly.

Pay Per Crawl, in six steps:

  1. Your site runs through Cloudflare, and is admitted. "Your domain must be using Cloudflare's nameservers, or have DNS records managed by Cloudflare" (Enable in account settings). The program is a closed beta (What is Pay Per Crawl). The plan table lists charging under Enterprise plans with Bot Management (Get started).
  2. You set one price and pick who pays it. Cloudflare lists the AI crawlers it knows, and you set each one to Charge, Allow or Block. Cloudflare says "you can only set a single price" for every crawler set to Charge (FAQ), and "The minimum price is $0.001 USD per crawl" (Set a price).
  3. The crawler registers. It signs its requests with Web Bot Auth, an IETF draft for signing bot requests. It passes Cloudflare's review and connects Stripe (Crawl pages).
  4. Cloudflare quotes. An unpaid request gets HTTP 402, Payment Required, with the price. The crawler retries with a signed agreement to it, or states a maximum up front.
  5. Cloudflare charges each successful fetch. "Error responses are not billed", and a crawler "will incur the cost set by the site owner" on every visit (FAQ).
  6. Cloudflare pays you. "Cloudflare acts as the Merchant of Record for pay per crawl" (What is Pay Per Crawl): it is the party that takes the crawler's payment. It pays monthly to a dedicated Stripe Connect account, "subject to settlement periods and minimum payout thresholds" (Manage payouts).

Pay Per Use reverses who sets the price (Pay Per Use, published 2026-09-30). An AI company "defines the use it will pay for and sets a price", such as returning an excerpt of your page to a customer. You accept offers in the dashboard, with "no origin change". "Usage is self-reported": the buyer reports its own uses, and Cloudflare charges the buyer and pays you monthly. Its reason for the program: "AI products fetch far more than they use".

Cloudflare in three numbers
$0.001
The lowest price you can set per crawl [17]
1
Price for every crawler you set to Charge, unless your origin or a Worker prices each page [12, 16]
20%+
Of the web behind Cloudflare's network, the vendor's own figure [19]
Who has been paid so far

No paying buyer named for Pay Per Crawl

Pay per crawl is currently in closed beta.

Cloudflare, What is Pay Per Crawl

Cloudflare names no AI company paying through Pay Per Crawl in any doc, post or press release. Press Gazette reported on 2026-08-17 that it "remains in closed beta testing" (Press Gazette). The publishers Cloudflare listed in 2025 were named "in support", not as participants. Participant counts and payout totals are not published.

For Pay Per Use, Cloudflare says it is "running experiments" with Ceramic.ai and You.com (Making AI search smarter, published 2026-07-01).

What it takes on your site

What your engineer sets up, and what can break

Pay Per Crawl needs your traffic routed through Cloudflare. Your other Cloudflare rules decide which crawlers it ever sees, and only crawlers on Cloudflare's list can be charged.

A crawler meets your DNS and proxy first (a hostname left unproxied never meets Pay Per Crawl), then your firewall rules and bot tools (a crawler they block is never offered a price), then Pay Per Crawl, where you set each listed crawler to Charge, Allow or Block; HTML isn't cached by default, so each crawl normally reaches your origin. The crawler gets the page if allowed, a 403 (refused) or a 402 (pay first) with your own message if blocked, a 402 (pay first) with the price if it is charged without an agreement, and the page, charged, if it agreed.
Your zone, in the order a crawler meets it.
Text version

A crawler meets your DNS and proxy first (a hostname left unproxied never meets Pay Per Crawl), then your firewall rules and bot tools (a crawler they block is never offered a price), then Pay Per Crawl, where you set each listed crawler to Charge, Allow or Block; HTML isn't cached by default, so each crawl normally reaches your origin. The crawler gets the page if allowed, a 403 (refused) or a 402 (pay first) with your own message if blocked, a 402 (pay first) with the price if it is charged without an agreement, and the page, charged, if it agreed.

What can go wrong

In Cloudflare's own docs

Where Cloudflare is ahead

Where Cloudflare is ahead of FORA

The differences

Eight questions a publisher asks of any licensing deal

Cloudflare runs one company's exchange inside its network; FORA is a protocol that any company can run an Exchange on. On your site they sell the same page, side by side, and they work together on bot detection and hosting. The price and money rows compare different things: one number per fetch against a menu of terms, and a company that moves money against a protocol that doesn't.

The questionCloudflareFORA
How do you say what's for sale? Dashboard settings: allow, charge or block per listed crawler, one price, free paths [12, 13, 16]. Preferences go in robots.txt as Content Signals, a line that states a preference with no price [30]. Cloudflare's docs describe no license in the 402. A file on your site names your Exchanges [34]. Each resource can carry several terms, up to 32: uses, buyers, places, caps, duties and a price [36].
How does a buyer find your content? A registered crawler asks Cloudflare for the domains that charge it, finds pages itself, and gets each page's price from a 402 [18, 20]. Pay Per Use: the domains that accepted its offer [4]. An agent asks your Exchange, or a Broker, which returns the signed offers whole (how a Broker orders offers) [37].
Who is the buyer, provably? A Web Bot Auth signature in an older draft's form, which deployed crawlers still send, plus a reviewed registration [18, 24, 40]. On the free plan, the user agent [22]. A Web Bot Auth signature in the form of the working group's first draft (-00), checked against a key on the buyer's domain [38].
What lets a given use happen? Cloudflare's edge, per request: a verified crawler set to Charge, with a signed price [18]. Use limits sit in robots.txt, or in each Pay Per Use program's terms [3, 4]. An offer the Exchange signs and the agent accepts and signs, then a signed link that expires, checked at your CDN [34].
Who sets the price, in what unit? You: one price per fetch for every charged crawler, or a price per page [12, 16]. Pay Per Use: the buyer sets a price per use; you accept or decline [4]. You, per term: free, per unit (a fetch, an access, a token) or flat. One article can be free for research and paid for commercial use. No revenue-share price [36].
How does money move? Cloudflare charges the buyer through Stripe and pays you monthly [2]. Its fee isn't published. Outside the protocol. See About FORA.
What do you see? Which crawlers visit, how often they're charged, robots.txt violations. "Your accrued earnings balance is not currently visible in the dashboard." (Monitor activity) A signed record of each sale, with its usage-report status, in the publisher console your Exchange operator runs [34]. No protocol call returns your sales.
What if the numbers are wrong? No dispute process is documented. Pay Per Use reports are the buyer's own; Cloudflare checks each names an enrolled site [4]. Defined in the protocol, handled under your operator's rules. See About FORA.

FORA pins Web Bot Auth -00; the working group published -01 on 2026-10-29 [38, 40].

Running both

Both on one site today

Works today Live

Doesn't fit

Settling a FORA sale through Pay Per Crawl. Pay Per Crawl charges when a crawler fetches a page. Its docs describe no way to bill an amount for a sale made elsewhere.

A Cloudflare price a FORA agent can verify. Cloudflare's docs describe no signed price or receipt. The crawler gets a header that states the amount charged.

What to do

If your site is on Cloudflare

  1. Decide per crawler first. AI Crawl Control allows or blocks each crawler on every plan, with no beta to wait for (Get started).
  2. Check your Cloudflare setup before you join. Every web hostname must run through Cloudflare's proxy. List the firewall and bot rules that would block a crawler you want to charge. Pay Per Crawl runs after Cloudflare's bot tools, so "you need to first turn off Block AI Bots" (AI Crawl Control with bots).
  3. Ask before you join Pay Per Crawl. Ask for the fee, the payout threshold and which crawlers pay; none is published. FORA itself is free, open-source software. Agree how often you'll get balance updates, and plan for a new Stripe account. For the terms and the exit, see For legal.
  4. Start your logs on day one. Logpush can't fill in the past, so set it up before the first charge.
  5. Sell to agents as well. Name your FORA Exchanges in fora.json, set terms per resource, and run FORA's edge on the paths you license.

Sources

Sources are checked on the date of publication.

Set your own terms for AI access

FORA is an open protocol and free, open-source software. Sell through any FORA marketplace, several of them, or your own: you set terms and a price per resource, and each Exchange signs your offers.

What is FORA? Run your own marketplace Licensing terms All content marketplaces