The power struggle over control of online content is entering a new era, and the stakes are high for both creators and artificial intelligence firms.
For years, website owners and publishers faced a blunt choice: either block AI bots entirely or let them take whatever they want without compensation. Cloudflare believes creators should have more agency over their work.
After countless conversations with news outlets, digital publishers, and major social networks, a common frustration emerged. Many want to allow automated crawlers, but only if there is fair compensation. Striking individual deals with AI companies, however, has always stacked the odds against smaller publishers who lack scale.
Cloudflare is stepping in with an attempt to rewrite these rules. Its new feature, pay per crawl, allows anyone who owns a website to charge automated bots for access to their content. The system taps a little-known web status code, 402 Payment Required, as its backbone.
Shifting the Balance for Online Creators
Publishers can now decide if AI bots are allowed to browse for free, pay a set fee per request, or be blocked outright. These options provide a dynamic middle ground, letting content owners dictate who can see their work and at what price.
“If a creator wants to block all AI crawlers from their content, they should be able to do so,” says a Cloudflare spokesperson. “But if they want to allow some or all AI crawlers full access for free, that should be a choice, too.”
The pay per crawl tool works using authentication and web standards compatible with current website defenses. When a bot tries to access a page, it signals if it’s willing to pay. If not, the website returns a 402 response that tells the bot the price for access. If the bot agrees, it retries the request and, upon payment confirmation, gets the content delivered.
Cloudflare handles both the billing between bot and publisher and the technical enforcement, keeping records of every paid request. The company also provides publishers with the ability to make exceptions: one bot might get complimentary access while another must pay, all set by simple rules.
The company addresses security challenges by using a digital signature technology that verifies bot identities and prevents impostors from sneaking past paywalls. Bots must register, publish secure credentials, and sign every request.
The payment process itself is flexible. A bot can either react to an access price after the fact or offer up a ceiling in advance, allowing for both negotiation and automation. If the publisher’s price is higher than the bot offers, access is denied and the negotiation begins again.
Publishers foresee this tool as a foundation for more sophisticated control. Dynamic pricing, licensing restrictions by type of use, and granular rules are all possibilities as websites seek to balance exposure with compensation.
The pay per crawl program is currently invite-only, but Cloudflare sees this as a first step toward an online world where creators and companies can define the terms of their digital relationships, as highlighted by the potential for AI to disrupt content revenue.
