Cloudflare has issued the AI industry a hard deadline. Starting September 15, 2026, the web infrastructure giant's default settings will block "mixed-use" crawlers — bots that blend traditional search indexing with AI training data collection and agent operations — from any pages that host ads.
The move is the most aggressive step yet by a major internet platform to force AI companies to pay for the content they use. CEO Matthew Prince framed it as a matter of ecosystem survival, noting that bots recently surpassed human traffic online for the first time — a milestone not expected until next year.
"Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," Prince said.
The policy specifically targets companies running a single crawler for both search and AI training — a practice Cloudflare says gives the "world's largest search engine" (a clear reference to Google) access to roughly twice as much information as competitors.
Cloudflare is also evolving its Pay Per Crawl marketplace into "Pay Per Use," letting publishers charge AI companies when their content creates value, not just when it's fetched. Initial partners include Ceramic.ai and You.com.
The company's data shows that over 50% of crawl traffic from AI bots is spent re-fetching unchanged pages, wasting publishers' bandwidth. The new defaults will apply to new Cloudflare customers, new sites from existing customers, and all existing free customers — though site owners can opt out.
For AI model providers dependent on web-scale data, the September 15 deadline marks a fundamental shift in the economics of training data acquisition.
