Cloudflare has announced that starting September 15, 2026, all new domains joining the Cloudflare network will have new default settings: AI Training and Agent crawlers will be blocked on ad-monetized pages, while Search crawlers remain allowed. The change affects approximately 20% of the web's domains that sit behind Cloudflare.
The update introduces a three-tier AI traffic taxonomy that gives website owners finer control: Search (crawling to index content for later queries), Agent (bots acting on a person's behalf in real time), and Training (crawlers taking content to train or fine-tune models). Each category can now be independently allowed or blocked.
A significant new feature is the content-use signal that can be added to robots.txt, with three levels: immediate (interact but store and reuse nothing), reference (index, excerpt, and link back), and full (summarize and reproduce). Bots that reproduce content in full lose their Verified status.
The change also introduces transitive trust, using the Forwarded header (RFC 7239) to let site owners trust an operator's identity even when traffic arrives through intermediaries. Cloudflare argues this creates accountability: losing trusted status across 20% of web domains is a deterrent with real consequences.
The move comes amid a broader industry shift toward giving publishers more control over how AI companies use their content, and reflects the growing tension between AI companies seeking training data and content creators seeking fair compensation.
