For two years, telling AI crawlers to stay away has meant maintaining a list: GPTBot, Google-Extended, ClaudeBot, CCBot, and a new one every quarter. On 30 October, four of the largest labs said they will accept a single line instead.
OpenAI, Google, Microsoft and Anthropic committed to honouring a shared policy token in robots.txt — usage_ai-training — that lets a publisher express one preference across all of them. The signal is defined in an individual draft at the Internet Engineering Task Force, written by Kevin Kelley, a technical lead for standards at Anthropic. Existing crawler-specific entries keep working alongside it.
The dates are the story. Anthropic says it has already started. OpenAI says it will roll the signal out 'in the coming months'. Google plans to adopt it sometime in 2027. Microsoft said it will begin on 1 February 2027. Meta's communications chief Andy Stone welcomed the effort without committing Meta to honouring it.
And the defaults have not moved. Content remains scrapable unless a publisher actively opts out, Microsoft said in a blog post. The signal covers future training runs, not crawling itself, so a site that opts out can still be fetched, indexed and summarised. Google and Microsoft both warned that opting out of training may reduce how often a publisher's material surfaces in AI answers — a trade-off that turns a rights question into a traffic question.
The backdrop is litigation and leverage. The New York Times sued OpenAI and Microsoft in December 2023 over training on its articles. Cloudflare has blocked AI crawlers by default on new customer sites since July 2025. Meta has twice delayed business terms that publishers read as a blanket scraping licence, now pushed to 20 November.
Read together, the announcement is a concession on protocol and a refusal on default. Publishers get one line to write instead of nine, in exchange for accepting that silence means consent and that the line will not be enforced across the industry for another 14 months.
What to do now: add the token if you want the preference on record, keep the crawler-specific rules you already have, and measure AI-referral traffic before and after — the visibility cost is real and is not uniform across the four providers.
