Four of the largest AI companies in the West have agreed on something, and it fits in a robots.txt file.
On October 30, OpenAI, Google, Microsoft and Anthropic each said they will honor `__usage_ai-training`, a shared "policy token" that lets a site owner express an AI training preference once instead of negotiating it vendor by vendor. The mechanism reuses robots.txt grammar:
``` User-Agent: __usage_ai-training Disallow: / ```
That single group tells every participating crawler that the site's content is off limits for training. Sites that want exceptions carve them out with `Allow:` lines.
The proposal comes from Kevin Kelley, a technical lead for standards at Anthropic, and is filed as an individual IETF Internet-Draft — "Machine-Readable Training Preference Signal" — dated October 30, with an expiry of May 3, 2027. An individual draft is not an IETF standard and carries no formal standing in the standards process. That detail is worth keeping in view, because the announcement reads like a standard and is not one yet.
What the token does and does not do
- It governs training, not fetching. As the draft's own framing puts it, the token "is not a directive for crawling behavior." A bot can still retrieve the page; it just should not learn from it. - It cannot reach backwards. Adding a line to robots.txt does not remove anything already collected into a dataset or baked into a model. - It is coordination, not a security control. OpenAI describes the mechanism in RFC 9309 terms: crawlers that do not wish to comply can ignore it, and the sites that most need leverage — small publishers — have the least of it.
Timelines are uneven, which is the practical story. Anthropic says it has already begun implementing support. OpenAI says it aims to implement the signal "in the coming months." Microsoft targets February 1, 2027, and plans to keep training and "grounding" — the retrieval of live pages into answers — as separate switches. Google targets 2027, and says that when a site opts out through the token without naming Google-Extended, it will read that as a disallow for Google-Extended too. Google's John Mueller framed it as helping sites "express their AI training preferences across multiple AI providers in a single step."
Meta has not joined. Communications head Andy Stone called a common signal "an important step" and said the company looks forward to working with other AI companies on standards and implementation — support in principle, with no commitment to honor it.
Why it matters anyway
For three years, the only way to opt out of AI training was to maintain a separate, differently spelled exclusion for each lab: Google-Extended here, a crawler-specific user agent there, an IP block when politeness failed. That asymmetry favoured the largest publishers, who could afford the operational work, and left everyone else guessing. A single token is a genuine reduction in the cost of saying no — and the participant list suggests the industry knows it: the draft's discussion includes Meta, Cloudflare, the Common Crawl Foundation, News Corp, the Financial Times, the BBC, the Electronic Frontier Foundation and Creative Commons.
It also lands in a busy month for the same fight. Cloudflare now blocks AI crawlers by default; the Really Simple Licensing standard lets publishers price crawler access. A voluntary token that four vendors promise to respect is a different approach to the same problem: not a wall, a convention.
The test is not the press release. It is whether the token reappears as a working-group document rather than an individual one, whether the four honour it against their own interest when a valuable site opts out, and whether anyone outside the signatories — Meta first among them — follows.
