The public web's quiet bargain — you may be crawled unless you say otherwise — is being rewritten in favour of the crawlers. According to Golem, Meta has changed its terms of service so that websites must allow AI access, turning a permission that sites used to withhold with a single line in robots.txt into something closer to an obligation baked into a contract.
The mechanics matter more than the wording. robots.txt is a convention, not a law: it is enforced by good manners and, increasingly, by infrastructure. When a platform writes AI access into its terms, it stops asking the crawler to behave and starts asking the site owner to accept the arrangement as a condition of using the platform's services at all. For an operator who depends on Meta's pixels, login buttons or advertising tools, refusing is not a technical setting — it is a business decision.
The move lands in the middle of an arms race over training data. Publishers have spent two years bolting up defences: pay-per-crawl schemes at CDNs, licences negotiated one counterparty at a time, and technical opt-outs favoured by the EU's text-and-data-mining rules, which let rights holders reserve their content. AI companies have argued in reply that the open web is open — and have grown bolder as the data appetite of models has outrun the supply of licensed text.
For site owners the practical question is not whether crawling is legal but whether they can detect it, price it, or stop it. Meta's terms, as reported, shift the default answer: instead of asking whether Meta may take, the contract presumes that it may. Expect similar clauses to surface elsewhere, and expect the argument about who holds the right to say no to keep migrating out of engineering forums and into contract law.
