One of the more seductive ideas in AI governance right now is that models should evaluate themselves. Instead of standing up an expensive external testing regime for every deployment, you ship the model with an evaluator inside it — a component that scores its own outputs, flags dangerous requests, and reports upward in real time. It is cheap, it scales with the number of deployments rather than the number of auditors, and it produces a continuous stream of telemetry where today there is an annual audit at best.
Hugging Face's policy chief is not convinced, telling TechCrunch that embedded evaluators will not be an easy solution to the oversight problem. The reasoning is worth taking seriously, because it is an argument about a structural flaw rather than a technical one.
The core objection is that an evaluator living inside the model shares the model's failure modes. A judge trained on the same data, by the same lab, with the same blind spots, will systematically miss the class of problems its sibling has. Evaluators also make attractive targets: anything that sits in the inference path can be prompted, fine-tuned around, or simply switched off by whoever operates the deployment. And a model that grades its own homework has an obvious incentive gradient if any part of its reward depends on the grade.
The second objection is accountability. External evaluation is imperfect, but it creates an artefact — a report, a benchmark result, a red-team finding — produced by someone with no commercial stake in the answer. An internal evaluator produces a log line, visible to the developer and, at best, summarised to the regulator. When something goes wrong, nobody outside the company can reconstruct what the safety layer saw or why it stayed quiet. Governance built on self-reporting tends to become a compliance ritual: a checkbox that proves the evaluator ran, not that it worked.
The third is that "embedded" quietly relocates a public decision into private code. Which categories of harm count, what threshold triggers a block, how false positives are traded against false negatives — these are policy choices. Putting them inside a model's weights makes them invisible, unversioned, and effectively unauditable.
None of this makes embedded evaluation useless. It is the right tool for continuous monitoring, for catching regressions after a deployment that no external benchmark would see, and for defence in depth behind independent testing. The mistake is treating it as a substitute rather than a supplement — and, in particular, letting it displace the slower, more expensive, more adversarial work of outside evaluation.
What would a workable regime look like? Probably a hybrid: mandatory independent evaluation before high-risk deployment, embedded monitoring afterwards, and a disclosure obligation that makes the internal evaluator's outputs available to a regulator on request. The cheap version is only cheap if you don't count the cost of not knowing.
