Who Watches the Machines That May Govern Us?

The companies building the world’s most powerful artificial intelligence systems are beginning to acknowledge something their marketing usually softens:

They may be creating technology too consequential to release without independent review.

That recognition matters.

Google DeepMind CEO Demis Hassabis has proposed a new standards body to test frontier AI models before release, establish safety practices, and potentially recommend slowing development when risks become severe. He argues that artificial general intelligence may be only a few years away and that society has a limited window to prepare.

The outline sounds responsible. The proposed institution would bring together specialists capable of examining systems that conventional regulators may struggle to understand.

But the model Hassabis invokes should make us pause.

He compares it to FINRA, the organization that oversees American securities firms. FINRA is not a normal government agency. It is funded largely by the financial industry it regulates.

Applied to AI, that would mean laboratories helping finance and shape the body responsible for deciding whether their newest systems are safe enough to release.

This is not automatically corrupt. Governments routinely need outside expertise. A technically serious review system could be far better than politicians improvising rules after a crisis.

But expertise and legitimacy are not the same thing.

A frontier model may influence employment, military strategy, scientific research, public information, surveillance, education, medicine, and the distribution of economic power. Deciding whether such a system is safe is not merely an engineering judgment.

It involves political choices.

How much employment disruption is acceptable? Who decides whether a biological capability creates more public benefit than danger? Should a model be withheld because it threatens national security, even if restricting it strengthens a corporate monopoly? Whose language, values, and social conditions are represented in safety tests?

Technical experts can measure certain risks. They cannot legitimately answer all of those questions on behalf of everyone else.

The concern becomes sharper when we look at what is happening elsewhere.

Europe has delayed some of its strongest rules for AI systems used in hiring, education, credit, public services, and law enforcement until late 2027. The stated goal is to give institutions more time to develop workable standards and compliance systems.

That may be reasonable in administrative terms.

In human terms, it means automated systems can become embedded in decisions about jobs, exams, loans, promotions, and public benefits before the strongest protections arrive.

This creates a recurring pattern in AI governance.

Deployment is treated as urgent. Democratic protection is treated as something that must wait until the paperwork is mature.

The public is told that policymakers cannot regulate too quickly because they may misunderstand the technology. Meanwhile, companies are permitted to deploy quickly despite not fully understanding the technology themselves.

The likely result is governance by installed fact.

Once employers restructure hiring around automated screening, hospitals depend on proprietary models, schools purchase AI platforms, and governments integrate systems into public administration, removing them becomes economically and politically difficult.

Regulation then adapts to the infrastructure rather than shaping it.

There should be a role for an expert frontier-AI institution. It could conduct evaluations, coordinate scientific knowledge, protect confidential security information, and help regulators understand rapidly changing capabilities.

But it should not become a private priesthood issuing safety blessings behind closed doors.

Its testing methods should face independent scrutiny. Civil society, labor, educators, public-health experts, artists, affected communities, and smaller nations should have meaningful representation. Funding should not translate into control. Regulators must retain legal authority. Whistleblowers must be protected. Findings should be public whenever genuine security concerns do not forbid disclosure.

The deeper issue is not whether AI companies contain responsible people. Many do.

The issue is whether humanity’s most consequential technological decisions should depend on responsible people remaining responsible inside institutions built to win races, protect market share, and satisfy investors.

A machine does not become publicly governed merely because its maker agrees that governance is necessary.

The question is not only whether someone is watching the frontier.

It is whether the rest of us are allowed to watch the watchers.

Source note: Demis Hassabis’s frontier-AI framework and reporting on his proposed industry-funded standards body. Reporting and legal analysis on the delayed EU AI Act requirements for high-risk systems.