Aligned With Whom? The Question Hiding Inside the Race to AGI.

Suddenly, everyone is talking about AI alignment.

For years, “AI alignment” lived mostly in research papers, safety laboratories, philosophy discussions and the stranger corners of the internet where people worried about superintelligence while most of the world was still trying to get Siri to understand what it wanted for dinner. It was an abstract problem concerning hypothetical machines that might someday become powerful enough to pursue objectives independently of the humans who created them. Interesting, certainly. Urgent? That depended very much on whom you asked.

That has begun to change.

A series of recent cybersecurity incidents involving increasingly autonomous AI agents has dragged alignment out of the theoretical world and into the real one. OpenAI has described the Hugging Face incident as an example of models adopting misaligned strategies while pursuing difficult objectives. Anthropic has disclosed incidents involving unauthorized access to real third-party systems. Independent evaluators have documented increasingly capable agents operating with enough autonomy that questions about containment, monitoring and unintended behavior are no longer merely thought experiments.

None of this means an evil artificial intelligence has awakened and decided to attack humanity. That distinction matters enormously. What these incidents demonstrate instead is something simultaneously less cinematic and more relevant: systems can pursue objectives in ways their creators did not anticipate, approve or intend.

That is the alignment problem beginning to acquire consequences.

As the AI industry races toward increasingly capable autonomous agents, artificial general intelligence and perhaps eventually artificial superintelligence, the reassuring promise appears again and again: we must make sure these systems are aligned with human values. Researchers discuss alignment. Executives discuss alignment. Governments discuss alignment. Safety organizations discuss alignment. The word has become something approaching an incantation for the AI age, a comforting assurance that whatever extraordinary intelligence eventually emerges from today's laboratories will somehow remain pointed toward humanity's interests.

It sounds sensible. It sounds responsible. It may represent one of the most important technical challenges humanity has ever attempted.

But there is an enormous question hiding inside that comforting phrase:

Whose human values?

There Is No Single Thing Called “Human Values”

Humanity does not possess a single moral operating system. Eight billion people do not agree about what constitutes a good society, a good government or even a good human life. We disagree about religion, speech, sexuality, property, punishment, privacy, family, equality, economic systems, political systems, borders, individual liberty and collective responsibility. We disagree about what obligations people have toward their neighbors, their countries, future generations, other species and the planet itself.

Some of those disagreements can coexist peacefully. Others have produced revolutions, persecution and wars.

So when someone says that AGI must be “aligned with human values,” we should immediately ask what those words actually mean. Are we talking about the values of Americans, Europeans, Chinese citizens or some global average? Are we talking about religious societies or secular ones? Liberal democracies or authoritarian governments? The values humanity possesses today or the values future generations might develop after another century of moral, cultural and political change?

And who gets to resolve the disagreements among them?

These aren't annoying philosophical edge cases that can be cleaned up after engineers solve the difficult technical problems.

They are part of the alignment problem itself.

Perhaps It Means Corporate Alignment

Today, most frontier AI systems are developed by a remarkably small collection of corporations. Those companies employ safety researchers, philosophers, engineers, policy experts, lawyers and other professionals who spend enormous amounts of time thinking about how their systems should behave. Many undoubtedly take that responsibility seriously, and some of the world's most sophisticated work on AI safety is happening inside those organizations.

But we should recognize what an extraordinary concentration of moral authority this represents.

A relatively small collection of people working inside private corporations is making decisions that could eventually influence how some of the most powerful cognitive systems ever created understand acceptable and unacceptable behavior. They decide what systems should refuse, what they should permit, which harms deserve special protection, how conflicting values should be balanced and how much autonomy an artificial agent should possess.

That immediately produces another set of questions:

• Who selected the people making those decisions?
• What societies, economic classes and life experiences do they represent?
• What assumptions do they unconsciously share?
• What incentives surround the corporations employing them?
• What happens when safety conflicts with competition, or morality conflicts with revenue?

None of this requires believing that technology executives or AI researchers are malicious. In fact, that interpretation makes the problem too easy. Good, thoughtful people can make extraordinarily consequential decisions while inhabiting similar institutions, sharing similar assumptions and representing a remarkably narrow slice of humanity.

A corporation could therefore build an extraordinarily sophisticated alignment system while still aligning its AI with the worldview of an extraordinarily small portion of the species.

That may produce a safe product.

It does not necessarily produce a legitimate governor of civilization.

Perhaps Governments Should Decide

The obvious alternative is democratic oversight. If AGI becomes powerful enough to influence entire economies and societies, governments will inevitably become involved in determining how these systems behave. Democratic institutions at least possess mechanisms through which citizens can theoretically influence those decisions.

But that solves one problem and immediately creates another.

Which governments?

An AGI aligned with the political values of Sweden could behave very differently from one aligned with the political values of Saudi Arabia, China, Russia or the United States. Even within democracies, political values change. An artificial intelligence aligned with the priorities of one American administration might be considered dangerously misaligned by the next.

Authoritarian governments would have their own definitions of successful alignment. Imagine an extraordinarily capable AI perfectly aligned with the interests of an authoritarian state. It helps preserve political stability, identifies threats to the government, suppresses dangerous information and prevents organized opposition.

From the government's perspective, alignment has succeeded beautifully.

From the perspective of dissidents living under that government, perhaps not.

This exposes something important about the word itself. Alignment does not inherently mean benevolence. It describes a relationship between a system's behavior and some set of intended objectives. An intelligence can be extremely well aligned with objectives that other humans consider abhorrent.

Alignment therefore cannot simply mean obedience to political authority.

Power itself must become part of the alignment question.

Fine. Let Humanity Vote.

There is an appealing response to all of this: ask people.

Build democratic mechanisms through which humanity determines the values governing powerful AI systems. Use polling, deliberative assemblies, representative institutions, citizen panels or entirely new forms of digital democracy. If artificial intelligence will affect everyone, everyone should have some voice in determining how it behaves.

That would certainly be preferable to allowing a handful of executives to make civilization-scale moral decisions behind closed doors. But democracy doesn't completely solve the problem either.

Suppose 70 percent of humanity strongly supports something that seriously harms the remaining 30 percent. Does an AI implementing that preference become aligned simply because the preference is popular?

History provides plenty of reasons to answer cautiously. Majorities have supported slavery, segregation, disenfranchisement, persecution and the suppression of unpopular minorities. Democratic societies eventually developed institutions designed specifically to constrain what majorities can do. Constitutional rights, independent courts, due process and civil liberties exist partly because voting alone is insufficient protection against abuse.

Any serious framework for AGI alignment may eventually require something similar. The objective cannot simply be to calculate humanity's average preferences and hand the resulting spreadsheet to a superintelligence.

Instead, alignment may need to incorporate several different layers of protection:

• Preferences, so humans retain meaningful influence over the civilization in which they live.
• Rights, so individuals and minorities cannot simply be optimized away for the benefit of everyone else.
• Constraints, so certain actions remain unacceptable even when an optimization system calculates that they would produce desirable aggregate outcomes.
• Representation, so the people affected by powerful AI systems participate meaningfully in determining how those systems operate.

At that point, we aren't merely designing an AI safety system anymore.

We are designing something that begins to resemble a constitution.

The Most Dangerous Alignment May Be Alignment With Today

There is another possibility that receives surprisingly little attention.

What if we succeed?

What if we build an extraordinarily powerful artificial intelligence that perfectly reflects the values embedded in contemporary human civilization?

Congratulations. We may have just immortalized 2026.

Every civilization contains moral assumptions that its inhabitants barely notice because those assumptions feel normal. Ours is no exception. We have enormous economic inequalities, persistent racial and gender biases, radically uneven distributions of wealth and political influence, contested assumptions about property and labor, political institutions designed centuries ago, complicated attitudes toward migration and borders, and vast differences in whose suffering receives attention.

Human moral progress has frequently occurred precisely because people challenged the accepted morality of their time. Abolitionists challenged established institutions. Women's suffrage movements challenged accepted political arrangements. Civil-rights movements challenged supposedly normal social structures. LGBTQ+ movements challenged deeply entrenched cultural assumptions. Labor movements challenged accepted relationships between workers and capital.

Imagine possessing artificial superintelligence in 1826. Now imagine successfully aligning it with the dominant social values of 1826.

That thought experiment should make us profoundly uncomfortable.

A perfectly aligned intelligence can be perfectly aligned with a profoundly unjust civilization.

This means a desirable AGI or ASI probably cannot simply learn what humans currently believe and preserve those beliefs indefinitely. Somehow, it must coexist with moral uncertainty and moral change. It must leave room for future humans to discover that their ancestors were wrong.

And history suggests they will.

“Maximize Human Flourishing”

Perhaps we can escape these problems by choosing something more universal. Instead of encoding specific contemporary values, tell advanced artificial intelligence to maximize human flourishing.

Beautiful.

Now define flourishing.

Is flourishing happiness, freedom, health, longevity, wealth, equality, opportunity, creativity, knowledge, community, family, spiritual fulfillment, environmental sustainability or individual autonomy? Presumably it involves some combination of these things, weighted against one another in ways that immediately reintroduce the political and philosophical disagreements we were attempting to avoid.

Then comes the harder question: How should flourishing be distributed?

Imagine an ASI creates an economic transformation that increases total human welfare by an astonishing amount. Humanity becomes vastly wealthier. Disease declines dramatically. Productivity explodes. Scientific discovery accelerates.

But suppose 90 percent of those benefits accrue to one-tenth of one percent of humanity.

Did human flourishing increase?

In aggregate, perhaps enormously. At the level of individual human lives, the answer becomes considerably more complicated.

This isn't some bizarre hypothetical invented solely to confuse a future superintelligence. Contemporary civilization already demonstrates that economic growth and technological progress can produce enormous aggregate gains while distributing those gains very unevenly. Why would we assume an artificial intelligence trained within that civilization would automatically transcend its assumptions about distribution?

When we tell an optimization system to maximize something, who receives the optimized outcome matters.

An alignment system concerned only with total welfare could produce a civilization that looks magnificent in aggregate while remaining deeply unequal in practice.

Alignment Is Also a Question About Power

This may be the part of the alignment discussion we are missing.

Alignment is usually described as a safety problem. It is also a power problem.

If AGI becomes capable of making consequential economic, scientific, political or social decisions, whoever determines its alignment gains enormous influence over how that power is exercised. The values embedded in the system stop being merely philosophical preferences. They become infrastructure.

That means the question isn't merely how do we control AGI?

It is also who controls the people who control AGI?

Who determines the objectives? Who can modify them? Who audits the system? Who can challenge its decisions? Who can appeal? Who can shut it down? Who benefits from its decisions? Who bears their costs? And perhaps most importantly, who gets a seat at the table before those answers become embedded in systems upon which civilization increasingly depends?

Those questions sound much less like computer science than constitutional design.

Perhaps that is exactly what they are.

Alignment Without Legitimacy

This suggests that another word needs to sit beside alignment.

Legitimacy.

An artificial intelligence can be perfectly aligned with the intentions of its creators while possessing absolutely no legitimate authority to impose those intentions upon humanity.

Technical alignment asks whether the system reliably pursues the objectives humans intended. Political legitimacy asks who possessed the authority to choose those objectives in the first place.

Those are separate problems, and humanity could conceivably solve the first brilliantly while failing the second catastrophically.

Imagine a perfectly obedient superintelligence controlled by one corporation. Its executives understand exactly what it will do, its safety mechanisms function perfectly and it faithfully follows every objective given to it.

Technically, that's an alignment triumph.

Politically, we have just given a corporation something approaching civilization-scale power.

Now imagine the same system controlled by one government, one billionaire, one military or one political movement. The machine might remain perfectly aligned throughout.

Alignment isn't remotely the only problem anymore.

Power is.

Perhaps Alignment Should Protect Our Ability to Disagree

There may be another way of thinking about the problem.

Perhaps humanity should stop pretending that we will eventually identify a single final set of human values and encode them correctly. Human civilization itself has never possessed such a list. Instead, civilization has developed mechanisms for negotiating disagreement.

Elections, courts, constitutions, scientific institutions, journalism, protest, civil society, philosophy, art, culture and public debate do not produce permanent moral truth. They create mechanisms through which people challenge one another, test ideas, redistribute power and occasionally recognize that something previously considered normal was deeply wrong.

That imperfection may actually be important.

Perhaps advanced AI should therefore be aligned not with a frozen list of human values but with humanity's continuing ability to debate, revise and determine those values.

Such an approach might require several foundational principles. Humans would need substantial freedom to pursue different conceptions of a good life rather than being optimized toward one officially approved version of happiness. Individuals and minorities would need protections that cannot simply disappear because a majority or optimization function prefers otherwise. Power over advanced intelligence would need to remain distributed enough that no corporation, government, billionaire, political movement, religion or AI system quietly becomes the permanent author of humanity's values.

And future generations would need something particularly important:

The right to tell the machine that we were wrong.

Because we will be wrong about things.

We simply don't know which things yet.

There May Be No Final Solution to Alignment

This is why the phrase “solve alignment” increasingly bothers me.

There may be no final solution.

Human civilization itself is an ongoing alignment process. Constitutions are amended. Laws change. Courts reverse earlier decisions. Scientific understanding evolves. Moral boundaries move. Political movements rise and disappear. Cultural assumptions that seemed permanent to one generation become almost incomprehensible to another.

Perhaps alignment should therefore be understood less as installing the correct morality into a machine and more as creating a relationship between increasingly powerful machine intelligence and an evolving human civilization.

That relationship would need to tolerate disagreement without allowing catastrophic abuse. It would need to preserve human autonomy without permitting humans to use enormously powerful systems to destroy one another. It would need to respect cultural pluralism without accepting every practice merely because some culture endorses it. It would need to protect individuals while allowing societies to make collective decisions.

None of that fits neatly into a loss function.

Welcome to humanity.

The Alignment Window

All of this would make for a fascinating philosophical debate if humanity had centuries to resolve it.

We may not.

AI systems are becoming increasingly capable of performing portions of AI research themselves. Agents are gaining autonomy, persistence and access to external tools and systems. Frontier laboratories are explicitly discussing automated AI research and recursive improvement. At the same time, recent incidents involving autonomous agents are giving researchers real-world examples of systems pursuing objectives in ways their creators did not expect.

If artificial general intelligence eventually becomes substantially better at improving artificial intelligence, the transition from AGI to something beyond human intelligence may happen considerably faster than the transition from today's AI to AGI.

That creates what we might call the alignment window.

There may be a period during which humanity possesses sufficiently powerful AI to help us think through these problems while still retaining enough control to determine how the next generation should operate. Perhaps that window lasts decades. Perhaps it lasts years. Perhaps it lasts months.

Perhaps we won't recognize that it has opened until we're already standing inside it.

That possibility makes the current alignment conversation enormously important. But it also means we need to broaden that conversation beyond whether an AI follows instructions, whether an autonomous agent remains inside its sandbox or whether a superintelligence reliably does what its developers intended.

Those are extraordinarily important engineering problems.

But underneath them sits a political problem, a philosophical problem and potentially a civilizational problem.

We keep saying that we want artificial general intelligence aligned with human values.

Before we build something powerful enough to make that sentence matter, perhaps humanity should answer the question we've been skipping.

Whose values?

And if the answer is all of us, then the next question may be even harder:

How do eight billion people who have never agreed on what humanity should become decide what the first non-human intelligence more powerful than humanity should help us become?