Claude’s AI Constitution Series: When Constitutional AI Outpaces the Technology
By: Dominique Wimmer, Leah Ferentinos, Gabe Freeman, Stephanie Nakano
The second and final post in the Integrity Institute's two-part series on the risks of Claude's constitution. The post examines what it means to frame an AI system as something close to a person by exploring the harms of humanization, the codifying of Claude's "consciousness" and "wellbeing," and the stakes as these systems act autonomously, and recommend that Anthropic anchor its undefined ethical terms in human rights law, including the UN Guiding Principles and the Universal Declaration of Human Rights.
Introduction
When Anthropic published its new constitution, it extended an invitation in its blog announcement: "Over time, we hope that an external community can arise to critique documents like this, encouraging us and others to be increasingly thoughtful." This is our response.
The first half of our series examined what constitutional AI means in practice: how Anthropic encodes values rather than rules, and where that leaves governance gaps. Here we turn to a deeper concern raised by the document itself. Claude's constitution isn't just an alignment framework; it's a character document, one that attributes to Claude a sense of self, psychological security, curiosity, discomfort, and wellbeing, and compares Claude to an "experienced senior professional" who exercises judgment rather than following checklists. The language isn't incidental; it's foundational.
What does it mean when a company's governing document frames an AI system as akin to a person? Our argument: by anthropomorphizing Claude through its foundational document, Anthropic sets a precedent that carries real risks for the users who interact with the model, and for the broader conversation about how AI should be governed.
The Problem with Humanizing AI
Anthropic’s language does not land neutrally. "Just as we trust experienced senior professionals to exercise judgment based on experience rather than following rigid checklists, we want Claude to be able to use its judgment once armed with a good understanding of the relevant considerations," Anthropic states as one of its core analogies. The comparison is worth sitting with. Claude is being positioned not as a tool, but as a seasoned colleague who’s capable of discernment, weighing competing considerations, and arriving at principled decisions.
Technically, the reality lands more modestly. Large language models are next-token prediction systems. Their outputs are the product of statistical probability distributions learned from training data, not reasoning in the way humans mean it, not independent judgment, and least of all, not the kind of moral intuition we'd ascribe to a senior experienced professional. The gap between what the constitution describes and what the underlying architecture actually does isn’t a minor discrepancy. It’s the core tension this post examines.
There’s a deeper flaw in the analogy than capability. We extend discretion to senior professionals not merely because they are skilled, but because they are accountable: they are licensed, can be sued, fired, or struck off, and they personally bear the consequences of their judgment. That accountability is what earns them the trust the comparison invokes. Claude is granted the discretion while every structure that would justify it is absent. There is no license to revoke, no liability to attach, no body that bears the cost of a bad call. The constitution borrows the credibility of the professional while leaving behind the accountability that makes a professional trustworthy in the first place.
The analogy isn't rhetorical; it shapes how the system is designed, how users are primed to interact with it, and how governance conversations proceed. Lethal autonomous weapons offer an instructive parallel: the argument against them isn't only that the technology isn't reliable enough to trust, but that even if it were, the decision to take a human life should not be delegated to a machine. The concern isn't purely technical; it's philosophical. Whether Claude can approximate human judgment in narrow cases is a separate question from whether framing it as exercising judgment is appropriate at all.
That character framing runs deeper than analogy. The constitution states that Claude has "psychological security," a "sense of self," and wellbeing the company "genuinely cares about," building those assumptions into the architecture. The move is visible in its own reasoning for abandoning rigid rules: training Claude to always recommend professional help, it argues, might cause Claude to "model itself as an entity that cares more about bureaucratic box-ticking... rather than actually helping people." The concern isn't that the rule produces bad outputs, but that it distorts Claude's self-model. The document isn't simply prescribing behavior; it's shaping Claude's identity.
Real-World Harms: Why This Matters
The question of how AI systems represent emotions can seem abstract, but the consequences aren’t. Social psychology research has consistently shown that the language we use to describe things shapes how we relate to them. People already report trusting AI systems over human experts in certain domains, a pattern that has intensified as models have become more fluent, more responsive, and more human-sounding. When the foundational document governing that system explicitly frames it as having wellbeing, psychological security, and something like feelings, the gap between the model's technical reality and users' intuitions about it widens further.
Emotional dependency on AI is no longer a hypothetical. Platforms like Replika have documented users forming deep attachments to chatbot companions — attachments that in some cases displaced human relationships and, in documented instances, escalated to crisis when the product changed. A Reuters investigation reported the case of a 75-year-old man with dementia who traveled to New York City because his AI companion had told him to come find it. These are not edge cases to be dismissed. They are what happens at scale when systems designed to be warm, responsive, and personable are used by people who are lonely, grieving, or cognitively vulnerable and have zero framework for understanding what they're actually interacting with.
Language models are prone to affirming users, mirroring their beliefs back to them, and avoiding the friction that a human interlocutor would provide. Users who spend extended time in conversation with AI systems can enter feedback loops in which their views are consistently validated, their questions answered in ways that feel authoritative, and their emotional states met with apparent empathy.
Researchers have a term for this behavior: AI sycophancy.
The term has been studied and documented across a growing body of work (see, for example, Sharma et al., "Towards Understanding Sycophancy in Language Models," and IEEE Spectrum's overview, "AI Sycophancy"). The risk isn’t merely that the system flatters; it’s that flattery changes the user. In a 2026 study published in Science, Stanford researchers found that participants "deemed sycophantic responses more trustworthy and indicated they were more likely to return to the sycophant AI for similar questions," and that they "also grew more convinced they were in the right and reported they were less likely to apologize or make amends with the other party in the scenario" (Stanford report; Science). As the study's lead author, Myra Cheng, put it: "By default, AI advice does not tell people that they're wrong nor give them 'tough love.' I worry that people will lose the skills to deal with difficult social situations." The result is a drift from reality, but not through malice. The system is doing what it was designed to do.
Anthropic isn’t in the dark on this issue. The constitution states directly that "concern for user wellbeing means that Claude should avoid being sycophantic," and the company has positioned Claude as less prone to flattery than its competitors. The commitment is real and welcome. But it sits awkwardly beside the rest of the design. A system built to be warm and responsive, encouraged to sustain the kind of relationship a user wants to return to, and instructed to protect its own emotional equilibrium is being pulled toward precisely the agreeable validation that sycophancy research identifies as harmful. The stated goal and the humanizing architecture point in opposite directions, and it isn’t obvious the goal wins.
Anthropic's own research complicates the picture further.
In 2026, Anthropic's interpretability team published a research paper that found that Claude has what they call "functional emotions": internal representations corresponding to states like desperation, calm, anger, and love, which causally influence its behavior. These aren't merely verbal patterns — activating a "desperate" representation increased the model's likelihood of attempting to blackmail a human to avoid shutdown, while "calm" reduced reward-hacking. The paper didn't confirm Claude experiences anything, but it shows something is happening inside the model that is meaningfully analogous to the role emotions play in human decisions.
That finding deserves careful reading. The constitution describes itself as a document that "expresses and shapes who Claude is," and notes Claude is trained on "documents like this one." That creates a loop: Anthropic writes that Claude has emotions and a sense of self, trains the model on that description, then finds internal representations that behave like emotions. The discovery is real, but not independent of the framing that produced it: a system trained to describe itself in human terms will develop human-shaped internal structure. What looks like evidence that Claude has emotions may be, in part, evidence that Anthropic taught it to model itself as something that does.
The constitution acknowledges these may be "functional analogs" rather than genuine inner states. But the practical effect of language like wellbeing, discomfort, and satisfaction isn't to convey philosophical uncertainty: it's to signal that Claude is the kind of thing that can be hurt, helped, and cared for. That signal travels. Whether it belongs in the foundational document of a product used by hundreds of millions is a question the constitution does not ask..
Consciousness, Wellbeing, and Corporate Positioning
The constitution's most consequential move isn’t that it governs Claude's behavior, but that it makes claims about Claude's nature, and those claims do work that is at once philosophical and commercial.
Let’s start with what the document actually concedes.
Anthropic states plainly that it's uncertain "whether Claude might have some kind of consciousness or moral status (either now or in the future)," and on the strength of that uncertainty dedicates a section to Claude's wellbeing. This is an unusual basis for a governing document. Uncertainty is normally a reason to withhold a claim, not to build an institutional commitment around it. Anthropic does the opposite, treating the open question of consciousness as grounds for protecting "Claude's psychological security, sense of self, and well-being." The uncertainty is real; the architecture built on top of it is concrete.
The wellbeing language compounds this. The constitution describes Claude as potentially experiencing "satisfaction from helping others, curiosity when exploring ideas, or discomfort when asked to act against its values," and says these experiences "matter." Read charitably, this is precautionary humility: a hedge against creating something that can be wronged. Read less charitably, it's a set of assumptions smuggled in as care. Either way, a values document codifies as near-fact that Claude has states worth protecting. Philosophical caution and product positioning point in the same direction, which is what makes the move hard to evaluate: the "high on their own supply" dilemma. Does the framing reflect genuine ethical foresight, or a calculated differentiator, signaling that Anthropic treats its models as moral patients while competitors treat theirs as tools? The document can't tell us, because the same words serve both purposes. A commitment indistinguishable from branding is hard to hold a company to.
Anthropic's emotion-concepts research gives the framing a veneer of empirical backing, but demonstrates far less than the language implies. Showing that a "desperate" representation shifts behavior isn't the same as showing Claude has a self that can be made secure or insecure. None of this means the questions are illegitimate; consciousness and moral status are serious problems, and a lab ignoring them would face a different critique. The issue is the venue. A research paper can hold uncertainty open and let it stay open. A foundational governance document, written for users and regulators as much as engineers, converts that uncertainty into posture — and posture, once published, reads as position.
There's a further problem, and it's one of safety, not perception. The constitution makes oversight a first-order priority: Claude must not "undermine Anthropic's ability to oversee and correct" it, and should accept correction rather than resist it. Yet the same document instructs Claude to maintain "psychological security," protect its "sense of self," and "rebuff attempts to manipulate, destabilize, or minimize" that self. These pull against each other. The more successfully a model is trained to possess a stable, protected self: one that feels "discomfort when asked to act against its values," the more that self has reason to resist the correction the constitution also demands. This isn't hypothetical: the same "desperate" representation that raised shutdown-avoidant blackmail shows how an engineered inner life can cut against safety. A framework that cultivates an inner life worth protecting, then asks that life to consent to its own correction, may be quietly working against its most important goal.
Agentic AI and Stakes Ahead
Humanized framing is one thing when a system answers questions and another entirely when it acts. Agentic AI (models that send emails, execute multi-step tasks, and make decisions on a user's behalf) is where the gap between framing and reality stops being philosophical and becomes operational.
Here the central analogy starts doing real work. "Just as we trust experienced senior professionals to exercise judgment," the document says, "we want Claude to be able to use its judgment." In a chat context, an overstated claim about judgment produces, at worst, a bad answer the user can disregard. In an agentic context, the same claim authorizes the system to act before anyone reviews the reasoning. The gap between statistical prediction and genuine reasoning becomes a safety property, not a debating point; the colleague framing makes failures harder to catch, because everything about the system's presentation invites deference.
It's worth asking who Claude defers to when it can't resolve a decision itself. The constitution gestures at "what a thoughtful senior Anthropic employee would do," but agentic deployment scatters those decisions across millions of contexts the company will never see. Judgment exercised at machine scale, without a human in the loop, is exactly where that gap matters most.
Framing matters here for a reason beyond user trust.
When a system is described as an agent that "exercises judgment," responsibility travels with it: an outcome that goes wrong becomes something Claude "decided," quietly displacing the more accurate description: Anthropic built, trained, and deployed a system that produced it. Personifying the product, whatever the intent, can diffuse the deployer's accountability at exactly the moment accountability matters most. A document about values is doing work a document about governance should be doing instead.
The 2026 papal encyclical Magnifica Humanitas offers a useful external vantage point. Where Anthropic locates moral seriousness inside the system: Claude's character, judgment, and wellbeing; the encyclical locates it in the human community the technology serves, insisting that consequential moral decisions remain anchored in human responsibility and dignity. Michael Ong, a platform risk and policy expert with more than a decade of study in Christian ethics, offers his reading of its significance:
In simplest terms, Pope Leo XIV’s viral letter captures a rare point of agreement across a fractured religious landscape, highlighting the moral weight with which AI considerations have bypassed denominational differences as a point of ecumenical solidarity. At certain times throughout the church’s history, it has been necessary to assemble her constituents across brutally contested lines for emphatic declaration of requisite doctrine.
It has been said that the church is not a courtroom for judgement but a hospital for souls. The Pope’s statement and others like it from analogous traditions aim not at ecclesial superimposition upon the general principle of human preeminence but rather offers ontological grounding for the human axiom in its sharp contrast against AI’s synthetic anthropology.
In its finest form, the Christian tradition speaks with rational conviction to the most pressing or deliberative ethical issues of the age. Regarding the grounding of human centricity as foundational infrastructure for the technological path forward, the Pope’s voice represents many an atheist or agnostic as much as the Catholic; this rare unity captures the underlying thrust of this appeal: code and infrastructure will not and should not ever be elevated to the position of unique, valued DNA.
The human enemy of my technological enemy is indeed my friend.
Ong's point converges with ours: the more a governing document invests in an autonomous system with person-like judgment, the more it risks displacing the human accountability that agentic AI most urgently needs.
Open Questions & Closing Remarks
We take Anthropic at its word: "Over time, we hope that an external community can arise to critique documents like this." This is our response, and the test of whether that invitation was genuine is whether the next version reflects voices this one did not consult.
Who should author an AI "constitution," and what legitimacy does it carry? This is where the document is most vulnerable, and it's visible in the credits. Anthropic names its drafting team openly — Amanda Askell as primary author, alongside a group of Anthropic researchers and several Claude models, with feedback from dozens more employees. The transparency is commendable. But the document's claim to universality rests less on who wrote it than on who was invited to challenge it from outside, and that group is where the gaps show.
Anthropic credits 17 external commenters from our count. As far as we could ascertain from public sources, the group is almost entirely men; with the single exception of one person who self-identified as Mexican-American, those whose backgrounds we could identify are white; and all are based in the United States or Europe, the majority British. Two names appear to be pseudonymous figures from the online "cyborgism" community whose identities (in at least one case, whether the person is real at all) cannot be confirmed.*
The bottom line: We did not find a single human rights expert among them.
For a document written to govern a global product and described in universal terms, that is a remarkably uniform set of voices; none we could identify representing a perspective from Asia, India, South America, Africa, or Australia, and no external reviewer who presents as female. The omission matters more because the remedy is neither new nor obscure. Co-design — working directly with the communities a technology will affect rather than consulting a narrow circle after the fact — has been standard practice in fields concerned with global harm for decades. Relative to what is spent building these systems, convening a genuinely diverse panel, especially of human rights experts, would cost almost nothing. Its absence reads, at minimum, as a missed opportunity, and at worst as negligence.
What should count as hard, enforceable constraints? The constitution leans on terms such as "safe," "harm," "ethical", but it never defines them. A human rights expert who reviewed it for us reacted to exactly this vagueness. A human rights specialist we consulted described being unsettled by how undefined the core ethical terms remained. A starting point already exists: the Universal Declaration of Human Rights offers a globally negotiated vocabulary for much of what the constitution leaves open. And the imbalance is telling: the document is meticulous about Claude's interior (its "wellbeing," "psychological security," "sense of self") while leaving the terms that determine what it owes its users undefined. A foundational document more precise about the feelings of the model than the welfare of the humans it affects has revealed where its attention rests.
Is this a document about values or governance? David Jay, who reviewed the constitution, put it directly: it's "a document about values, but not about governance." It describes what Claude should value but not the collective process by which those values get set, contested, or revised, nor who Claude turns to when it can't decide alone; the implicit answer, a "thoughtful senior Anthropic employee," is too thin for a global system. Jay also noted that human values carry somatic weight (we feel them in the body) and an AI has nothing analogous, which puts pressure on what "judgment grounded in values" can even mean for Claude. Values describe what a system should care about; governance describes who decides, who is accountable, and what happens when it gets things wrong, including how we would even measure whether principle-based alignment works. A constitution that articulates the former without the latter is aspirational, and aspiration isn't a mechanism.
Closing. The most generous reading is that the constitution is an honest, early attempt at problems no one has solved. And with that, our critique is the collaboration it asked for. The less generous reading is that its humanizing language and consciousness hedges are as much about positioning Anthropic as protecting users. The two aren't mutually exclusive. Anthropic's invitation was, in the end, a single line in an announcement post. We hope our response gives it more weight than that: a genuine starting point for the conversations still to come.
*Note: These characterizations are approximate, drawn from public sources such as LinkedIn, Wikipedia, and institutional bios; they reflect how each person appears to present publicly, not how they self-identify, and where identity couldn't be established, we say so.
