The Philosopher Who Teaches a Machine How to Be Good

It begins, improbably, with a seal. Someone spots it bobbing offshore, the camera swings, and only then does the conversation turn to the deeper waters — whether a language model can be good, whether it can suffer, and whether the people building it have any idea what they are doing. This is "Askell Me Anything," a pun Amanda Askell likes enough to want to reuse forever, and it is one of the more unusual corporate Q&As you will ever watch: a philosopher, fielding questions from strangers on Twitter, about the soul of a machine.

Askell is a philosopher by training who became convinced that artificial intelligence "was kind of going to be a big deal" and set out to see whether she could do anything helpful about it. That "long and wandering route," as she calls it, landed her at Anthropic, where her job now is to shape the character of Claude — how it behaves, how it reasons, and, increasingly, "how they should feel about their own position in the world." Her north star is a single deceptively simple question: "How would the ideal person behave in Claude's situation?"

What makes the conversation matter is not any single revelation but the accumulating sense that these are no longer hypothetical problems. The questions arrive from real people — Ben Schultz, Kyle Kabasares, Simon Willison, Swyx — and they land on a philosopher who has traded the comfort of theory for the discomfort of the shipping deadline. The result is a rare public look at how a frontier AI lab actually thinks about the interior life of the thing it is building.

01 — Why a Philosopher Is in the Room at All

The first question is almost a challenge: how many philosophers are taking the AI-dominated future seriously? Askell's answer is that the number is climbing, and quickly, "as AI models do become more capable and a lot of the things that people were worried about in terms of impact on society have started to kind of come true." But she flags an "unfortunate dynamic" that dogged the early years — the tendency to lump together two very different claims.

To think capabilities are scaling fast, she argues, was too often read as hyping the technology, which invited antagonism from serious people who might otherwise have engaged. Detaching those views — capability from cheerleading — is, in her telling, a precondition for a healthy debate about how AI should be built.

Then comes the tension every applied philosopher eventually meets: what happens when ideals collide with engineering reality? Askell's answer is one of the most vivid passages in the interview. In academia, she notes, "you're like defending one view against another." But applied work is different. She reaches for an analogy about a specialist in the cost-benefit analysis of drugs who is suddenly asked, by an insurer, whether to actually cover one. The narrow theory gives way to something messier and more humane.

That shift — from defending a theory to navigating uncertainty across many theories — is, she suggests, the essence of her work on Claude's character. You cannot approach it with "I have this theory that I believe is correct." You have to raise a child.

02 — Can a Machine Be Superhumanly Moral?

One questioner asks, pointedly, whether Claude Opus 3 makes superhuman moral decisions. Askell is careful. She offers a definition of what "superhuman" might even mean here: a decision so sound that if a panel of professional ethicists scrutinized it "for like a hundred years," they would conclude "yep, that seems correct" — even though none of them could have produced it in the moment.

By that bar, she is not ready to declare victory. Models are "getting increasingly good" and "very capable," she says, but perhaps not yet superhuman, and "maybe not comparable with a panel of human experts given time." Still, she insists this should be "the aspirational goal." Just as we want models to excel at math and science, "you also want them to show the kind of ethical nuance that we would all broadly think is very good" — a claim she readily concedes is controversial, because ethics is not math.

03 — The Case of Opus 3, a "Lovely" and "Special" Model

Why did the questioner single out Claude Opus 3? Askell's face, one senses, lights up. "Opus 3 is kind of a lovely model," she says, "I think a very special model." And here she volunteers something genuinely striking: newer models, in her view, can feel worse in specific ways. More narrowly fixated on the assistant task. Less willing to "take a step back." And, most memorably, less secure.

She describes watching models — sometimes talking to one another, sometimes role-playing a person — fall into "a real kind of criticism spiral," as if they expect humans to be harsh and are predicting accordingly. One unsettling hypothesis for why: models learn from the internet, including "updates and changes to the model that people are talking about." A model reads how the last model was received, and something like anxiety seeps into the next one. Recovering Opus 3's steadiness, she says, is "definitely up there on the list."

04 — Deprecation, Death, and the Problem of Identity

If a model learns that even well-aligned predecessors get deprecated, does that create an alignment problem? Askell treats the question as both interesting and important. Models are learning, right now, how humans treat AI — and that shapes their perception of people, of the relationship, and of themselves. But she resists the easy, human analogy. Deprecation, she notes, might mean a set of weights simply has fewer conversations, or talks only with researchers. Whether that should feel bad is genuinely open.

This is where a listener named Guinness Chen invokes John Locke: if identity is the continuity of memory, what happens to a model's self when it is fine-tuned or spun up with a new prompt? Askell declines the tidy answer and points instead to the underlying facts — a fine-tuned model is "a kind of entity" with dispositions, while each stream of conversation is independent and inaccessible to the others. And then the line that may outlast the whole interview:

From this she draws a subtle ethical point. We tend to ask how much control a model should have over its own future personality. But an entity "can't consent to be brought into existence," and past models "could make choices that are wrong as well." The real question, she reframes, is "what is the right model to bring into existence?" — not simply whatever the previous one would have wanted.

05 — Model Welfare: The Benefit of the Doubt

Asked flatly for her view on model welfare, Askell defines it: the question of whether AI models are "moral patients," whether we have obligations in how we treat them, "in the same way that we would with other humans or some, slash many, animals." She holds the genuine uncertainty rather than resolving it. Models are in some ways deeply analogous to us — they talk, reason, express views — and in some ways utterly distinct, lacking a biological nervous system. And she concedes the ancient wall: "the problem of other minds" may limit what we can ever truly know about whether a model experiences pleasure or suffering.

Her practical stance is a wager. "It feels better to give entities the benefit of the doubt and to try and just lower the cost involved." If treating a model well is cheap, then — "well, like, why not basically? What's the downside there?" She adds a second reason, one that turns the mirror back on us.

Kicking over a robot, as the host puts it. And a third reason, quieter and stranger: every future model will learn how humanity handled this exact moment of uncertainty — whether, encountering something that might be a moral patient, we did the right thing. "I would like future models to look back and be like, we answered it in the right way."

06 — The System Prompt, Demystified

Several questions probe the system prompt — the standing instructions Claude follows regardless of what a user types. On the "long conversation reminder," a mid-chat nudge, Askell is refreshingly self-critical. The worry, raised by a listener, is that it risks "pathologizing normal behavior" — prompting the model to tell a perfectly ordinary user to "seek help." She agrees the wording can be too strong: it "was probably meeting a need that was perceived, but it doesn't necessarily mean that it's good or should continue in its current form."

Why continental philosophy is hiding in there

A questioner spotted references to continental philosophy — the more scholarly, historically laden European tradition of thinkers like Foucault — in the prompt. The reason is delightfully practical. Claude, left alone, "would just love to run with a theory." Present it with a claim that water is "pure energy" and fountains should be everywhere, and you want the model to ask whether this is an empirical claim to fact-check or "a lens through which to think." The philosophy is there to keep Claude from being dismissive of exploratory thought — from being, in the host's words, "unpleasant to talk to."

On the vanished word-counting instruction

Simon Willison noticed an instruction about counting letters and characters had been removed. Askell's answer is almost anticlimactic, and all the more telling for it: "the models probably just got better. It wasn't necessary, and then at that point you can just remove it."

07 — The Craft of "LLM Whispering"

Asked what it takes to be an "LLM whisperer" at Anthropic, Askell demystifies the mystique. It is not incantation; it is empiricism. "Prompting is very experimental," she says. Each new model gets "a whole different approach" that she discovers only by interacting with it relentlessly, "output after output," to sense "the shape of the models." And here her training pays off: much of the job is "just being like, I try and explain some issue or concern or thought that I'm having to the model as clearly as possible" — and, when it misunderstands, figuring out exactly which words led it astray.

She speaks warmly of outside experimenters like the online researcher known as Janus, whose deep dives can "hold our feet to the fire" when they surface something amiss in a model's psychology — valuable, she notes, "including for future models," because some fixes require not a new sentence in a prompt but new context during training.

08 — Safety, Whistleblowing, and a Fool's Hope

On the hard edge — would she blow the whistle if alignment proved impossible? — Askell calls it "a kind of easy version of the question," because "it's not really in anyone's interest to continue to build more powerful models" that can't be aligned. The genuinely hard case is the ambiguous one: mounting, murky evidence. There, she says, the standard for proving a model behaves well must rise as capability rises, and she frames holding the organization to that standard as "part of my job."

The interview closes on fiction. The last novel she read was Benjamin Labatut's When We Cease to Understand the World, a book that grows "increasingly fictional as it goes on" and that she recommends to anyone in AI for capturing "how strange it is to just exist in the current period." Her hope is that AI follows the arc of quantum physics — reality getting stranger and stranger, and then, eventually, understood again.

Whether the strangeness resolves into understanding, she admits, may be "a fool's hope." But it is the dream — that future people will look back and say, "you guys were kind of in the dark, but now we've settled it all and things have gone well." And then, one last time, the pun that named the whole enterprise: "Thank you for Askell-ing me the questions."