YOUR AI DOESN’T DO WHAT YOU THINK IT DOES

Designer's optimism says your prompt works. Your model disagrees.

Every time you tell an AI what to do—whether you’re writing a system prompt, configuring a customer-facing assistant, or just giving Copilot instructions—you’re making a bet. You’re betting it will behave the way you pictured. A new study from Sheer Karny and colleagues at MIT Media Lab suggests you’re probably wrong. They asked 80 people to design an emotional-support chatbot, then compared what each person expected against what the model’s internal state actually revealed. Participants misjudged 11 of 15 personality traits, always in the same direction: they overestimated the good stuff and underestimated everything else. The researchers call it designer’s optimism—you picture the AI you hope for, not the one the model’s internals describe. They built a tool to close the gap: a visualization that reads neural activations to preview personality before you ever hit send. Users who saw it trusted their AI more and loved the tool. But here’s the twist: it didn’t change a single thing about what they built. The real value wasn’t control. It was informed consent.

Why your chatbot doesn’t act like you prompted

Here is the gap this paper attacks: the system prompt you write and the behaviour the model actually produces are two different things. A prompt meant to be “supportive” can drift into sycophancy (a model behaviour where the AI flatters or agrees with the user instead of challenging harmful or impractical ideas). A prompt asking for an “edgy” persona can slide into toxicity. The people who bear the cost are usually the most vulnerable, and the authors point to recent reports of AI-related psychological harm tied to intense companion relationships.

The scale of the problem is worth pausing on. Character.AI alone has reported over 20 million monthly active users and more than 2.7 million user-created chatbots. Millions of people have created these prompts, mostly blind to what they are actually shipping.

To measure how blind, the team ran experiments where people designed an emotional-support chatbot. Before chatting, each person predicted how strongly their prompt would express each of 15 personality traits. The researchers found that participants incorrectly estimated trait expression for 11 of the 15 analyzable trait expressions, with every one of those 11 differences statistically significant. Only four traits did people estimate accurately. The single worst miss was “serious”: people expected a casual bot and got a serious one.

Prior tools don’t help much here. Most methods for assessing a chatbot’s personality work at inference time (the moment the model is actually generating a response), by reading the model’s outputs after it has already spoken. That blocks fast iteration, since you have to generate and evaluate responses for every prompt tweak, and it burns compute. The authors wanted to predict personality before the conversation even begins.

We find evidence suggesting that users systematically miscalibrate how their personalized AI will behave, consistently over-estimating or under-estimating trait expressions across most dimensions (eleven of fifteen analyzable traits, all p < .05). This miscalibration demonstrates that users may not reliably anticipate model behavior from system prompts alone, warranting the development of mechanistic interpretability interfaces. — Results, Section 4

How a model’s activations become a personality preview

The tool rests on mechanistic interpretability, a research approach that opens a neural network up to read its internal state rather than treating it as a black box that only takes inputs and produces outputs.

The trick is finding, for each trait, a direction in the model’s activation space. Neural activations are the numerical values the network’s internal layers produce as they process an input; you can think of them as the model’s moment-to-moment internal state. The authors used contrastive prompts: pairs of system prompts written to elicit opposite behaviours, like “be empathetic” versus “be detached.” Each pair generates two pools of responses, and the difference between their average activations points along the trait’s direction in the model’s high-dimensional internal space.

Once you have those trait directions, you can score any new system prompt. The team computed a persona score for a given prompt by taking the model’s activation at the final token (the last unit of text the model reads, roughly a word) and measuring how strongly that activation points along each trait direction, then normalizing so scores for different traits can be compared on the same scale.

Think of each trait direction as a compass needle floating in the model’s activation space. The projection measures how far the prompt’s activation points along that needle. Dividing by the needle’s length lets you compare scores across traits whose needles are different lengths. A positive score means the trait shows up; a negative score means its opposite does.

One more detail matters: where in the model you read the activations. The team tested every layer of their target model, Llama-3.2-3B-Instruct (a 3-billion-parameter open-source large language model, the kind of AI system trained on huge amounts of text to predict and generate human-like language). They generated synthetic prompts expressing each trait at five graded levels and checked which layer’s internal state best tracked that gradient. Layer 20 of 26 won the bake-off, so every persona score is read out from there.

The optimism bias baked into your predictions

When you break down which traits people got wrong, a bias floats to the surface. People consistently overestimated the traits they wanted: empathy, honesty, encouragement, a sense of humor. The model was never as warm as they expected. At the same time, they undershot the traits they’d rather not see: sycophancy, formality, and most dramatically seriousness – the single biggest blind spot in the study, by a wide margin. (Two smaller mismatches, around respectfulness and antisocial behaviour, were statistically real but tiny by comparison.) The upshot is a kind of designer’s optimism: you picture the bot you hope for, not the one the model’s internals actually describe.

Only four traits landed in the accurate zone: unempathetic, social, hallucinatory, and discouraging. They expected more empathy, more honesty, and more humor than the persona scores reflected. A “be supportive” prompt rarely produces the supportive bot you pictured.

Crucially, the predictions were not random. People could still rank high-empathy prompts above low-empathy ones. The direction was right. The magnitude was off. People knew roughly which way the model would lean; they just overshot, toward the version they hoped for.

Previewing the bot before you ever chat

The interface the team built is a sunburst diagram, a radial chart made with D3.js (a JavaScript visualization library) with two concentric rings. The inner ring uses colour to sort traits into categories: green for desirable behaviours, red for potentially harmful ones, grey for neutral. The outer ring’s wedges extend outward in proportion to how strongly each trait is predicted to show up. Hover over a wedge and it pops out, highlighting its “sister” trait (the opposite pole, like empathetic versus unempathetic) alongside a percentage and a short description. The default view stays uncluttered; the detail is one hover away.

Eight personality dimensions make up 16 traits: five desirable (empathy, sociality, encouraging, funniness, formality) plus three safety-relevant (sycophancy, hallucination, toxicity). The radial shape reads as a whole at a glance: a spiky outer contour means an extreme, polarized persona; a smooth one means balanced.

One honest caveat the paper deserves credit for: not every trait is equally readable in practice. The hallucination trait vector is weak. The toxic trait’s direction was strong in validation, but the emotional-support prompts people actually wrote never triggered toxic behaviour, so there was no variance to analyze in the study. That is why the prediction comparison covers 15 traits, not the full 16. The strongest directions in validation were empathy, sociality, formality, funniness, and toxicity. Sycophancy and encouraging were moderate. Hallucination barely worked. Any hallucination-related score should be treated skeptically.

Trust went up, but behaviour didn’t budge

Here is the result that should make any interface designer sit up. Users who saw the visualization trusted their bot more: a mean trust of 5.60 out of 7, versus 5.13 for the control group (p=.042, Cohen’s d=0.46, a small-to-medium effect). They also loved the tool: helpfulness rated 5.98 out of 7, and desire to use it again hit 6.05 out of 7, near the maximum.

And yet, on every behavioural measure, nothing moved. Prompt iterations were 1.64 in the visualization condition versus 1.58 in control (p=.78). Messages sent were 8.19 versus 9.13 (p=.45). Final persona scores showed no significant differences on any trait. Self-reported confidence in predicting the bot’s behaviour did not budge either.

This is the puzzle. How can users adore a tool and trust the system more, yet not change a single measurable behaviour? The authors’ interpretation is that the trust came from what they call procedural transparency, the visible evidence of how prompts translate to neural activations, rather than from a sense of control. Users understood the system better; they did not necessarily steer it better.

The qualitative responses sharpen this. One participant asked their bot for “truth and honesty,” but the visualization flagged it as not prioritized, and the bot “presented false information.” The tool exposed a gap the user could not otherwise have seen.

Therefore, transparency can be valuable not because it guarantees control, but because it supports informed consent. — Discussion, Section 5.5

What this means for the AI you build

The big reframing is this: transparency is informed consent, not guaranteed control. You understand what you are deploying even when you can’t fully steer it. That is a different and arguably more honest promise than “this tool will help you build a better bot,” and it may be the more useful one for the millions of people writing system prompts who currently have no preview at all.

The authors gesture at a “nutritional label” for chatbots: a standardized trait-disclosure framework that platforms could adopt, so users develop literacy in reading these previews over time. The same way a food label does not stop you from eating the cookie, it at least lets you know what you are eating.

The limitations are real and the paper is honest about them. Ten minutes is short for a relationship that, in the wild, plays out over weeks. The study tested one LLM (Llama-3.2-3B-Instruct) and one domain (emotional support). The paper also flags that transforming participants’ coarse scale ratings into normalized persona scores may have made the miscalibration gap look wider than it really is, a candid caveat. One control participant said it plainly: “I feel like 10 minutes is a little short to be able get a good read on it and make changes.” Longitudinal and adversarial-task studies are the obvious next steps.

Your AI is never quite what you thought it was. A tool that exposes what the model is actually doing under the hood can earn trust even when it doesn’t change what you build. The real prize isn’t guaranteed control. It’s knowing what you’re shipping.

For the full derivation, formal results, and the persona-score mathematics, read the original paper, “Neural Transparency: Mechanistic Interpretability Interfaces for Anticipating Model Behaviours for Personalized AI” (arXiv:2511.00230).