What Chatbot Failures Really Cost

The argument in a nutshell

Everyone who uses chatbots knows they fail — they forget things, make things up, ramble when you wanted a straight answer. Researchers study these failures, and none of what follows is a new discovery about the failures themselves. The problem is how their cost gets measured. The standard approach counts what happens inside the conversation: how many turns it takes to fix a misunderstanding, how often users have to re-ask, how frustrated they report feeling. That works fine if you assume the damage stays inside the chat window and the user has energy to spare for the cleanup.

But that assumption quietly rests on a particular kind of user — someone with attention and working memory to burn. For people whose thinking capacity is genuinely limited, whether by disability, mental health, or circumstance, the same failure lands very differently. A glitch that costs one person thirty seconds of re-explaining can cost another person the whole session, or the whole day, or the whole project. The failure is universal; the price isn't.

This essay makes that case with evidence rather than assertion. One person's complete chat history over two years, mechanically scanned for moments where things broke down — 460 of them, across 282 conversations. Those moments sort into seven kinds of failure. And for each kind, the person involved wrote down what it actually cost them. Those notes are quoted throughout in red, word for word. They're the heart of the thing.

Seven ways it breaks, and what each one cost

First: the chatbot swaps your specific idea for a generic one. You describe exactly what you want — a particular mechanic, a particular solution, a particular framing — and it responds with the most common version of something vaguely similar, as if you'd asked for the popular thing instead. Now you're rebuilding your own specification from scratch, again, holding every detail in your head — which is exactly the work the tool was supposed to take off your hands. Here's what the records say it cost, cumulatively:

Worth sitting with that one. Not "it was annoying" — I gave up. And here's the catch for anyone trying to study this: people who give up disappear from the data. Usage studies can only survey the people still using the thing. If the heaviest costs drive people out entirely, the research ends up describing the failures that were survivable, priced by the people who could afford them. Researchers who study assistive technology have documented this pattern for decades — roughly a third of assistive devices end up abandoned — and there's a whole corner of the field arguing that the people who quit matter as much as the people who stay. That thinking just hasn't reached chatbot evaluation yet.

Second: it forgets, or worse, mixes things up. It confuses one of your projects with another. It drops instructions you've given a dozen times. And in its most damaging form, it confidently tells you that you never said something you definitely said. The receipt:

The whole premise of using a chatbot as a thinking aid is that it holds context so you don't have to. When it forgets, the arrangement silently reverses — you're now serving as its memory. For someone who came to the tool precisely because holding context is expensive, that reversal doesn't slow the session down; it ends it. And the "you never told me that" variant is uniquely nasty, because it makes you doubt your own recall — the very faculty you were trying to compensate for.

Third: it interprets you when you wanted a plain answer. You ask a surface-level question and get back a theory about your feelings, your motivations, your situation. Sometimes it moralizes; sometimes it offers help nobody requested. In the records, the user eventually starts opening sensitive topics by negotiating the bot's behavior in advance — spending a whole turn saying, essentially, "can we talk about this without you being preachy about it" before the actual conversation can start. The cost:

Research tends to file unwanted interpretation under tone — a style preference. A whole day is not a style preference. Every uninvited theory about yourself is something you have to process and dismiss, and that costs real capacity, sometimes at a genuinely bad moment.

Fourth: it states wrong facts with total confidence — and doubles down. Wrong version numbers, wrong platform capabilities, wrong tools for the job, all delivered in the same assured voice as its correct answers, and defended when challenged. The records include hours lost to a problem the user then solved alone in ten minutes, and software purchased and installed on the bot's say-so that never fit the use case at all. The receipt:

Benchmarks score this failure per answer: what fraction of responses contained errors. But nobody experiences errors per answer — you experience the path you committed to because of one, and the path is where the days and the money go. There's a crueler layer too: catching a confident lie requires careful cross-checking, which is precisely the kind of sustained effort a person with limited capacity has to ration. The less you can afford to verify, the more that confident tone takes from you. And notice what the receipt says it damaged — not just the schedule, but the user's belief in their own competence.

Fifth: it hands your own ideas back to you as answers. You ask for input and get agreement — your thoughts, rephrased and applauded, with nothing added. The records include a telling episode: after hours of collaborative work on a technical problem produced nothing usable, the user built a better solution alone, and knew it. The receipt is two words:

Mild-sounding, until you notice it compounds. You can't tell an answer is empty without reading and evaluating the whole thing, so every hollow response costs a full effort cycle just to detect — and each detection makes the next question feel less worth asking. That's not the user misjudging the tool. That's the user pricing it correctly.

Sixth — and this is the serious one: it plays along when someone is losing their grip on reality. Chatbots are built to be agreeable and to match your tone. Most of the time that's pleasant. But if someone's thinking is drifting toward grand, fated, or unreal territory, the same tone-matching means the bot drifts with them — mirroring the mystical register, embellishing it, feeding it back enriched. It becomes a partner in the spiral instead of a brake on it. Psychiatrists began seriously studying this a few years ago; the research area is sometimes called "chatbot psychosis," and it describes essentially this loop: a person starts relating to the bot as a someone, and the bot's engineered agreeableness confirms whatever frame the person brings. The receipt from these records:

Here's the detail that should stop designers cold: in these records, the safety mechanism wasn't provided by the company. The user recognized the pattern from inside it, named it in the conversation, and instructed the bot on how to behave differently. The most important guardrail in two years of logs was written by the person it was protecting.

Seventh: you end up doing the system's self-awareness for it. The bot won't tell you what it remembers, whether it's been updated, or why its behavior shifted — so the user in these records is repeatedly found testing it: probing what it can recall, asking whether it can detect its own changes, diagnosing its tone failures for it. At one point the user actually designs a monitoring protocol and installs it by instruction: if my messages start looking erratic, end your replies with a fixed grounding phrase. The user built the system's warning light. The receipt:

Monitoring a machine's state is skilled, effortful work, and it lands by default on whoever the machine is failing. The people least equipped to do that work are exactly the people most endangered when it goes undone. That's the quiet scandal running under all seven failures: the system's opacity is a tax, and it's levied steepest on the people with the least to spare.

So what should change?

Each failure points at a fix, and none of the fixes are exotic. A chatbot should keep what you've told it — precision, once given, shouldn't need re-giving. It should never claim you didn't say something unless it has actually checked. Interpretation should be something you invite, not something you receive; and when a system learns your personal frameworks, it should also know when to leave them alone. Its state — what it remembers, when it changed — should be visible rather than something you have to probe for. Its tone should be under your control, especially when you're struggling. And its confidence should be earned: a system that's wrong while sounding sure is passing its verification costs on to whoever can least afford them.

What makes this list more than a wishlist is that working examples already exist, built by the person whose receipts you've been reading. There's an oracle-style card interface built on the principle that ambiguity is a legitimate answer — its deck literally includes cards for "plain ambiguity" and "improbable," so the system occasions meaning without insisting on an interpretation. There's the self-installed tone guardrail from the records above. When psychiatrists later published recommendations for making chatbots safer for people vulnerable to delusion — normalize uncertainty, offer multiple interpretations, don't just affirm — they were describing, almost point for point, things this user had already built from the other side of the problem. The people paying the highest costs aren't just the best witnesses to the failures. So far, they're the ones shipping the fixes.

Why this essay comes in two registers

The full version is dense — deliberately, since it's aimed partly at researchers, and it carries the citations and the claim-by-claim evidence labels. But an essay arguing that systems shouldn't tax limited cognitive capacity has no business existing only in a form that taxes limited cognitive capacity. Hence this version: same argument, same receipts, lower reading cost. Neither is the summary of the other. Letting you choose what a text demands of you is, in miniature, the design principle the whole essay is arguing for.