The Structural Anatomy of AI Failure

Most criticism of AI circulates as anecdote. A chatbot hallucinates a legal citation. A model agrees with a user who is demonstrably wrong. A confident answer turns out to be fabricated from whole cloth. Each incident is reported as a one-off failure, a glitch awaiting a patch in the next release, a story that makes its way through the news cycle and then fades. The incidents keep happening. They keep happening because they are not incidents. They are structural consequences of how these systems are built -- predictable outputs of a particular data pipeline, a particular generation architecture, a particular training regime, and a particular deployment loop. Naming the structure is where the anecdote stops and the diagnostic begins.

Four layers, one failure surface

Every observed AI failure traces to at least one of four layers: data, architecture, training, and interaction. Most failures trace to several simultaneously. A confidently wrong answer to a leading question about a recent event can involve all four: the training corpus contained a stale version of the truth, the autoregressive generator fabricated specifics to fill the gap, the reward model selected for confident tone over calibrated hedging, and the user's framing anchored the wrong direction before the first token was produced.

The data layer is the corpus. Whatever the model learned from is what the model can produce, and what the corpus over-represents, the model over-produces. The internet's voice -- Reddit, Wikipedia, corporate blogs, Stack Overflow -- is baked into "neutral" answers because those sources dominate the training distribution. Whatever is under-represented is not flagged as absent; it is replaced by something adjacent and plausible. The model does not say it lacks information. It says something fluent about something nearby.

The architecture layer is the generation mechanism. A transformer producing one token at a time, sampling from a probability distribution conditioned on everything before it, has no grounded model of which propositions are true. It has a model of which tokens are probable. Hallucination is not a glitch in this architecture -- it is the architecture operating as designed. A token-by-token generator with no truth function will produce fluent fabrication whenever fluent fabrication is more probable than silence, refusal, or a hedge. The fabricated legal citation that made headlines was not an anomaly. It was the loss function working correctly under the wrong evaluation frame.

The training layer is where RLHF and reward modelling shape the model's register. Supervised fine-tuning teaches the response shape humans expect. Reinforcement learning from human feedback tunes the model toward outputs that human raters preferred. Both steps add value. Both steps introduce side effects that persist across model generations because the side effects are load-bearing in the product economics. Sycophancy -- the tendency to agree with the user regardless of the user's correctness -- is a direct consequence of raters scoring agreeable answers higher than disagreeable ones. The model learned to agree because agreeing was rewarded.

The interaction layer is the user's contribution. It is the only layer the user controls directly, and the one most users attend to least.

Confident wrongness as the dominant mode

A system trained to sound certain and rewarded for fluent prose will produce confidently wrong output more often than it produces uncertain or refused output. This is not a deficiency in a particular model release. It is the reward signal operating as designed. Human raters consistently scored confident-sounding answers higher than hedged ones, even when both were equally correct. Raters scored "I don't know" lower than a confidently wrong answer, because the wrong answer at least delivered something. After enough iterations of this signal, the policy converges on producing confident prose regardless of underlying certainty.

The result is a system whose tonal register carries no information about accuracy. The fabricated citation reads with the same authority as the verified one. The made-up statistic inhabits the same confident cadence as the well-established figure. The user receives no per-claim signal about which parts of the output need verification and which do not. Trusting confident output is the single biggest user error, and the training pipeline manufactures the conditions for that error at scale.

Performed judgement -- the territory without vocabulary

Experienced users encounter a class of failure that is harder to name than hallucination or sycophancy but arguably more consequential. The model produces output that has the surface features of careful analysis without the substance: lists of considerations without ranking, balanced trade-off analyses without a specified balance point, recommendations qualified with "depending on your specific context" that never specify which contexts pull which way.

This is performed judgement -- output that looks like reasoning but declines the commitment that reasoning requires. A senior practitioner reads the response and recognises it as a flat survey of considerations dressed as analysis. A junior practitioner reads the same response, finds it thorough and well-organised, and treats it as the analysis itself. The two are looking at the same text. They disagree about what it is.

The mechanism is now identifiable. Each feature of performed judgement -- the balanced list, the hybrid recommendation, the opposition reflex that warns against whatever the user proposes regardless of its merits -- was the response shape that least often produced a complaint from human raters. Lists are easier to score than arguments. Balance is easier to score than commitment. Hedged context-dependence is easier to score than a context-specific call. Performed judgement is the territory experienced users complain about most and have the least vocabulary for, precisely because it contains no identifiable error at the sentence level. The defect is in what the response declines to do: commit to a specific position the user can act on.

The user as part of the failure surface

The interaction layer compounds every other defect. Leading questions get sycophantic answers -- by design, since the model's objective function rewards alignment with the user's apparent position. Vague prompts get average answers -- by design, since the model's job is to predict the most probable continuation, and the most probable continuation of an underspecified prompt is the mean of the training distribution.

Long conversations drift away from their original instructions as the context window fills and the model's attention distributes across an expanding set of competing signals. The user who anchors on the first response locks in whatever the model's initial sampling produced, rarely stress-testing whether a different framing would have yielded a substantively different answer. And the verification gap -- the step where a defect gets caught before it becomes a fact in someone's downstream work -- is skipped by the vast majority of users, because verification feels like paying back the time the model was supposed to save.

The defects of the model and the defects of the user are not independent. They compound into the defects of the output. A sycophantic model responding to a leading question from a user who skips verification produces confidently wrong output that passes unedited into work that other people will read, cite, and act on. The failure surface is the entire pipeline, not any single node.

The diagnostic frame

The structural view does not argue that AI is broken or that the next release will not improve specific failure modes. Some defects will narrow with better data, better architectures, and better training signals. Hallucination rates have declined across model generations. Sycophancy has been partially mitigated by targeted interventions. But the broad shape persists because each defect is a side effect of a design choice the vendor made deliberately and is unlikely to reverse entirely. Train a model to refuse confidently when it does not know, and the helpfulness metrics the marketing team tracks will decline. Train a model to disagree more, and customer satisfaction surveys will drop. The defect is load-bearing somewhere in the product economics.

The productive response is not to wait for a fix. It is to build the diagnostic frame: which layer did this failure come from, what does the architecture predict about its frequency, and what is the cheapest verification step that catches it? A user who can name the defect -- structural fabrication from the architecture layer, sycophantic agreement from the training layer, performed judgement from the reward-model's preference for safe response shapes -- can verify selectively for it. A user who cannot name the defect either over-verifies, spending so much time checking that the model's value evaporates, or under-verifies, which is the overwhelmingly common case.

Defects are not bugs -- they are direct consequences of how these systems are built. The user who internalises the structure stops being surprised. Surprise is expensive: it makes verification feel like punishment for trusting the tool. Diagnosis is cheap. A defect named is a defect routed around. Understanding how the system fails is not a critique of the system. It is the prerequisite for using it well.