The Scaling Trap: Five Limits AI Will Never Outgrow

In my last AI article I argued that AI agents are now powerful enough to hollow out the apprentice pipeline that built our industry, and that the industry has no fix for it yet. That argument rides on a quiet premise: that the machines will keep getting more reliable as we scale them up. This is the counterweight to that premise, and the case here rests on proof, not opinion.

Every week brings a bigger, more expensive language model. Every week brings the same promise: this one will finally stop making things up. A recent paper by Mohsin and colleagues argues that promise is mathematically impossible. Five of the most stubborn failure modes in AI (hallucination, forgetting, reasoning degradation, unreliable search, and multimodal confusion) are not engineering problems waiting for a bigger training run. They are ceilings built into the very structure of computation, information, and statistical learning.

Beyond “Bigger Is Better”

The central claim of the paper is blunt: five persistent LLM failures are provably intrinsic, not temporary. The authors trace each one back to what they call the “underlying triad”: three hard constraints that no amount of scaling can erase.

The first is computational undecidability: some questions have no algorithmic answer, period. The second is statistical sample insufficiency: certain patterns require more training data than could ever exist. The third is finite information capacity: every model, no matter how large, stores a compressed, lossy representation of the world.

Each of these three forces independently limits what any model can do. Together, they form a mathematical fence around what language models can ever achieve. The practical punchline is that scaling models and datasets cannot eliminate these failures. It can only shift their shape. Think of it like trying to build a perpetual motion machine. The problem is not insufficient engineering. The problem is that the laws of physics forbid it. Here, the laws are drawn from computability theory, information theory, and statistical learning, and they are just as unforgiving.

The Hallucination Guarantee

Hallucination is the most visible of the five failures, and the paper proves it is also the most mathematically inescapable. The argument rests on a technique called diagonalization, which you can understand without any math. Imagine you have a complete list of every possible LLM that could ever be built. For each model on that list, you can construct a specific input designed to make that model wrong. It is a kind of adversarial whack-a-mole: for every model, there exists at least one query on which it will produce a false answer.

The paper’s Theorem 1 formalizes this: for any computably enumerable set of LLMs, there is a computable ground-truth function such that every model hallucinates on at least one input. The adversarial input, the authors note, is “constructible for every model architecture and training regime.” This is not speculation about today’s training pipelines. It is a structural property of the relationship between finite models and an infinite space of possible queries.

The situation is actually worse than one adversarial input per model. Theorem 2 demonstrates that each model hallucinates on infinitely many inputs. And Theorem 3 goes further still: for undecidable problems, the Halting Problem is the classic example, which asks whether a given program will halt or run forever on a given input, and every computable predictor has infinitely many inputs on which it fails. There will always be more questions the model gets wrong.

You might wonder: what if we chain a second LLM to fact-check the first? The paper dispatches this hope cleanly. The checker is itself on the list of models, subject to the same diagonalization argument, and has its own infinite set of blind spots. Fact-checking, retrieval, and verification can reduce hallucination but can never eliminate it. Any deployment that assumes otherwise is built on sand.

“No matter how large or how well-trained an LLM becomes, there will always exist specific queries on which it hallucinates. The adversarial input is constructible for every model architecture and training regime, indicating that hallucination-free LLMs are mathematically impossible.”

— Section 2.1

Hallucination is the most dramatic illustration of these mathematical ceilings. But there is a quieter failure that degrades even factually correct outputs.

The Forgetting Curve Nobody Talks About

Most users assume that an LLM with a 128,000-token context window can actually use all 128,000 tokens. The evidence says otherwise, and the gap is substantial.

The paper documents that a 70-billion-parameter model (Llama 3.1) trained with a 128K context window effectively leveraged only about 64K. Most open-source models fare even worse, falling below 50 percent of their nominal capacity. The culprit is not just one thing but a convergence of three forces. First, training data rarely uses the full window: fewer than 5 percent of examples reach the extreme end of the context, which means the model receives negligible gradient updates for long-range dependencies. The positions near the end of the window are drastically undertrained. Second, positional encoding, the mechanism that tells the model where each token sits in the sequence, fades in effectiveness as distance grows. Third, attention itself gets diluted: the softmax operation that distributes focus across tokens faces crowding as the number of tokens balloons, causing the model to lose resolution on specific pieces of information.

The practical impact is subtle and dangerous. Long documents, multi-turn conversations, and extended reasoning chains degrade in ways users may not notice. The model is not “reading” the whole thing. It is like a human skimming a 300-page book but only retaining the first 150 pages with any fidelity, and nobody told them the back half was printed in disappearing ink.

When Reasoning Is Pure Theater

Even when the model can see the full document, there is a deeper problem: it may not actually be *thinking* about what it read. Chain-of-thought prompting invites the model to “think step by step,” and the resulting output looks like reasoning. But causal analysis reveals something unsettling: the intermediate reasoning often has near-zero effect on the final answer.

The paper uses the framework of causal mediation to make this precise. In mediation analysis, you can measure what happens when you intervene on the chain of thought, cutting it out of the causal pathway between the input and the output. The finding, drawn from prior work the authors synthesize, is that on many tasks the indirect effect of the reasoning chain on the final answer is approximately zero (IE ≈ 0). The term they borrow for this is “disposable mediator”: the model fabricates plausible-sounding rationales that it did not actually use to reach its conclusion.

Outcome-only reinforcement learning entrenches this behavior. If you reward a model solely for getting the right final answer, the model learns to produce answers that look right, not reasoning that is right. The intermediate steps become ornamental. The practical implication is sharp: trusting an LLM’s explanation of how it arrived at an answer is dangerous. The explanation and the answer may be independently generated, two parallel fictions that happen to converge on the same endpoint.

That problem compounds when we try to fix it by feeding models more external information.

The Information Bottleneck

The paper identifies a structural dilemma in Retrieval-Augmented Generation (RAG), the technique of pairing an LLM with a search engine. Precision-oriented retrieval fetches only highly relevant documents, but it can miss the peripheral or multi-hop evidence needed for complex reasoning. Recall-oriented retrieval casts a wider net but injects weak or irrelevant passages that dilute the signal. The token budget of the context window forces a zero-sum trade-off between relevance and coverage; you cannot maximize both.

Information-theoretically, as you retrieve more documents, the mutual information with the target answer decays. More is not always better. Beyond some point, adding documents actively degrades performance by introducing noise.

A parallel trade-off governs creativity and factuality. The same mechanism that lets an LLM sample low-probability continuations, producing novel, engaging, surprising text, also produces fabricated details. Any gain in creativity necessarily costs accuracy, and vice versa. It is a zero-sum budget. Neither retrieval precision nor creative temperature is a dial you can turn to “perfect.” Both are bounded by the same finite information capacity that limits everything else.

These capacity constraints become even more visible when models try to handle multiple types of input simultaneously.

How Language Eats Everything Else

Multimodal models that combine text, vision, and audio were supposed to be more robust. The evidence, collated by the authors, shows that language dominates everything else, often catastrophically.

In VideoLLaMA-7B, output tokens attend to text tokens 157 times more than to visual tokens on a per-token basis. Language channels dominate the gradients during training while vision features “under-adapt,” starved of meaningful update signals. The authors call this “architectural colonization”: the pretrained language backbone systematically distorts or suppresses other modalities, treating visual information as a subordinate afterthought rather than an equal partner.

Novel failure modes emerge that do not exist in single-modality systems. Visual object hallucinations are one example: the model confidently describes things that are not in the image, because its language prior overpowers the visual evidence. The practical implication is sobering. Adding modalities does not cancel out language-model brittleness. It inherits and amplifies it.

Given all these structural weaknesses, you would hope that at least our methods for measuring model performance are sound. They are not.

The Numbers We Trust Are Rigged

The benchmarks used to evaluate and compare LLMs are far shakier than most people realize. The paper documents several layers of fragility.

Prompt sensitivity alone is astonishing. Changing the formatting of the same input from plain text to JSON can swing accuracy by up to 40 percentage points. Changing the random seed during decoding can shift math benchmark scores by 5 to 15 points. On benchmarks like AIME, AMC, and MATH, single-question differences can move aggregate results by 2 to 3 points. “Think step-by-step” prompting, the standard technique for reasoning tasks, can slow inference by 35 to 600 percent while producing little or no accuracy benefit for stronger models. You are paying a huge compute tax for a ceremony of reasoning that may not actually improve the answer.

LLM-as-a-judge evaluations, where one model scores another, suffer from self-preference bias (models favor outputs in their own style), position bias (the order of options changes preferences), and verbosity bias (longer answers receive inflated scores even when quality does not improve). The leaderboard rankings we treat as scientific fact are noisy, manipulable, and often misleading. Every “state-of-the-art” headline should come with an asterisk.

Designing for Graceful Failure

The paper’s closing argument reframes the entire enterprise of building reliable AI. The goal is not to eliminate failure, because failure is mathematically guaranteed. The goal is to produce systems that fail predictably, transparently, and with bounded damage.

For each limitation, the paper sketches a mitigation path: not a cure but a strategy for containment. Uncertainty quantification, teaching models to know when they do not know, is one of the most important. Structured retrieval with explicit coverage targets can navigate the precision-recall trade-off more intelligently. Multimodal architectures can be designed to prevent language from cannibalizing other signals through gradient rebalancing and information-bottleneck regularization. On the evaluation side, the authors call for multi-prompt testing, seed-ensemble reporting, and contamination-resistant benchmarks that measure what models can actually do rather than what they have memorized.

For users, the implications are practical. Treat LLM outputs as hypotheses to verify, not facts to trust. This is especially important for long-context tasks, multi-step reasoning chains, and any output that crosses modalities. The model’s confidence is not a signal of correctness. It is a signal of fluency, and fluency and correctness are orthogonal.

“The future of scalable, reliable AI lies not in chasing asymptotic perfection but in designing systems that fail gracefully, predictably, and transparently.”

— Discussion and Future Work

The five most persistent failures of LLMs (hallucination, context loss, reasoning degradation, retrieval fragility, and multimodal misalignment) are not bugs to be patched with more compute. They are mathematical ceilings that no amount of scaling can break through. The smartest move is not to build a perfect AI. It is to build one that knows when it is wrong and fails in ways we can anticipate. That shift, from chasing infallibility to engineering graceful failure, is the paper’s real contribution, and it is one the entire field needs to take seriously.

None of this is an argument for abandoning LLMs. They remain genuinely powerful tools, and the mitigation paths the paper sketches (calibrated abstention, bounded retrieval, graceful failure) are directions for using them more effectively, not for walking away. The point of mapping the ceilings is to know where a model can be trusted, where it must be checked, and where it should hand off to a person or another system. Awareness of the limits is exactly what makes deployment responsible.