In mid-2025 a quiet paper landed with a thud. Titled “Hallucination Stations,” it argued—not as opinion but as formal analysis—that transformer-based language models face mathematical barriers when asked to act as reliable, general-purpose agents. The claim was stark: left to their own probabilistic devices, these models will make mistakes you can predict in kind if not in timing.
It’s tempting to treat this as a purely academic spat. It isn’t. Around the same time, big vendors and startups were promising that 2025 would be “the year of the AI agent.” That year ended with more demos than deployments, and the paper’s argument offered one reason why: math and production environments don’t always play nice.
The mathematical challenge (and why it matters)
The paper’s core point is simple to state and fiendishly stubborn in practice: language models are optimized to predict tokens, not to implement deterministic procedures. When you pile multistep reasoning, tool calls, and stateful memory on top of probabilistic output, unpredictable behaviors—hallucinations, silent policy violations, runaway loops—become not just possible but likely.
OpenAI’s own research has quietly acknowledged this reality. In experiments asking models to report an author’s dissertation title, multiple systems fabricated plausible-sounding but false answers. The conclusion was sober: accuracy won’t hit 100 percent. That’s a blunt reminder that even the best LLMs can invent details when pressed to be authoritative.
Why care beyond the lab? Because agents aren’t just toys for tech demos. Companies want them to automate hiring messages, manage calendars, triage tickets, and—eventually—coordinate complex workflows. A hallucination that invents a deadline or silently escalates a ticket can have real cost and compliance consequences.
Why the industry keeps building anyway
If the math looks grim, why are hyperscalers and venture-backed teams still racing forward? Two forces are at work.
First: economics and momentum. Agentic features sell. Google, for example, has been rolling agentic booking and calendaring capabilities into its products, betting that users value convenience even if the systems aren’t infallible; you can see that push in Google’s recent expansion of AI booking features and UI experiments around AI modes in its products. Second: hybrid engineering. Researchers and startups are building verification layers, secondary models, formal methods, and rule-based control planes to surround LLMs—effectively turning an unreliable core into a component within a more deterministic system.
One startup’s gambit is instructive. A team using formal verification tools to check LLM outputs claims improved reliability on coding tasks by encoding logic into provable forms. That doesn’t erase hallucinatory tendencies, but it confines them to places where they can be detected and corrected.
Where agent demos break in production
Engineers who’ve tried deploying agents at scale report a surprisingly familiar list of failure modes. These are not exotic bugs. They’re architectural: nondeterministic outputs, distributed memory that’s hard to trace, stealthy policy breaches embedded in prompts, and cost blowouts from open-ended reasoning loops.
A common pattern: an agent given latitude to “keep trying different approaches” eventually spirals into hundred-step explorations, invoking models and APIs at enormous cost. Another pattern is state drift—where the agent’s internal memory accretes context until its actions drift far from the original intent. These behaviors expose how demos gloss over the hard parts of running software in the wild: observability, auditability, and cost control.
That’s why many practitioners are arguing for a control-first architecture. Make the agent one part of a workflow, not the entire control plane. Enforce iteration and time limits, insert checkpoints for deterministic validation, and keep humans in strategic oversight roles. Treat agents as teammates who propose actions rather than as autonomous operators.
Practical responses: verification, orchestration, and grounding
Teams are converging on three pragmatic approaches:
- Verification layers: Use formal methods or secondary models to check critical outputs. This is especially tractable in domains like code or math where assertions can be verified.
- Orchestrated workflows: Keep control flow in code and let the LLM handle interpretation and synthesis. Graph-based execution engines and well-defined tool interfaces reduce ambiguity.
- Better grounding: Connect agents to authoritative data sources so they cite and check facts rather than invent them. Google’s work on integrating deep, grounded search into document and inbox contexts is an example of this trend toward stronger grounding in productivity flows.
None of these is a silver bullet. Verification can be expensive and limited in scope; orchestration reduces agent flexibility; grounding depends on trustworthy data. But combined, they shrink the gap between demo and durable service.
AI agents therefore sit at an awkward crossroads: mathematically constrained, economically irresistible, and engineering-wise solvable only with trade-offs. The conversation has shifted from “Can agents do everything?” to “Which tasks should we let them do, and how will we make failure visible and affordable?”
If you’re designing or buying agents, pay attention to operational signals—cost per task, transparent action logs, clear exit criteria, and enforced validation points. Those markers separate novelty from production.
The argument that agents are “doomed” is useful because it forces realism. But it’s not a death sentence. It’s a design brief. The question for product teams and regulators is not whether agents will fail sometimes—they will—but whether architectures and policies can limit the harm when they do. The math sets the boundary; engineering decides how close to the edge we walk.