The demo works perfectly. The agent receives a task, calls three tools, and produces exactly the right answer. You ship it. In production, with real inputs and real APIs, it fails 30% of the time. Here's why, and it's not the model's fault.
LLM agents have a reliability problem that is structural, not incidental. It comes from the combination of multi-step execution, external dependencies, and the inherent non-determinism of language model outputs. You need to understand the structure to engineer your way out of it.
Why demos work and production doesn't
Demos use clean, controlled inputs. The task is well-scoped. Tool responses are predictable, usually because you've hardcoded them or chosen a demo scenario that avoids the cases where tools behave unexpectedly. The workflow fits neatly into a handful of steps. Nothing goes wrong because the demo was designed around the paths where nothing goes wrong.
Production has malformed inputs. APIs return 429s because you've hit a rate limit, or 500s because someone deployed a bad version, or responses with schema changes because the vendor updated their API without telling you. Users phrase requests ambiguously. Multi-step tasks accumulate errors: a wrong decision in step 3 derails steps 4 through 10. Edge cases you didn't anticipate turn out to be common in real traffic.
Demos are optimistic paths. Production is the full distribution. The gap between them isn't a failure of imagination during demo design. You simply can't enumerate all the ways things go wrong until they start going wrong.
The compounding reliability problem
The math of multi-step agents is uncomfortable. Take a task that requires ten sequential steps. If each step succeeds 95% of the time, which is a reasonable assumption for a well-prompted model on a well-defined step, the probability that all ten steps succeed is 0.95 raised to the power of 10, which is about 0.60. A 60% end-to-end success rate on a task that works 95% of the time per step.
To achieve 95% end-to-end success on a ten-step task, you need each step to succeed approximately 99.5% of the time. Going from 95% per step to 99.5% per step sounds like a small improvement. In practice, it's the difference between a system that's barely usable and one that's production-ready.
Reliability engineering for agents is multiplicative, not additive. A small improvement in per-step reliability has outsized impact on end-to-end success. Conversely, a single poorly-defined step that fails 20% of the time will drag down the entire pipeline regardless of how good the other steps are.
Tool call failures
Agents interact with the world through tool calls: API requests, database queries, file operations. Tools fail. This is not an edge case; it's routine. APIs return errors. Rate limits kick in. Responses don't match the schema the agent was expecting because the upstream service changed something.
Agents that don't handle these failures gracefully have a few default behaviors, all bad. They loop indefinitely, retrying the same failing call until they hit a timeout (consuming context window the whole time). They hallucinate: deciding that the tool probably returned something reasonable and inventing what that something was. Or they abort with an unhelpful error that exposes the implementation detail to the user without explaining what went wrong or how to fix it.
Good tool definitions include explicit error states. A tool definition that can return "rate_limited: true, retry_after: 60" gives the agent something to reason about. A tool that returns a structured error with a message and an error code gives the agent what it needs to decide: retry, abort, or take an alternative path. A tool that throws an unstructured exception gives the agent nothing to work with.
Design failure handling into the agent's instructions from the start. Specify what can go wrong on each step and how the agent should respond to each case.
Context management
Long-running agents have a context window problem. Language models have a fixed context window, a maximum amount of text they can attend to at once. As a multi-step agent accumulates tool call results, intermediate reasoning, and prior steps, it fills that window. When the window fills, the model has to make tradeoffs: truncate the oldest content, compress it, or fail.
The failure mode looks like this: the agent establishes constraints or state early in the task. "The output should be in Spanish." "Do not modify files outside the specified directory." After enough steps have elapsed, those constraints fall outside the effective context window. The agent starts making decisions that contradict the original instructions, not because it's ignoring them, but because it can no longer see them.
There are a few mitigation options, each with costs. Structured summarization of completed steps compresses earlier history while preserving the key state the agent needs, at the cost of possibly losing detail. External memory stores (writing intermediate state to a database the agent can query) unbundle state from context, at the cost of adding another system dependency. Sub-agent delegation (handing off discrete sub-tasks to agents with fresh context) keeps individual context windows short, at the cost of coordination overhead and the risk of losing coherence between sub-tasks.
None of these is a clean solution. Picking the right one depends on the task structure. Context management is a design problem, not something that resolves itself.
Non-determinism
Language models are non-deterministic. The same input can produce different outputs on different runs. This is inherent to the sampling process: the model draws from a probability distribution over possible next tokens rather than computing a single deterministic output. For creative tasks, this is a feature. For workflows where you need consistent, repeatable behavior, it's a problem.
Temperature is the main lever. Lower temperature makes the model more deterministic by concentrating probability mass on the most likely tokens. Temperature zero produces near-deterministic outputs (though not perfectly deterministic due to floating-point parallelism). This helps, but it doesn't eliminate variance.
Structured output formats constrain the model's decision space. If the model must respond in JSON matching a specific schema, the space of possible outputs is far smaller than if it's free to respond in any format. Fewer choices means less variance. Tool use with well-defined signatures has a similar effect: the model must produce a call that matches a specific function signature, which rules out a large fraction of possible failure modes.
For every step in your agent that needs to be deterministic, use structured outputs, lower temperature, and explicit format constraints. Leave flexibility for the steps where variation is acceptable or desirable.
Evaluation for agents
Agent evaluation is harder than single-call evaluation because you're measuring an entire trace of decisions, tool calls, and state transitions, not a single output.
The metrics that matter: task completion rate (did the agent achieve the stated goal?), step efficiency (did it do it in a reasonable number of steps, or did it take 40 steps for a 5-step task?), and tool call accuracy (did it call tools with valid arguments, in valid sequences?). Each is independently informative. An agent that completes tasks but takes ten times as many steps as necessary is expensive and probably brittle. An agent with high task completion on simple inputs but tool call errors on complex ones has a reliability gap in the exact scenarios where reliability matters most.
Trace analysis is the debugging primitive for agent failures. A failed agent run needs to be reproducible and inspectable: every step, every tool call, every model response, in order. That's the same requirement as distributed system debugging. You need a complete record of what happened to understand why it went wrong. Tools like LangSmith or generic distributed tracing systems (Honeycomb, Jaeger) provide this, with different tradeoffs around LLM-specific features versus general observability capabilities.
The model is usually not the bottleneck
When an agent fails, it's tempting to look at the model as the problem: try a smarter model, adjust the prompt, add chain-of-thought. Sometimes that's right. More often, the failure is in the system around the model: tool definitions that don't expose error states, missing retry logic, context management that wasn't designed for long runs, test coverage that only covered the happy path.
An LLM agent is a distributed system where the orchestrator happens to be a language model. The reliability engineering is the same as for any other distributed system: assume failures happen, handle them explicitly, instrument everything, measure what's actually failing, and improve iteratively. The model is one component of that system, capable and flexible, but it cannot compensate for an unreliable system around it any better than a good database can compensate for a network that drops packets.
The demo worked because the demo didn't test the system. Production does.
Back to writing