LLM agent architecture patterns for production

You're building an agent. The LLM picks tools, calls them in sequence, gets results, and reasons forward. Sounds simple. Then you ship it and discover that agents are stateful systems with feedback loops, and every loop introduces a place where things can fail silently.

This post covers the architectural patterns that actually work in production, the tradeoffs between them, and what breaks when you get them wrong.

The basic loop and where it falls apart

The simplest agent loop is: prompt → LLM → parse tools → execute → loop back.

User input → LLM (with tools) → Parse tool calls → Execute tools → 
Append results to context → LLM again → User output

This works for simple cases. It breaks when:

• The LLM hallucinates tool calls that don't exist or misunderstands arguments
• A tool fails mid-execution and you have no graceful degradation
• The context window fills up and you've lost the reasoning trail
• You need to enforce that certain tools run before others
• You want to retry a failed tool without re-invoking the LLM

The pattern you need depends on how much structure your domain requires.

Pattern 1: Agentic loop with explicit state

This is the workhorse pattern. You maintain an explicit state machine where the agent is always in one of a few known states: planning, executing, waiting, done.

class AgentState:
    current_step: int
    plan: list[str]  # tool names in intended order
    results: dict[str, Any]
    failures: dict[str, str]
    context_tokens: int

def step(state: AgentState, user_input: str) -> AgentState:
    if state.current_step == 0:
        # Planning phase: ask LLM to decompose
        plan = llm.plan(user_input)
        state.plan = parse_tools(plan)
        state.current_step = 1
    
    elif state.current_step < len(state.plan):
        # Execution phase
        tool_name = state.plan[state.current_step]
        try:
            result = execute_tool(tool_name, state.results)
            state.results[tool_name] = result
        except ToolError as e:
            state.failures[tool_name] = str(e)
        state.current_step += 1
    
    else:
        # Synthesis phase: ask LLM to answer
        state.answer = llm.synthesize(
            user_input, state.results, state.failures
        )
    
    return state

The key insight: the LLM doesn't decide execution order on every turn. You ask it to plan once, then execute deterministically. This gives you:

• Observability: you can see exactly which tool ran when
• Retry semantics: a failed tool can be retried without re-planning
• Bounded cost: planning happens once per query, not per tool
• Debuggability: the execution trace is a simple sequence

The downside: the agent can't adapt its plan based on intermediate results. If the first tool fails, it doesn't pivot. You need to decide whether that's acceptable for your use case.

Pattern 2: Reactive loop with tool-level decision gates

If your domain requires adaptation—"if this search returns nothing, try a different approach"—you need reactive execution. The LLM makes a decision after each tool.

LLM decides next tool → Execute → LLM sees result → 
LLM decides whether to continue or pivot → loop

This is more flexible but more expensive and harder to debug. Every tool execution triggers another LLM call. The LLM can hallucinate differently on each turn. Context bloat accelerates because you're appending results repeatedly.

Use this pattern when:

• Tool outcomes are genuinely unpredictable and require real reasoning
• You have budget for multiple LLM calls per query
• You can afford to lose observability (execution traces become branching trees)

Most teams start here because it feels natural, then migrate to pattern 1 because the costs and failure modes become untenable.

Pattern 3: Hierarchical agents with delegation

For complex domains, nest agents. A high-level agent routes to specialized sub-agents.

flowchart TD
    A["User query"] --> B["Router agent"]
    B --> C["Search agent"]
    B --> D["Analysis agent"]
    B --> E["Planning agent"]
    C --> F["Execute search tools"]
    D --> G["Execute analysis tools"]
    E --> H["Execute planning tools"]
    F --> I["Aggregate results"]
    G --> I
    H --> I
    I --> J["Synthesize answer"]

This works well when you have distinct domains with different tool sets. The router agent is lightweight—it just needs to classify the query. Each sub-agent has a focused tool set and can be optimized independently.

The tradeoff: more moving parts, more places to fail, harder to trace execution across boundaries. You need explicit contracts between agents (what inputs each expects, what outputs it produces).

Handling tool failures and retries

Tools fail. Networks timeout, APIs return errors, tools are called with invalid arguments. You need a strategy.

Immediate retry with exponential backoff: Good for transient failures (network hiccups). Implement at the tool execution layer, not in the agent logic.

Fallback tools: If a search fails, try a different search provider. Declare fallback chains explicitly in your tool registry.

Graceful degradation: If optional tools fail, continue without them. Mark tools as required or optional; fail the agent only if a required tool fails.

LLM-driven retry: Ask the LLM to revise its tool call. "That search returned no results. Try a different query." This works but costs another LLM call and can loop infinitely if the LLM keeps making the same mistake.

In practice, combine these. Retry transient failures automatically. For semantic failures (bad arguments, tool doesn't exist), log and either fail fast or ask the LLM to recover once.

Context and token budgets

Agents blow through tokens. Every loop appends results to context. Every retry or fallback adds more. You need hard limits.

Track tokens explicitly. Maintain a budget per query. When you're 80% through the budget, stop accepting new tools and move to synthesis. When you hit 100%, truncate context aggressively (keep the original query and latest results, drop intermediate reasoning).

def should_continue_loop(state: AgentState, budget: int) -> bool:
    tokens_used = estimate_tokens(state.context)
    tokens_remaining = budget - tokens_used
    # Reserve 20% for synthesis and safety margin
    return tokens_remaining > budget * 0.2

This prevents runaway costs and keeps your system predictable.

Observability and debugging

Production agents are black boxes unless you instrument them. Log:

• Every LLM call: prompt, model, temperature, tokens used, latency
• Every tool call: tool name, arguments, result, latency, error if any
• State transitions: which state the agent moved to and why
• Token usage: running total per query

Use structured logging (JSON, not strings). Include a trace ID so you can follow a single query through retries and sub-agents.

When an agent gives a wrong answer, you want to replay it. Store the full state and context at each step. You should be able to deterministically re-run a query and get the same trace.

Choosing a pattern

Start with pattern 1 (explicit state, deterministic execution) unless you have a specific reason not to. It's simpler, cheaper, and easier to debug.

Move to pattern 2 (reactive loop) only if you've hit a wall where the agent genuinely can't solve problems without mid-execution adaptation.

Use pattern 3 (hierarchical) when your tool set is large enough that routing becomes a bottleneck, or when you have distinct subdomains that benefit from specialization.

The honest truth: most production agents are a hybrid. You'll have a high-level deterministic plan with reactive fallback for specific tools. You'll retry transient failures automatically and ask the LLM to recover from semantic failures once. You'll maintain explicit state and log everything.

Build for observability first. The architecture matters less than being able to see what went wrong.