Multi-agent orchestration patterns for production systems

You have multiple agents. One handles customer support, another does data retrieval, a third processes payments. They need to coordinate without becoming a tangled mess of callbacks. You also need to know what happened when something breaks at 3am.

Most engineers start by treating multi-agent systems like they're building a microservice architecture. That's partially right and partially wrong. The difference is that agents are stateful, their outputs are probabilistic, and failure modes cascade differently.

The core problem

Single-agent systems are straightforward: prompt → LLM → tool use → response. Multi-agent systems introduce questions that don't have clean answers:

• Who decides what work goes where?
• What happens when agent A's output doesn't match agent B's expectations?
• How do you prevent infinite loops or contradictory instructions?
• Where does the user sit in this flow?

The pattern you choose shapes your debugging experience, your latency profile, and how hard it is to add a new agent later.

Three patterns and their tradeoffs

Pattern 1: Centralized orchestrator

One agent (or service) acts as a router. It receives requests, decides which specialized agents to invoke, aggregates results, and responds to the user.

How it works: The orchestrator is a planner. It has visibility into all available agents, their capabilities, and the current task. It breaks work into steps, calls agents sequentially or in parallel, and handles the final response.

Advantages:

• Single point of observability. You can log every decision.
• Easy to add retry logic, timeouts, and fallbacks at one layer.
• Clear semantics: the orchestrator owns the flow.

Disadvantages:

• The orchestrator becomes a bottleneck and a single point of failure.
• If the orchestrator hallucinates about which agent to use, the whole request fails.
• Latency is additive: each agent call waits for the previous one (unless you parallelize, which adds complexity).
• The orchestrator needs detailed knowledge of every agent's interface, which couples them together.

When to use: Small number of agents (2-5), well-defined request types, latency isn't critical, and you want strong guarantees about correctness.

Pattern 2: Choreography

Agents communicate directly. Each agent knows which agents it might need to call next. No central controller.

How it works: Agent A receives the request, does its work, then decides if it needs Agent B. If it does, it calls B directly. B does the same. Responses flow back up the chain.

Advantages:

• No bottleneck. Agents can call each other in parallel.
• Scales to many agents without a central coordinator.
• Each agent is independent; adding a new agent doesn't require changing the orchestrator.

Disadvantages:

• Observability is a nightmare. You need distributed tracing to understand what happened.
• Agents need to know about each other's interfaces. That's coupling.
• Failure modes are hard to reason about. If Agent B fails, does Agent A retry? How many times? Who decides?
• Debugging a bug in the choreography means reading agent code across multiple systems.
• Easy to create circular dependencies or infinite loops.

When to use: Many agents, they're relatively autonomous, you have strong observability infrastructure (structured logging, tracing), and you can tolerate eventual consistency.

Pattern 3: Hybrid (hierarchical with delegation)

Most production systems end up here. You have a top-level orchestrator that handles high-level routing, but agents below it can delegate to each other.

How it works: The top orchestrator has a small, well-defined set of responsibilities. It routes to specialist agents. Each specialist agent has its own sub-agents it knows about. The specialist handles coordination within its domain.

Advantages:

• Observability at two levels. You see the top-level flow clearly, and specialists own their own sub-flows.
• Scalability. You can add agents at the specialist level without touching the orchestrator.
• Failure isolation. If a specialist's sub-agent fails, the specialist can handle it (retry, fallback, etc.) without bubbling up.
• Reasonable latency. Specialists can parallelize their sub-agents.

Disadvantages:

• More complex to reason about. You need to understand two levels of routing.
• Specialists need to be well-scoped. If they're too broad, they become mini-orchestrators.
• Debugging still requires understanding multiple agents, but it's localized.

When to use: Most production systems. You have multiple domains of work, enough agents that a single orchestrator would be a bottleneck, but not so many that you need full choreography.

Practical implementation details

State management

Multi-agent systems need to carry state through the flow. Each agent sees the conversation history, relevant context, and decisions made so far.

Use a structured format:

{
  "request_id": "uuid",
  "user_input": "...",
  "context": {
    "customer_id": "...",
    "session_data": {...}
  },
  "decisions_made": [
    {"agent": "router", "decision": "route to support", "timestamp": "..."},
    {"agent": "support", "decision": "retrieve FAQ", "timestamp": "..."}
  ],
  "current_state": "awaiting_specialist_response"
}

This becomes your audit trail. When something goes wrong, you can replay it.

Failure handling

Assume agents fail. Not sometimes. Always.

For each agent call, define:

• Timeout: How long before you give up. 5-30 seconds depending on the agent.
• Retry logic: Exponential backoff, max 2-3 retries. Don't retry on semantic failures (agent returned a valid response, just a bad one).
• Fallback: What happens if the agent fails completely. Return a default? Escalate to a human? Fail the whole request?

async def call_agent(agent, input_data, timeout=10, max_retries=2):
    for attempt in range(max_retries):
        try:
            result = await asyncio.wait_for(
                agent.invoke(input_data),
                timeout=timeout
            )
            return result
        except asyncio.TimeoutError:
            if attempt == max_retries - 1:
                return {"error": "timeout", "fallback": default_response}
            await asyncio.sleep(2 ** attempt)
        except Exception as e:
            log_error(e)
            if attempt == max_retries - 1:
                return {"error": str(e), "fallback": default_response}

When you first deploy this pattern, expect your fallback to trigger more often than you'd like. That's normal. Monitor which agents are timing out and which are failing semantically. Use that data to tune your timeouts and retry counts. After a week, you'll have real numbers on what works for your specific agents and infrastructure.

Observability

Log every agent invocation with:

• Request ID (trace it end-to-end)
• Agent name and version
• Input (truncated if large)
• Output (truncated)
• Latency
• Success/failure
• Tokens used (if billing matters)

This is non-negotiable. You will debug production issues at 3am. Make it possible.

When you're setting up logging, resist the urge to log everything to a single stream. Use structured logging with fields that let you filter. You want to be able to query "show me all failures for the support agent in the last hour" without drowning in noise. In the first week, you'll discover which log fields actually matter. Add more as you learn what breaks.

Preventing infinite loops

If agents can call each other, set a depth limit. Track the call stack. If you hit a nesting limit, stop and return an error.

def invoke_with_depth_check(agent, input_data, depth=0, max_depth=10):
    if depth > max_depth:
        return {"error": "max depth exceeded"}
    input_data["_depth"] = depth + 1
    return agent.invoke(input_data)

This is crude but effective. In practice, you'll rarely hit this limit unless you have a bug. But when you do, you'll be grateful it exists. The alternative is a runaway agent that consumes your entire token budget in seconds.

Choosing your pattern

graph TD
    A["How many agents?"] -->|2-5| B["Centralized Orchestrator"]
    A -->|5-20| C["Hybrid Hierarchical"]
    A -->|20+| D["Choreography"]
    B --> E["Log everything at one point"]
    C --> F["Two levels of observability"]
    D --> G["Distributed tracing required"]
    E --> H["Start here"]
    F --> H
    G --> I["Only if you have the infrastructure"]

Start with centralized orchestration if you have fewer than five agents and request patterns are predictable. It's easier to debug and reason about. You'll spend your first week tuning timeouts and watching logs. That's the right place to spend effort.

Move to hybrid orchestration as you add agents. Define clear domains and let specialists own their sub-agents. The transition happens naturally: you'll notice your orchestrator getting complicated, and splitting it into domains will feel like relief, not complexity.

Only adopt full choreography if you have many agents, they're genuinely autonomous, and you have the infrastructure to trace distributed flows. This means structured logging, distributed tracing tools, and a team that knows how to read them. If you're not sure you have that, you don't.

The worst mistake is over-engineering early. A simple orchestrator that works is better than a clever choreography system that you can't debug.