Adding guardrails to AI agents in production

Your agent is live. It's calling tools, making decisions, and sometimes it does something you didn't intend. Maybe it calls the same tool fifty times in a loop. Maybe it hallucinates a parameter and passes garbage to your database. Maybe it spends $200 on API calls for a $5 user request.

Guardrails aren't a single feature. They're a layered system. You need them at multiple points: before tool execution, during the agent loop, at the model level, and in your infrastructure. Each layer catches different failure modes.

The problem with guardrails

The core tension: you want to prevent bad behavior without crippling the agent's capabilities. A guardrail that's too strict becomes a filter that blocks legitimate requests. Too loose and you're back where you started.

Most engineers reach for prompt injection first—adding instructions to the system prompt. This doesn't work. It's the first thing to fail under adversarial input or when the agent gets confused. You need structural, not instructional, constraints.

The other common mistake is treating guardrails as a binary gate. You either let the request through or you reject it. In practice, you want to degrade gracefully: warn, retry with constraints, fall back to a safer mode, or ask the user before proceeding.

Layer 1: Tool-level constraints

Start here. This is where you have the most control and the most information.

Define explicit limits on each tool:

• Rate limits (calls per minute, per session)
• Parameter validation (ranges, allowed values, string length)
• Cost budgets (for external API calls)
• Preconditions (what state must be true before calling)

Example: a database query tool shouldn't accept queries that would scan more than 100k rows without explicit user confirmation. A code execution tool shouldn't write to certain directories.

class ToolConstraints:
    def __init__(self, name: str):
        self.name = name
        self.call_count = 0
        self.max_calls_per_session = 20
        self.cost_so_far = 0.0
        self.cost_limit = 5.0
        self.last_call_time = None
        self.min_interval_seconds = 0.5
    
    def check_before_execution(self, params: dict) -> tuple[bool, str]:
        if self.call_count >= self.max_calls_per_session:
            return False, f"Tool {self.name} called {self.call_count} times"
        
        if self.cost_so_far > self.cost_limit:
            return False, f"Cost limit exceeded: ${self.cost_so_far}"
        
        if self.last_call_time:
            elapsed = time.time() - self.last_call_time
            if elapsed < self.min_interval_seconds:
                return False, "Rate limit: wait before retry"
        
        return True, ""

These are not suggestions. They're hard stops. The agent cannot proceed without satisfying them.

Validate parameters strictly. If your tool expects an integer ID between 1 and 10000, check that before passing it to the underlying system. If it expects a date, parse and validate. This catches hallucinations early.

Layer 2: Agent loop constraints

The agent itself needs guardrails. Common failure modes:

Infinite loops: Agent keeps calling the same tool or cycling between tools. Set a hard limit on total steps (typically 10-30 for a single request) and on consecutive calls to the same tool (typically 2-3). When the limit is hit, stop and return what you have.

Token explosion: The agent's context window grows as it accumulates tool outputs. Implement a maximum context size. When you're near it, summarize old steps or truncate them. Don't let the agent see its full history if it's too large.

Hallucinated tool calls: The agent tries to call a tool that doesn't exist or with parameters that don't match the schema. Your framework should reject this immediately. Don't let it go to the tool layer. Respond with "that tool doesn't exist, here are the available tools" and let the agent retry.

def run_agent_step(agent_state, max_steps=20, max_context_tokens=8000):
    step_count = 0
    consecutive_same_tool = 0
    last_tool_name = None
    
    while step_count < max_steps:
        if len(agent_state.context_tokens) > max_context_tokens:
            agent_state.summarize_old_steps()
        
        tool_call = agent_state.next_tool_call()
        
        if tool_call.name == last_tool_name:
            consecutive_same_tool += 1
            if consecutive_same_tool > 2:
                return "Max retries on same tool"
        else:
            consecutive_same_tool = 0
        
        last_tool_name = tool_call.name
        result = execute_tool(tool_call)
        agent_state.add_observation(result)
        step_count += 1
    
    return "Max steps reached"

Layer 3: Model-level constraints

Some models support structured output or function calling with schemas. Use this. It reduces hallucinations significantly.

If your model supports it, define tool schemas precisely. Include descriptions, required vs. optional parameters, and valid value ranges. The model's function-calling layer will validate before it even generates the call.

For models that don't support this, you need to parse the output yourself and validate against your schema. Reject malformed calls and ask the agent to retry.

Consider using a smaller, cheaper model for routing decisions and a larger model only when needed. A guardrail can be "this request is too risky for Claude 3.5 Sonnet, route to GPT-4o instead" or "this is straightforward, use Claude 3.5 Haiku to save cost."

Layer 4: Semantic guardrails

This is harder and more fragile, but sometimes necessary.

Use a separate classifier to detect when the agent is about to do something you don't want:

• Attempting to access sensitive data it shouldn't
• Making a decision that contradicts business rules
• Asking for information that would violate privacy

def should_block_request(agent_intent: str, user_context: dict) -> bool:
    # Example: don't let the agent delete data for users
    # who haven't explicitly opted in
    if "delete" in agent_intent.lower():
        if not user_context.get("deletion_confirmed"):
            return True
    return False

This is a classifier, not a prompt. You can use a small model or even a keyword/regex approach depending on your risk tolerance. The point is to make it independent of the main agent model.

The tradeoff: semantic guardrails catch more sophisticated failures, but they have false positive rates. You need to monitor them and tune them over time.

Observability and feedback loops

Guardrails only work if you can see when they're triggered.

Log every guardrail event:

• Which tool was blocked and why
• What the agent was trying to do
• Whether the user retried or abandoned

{
  "timestamp": "2024-01-15T10:23:45Z",
  "user_id": "user_123",
  "guardrail_type": "tool_rate_limit",
  "tool_name": "database_query",
  "reason": "5 calls in 10 seconds",
  "agent_state": "waiting_for_retry"
}

Set up alerts for:

• Guardrails triggering more than expected
• New patterns of guardrail violations
• Legitimate use cases being blocked repeatedly

When you see a pattern, adjust the guardrail. If the cost limit is too low and users are hitting it legitimately, raise it. If a tool is being called in loops, lower the consecutive call limit or add a precondition.

The architecture

flowchart TD
    A["User Request"] --> B["Agent Starts"]
    B --> C["Generate Tool Call"]
    C --> D{Valid Schema?}
    D -->|No| E["Reject, Ask Retry"]
    E --> C
    D -->|Yes| F["Tool Constraints Check"]
    F --> G{Passes?}
    G -->|No| H["Block, Log Event"]
    H --> I{Retry Allowed?}
    I -->|Yes| C
    I -->|No| J["Return Error"]
    G -->|Yes| K["Execute Tool"]
    K --> L["Add to Context"]
    L --> M{Loop Limit Hit?}
    M -->|Yes| N["Return Result"]
    M -->|No| O{Done?}
    O -->|No| C
    O -->|Yes| N

What you'd actually do

Start with tool-level constraints. They're easy to implement and catch most problems. Add rate limits, parameter validation, and cost tracking. Monitor for a week.

Then add loop constraints. Set step limits and context size limits. These are almost free and prevent runaway behavior.

Add semantic guardrails only if you have a specific risk you're worried about. Don't add them preemptively.

Test your guardrails by deliberately triggering them. Does the agent recover? Does it retry with different parameters or does it get stuck? Does the user see a clear error message?

Expect to tune guardrails over time. The first settings won't be right. You'll discover edge cases where legitimate requests get blocked or where the agent finds a way around your constraints. That's normal. Keep logs, monitor, adjust.

The goal isn't to prevent all bad behavior. It's to make bad behavior expensive or obvious enough that you catch it before it becomes a production incident.