How to add tool use to an LLM

You want your LLM to call functions. Not hypothetically. You're building something that needs to query a database, hit an API, or run code, and you want the model to decide when and how to do it.

The basic flow is straightforward: you describe your tools to the model, it decides to use one, you parse its response, execute the tool, and feed the result back. But the details matter. Parsing fails. Models hallucinate tool names. Execution errors cascade. You need to know where the failure modes live and how to defend against them.

The shape of tool use

Here's what actually happens:

1. You send the LLM a prompt plus a list of available tools (their names, descriptions, parameters)
2. The model outputs a response that includes a tool call (format depends on your API—OpenAI uses tool_calls, Anthropic uses <tool_use> blocks, open models vary)
3. You parse that output
4. You validate the parameters
5. You execute the tool
6. You send the result back to the model in a specific format
7. The model continues reasoning with that information

This is a loop. The model might call multiple tools in sequence, or call one tool, get a result, and decide it needs to call another. You're managing state across multiple round-trips.

sequenceDiagram
    participant User
    participant LLM
    participant Agent
    participant Tool

    User->>Agent: Query
    Agent->>LLM: Prompt + tool definitions
    LLM-->>Agent: Tool call (name, params)
    Agent->>Agent: Parse & validate
    Agent->>Tool: Execute with params
    Tool-->>Agent: Result or error
    Agent->>LLM: Result (in tool format)
    LLM-->>Agent: Next action or response
    Agent-->>User: Final response

What you need to decide first

Tool definition format. OpenAI's tools parameter expects JSON Schema. Anthropic's tool_use expects XML with specific fields. Open models via Ollama or vLLM might not support tool calling at all—you'd need to use prompt engineering or fine-tuning. Pick your model first, or accept that you'll need adapter layers.

Synchronous or asynchronous execution. Most production setups execute tools synchronously—you call the tool, wait for the result, feed it back. This is simpler to reason about and easier to debug. Async matters if your tools are slow (API calls with high latency) and you want to parallelize multiple tool calls. For most cases, start synchronous.

Single-turn or multi-turn. Can the model call multiple tools before you interrupt? Most frameworks support this. It's more powerful but harder to control. You need better error handling and clearer stopping conditions.

Who owns retries. If a tool call fails (bad parameters, execution error), do you retry the tool, or do you return the error to the model and let it retry? The latter is more flexible but burns tokens. The former is faster but you need good error messages so the model learns what went wrong.

Parsing: where it breaks first

The model's output format is not guaranteed to be valid. Even with structured outputs (which some APIs now offer), you need defensive parsing.

# Pseudo-code: what you're actually doing
response = client.chat.completions.create(
    model="gpt-4-turbo",
    messages=messages,
    tools=tool_definitions,
)

# response.choices[0].message might have tool_calls
# or it might not. It might have text instead.
# or both.

if response.choices[0].message.tool_calls:
    for tool_call in response.choices[0].message.tool_calls:
        tool_name = tool_call.function.name
        tool_args = json.loads(tool_call.function.arguments)
        # Now validate tool_name is in your registry
        # and tool_args match your schema
else:
    # Model decided not to use tools, or hit a limit
    return response.choices[0].message.content

Common failure modes:

• Model outputs a tool name that doesn't exist. You need a registry. Check membership before executing.
• Model outputs invalid JSON for parameters. Wrap the parse in try-except. If it fails, you can either return an error to the model or retry with a corrected prompt.
• Model outputs parameters of the wrong type. JSON Schema validation catches some of this, but not all. Validate at execution time.
• Model gets confused and outputs multiple tool calls when you only expected one, or outputs a tool call and text in the same response. Handle both gracefully.

If you're using an API that supports structured outputs (OpenAI's response_format with JSON Schema, Claude 3.5's native tool use), use it. It reduces parsing failures significantly. But don't assume it's bulletproof.

Parameter validation and execution

Once you've parsed the tool call, validate before you execute.

def execute_tool(tool_name: str, tool_args: dict) -> str:
    if tool_name not in TOOLS:
        return f"Error: Unknown tool '{tool_name}'"
    
    tool_def = TOOLS[tool_name]
    schema = tool_def['schema']
    
    # Validate against schema
    try:
        validate(tool_args, schema)
    except ValidationError as e:
        return f"Error: Invalid parameters: {e}"
    
    # Execute
    try:
        result = tool_def['fn'](**tool_args)
        return str(result)  # Convert to string for LLM
    except Exception as e:
        return f"Error executing {tool_name}: {str(e)}"

Key decisions:

• Error messages matter. When a tool fails or parameters are invalid, the error message goes back to the model. Make it specific enough for the model to correct itself. "Invalid parameter" is useless. "Expected 'limit' to be an integer between 1 and 100, got 'abc'" is actionable.
• Timeouts. Set them. A tool that hangs will hang your entire agent. Use signal (Unix) or timeout libraries, or wrap execution in a thread with a timeout.
• Return type. Convert results to strings. The model reads text. If you return a Python object or raw JSON, it gets serialized unpredictably. Explicit is better.

The loop: feeding results back

After execution, you need to return the result to the model in a format it understands. This is API-specific.

For OpenAI, you append a tool_result message to the conversation:

messages.append({
    "role": "assistant",
    "content": "",
    "tool_calls": [tool_call]
})

messages.append({
    "role": "user",
    "content": [{
        "type": "tool_result",
        "tool_use_id": tool_call.id,
        "content": result_string
    }]
})

# Continue the conversation
response = client.chat.completions.create(
    model="gpt-4-turbo",
    messages=messages,
    tools=tool_definitions,
)

For Anthropic, it's similar but the format differs slightly. For open models, you might need to construct the message manually.

Stop conditions: You need a way to break the loop. Common approaches:

• Model stops outputting tool calls (returns text instead)
• You hit a maximum number of iterations (default 10 or so)
• You hit a token budget
• A tool returns a terminal error

Without these, you can loop forever or burn through your budget.

What you actually need to build

A minimal production setup:

1. Tool registry. A dict or database mapping tool names to definitions and implementations. Make it easy to add tools without changing core logic.
2. Validation layer. JSON Schema validation, type checking, range checking. Fail fast and clearly.
3. Execution wrapper. Handles timeouts, error catching, result serialization.
4. Loop controller. Manages the conversation state, stops conditions, token budgeting.
5. Observability. Log every tool call, every parameter, every result. When it breaks, you need to see what happened.

Start simple. A single synchronous loop with error handling and logging will get you most of the way. Add complexity (async, multi-tool parallelization, caching) only when you hit concrete problems.

Common tradeoffs

Prompt engineering vs. structured outputs: Structured outputs are more reliable but only available on newer models. If you're using an older or open model, you'll write more prompt engineering and handle more parsing failures.

Flexibility vs. safety: The more tools you expose, the more ways the model can go wrong. Constrain your tool set to what you actually need. Better to have five tools the model uses correctly than fifty it hallucinates.

Token cost vs. quality: Detailed tool descriptions and examples help the model use tools correctly, but they consume tokens. Find the balance for your use case.

The honest version: tool use is powerful but not magic. You're still fighting the model's tendency to hallucinate, misunderstand parameters, and take wrong turns. Good error messages, defensive parsing, and tight observability are not optional.