Structured output and tool calling
The moment a model's answer feeds into code instead of a human, prose becomes a liability.
Your pipeline expects {"category": "refund"} and at 2am it receives "Sure! Here is the
JSON you asked for:" followed by the JSON, followed by a friendly note about it. The
JSON.parse call throws, and a queue backs up behind it.
This chapter covers the two features that turn a chatty model into a component: output with a guaranteed shape, and tool calling, which is how a model that can only write text ends up checking an order or issuing a refund.
A support bot 'checks your order status' by calling a tool. Which program actually connects to the orders database and runs the query: the model, or something else? Who decides whether it is allowed?
Three ways to get JSON#
Ask for it in the prompt. "Reply only with JSON in this shape." It works most of the time. Most of the time is not good enough for a pipeline that runs a million times a day.
JSON mode. A flag on the request that guarantees the output parses as JSON. It does not
guarantee the right keys. You can get perfectly valid JSON that has categry instead of
category.
Schema constrained output. You send a JSON Schema and the provider restricts which tokens the model is allowed to produce at each step, so the output cannot leave the schema. Many APIs call this structured outputs. It is the one to use whenever code consumes the answer.
{
"type": "object",
"properties": {
"category": { "enum": ["refund", "late", "wrong_item", "account", "other"] },
"order_id": { "type": ["string", "null"] },
"summary": { "type": "string", "maxLength": 280 }
},
"required": ["category", "order_id", "summary"]
}Note the enum. A closed list of values is the single most useful thing in a schema,
because it turns "the model wrote something" into "the model picked one of five things my
code already handles".
A schema guarantees structure, not truth. {"category": "refund", "order_id": "44812"}
passes every check even if the customer wrote 44821. Validate meaning in code where you
can (does this order exist, does it belong to this user), and when validation fails, send
the error back to the model once and let it correct itself. One retry fixes most
failures. Five retries means the prompt is wrong.
How tool calling works#
A model cannot run code, open a socket or read a database. It can only produce tokens. Tool calling is an agreement about which tokens mean "please run this for me".
You describe your tools in the request: a name, a plain English description, and a JSON Schema for the arguments. When the model decides a tool would help, it stops writing prose and returns a structured tool call instead. Your code runs the tool, sends the result back, and the model carries on with that result in its context. Watch one request go round.
Every tool call is a request from text that anyone can influence: the customer, a document you retrieved, a web page. So treat tool arguments exactly like user input. Validate them. Check permissions against the real user, never against something the model claims. And for anything that moves money or deletes data, put a confirmation step in front of it.
What makes a tool easy to use correctly#
The model chooses tools by reading your descriptions, so the descriptions are part of your
prompt and deserve the same care. A tool called query with the description "queries data"
will be called at the wrong times with the wrong arguments. A tool called
get_order_status described as "Look up the delivery status and ETA of one order by its
order id. Use when the customer asks where their order is." mostly will not.
Keep each tool narrow, give arguments obvious names and closed lists where possible, and return results the model can read: short, specific, and with a clear error message when something fails. There is a full chapter on this: Tool design and MCP.
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Prompt for JSON | Works with any model and any provider. | Fails some small fraction of the time, which at scale is a steady stream of errors. | Prototypes, and models that offer nothing better. |
| Schema constrained output | The output always matches the schema, so parsing never fails. | Some schema features are not supported, and values can still be wrong. | Anything a program consumes. The default for production. |
| Tool calling | The model can fetch real facts and take actions mid answer. | Every tool is an attack surface, and every call is another model round trip. | When the answer depends on live data or the task needs an action. |
Checkpoint#
1. You switched on JSON mode, but sometimes the output is missing the "category" key. Why?
2. In tool calling, who executes get_order_status?
3. A model can call issue_refund(order_id, amount). What is the most important safeguard?
- JSON mode guarantees parsing. A schema guarantees shape. Neither guarantees truth.
- An enum in the schema turns free text into one of the cases your code already handles.
- The model requests a tool call. Your code runs it, after validating and checking permission.
- Retry once with the validation error. If it needs five, fix the prompt.
When code consumes a model's output, I use schema constrained output so the shape is guaranteed, with enums wherever there is a closed set of answers, and I still validate meaning in code and retry once with the error. Tool calling works by describing tools with a name, a description and an argument schema. The model returns a structured call instead of text, my code validates the arguments, checks permission against the real user, runs it, and sends the result back so the model can continue. The model never touches the system directly, so anything that moves money or deletes data gets limits in code and a confirmation step.