Structured LLM Outputs, Decoding and Tool Boundaries

Three boundaries to keep separate

Decoding controls how the model chooses tokens. Temperature and sampling can increase variation, while greedy or constrained decoding can make output more repeatable. Structured output defines a machine-readable contract. Tool execution performs an action. None of these makes the others safe automatically.

Use a schema that rejects unknown fields, validates types and limits values. Parse into data, never into executable code. Map an approved action name to a local function, enforce the authenticated user’s permission, validate arguments again and require confirmation for irreversible operations.

Generation still needs a budget

Reserve context space for the output schema, trusted instructions and expected response. Long schemas consume tokens. If the request plus evidence approaches the context limit, reject it or reduce evidence using a documented method. Silent truncation can remove the very rule that constrains the output.

Build a validator and tool dispatcher

Save this as safe_tool_request.py. It simulates model output so the security boundary can be tested without an API key.

python -m pip install jsonschema
python safe_tool_request.py

# safe_tool_request.py
import json
from jsonschema import Draft202012Validator

SCHEMA = {
    "type": "object",
    "additionalProperties": False,
    "required": ["action", "ticket_id", "reason"],
    "properties": {
        "action": {"enum": ["summarize_ticket", "request_review"]},
        "ticket_id": {"type": "integer", "minimum": 1},
        "reason": {"type": "string", "minLength": 5, "maxLength": 120},
    },
}

def summarize_ticket(ticket_id, reason):
    return {"status": "preview", "ticket_id": ticket_id, "reason": reason}

def request_review(ticket_id, reason):
    return {"status": "queued", "ticket_id": ticket_id, "reason": reason}

TOOLS = {
    "summarize_ticket": summarize_ticket,
    "request_review": request_review,
}

def dispatch(raw, allowed_actions):
    data = json.loads(raw)
    Draft202012Validator(SCHEMA).validate(data)
    action = data.pop("action")
    if action not in allowed_actions:
        raise PermissionError("action is not allowed for this user")
    return TOOLS[action](**data)

model_output = json.dumps({
    "action": "request_review", "ticket_id": 42,
    "reason": "Refund policy needs a human decision",
})
print(dispatch(model_output, allowed_actions={"request_review"}))

Expected result

{'status': 'queued', 'ticket_id': 42, 'reason': 'Refund policy needs a human decision'}

Try adding an unknown field, changing ticket_id to text or removing the action from the permission set. Each case should stop before a tool runs. In production, record a request ID and outcome without logging secrets or unnecessary personal data.

Decoding choices by task

A deterministic configuration can repeat a wrong answer. A valid JSON document can contain a false claim. Validate syntax, semantics and authority as separate gates.

Common implementation failures

• Schema drift: version the schema and test old clients.
• Prompt-only permissions: move authorization into application code.
• Tool name injection: use a fixed allowlist rather than importing a supplied function name.
• Retry storm: cap repair attempts and expose a clear failure state.
• Secret leakage: pass only the minimum tool result back to the model.

Test the contract at three layers

Parser tests should reject malformed JSON and extra fields. Semantic tests should reject values that are syntactically valid but impossible for the business process, such as a closed ticket ID. Authorization tests should call every tool with missing, wrong and expired permissions. Add one test proving that text inside a reason field cannot become a tool name or bypass confirmation. Keep the raw model response for debugging only when privacy policy permits it, and redact secrets before any log leaves the request boundary.

Knowledge check

Standards and security references

• JSON Schema 2020-12 core
• OWASP Top 10 for LLM Applications
• NIST Generative AI Profile