LLM Observability in Production
Most teams treat an LLM integration like a function call: send a prompt, get text back, log errors. That works in a demo. In production, the model is your infrastructure — and infrastructure needs the same discipline as any other service.
I spent time deploying and monitoring Claude and Llama workloads on AWS Bedrock. The lesson that stuck: you are not debugging a model, you are debugging a pipeline — retrieval, routing, tool calls, retries, and fallbacks all show up in the same user-facing latency.
Start with one trace per request
Every user-facing completion should become a single trace with child spans:
- Retrieval — embedding lookup, rerank, chunk count
- Router — which model, why (cost cap, latency SLO, capability)
- Generation — time-to-first-token, total tokens, finish reason
- Tools — each invocation as its own span with input hash and outcome
- Post-processing — JSON parse, validation, guardrails
If you cannot reconstruct a bad answer from the trace alone, your observability is decorative.
Metrics that actually matter
Dashboard vanity is easy. These numbers change decisions:
| Metric | Why it matters |
|---|---|
| p95 time-to-first-token | Users feel streaming latency before total completion time |
| Tokens in / out per route | Cost and context-window pressure differ by model |
| Retry rate by provider | Early signal of throttling or regional degradation |
| Tool failure rate | Often dominates tail latency more than the model itself |
| Fallback rate | How often you had to downgrade model or skip a step |
Track cost per successful task, not per request. A retry storm looks cheap until you multiply by three.
Logs are not enough
Structured logs help, but LLM failures are often semantic — the response parsed, validated, and still violated your intent. Capture:
- Prompt template version and variable snapshot (redacted)
- Model ID and inference parameters
- Raw output before parsing, truncated and stored with retention policy
- Evaluator score when you have one — even a lightweight rubric
When someone asks "why did it say that Tuesday?", you need the artifact, not a stack trace.
Alerts worth paging on
Skip alerting on average latency. Page on:
- Error budget burn on successful task completion (not HTTP 200)
- Circuit breaker open on a primary model route
- Sudden spike in empty or truncated outputs (
finish_reason: length) - Cost anomaly vs. trailing 7-day baseline
Everything else can wait for the daily review.
The habit
Run a weekly "bad trace review" — thirty minutes, five worst traces, one fix shipped. Observability for LLMs is less about buying a platform and more about building the reflex to treat every weird answer as a systems problem.
More on the client side of reliability — retries, jitter, circuit breakers — in the Retry & Backoff Playground.