AI Observability

In one line: Production AI features need their own observability layer — prompts, responses, costs, latency, errors, and quality metrics — because the underlying model is non-deterministic and silently changes over time.

In plain English

With normal code, "did it work?" is a yes-or-no question. With AI, the response was probably correct enough but might be slightly wrong in a way no exception will catch. Observability for AI means logging every prompt and response, tracking cost and latency per call, and running eval datasets continuously — because that's the only way to notice your AI silently getting worse.

Production AI requires its own observability layer.

What to track

• Every prompt and response. For debugging, eval, and audit.
• Cost per call. Token counts in and out, model used, cost.
• Latency. Time to first token, total time.
• Errors. Rate limit hits, timeouts, malformed responses.
• Quality metrics. User feedback (thumbs up/down), task success rates.
• Drift. Are responses getting worse over time?

Tools

• Langfuse — Open-source, comprehensive AI observability.
• Helicone — Drop-in proxy that adds observability.
• LangSmith — LangChain's own.
• Braintrust — Eval-focused.
• OpenLLMetry — OpenTelemetry-based.

Evaluation

Beyond observability: how do you know your AI feature is good?

Eval datasets. A set of test inputs + expected outputs. Run regularly.

LLM-as-judge. Use a stronger model to evaluate the output of a weaker one.

Human review. Sample a percentage of production interactions for human grading.

A/B testing. Compare two prompt versions in production; measure user behavior.

Evaluation is genuinely hard for open-ended generation. Quality is often subjective; metrics like BLEU/ROUGE don't capture what matters. Most teams end up with custom human-graded rubrics.

Worked example: catching silent quality drift

A team ships a customer-support bot. Over two months, response quality slowly degrades but no errors are thrown — they only find out when customer complaints rise.

After adding proper observability:

1. Logged every prompt and response to Langfuse.
2. Built an eval set of 50 representative customer questions, with human-graded "good" answers.
3. Ran the eval set nightly against the current prompt + model.
4. Set up an alert for any drop in eval score > 5%.

Two weeks later, the alert fires: the provider had silently updated the model. The team's eval surfaces a measurable regression on tickets about refunds. They pin the previous model version, tune the prompt, and the eval score recovers — before customers complain.

Without the observability + eval setup, they'd have learned about the regression from a wave of support tickets two weeks later.

Highlight: evals are tests for non-deterministic code

Treat evals like unit tests for AI features:

• Version-controlled in the repo.
• Run on every prompt change in CI.
• Block merges on regressions beyond a threshold.
• Owned by the team that ships the feature — not "the ML team."

A team without evals can't safely change a prompt. You can't see whether you made things better or worse. An eval set of even 30–50 examples beats no eval set by a huge margin.

Common mistakes

Where people commonly trip up

• Logging only the final response. When a user complains "the bot said something wrong," you also need the system prompt, the retrieved chunks, the tool calls, and the model + version that ran. Log the whole envelope — without it, every bug report is unreproducible.
• Storing prompts and responses with PII in plaintext forever. Your observability store is now a privacy liability and a prime exfiltration target. Redact PII at the logging boundary, set a retention window, and treat the trace store with the same access controls as your production DB.
• Running evals once at launch and never again. Provider models silently update, your prompt changes, your data shifts — a one-shot eval at launch tells you nothing six weeks later. Wire evals into CI on every prompt change and run a scheduled regression nightly.
• Eval set hand-picked by the prompt author. If the same person wrote the prompt and the eval set, the eval scores the prompt's blind spots out of existence. Build the eval set from real production traffic (sampled and labeled by someone else) and refresh it as patterns shift.
• LLM-as-judge grading itself. Using the same model under test to also grade the outputs leaks the model's biases into the score. Grade with a stronger, different model — and spot-check the judge against human ratings periodically to make sure it's still calibrated.

Page checkpoint

Checkpoint Quiz

Did AI observability stick?

Required

What's next

→ Continue to Costs and Optimization — the techniques that keep LLM bills from spiraling.