The three measurements every LLM production system needs

A production LLM system can survive a lot of imperfection. It can survive intermittent latency spikes, occasional generation failures, and even meaningful drift in cost-per-request. What it can't survive is operating without measurement — the kind of measurement that surfaces drift before customers notice it.

Most teams measure the wrong things. They track latency p99 and monthly token cost; they don't track the things that determine whether the system is actually working. Here are the three we insist on.

Measurement 1: Eval score over time

The score from your evaluation harness, tracked per deployment, with a clear baseline.

What it actually measures. Whether the model is doing what it's supposed to do. Whether prompt changes helped or hurt. Whether the underlying model has drifted (in API contexts where the vendor updates the model).

What it doesn't measure. Anything about whether users are happy. Anything about cost. Anything about latency. It's a quality signal only.

How to track it. Run the eval harness on every deploy. Store the score. Plot over time. Alert when it drops more than 2pp from the trailing 4-week mean.

Why teams skip it. Eval harnesses are expensive to build and maintain. Hand-graded examples take time. Teams launch deployments without one because "we'll add it later." Later doesn't come, and the system runs blind for years.

Common shape. A score on a domain eval (the headline number), a score on a general capability eval (catches regressions), and a score per query category (catches narrow regressions hidden in the aggregate).

We've shipped engagements where the only deliverable that mattered six months later was the eval harness. The model gets replaced; the harness compounds.

Measurement 2: Failure-mode distribution

The categorical breakdown of how the model fails, tracked over time.

What it actually measures. Where the bad outputs cluster. Whether new failure modes are emerging. Whether existing ones are getting worse.

What it doesn't measure. How often the model fails (that's eval score). It measures the shape of the failures.

How to track it. Every failed eval example gets a tag describing the failure type: "hallucinated fact," "wrong format," "refused valid query," "incomplete answer," "tone problem," "factual error caught by RAG." Each new failure during ongoing eval picks from this taxonomy (or extends it).

The output is a stacked bar chart over time. You can see at a glance:

• Are factual errors growing?
• Did the recent prompt change introduce new format failures?
• Is the model refusing more queries than it used to?

Why this matters. Aggregate eval score may stay flat while the kind of failures shifts. A model that hallucinates less but refuses more is a different model than the one you deployed. Detecting the shift early lets you address it before it becomes a complaint pattern.

Implementation. Easy version: an LLM-as-judge categorizes each failure with a fixed taxonomy. Cheap, noisy. Manual version: human reviewer tags 100 failures per week, fed back into the dashboard. More expensive, cleaner signal. Most production setups blend both.

Measurement 3: Distribution shift in inputs

How the queries your system receives are changing over time.

What it actually measures. Whether users are sending different queries than they did three months ago. Whether the model is being tested on inputs the eval harness doesn't cover. Whether the system is drifting outside its design envelope.

What it doesn't measure. Anything about model output. It's an input signal only.

How to track it. Embed every input query. Cluster embeddings into ~50 buckets (k-means or similar). Track the bucket distribution week over week.

When buckets shift — a previously rare cluster suddenly dominates, or a new cluster appears that wasn't in the original eval set — you know your input distribution has changed. The model may handle it fine, but you don't know that until you measure.

The dashboard shows:

• Bucket A: 32% of traffic (last week 28%, last month 22% — growing)
• Bucket B: 18% of traffic (steady)
• Bucket C: NEW (5% last week, didn't exist before)

Action on Bucket C: pull 50 random examples, label them, evaluate the model on them, add them to the eval set. The system either handles it or surfaces a gap.

Why this matters. Models drift relative to traffic, not relative to time. If your traffic is stable, your model stays good. If your traffic shifts, your model may need adjustment even though the model itself hasn't changed.

We've seen production systems where the underlying model was unchanged for a year but the failure rate climbed 15% — because the user base had shifted to a new query pattern the model wasn't designed for. The input drift was the leading indicator; the failure rate was the lagging one.

What teams typically measure instead

The dashboard most production LLM systems have:

• Latency p50, p95, p99
• Monthly token cost
• Request volume
• Error rate (HTTP 4xx, 5xx)

These are operational metrics. They tell you whether the service is up and what it's costing. They don't tell you whether it's good. A system can have perfect operational metrics while quality silently degrades.

The three measurements above don't replace the operational ones. They supplement them. The full dashboard has both.

What goes wrong without these

A pattern we've seen on engagement audits:

Without eval score over time: Model versions get swapped because someone read about a new release. Quality changes silently. Customers notice; the team doesn't.

Without failure-mode distribution: Customer complaints arrive. The team investigates individual complaints. They miss the pattern that all the complaints are about the same new failure mode that emerged with the last prompt change.

Without input distribution shift: The system performance "feels different" but no one can articulate why. Eyeballing recent traffic shows nothing obviously wrong. The actual answer — that 20% of queries are now a kind the eval set never tested — never surfaces.

All three failure modes are recoverable but expensive when discovered late.

What "ready for production" looks like

Our engagement deliverables include a production-readiness checklist. The measurement layer:

• Eval harness runs on every deploy. Score is logged.
• Eval set has at least 500 examples across the categories the system handles.
• Failure-mode taxonomy is documented. Failures get tagged.
• Input clustering pipeline is running. Distribution shift dashboard exists.
• Alerts are configured for score drops > 2pp or new clusters > 5% of traffic.
• Weekly review meeting is on the calendar. Someone's job to attend it.

The last item is the load-bearing one. Dashboards no one looks at don't help. Engagement effort is justified by knowing what to measure; ongoing value comes from someone owning the measurement.

What the measurements pay for

A measured production system:

• Catches regressions in the first day, not the first month
• Surfaces leading indicators before they become customer complaints
• Justifies model updates with data ("we're confident this beats the current model on N examples")
• Gives ops the dashboards they need to defend the system in compliance reviews
• Lets the team experiment safely — try changes, measure impact, revert if needed

Unmeasured production systems are fragile in ways that aren't visible until something breaks. By then the cost of recovery is much higher than the cost of measurement would have been.

We won't sign off on engagement completion without these in place. It's the part of the deliverable that compounds over years.