Why we route, calibrate, and compress (and what each one buys you)
Three optimization patterns we apply on most production LLM engagements: semantic routing, custom calibration, prompt compression. Each addresses a different bottleneck. Stacked, they typically deliver 5-10× cost reduction without quality loss.
Most production LLM deployments we audit are running unoptimized. The team is paying for capability they're not using, sending tokens that don't carry information, and using a single model for queries that range from trivial to genuinely hard. Three patterns address the three biggest bottlenecks.
This is the order we apply them on engagements, and what each one earns.
Pattern 1: Semantic routing
What it does. Sends each request to the smallest model that can handle it well. A small classifier inspects the query and decides between (typically) a 3B-class model for routine traffic and a 70B-class model for harder queries.
What problem it fixes. Production LLM traffic is bimodal. Most queries are easy — factual lookups, simple summarization, basic intent classification. Some are hard — multi-step reasoning, complex code generation, nuanced judgment calls. Sending everything to the big model means paying 5-10× more per request than necessary on the easy queries (the majority).
What it earns. 5-7× cost reduction is typical when 60-80% of traffic routes to the small model. Latency also drops on the routed traffic because small models are faster.
Trade-off. Quality drops slightly on borderline queries — the router occasionally sends a query to the small model that the big model would have handled better. Mitigation: tune the confidence threshold to err toward escalation on uncertain calls.
When it doesn't help. Workloads where every query is genuinely hard (heavy reasoning, code generation only). Routing has no easy queries to siphon off.
A reasonable implementation: a fine-tuned BERT classifier trained on a few thousand labeled "easy" vs "hard" examples from the client's actual traffic. Latency overhead under 10ms; throughput at 5K+ queries/sec on CPU.
Pattern 2: Custom calibration
What it does. Re-runs the model's quantization step using a calibration corpus drawn from the client's actual production traffic, rather than the generic web-text corpus the off-the-shelf quants use. The result is a same-size GGUF that scores higher on the client's workload.
What problem it fixes. Off-the-shelf 4-bit quants are calibrated against general web text. They allocate quantization bits based on what matters for that distribution. If the client's distribution is different — SQL queries, contract clauses, support transcripts — the bit allocation is suboptimal.
What it earns. 5-10 percentage point accuracy lift on domain-specific tasks. No inference cost change — same file size, same kernels, same throughput. The lift is entirely on the quality side.
Trade-off. General capability drops slightly (typically 1-3pp on MMLU). If you need both domain accuracy and general capability, this is mostly fine.
When it doesn't help. Workloads where the generic calibration was already well-matched (general English chat). Models running at Q8 or higher (less room for calibration to improve). Models that are fine-tuned aggressively (calibrate after fine-tuning, not before).
Engagement effort: half a day to a day if the calibration corpus already exists; longer if we need to build it from logs.
Pattern 3: Prompt compression
What it does. Uses a small compression model (LLMLingua-2 or similar) to identify and drop low-information tokens in the prompt. A 4,000-token system prompt typically compresses to 800-1,200 tokens with measured quality retention.
What problem it fixes. Production prompts include a lot of static content — system prompt, few-shot examples, RAG context. Most of those tokens are repeated across requests and are billed every time. Compression cuts the repeated cost.
What it earns. 60-75% reduction in input tokens. On API deployments this is direct cost savings. On self-hosted, it's TTFT reduction (less prefill) and throughput improvement (more requests fit on the same hardware).
Trade-off. Quality drops 1-3pp typically; up to 10pp at aggressive compression ratios. Measure on a held-out set before deploying; tune the compression ratio based on your tolerance.
When it doesn't help. Structured prompts (JSON, exact format templates) — compressing breaks the structure. Short prompts (under 500 tokens) — overhead exceeds savings. Code generation prompts where every variable name matters.
Implementation: load the LLMLingua-2 model in the application; compress static prompts once at deploy time, store the compressed versions, serve those.
The stack effect
These three patterns address different bottlenecks. They stack:
- Routing reduces the fraction of traffic going to expensive models.
- Calibration improves the quality at any given cost level.
- Compression reduces the per-request cost of every model.
A typical engagement applying all three to a previously-unoptimized deployment:
| Layer | Cost reduction |
|---|---|
| Calibration (no cost change, ~+5pp accuracy) | enables smaller models in the next layer |
| Routing (60% of traffic to a small model) | ~5× on routed traffic |
| Compression (~3× input token reduction) | ~3× input cost on remaining traffic |
| Combined | 8-12× total cost reduction |
The combined number is multiplicative because each pattern compresses a different dimension.
Order matters
We apply them in this order because each one changes the input for the next:
- Calibration first. It enables routing — calibrated small models can handle more queries that previously had to escalate.
- Routing second. Once the small model is more capable, more traffic safely routes to it.
- Compression third. Apply to whichever model variants are now active.
Doing them in reverse order works less well. Compressing first means you're optimizing prompts for models you'll later swap out. Routing first without calibration means the small model handles too few queries to justify the routing complexity.
What we don't apply
A few patterns that get talked about but we don't reach for first:
Speculative decoding. Reduces decode time. Cool but heavy engineering investment for a typically modest gain. We apply only when latency is the binding constraint.
KV cache quantization to Q4. Saves VRAM. Useful but the quality cost is non-trivial. Try Q8 KV first; only drop to Q4 if Q8 isn't enough.
Multi-step CoT prompts. Improves quality on reasoning-heavy tasks at the cost of latency and tokens. Useful for specific tasks; not a general-purpose pattern.
Self-correction loops. Have the model critique and revise its own output. Doubles or triples token cost. Apply selectively where quality matters intensely.
These all work; they just don't have the cost/effort ratio of the three primary patterns.
What this looks like on a typical engagement
A representative cost trajectory from a deployment we've audited:
- Pre-engagement: $34K/month
- After calibration (week 3): same cost, +6pp accuracy. Enables routing.
- After routing (week 5): $14K/month (-58%), accuracy still up vs baseline.
- After compression (week 7): $9K/month (-74%), accuracy still up vs baseline.
The engagement fee was $40K. Payback period: under 4 weeks. The patterns aren't magic — they're well-known techniques applied with measurement discipline.
What we hand over
Each pattern includes operational artifacts the client owns afterward:
Routing: the trained classifier, the labeled training data, instructions for re-training as traffic distribution changes.
Calibration: the calibration corpus, the imatrix file, the custom GGUF, instructions for re-calibration on future base model updates.
Compression: the compressed prompts (deployed), the compression script (can re-run on prompt changes), the eval set used to verify quality.
These compound. Six months after the engagement, the client can re-run any of them as their workload shifts. We've watched clients keep extending the patterns over years without needing us back for the routine work.
What we engage for after these patterns
When clients call us back after the initial engagement:
- New use cases (we apply the same patterns to a new workload)
- Model generation changes (re-calibrate, re-train routes, often re-compress)
- New compliance requirements (architecture changes that touch the optimization patterns)
- Performance issues that the standard patterns don't address (specialized work)
The base engagement is a credit on the relationship. The retainer or follow-up engagements are smaller and more targeted, which is what most clients want.