A 70% prompt-token reduction without losing the answer (anonymized)

A recent engagement, anonymized. Mid-market SaaS company, customer-support deflection chatbot, sustaining ~250K customer interactions per month through an OpenAI deployment. Monthly bill: $34K, climbing about 6% month over month as adoption grew.

The internal team had already tried the obvious optimizations: switched to GPT-4o-mini for routine queries, added basic caching at the application layer, set per-customer rate limits. Bill kept climbing because volume kept growing faster than the optimizations could absorb.

The engagement ran six weeks. The bill dropped to $11K/month, with no measured change in customer-perceived response quality.

The shape of the problem

We started with the cost audit (see Hidden costs of running OpenAI at production scale). The audit showed:

• 4,200-token system prompt per request (persona, guidelines, escalation triggers, formatting rules)
• 1,800-token retrieved RAG context per request (top-5 chunks at ~360 tokens each)
• ~50-200 tokens of actual customer message
• ~150-400 tokens of response

So 80%+ of every billable token was the static system prompt + retrieved context. The customer's actual message was a small minority of the input.

This is extremely common. We've audited a dozen production deployments and most have a similar profile.

What we did

Four interventions, in order of impact:

1. LLMLingua compression on the system prompt

The 4,200-token system prompt compressed to 980 tokens via LLMLingua-2, with measured accuracy retention. We ran 200 held-out customer queries through both the original-prompt and compressed-prompt setups and compared response quality (graded by GPT-4 against a rubric). Score difference: 0.3 points on a 10-point scale.

Token reduction: 3,220 per request. At 250K requests/month, that's 805M tokens/month saved on the input side.

2. RAG context re-ranking with summary-aware retrieval

The 1,800-token context was top-5 chunks at 360 tokens each. We added a small re-ranker (BGE-Reranker-Large) that scored the 5 chunks and dropped the lowest-quality ones below a threshold. Median post-ranking context: 3 chunks, ~1,080 tokens.

We also experimented with chunk summarization — replacing each retrieved chunk with a 100-token summary unless the query specifically asked for verbatim text. Cut context further to ~450 tokens average.

Token reduction: 1,350 per request. Another 337M tokens/month.

3. Routing to smaller models for simple queries

A small BERT classifier predicted, per query, whether it was a "lookup" question (small model can handle) or a "judgment" question (needs the bigger model). Trained on 2,000 labeled queries.

70% of traffic routed to a much cheaper model variant. Quality scored within 0.4 points on the routed set.

Cost reduction: about 40% of total inference cost for the routed traffic.

4. Better prompt cache hit rate

The system prompt was getting hit by the prompt cache (~75% hit rate). After compression, the cached prefix was much smaller and hit rate climbed to 96% because the team also canonicalized whitespace and ordering (small mistakes that had been killing hits at the margin).

Doesn't reduce token count but reduces prefill cost on the server side, which translated to ~15% reduction in dollar billing because of how the API priced cached prefix tokens.

Results

Six weeks of work. Final numbers:

The engagement paid back in the first month after the changes shipped. The annualized saving is ~$275K against an engagement fee of $35K.

What we measured before signing off

Quality was the critical risk. A 70% cost reduction is worthless if the chatbot starts giving worse answers.

Three measurement layers:

1. Held-out eval set: 1,200 historical customer interactions with reference responses. Pre-change: 86.4% rubric score. Post-change: 85.9%. Within noise.

2. A/B test on live traffic: 5% of customers got the new prompt setup; 95% got the original. Two weeks. CSAT, deflection rate, escalation rate — no meaningful difference.

3. Manual review by client's support leads: 50 random conversations per week reviewed by the client's experienced support staff. They couldn't reliably tell which conversations used the new vs old setup.

Once these signals all said "no measurable difference," we shipped the new setup to 100% of traffic.

What the client owns now

The deliverable:

• Documented prompt compression pipeline they can re-run on future prompt changes
• Re-ranker model trained on their data, hosted by them
• Router model + the training data we built it from
• Eval harness with the 1,200 examples and the rubric
• A runbook for re-evaluating quality whenever they change the model variant or the prompt

The asset is the eval harness more than the model. The harness lets them measure future changes; without it they'd be back to flying blind.

What we'd do differently

A few things we'd improve on the next engagement of this shape:

• Run the cost audit first, before scoping. We did it as week-one work; if we'd done it pre-engagement we'd have set expectations more accurately about the upside.
• Build the router earlier. We did it in week 4; should have been week 2. The biggest individual savings were from routing.
• Push harder on chunk summarization. We were conservative; the experiments showed we could compress context further with acceptable quality loss.

Why it worked

Three factors that aren't universal:

1. The client had clear quality criteria. Specific metrics, rubric-graded eval, willingness to accept measured trade-offs. Without that, the engagement would have been "we can't tell if quality dropped" and the changes wouldn't have shipped.

2. The prompts had a lot of slack. A 4,200-token system prompt that compressed 4× tells you the original was verbose. Some prompts genuinely need every token; this one didn't.

3. The team had operational discipline. They were willing to A/B test, willing to wait two weeks for confidence, willing to run the eval harness on every change going forward. Engagements without that discipline tend to revert as soon as we leave.

Where this pattern transfers

Any production LLM deployment with:

• Large system prompts (2K+ tokens)
• RAG context that varies in actual relevance
• Mixed query difficulty
• Measurable success criteria

We've shipped this shape on 4 engagements now. The headline number varies (40-75% cost reduction) but the playbook is consistent. The biggest variation is the quality bar — clients with strict regulatory requirements (legal, medical) need tighter quality monitoring and accept smaller cost reductions in exchange.

For the SaaS client in this case: the engagement made one operational expense category drop by 65% with no other moving parts. Easiest budget conversation we've ever helped a client have.