Production audit: finding $18k/month in API spend that LLMLingua erased

An EdTech company — anonymized — was running document Q&A across student-uploaded study materials. The product: students upload textbook chapters and lecture notes; an AI answers questions about them. Volume: ~480,000 question/answer pairs per month. Backend: GPT-4o. Bill: $24,200/month, growing 8% monthly as student adoption expanded.

The CFO wanted a path to lower the bill before the next budget cycle. The engineering team had tried switching to GPT-4o-mini for routine queries but the accuracy dropped too much for student trust. They needed a different lever.

We ran a 3-week engagement. The headline outcome: the bill dropped from $24,200 to $6,100 per month, accuracy held steady, total engagement fee was $32,000.

What we found in the audit (week 1)

The cost audit (described in Hidden costs of running OpenAI) surfaced:

• The system prompt was 1,400 tokens. Reasonable.
• Each query retrieved 8 chunks at 400 tokens each = 3,200 tokens of context. Suspicious.
• The student's question was ~30 tokens.
• The response was ~250 tokens.
• Total input tokens per request: 4,630.
• Total output tokens per request: 250.

So input was 95% of the request size; output was 5%. Cost-wise, with OpenAI's roughly 5× higher output token pricing, input was still ~80% of the dollar cost.

Three more findings:

1. Retries were 18% of total requests. The team didn't know. Rate limit edges + transient 5xx errors were doubling some requests.

2. The retrieved chunks were not deduplicated. The same study material appeared multiple times in the 8 chunks for many queries because the retriever wasn't filtering near-duplicates.

3. Many chunks were marginally relevant. Manual review showed the top 3 chunks were usually high quality; chunks 4-8 were often filler that the model treated as noise.

What we shipped (weeks 2-3)

Four interventions, in order of impact:

1. Retrieved-chunk re-ranking.

Added a cross-encoder (BGE-Reranker-Large) that scored the 8 retrieved chunks and dropped any below threshold. Median post-ranking context: 3 chunks, ~1,200 tokens.

Token reduction: ~2,000 per request.

2. Per-chunk summarization.

For chunks below a high relevance threshold (the bottom-ranked ones that survived re-ranking), we replaced them with 80-token GPT-4o-mini-generated summaries instead of full text. Surprise: this actually improved answer quality slightly, because the summary highlighted what was relevant.

Token reduction: another ~500 per request.

3. LLMLingua compression on the system prompt.

The 1,400-token system prompt compressed to 380 tokens via LLMLingua-2. Quality measurably unchanged on a 200-question held-out set.

Token reduction: 1,020 per request.

4. Retry hygiene.

Implemented exponential backoff with jitter. Added a circuit breaker for sustained 5xx error periods. Retry rate dropped from 18% to 4%.

Cost reduction: ~14% of total spend (eliminating spurious retries).

Combined effect

Per request after changes:

• System prompt: 380 (was 1,400)
• Retrieved context: ~1,000 (was 3,200)
• Student question: 30 (unchanged)
• Total input: ~1,410 (was 4,630). 70% reduction.

Plus 14% reduction from retry hygiene.

Total monthly bill: dropped from $24,200 to $6,100. 75% reduction.

Quality measurement

We measured quality at every step:

End-to-end accuracy: 84.0% vs 84.1% baseline. Effectively unchanged.

Results six months later

What the client owns

End of engagement:

• The chunk re-ranker model and integration
• The chunk summarization pipeline (uses cheap GPT-4o-mini calls)
• The compressed system prompt + the LLMLingua deployment that produced it
• A runbook for re-compressing the prompt when they change it
• The eval harness with 200 questions
• A monitoring dashboard tracking accuracy and cost weekly
• Retry/backoff code patterns documented for their other AI endpoints

Most of these compound. The eval harness alone has gotten reuse across two other product features the client built in the months after the engagement ended.

What we'd do differently

Honest learnings:

• The audit alone was worth $32K to the client. We could have charged for just the audit (~$10K) and let the client implement the changes themselves. We chose to bundle implementation because the client wanted speed; they could have done it themselves over 6-8 weeks instead.
• We could have introduced semantic routing on top of this. The audit revealed that ~30% of queries were factual lookups answerable by a much smaller model. We didn't do it because the engagement was scoped to compression and the timeline was tight. The client built that themselves three months later, using the same eval harness.
• Initial estimates were 50% cost reduction. We came in at 75% because the retry finding was a surprise. We tell clients up-front: cost audits often find surprises. The engagement scope should allow room for what we find.

What this transfers to

Any production LLM deployment with:

• Large per-request input tokens (3,000+)
• Retrieval-augmented architecture
• Static system prompt
• Measurable quality criteria

We've shipped variants of this on six client engagements. EdTech, customer support, internal employee Q&A, healthcare patient Q&A, legal contract Q&A. Numbers vary from 40% to 80% cost reduction; the methodology is consistent.

For most companies running an OpenAI-based deployment they think is "well-optimized," there's still 40-70% in headroom. The audit costs $5-15K; the implementation typically pays back in under a quarter. We've never run an audit that didn't find meaningful savings.