Calibration engagement: $15k, 2 weeks, 11% accuracy lift on a SQL benchmark
Anonymized case study. SaaS company with a SQL-from-natural-language feature shipped on GPT-4o. We re-calibrated their on-prem 7B and beat the API baseline by 11pp at 1/10 the cost. Two weeks of work, $15k engagement fee.
A SaaS company — anonymized — had a SQL-from-natural-language feature in their product. Users ask questions in English; the AI generates SQL; the SQL runs against the user's connected database. Volume: ~600K SQL generations per month. Backend: GPT-4o, $0.30 per million tokens blended. Bill: $4,800/month.
The product team wanted lower inference cost and better latency, but the lift had to be measurable — they'd been burned by previous "let's just swap models" experiments where the user-perceived quality dropped without measurement to back it up.
We ran a 2-week engagement. $15,000 fee. Below is the narrative.
The starting position
The team had:
- A working production deployment on GPT-4o
- About 12,000 historical natural-language → SQL pairs (validated by users in production through "was this query correct?" feedback)
- Internal interest in moving to self-hosted but no operational confidence
- A 24GB A6000 in a developer's workstation, available for experimentation
The product baseline (GPT-4o): 76.3% execution accuracy on their internal test set of 500 SQL queries.
What we set up
Day 1-2: Stood up Qwen 2.5 Coder 7B Instruct on the A6000. Off-the-shelf Q4_K_M from bartowski. vLLM serving. Reproduced the existing prompts against the new model.
Baseline (off-the-shelf 7B Q4): 62.1% execution accuracy. Substantially below GPT-4o. Expected — the 7B doesn't have the same capability ceiling.
Day 3-5: Built a calibration corpus from 3,000 of the client's natural-language → SQL pairs (their user-validated set). Stripped any literal customer values from the SQL (anonymized). Ran llama-imatrix against the unquantized Qwen 2.5 Coder 7B base. Generated a custom Q4_K_M.
Day 5 result (custom-calibrated 7B Q4): 71.4% execution accuracy. Recovered 9pp from the off-the-shelf baseline. Still below GPT-4o but closer.
Day 6-9: Trained a LoRA adapter on the same 3,000 examples using QLoRA. Rank 16, 3 epochs, default hyperparameters. The training run took an evening.
Day 9 result (custom-calibrated 7B + LoRA): 87.1% execution accuracy. Above GPT-4o by 10.8pp.
Day 10-12: Built the eval harness around the 500-example test set. Re-ran both the on-prem deployment and the GPT-4o baseline through it. Documented results. Set up the deployment to run alongside production (shadow mode for the client's team to inspect outputs).
Day 13-14: Final review with the client. Decision: proceed to cutover, but A/B test for two weeks first.
Two weeks later
A/B test: 30% of traffic to the on-prem deployment, 70% to GPT-4o. Two weeks of production traffic.
Results:
GPT-4o accuracy
76.3%
production baseline
On-prem accuracy
87.4%
+11.1pp
GPT-4o avg latency
1.8 s
TTFT
On-prem avg latency
0.6 s
no network round trip
User-reported issues
3
vs 11 baseline
User "was this correct?" yes-rate
+8pp
on-prem cohort
The A/B confirmed the synthetic eval. We moved to 100% on-prem at week 16.
Cost outcomes
The math at six months post-cutover:
| Item | Before | After |
|---|---|---|
| OpenAI bill | $4,800/mo | $0 |
| On-prem hardware amortized | $0 | ~$120/mo (workstation already owned) |
| Ops time | minimal | ~2 hr/week |
| Net change | -$4,680/month | |
| Engagement fee | $15,000 one-time | |
| Payback period | ~3.2 months |
The annualized saving is roughly $56K against a $15K engagement fee. The hardware was already owned (the A6000 was sitting on a developer's desk); a new build would have moved payback to ~9 months.
What the client owns
End of engagement:
- Custom-calibrated Qwen 2.5 Coder 7B Q4_K_M GGUF
- LoRA adapter trained on their 3,000 SQL pairs
- The 500-example eval harness with the execution-accuracy runner
- Documentation of the calibration corpus and how to refresh it
- Runbook for re-calibrating when they update the base model (Qwen 2.5 → 3.0 etc.)
- A weekly review template they can use to track production accuracy over time
The asset that compounds is the eval harness. The model will be superseded; the harness lets them evaluate the next model on the same benchmark.
What we'd do differently
A few honest learnings:
- We sequenced LoRA after calibration. With more data we'd have tried LoRA-only first to see how much of the lift came from each. As is, we can't fully separate the two contributions.
- The A/B test was 30/70. A 50/50 split would have given a tighter signal earlier. The client wanted to be conservative; in retrospect we'd have argued harder for 50/50.
- We didn't measure latency variance, only mean. After cutover the team asked about p99 latency for their internal alerting. We had to backfill that measurement.
What this transfers to
Any engagement where:
- The workload is narrow (SQL is narrow; general chat is not)
- Labeled training data exists (the client's user feedback set was the load-bearing asset here)
- The current setup is on a third-party API at meaningful cost
- The on-prem hardware exists or is justifiable
We've shipped variants of this on five engagements. SQL generation, contract clause classification, intent triage, code completion for an internal language, log line classification. Numbers vary; the playbook is consistent.
The headline: custom-calibrated 7B + LoRA on 3,000-5,000 examples often beats the API baseline on the trained task at a fraction of the inference cost. The 2-week engagement is enough time to prove it on the client's data.