Replacing a $42K/mo OpenAI bill with on-prem 70B (anonymized)

A regional law firm — anonymized — was running a document classification and clause extraction pipeline through OpenAI's GPT-4o. Volume: about 80M tokens/month. Bill: $42,000/month, with strict per-matter data handling requirements that made the OpenAI usage uncomfortable for partners and inside counsel.

We migrated to on-prem Llama-3.3 70B with custom calibration. Hardware capex: $52,000 for a workstation with 2× A100 80GB. Amortized over three years plus power and ops: $4,200/month equivalent. Payback period: 9 months. Quality measurably better than the OpenAI baseline on the firm's domain-specific eval.

This is what the engagement looked like.

The original setup

Two distinct workloads:

Document classification. Incoming engagement letters, contracts, court filings — all needed automatic classification into ~30 case types so they could be routed to the right practice group. Roughly 60% of total traffic.

Clause extraction. Once a contract was classified, specific clauses (governing law, indemnification, termination, change-of-control) needed extraction with citation. Roughly 40% of traffic. Higher-stakes — extracted clauses fed downstream review workflows where lawyers acted on them.

Both ran through GPT-4o with carefully tuned prompts. Volume was steady around 80M tokens/month. The system worked; nobody was complaining about quality.

Why they wanted to migrate

Three converging pressures:

1. Cost trajectory. Bill was $42K/mo and growing 4-5% monthly as the firm grew. Trajectory: $65K/mo within 18 months, $100K/mo within 36 months. Unsustainable as an operating expense for a regional firm.

2. Data handling discomfort. Even with OpenAI's enterprise data handling commitments, partners weren't comfortable sending un-redacted client documents to a third-party API. Some matters involved sealed litigation or active arbitration where confidentiality was a hard requirement.

3. Compliance momentum. Bar association guidance was moving toward "if cloud, it must be your cloud" for legal AI. Not a regulation yet but obviously where it was heading.

The engagement scope: replace the API with an on-prem deployment, match or exceed quality, document everything for the firm's compliance program.

What we built

Phase 1 — measurement (week 1-2): Built an eval harness with 800 documents the firm had manually classified and clause-extracted. This became the ground truth. Ran the existing OpenAI system against it; recorded baseline accuracy.

Baseline:

• Classification accuracy: 91.2%
• Clause extraction F1: 84.6%

Phase 2 — hardware procurement (week 3-4): Specced and ordered a workstation: dual Xeon, 256GB RAM, 2× A100 80GB, redundant power, on-prem in the firm's existing server room. Total: $52,000 including a 3-year warranty.

Phase 3 — base model deployment (week 5-6): Stood up Llama-3.3 70B Instruct on the new hardware. Off-the-shelf Q4_K_M from bartowski. Wired to vLLM with continuous batching. Reproduced the existing prompts against the new model.

Baseline (stock Llama-3.3 70B):

• Classification accuracy: 86.4%
• Clause extraction F1: 78.1%

Substantially below GPT-4o. Expected — Llama-3.3 70B is genuinely competitive but not always a drop-in replacement.

Phase 4 — custom calibration (week 7-9): Built a calibration corpus from 3,000 historical documents the firm had archived. Stripped PII. Ran llama-imatrix against the unquantized base. Generated a custom Q4_K_M.

After custom calibration:

• Classification accuracy: 89.7%
• Clause extraction F1: 81.4%

Improved meaningfully. Still below GPT-4o, but closer.

Phase 5 — LoRA fine-tuning (week 10-13): Trained two task-specific LoRA adapters using the firm's labeled data. 4000 examples for classification, 2500 for clause extraction.

After LoRA fine-tuning:

• Classification accuracy: 93.8%
• Clause extraction F1: 87.1%

Both metrics now above the GPT-4o baseline. The fine-tuning closed the capability gap because the firm's specific domain (regional contract law, specific practice areas) was narrower than the general training distribution.

Phase 6 — production cutover (week 14-16): Shadow mode for 2 weeks: production traffic ran through both systems, results compared offline. No customer-facing change.

A/B test for 1 week: 20% of new documents routed to the on-prem system, others to GPT-4o. Quality verified manually by senior attorneys.

Full cutover at week 16.

Results

Production metrics six months post-cutover:

The custom model is actually more accurate than the API baseline because LoRA fine-tuning on domain-specific data beats a general-purpose model. Latency dropped by half because there's no network round-trip to OpenAI.

What the firm owns

End of engagement, the firm has:

• The on-prem hardware, fully operational
• The custom-calibrated Q4_K_M GGUF
• Two LoRA adapters (classification + clause extraction)
• The 3000-document calibration corpus (with PII strip pipeline documented)
• The 6500-example training set for the LoRAs
• The 800-document eval harness
• Runbooks for re-training, re-calibrating, and re-evaluating on a 6-monthly cycle
• A compliance memo describing the data handling for the bar association

The asset that compounds is the eval harness and the labeled data. The model itself will need updating in 18-24 months as Llama-3.3 is superseded; the data assets survive that transition.

What we measured to convince leadership

Three things had to be true for the partners to sign off on the cutover:

1. Accuracy at or above the baseline. Measured against the 800-document eval set. Custom + LoRA setup beat it. Verified.

2. Manual review by 3 senior attorneys. 60 random documents per attorney, comparing GPT-4o output to the on-prem output. Attorneys couldn't reliably pick which was which. Verified.

3. Real-world latency acceptable. Shadow mode confirmed 1-2 second response times under production load. Verified.

The cost case was secondary to these — they wouldn't have moved if quality dropped, regardless of savings.

What goes wrong on this kind of engagement

A few honest failure modes from related engagements:

Insufficient labeled data. A firm without a labeled training set for the LoRA path is stuck at the calibration-only quality level, which sometimes doesn't beat the API baseline. We were lucky here — the firm had been hand-labeling for years.

Operational immaturity. On-prem hardware requires someone watching it. Firms without internal IT capacity end up calling us at 2 AM when the service is down. We now require clients to demonstrate operational capacity before the cutover.

Model deprecation cycle. Llama-3.3 will be Llama-4.5 in 12-18 months. Engagements that don't include a re-training plan get stranded on an old model. Our standard hand-over now includes a "re-train every 9 months" schedule and the tooling to do it without us.

What this pattern transfers to

Any engagement with:

• High-volume, narrow-domain LLM usage (80M+ tokens/mo)
• Sensitive data
• Labeled training data available
• Compliance pressure pushing toward control
• Operational maturity sufficient to run on-prem

We've shipped variations of this on 5 client deployments now. Healthcare imaging report classification, financial KYC document review, insurance claims triage. Numbers vary; the playbook is identical.