LoRA fine-tune for contract review: from 64% to 91% on the firm's eval set

A regional commercial law firm — anonymized — wanted to automate the first-pass review of incoming contracts. The workflow: an associate would scan a new contract, identify and tag specific clauses (governing law, indemnification, limitation of liability, termination, change of control, IP assignment, etc.), and flag any unusual provisions for partner review.

The firm wanted to replace the first-pass tagging with AI. The associate review would still happen, but only on the AI-flagged clauses rather than the full document.

They had ~3,500 hand-tagged contracts from the last three years — about 41,000 individual clause labels. We used this to fine-tune a Llama-3.3 8B Instruct via QLoRA.

The starting position

Base model accuracy. Off-the-shelf Llama-3.3 8B Instruct, given a contract clause and asked to classify into the firm's 16 categories: 64.3% F1 score on a 500-example held-out test set. Workable as a starting point but not deployable.

API baseline. GPT-4o on the same test set with a carefully tuned prompt: 82.1% F1. Better than the base 8B, but the firm wanted to avoid sending contract text to a third-party API for the privilege reasons discussed in our legal industry article.

Firm requirements.

• On-prem deployment only
• Inference latency under 500ms per clause
• Quality measurably above the off-the-shelf 8B
• Eval harness handed over for ongoing use
• Compliance memo for the firm's IT and risk committees

The engagement

Week 1: Data preparation.

The 41,000 labeled clauses needed cleaning. We:

• De-duplicated near-identical clauses (MinHash, threshold 0.7) — kept ~22,000
• Stratified by clause type — some categories had 6,000 examples, others had 300
• Down-sampled over-represented categories — final training set of 8,000 examples balanced across the 16 categories
• Held out 500 examples (stratified by category and contract type) as the eval set, locked away

PII strip wasn't needed — clauses themselves don't contain customer-identifying information. The firm reviewed a sample to confirm.

Week 2-3: Training infrastructure + initial fine-tune.

Set up a workstation with a 4090 (24GB) — the firm's existing development hardware. QLoRA configuration:

LoRA rank: 16
LoRA alpha: 32
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
LoRA dropout: 0.05
Optimizer: paged_adamw_8bit
Learning rate: 2e-4 cosine
Epochs: 3
Effective batch size: 16
Max sequence length: 1024

3 epochs took ~85 minutes on the 4090.

After training:

• Train loss epoch 1: 0.91 → 0.62
• Train loss epoch 2: 0.62 → 0.43
• Train loss epoch 3: 0.43 → 0.35
• Eval loss tracked train loss within 0.04 — no overfitting

Eval set F1 score: 88.7%. Already above GPT-4o; below where we wanted to land.

Week 4: Iteration.

Hyperparameter sweep:

• Rank 8: 86.2% (worse)
• Rank 16: 88.7% (baseline)
• Rank 32: 88.9% (no real improvement)
• Same rank but 5 epochs: 89.8% (mild improvement)

Data ablation:

• Removing the 1,200 examples from the "limitation of liability" category (the noisiest tagged category in the source data): F1 climbed to 90.4%
• Re-tagging those 1,200 with a senior attorney's review: F1 climbed to 91.1%

The 1,200 cleaner examples were a bigger lift than any hyperparameter change. Data quality > hyperparameters; this is the regular lesson.

Final configuration: rank 16, 5 epochs, cleaned training data.

Week 5: Deployment + eval harness.

Set up inference on the same 4090, serving via llama.cpp with the merged GGUF (LoRA merged into base, single 4.4GB file). OpenAI-compatible API on localhost:8080.

Built the eval harness: take a contract clause, get the model's classification, compare to ground truth. F1 per category, confusion matrix, sample of failures for human review.

Set up the production integration: the firm's document management system POSTs clauses to the local endpoint; results stream back. Latency: ~280ms per clause on the 4090.

Week 6: Production cutover + handover.

Shadow mode for the first week — production traffic ran through both the model and the human first-pass review; outputs compared offline. Confidence built; agreement was 89% on the human-AI pairs.

Full cutover at end of week 6. Associate review now only happens on AI-flagged clauses (the model's "uncertain" outputs based on confidence threshold) and on partner-requested deep dives.

Six-month outcomes

The headline metric the firm cared about was associate time. The 58% reduction in first-pass review time freed associates for higher-value work. At the firm's billable rate, the time savings annualized at ~$280K against the $48K engagement fee — paid back in roughly 2 months on time savings alone.

What the firm owns

End of engagement:

• The fine-tuned Llama-3.3 8B merged GGUF
• The 8,000-example training set (cleaned, stratified)
• The 500-example eval harness with per-category F1 reporting
• Confusion matrix tracking over time
• Runbooks for re-training when the firm's clause taxonomy evolves (it has, twice in six months)
• The IT compliance memo

The firm's IT runs the deployment internally. We're on retainer for major changes (re-training, new clause categories, model version upgrades) but the routine operation is theirs.

What goes wrong

A few honest failure modes from related engagements:

Data quality dominates. The single biggest lift came from cleaning 1,200 noisy examples, not from hyperparameter tuning. If the firm's source data is more noisy than ours was, the engagement budget shifts from "training compute" to "data cleaning labor."

Drift in the firm's own definitions. Six months in, the firm's senior partner decided to split "indemnification" into two sub-categories. The model didn't know about the new split. Re-training was needed; the eval harness made that change low-friction.

Inter-partner disagreement. Two senior partners disagreed on the correct classification of a borderline clause type. We had to mediate the eval set. The lesson: get partner consensus on the eval set BEFORE training.

What this transfers to

Any engagement where:

• Domain-specific classification or extraction is the task
• The firm has labeled training data (~3,000+ examples)
• On-prem deployment is preferred (legal, healthcare, finance)
• Latency requirements are reasonable (sub-second is fine)

We've shipped this pattern on three legal-tech engagements, two healthcare, one financial-services KYC. The numbers shift based on data quality; the structure is consistent.

For firms in the position of "we have hand-labeled data and we want to move to AI," this is the most reliable engagement shape we offer.