Self-hosted LLMs for legal: privilege, retention, and audit

Legal-tech LLM deployments collide with three constraints that don't apply (or apply less) to other industries:

Attorney-client privilege — the privilege protecting communications between lawyer and client
Retention obligations — both required-retention (for matters) and forbidden-retention (after settlement, in particular jurisdictions)
Audit and discovery readiness — whoever asks for records about your AI system needs an answer that holds up

API providers handle one and a half of these well. Self-hosted deployments handle all three. This article walks through why.

Privilege

Privilege protects communications. The classic rule: a client's confidential communication to their attorney for the purpose of seeking legal advice is privileged, and the privilege can be waived by disclosure to a third party.

The hard question: does sending a privileged document through OpenAI's API constitute disclosure to a third party?

The answer depends on jurisdiction and contract, but the conservative legal opinion is yes, potentially. OpenAI's enterprise terms include data handling commitments (no training, deletion after 30 days), but:

The communication has been transmitted to OpenAI's servers
OpenAI's employees and systems can technically access it (even if contractually they don't train on it)
Subpoenas can reach OpenAI's records, including the API logs they retain for that 30 days

For most matter types this is manageable risk. For high-stakes matters (sealed litigation, regulatory investigation, mergers and acquisitions where deal terms are confidential), the conservative posture is "don't send it to the API."

Self-hosting eliminates the disclosure question. The communication never leaves your environment.

The practical impact: firms running OpenAI-based legal AI typically restrict it to non-privileged or pre-redacted material. The 30% of matters where privilege is most acute have to use a different workflow — usually manual review without AI augmentation. That's the cost of the API path.

Retention

Two flavors of retention obligation:

Required retention. Some jurisdictions and ethics rules require that materials related to client representation be retained for specific periods (e.g., 7 years after matter closure). If the AI processes a client document, the AI processing record may be part of the matter file and need retention.

Forbidden retention. After settlement, some matters require deletion of related materials. NDAs, sealed settlements, regulator-ordered destruction. If the AI has logged the document content somewhere, that logging needs to be defensible against the deletion order.

API providers have their own retention policies. OpenAI's 30-day default works for some matters and conflicts with others. Negotiated enterprise terms can adjust retention, but:

You're delegating retention control to a third party
Discovery requests against the third party can reach materials you thought were deleted
Audit trails of what was processed and when are partial — you have your side of the conversation but not necessarily OpenAI's processing logs

Self-hosting means you set retention. Every record about every AI interaction lives in your infrastructure. You delete when you delete; you retain when you retain. Audit trails are complete because you generate them.

For firms with mixed matter types (some require deletion, some retention, some both for different documents), the operational simplicity of self-hosting is meaningful. You write one retention policy that applies to AI logs the same way it applies to email and document management.

Audit and discovery

In litigation, opposing counsel can request records about your AI use:

What documents did your AI system process?
What prompts were sent?
What outputs did it produce?
Were any AI-generated outputs incorporated into work product?
What model version was running at the time?

These questions can be answered. The question is whether your records are complete.

With a third-party API:

You have your side of the request (the API call you made)
You have the response you received
You may not have OpenAI's internal logs (and may not be able to obtain them quickly)
Model versions change without your control (OpenAI's "gpt-4o" today is not the same weights as last month)
Discovery requests against OpenAI directly are slower and may not produce useful records

With self-hosting:

You have your side
You have OpenAI's-equivalent side (because you're OpenAI here)
Model versions are pinned because you control the model
All records are in your custody and producible to the same timeline as any other firm document

For an audit by your bar association, an internal audit by general counsel, or discovery in active litigation, complete records produced on a normal litigation timeline matter. Self-hosting delivers them.

What changes when you self-host

A few things become harder:

You operate the infrastructure. Power, cooling, monitoring, replacement parts. The firm needs IT capacity (internal or contracted) that didn't exist before.
You manage the model lifecycle. Updates, security patches, performance regression — your problem.
You bear the capex. $30-100K of hardware capex for a serious deployment. Not nothing, especially for smaller firms.
You answer to your malpractice insurer. "Is your AI system properly configured" is a question they will ask. You need a clear answer.

A few things become easier:

Privileged matters can use AI. No disclosure question.
Retention policy is unified. One policy across all firm records.
Discovery responsive. Complete records, controlled production timeline.
Bar association readiness. When the rules tighten (they will), you're already aligned.
No vendor lock-in. Model behavior won't change because of a vendor's product decisions.

Common deployment shape

The pattern we've shipped on legal engagements:

On-prem hardware in the firm's server room. Not cloud. The privilege analysis is cleaner when the weights physically reside in your environment.
Per-matter encryption keys in addition to disk encryption. Each matter has its own key; deletion is cryptographic shredding of the key.
No external network egress from the AI server during inference. Updates flow through a controlled gateway during scheduled windows.
Full audit logging of every prompt and response, indexed by matter ID, retained according to matter retention policy.
Model version pinning with documented updates. Each matter is associated with the specific model version that processed it.

This shape doesn't preclude using the latest models or techniques. We've shipped custom-calibrated Llama-3.3 70B with LoRA fine-tuning in deployments exactly like this. The compliance posture is additive, not subtractive.

When the API still wins

Despite all this, the API path is sometimes right for legal:

Solo practitioners and small firms. Capex is prohibitive. They use API tools for non-privileged work and accept the trade-off.
Non-privileged matters. Some practice areas (estate planning, traffic violations, document review for public-record matters) involve less privilege concern. API is fine.
Pre-redacted workflows. If you can reliably strip identifying information before sending, the privilege question is reduced. Operationally hard to do consistently.
Specific use cases the open models don't handle well. Multimodal review of images, very long context (300K+ tokens). API leads, on-prem catches up over 6-12 months.

The reasonable shape for most mid-sized firms: on-prem for privileged matters and general AI use, API selectively for non-privileged work if any.

A firm asking about LLM adoption gets the same three triage questions:

What matter types? (Determines privilege exposure.)
What practice areas? (Determines retention requirements.)
What's your existing IT capacity? (Determines operational viability.)

If any answer points at "privilege matters, retention complex, IT capacity sufficient," self-hosting is the right path. We'd help build it. If IT capacity is the gap, we'd recommend a managed-on-prem path or a firm-hosted SaaS that retains control.

The honest mid-market story in 2026: most firms above 30 attorneys can support on-prem. Below that, the cost case is harder and the practice is moving slowly toward shared firm-side infrastructure.

For any size, the API path is becoming harder to defend on privileged matters. The trajectory is clear; better to plan the migration than be caught by it.