On-prem AI for financial services: SEC, FINRA, and the GPU bill

Financial services LLM deployments operate inside a regulatory perimeter that doesn't relax because OpenAI has good marketing. The SEC, FINRA, and equivalent state regulators expect specific posture around customer data, recordkeeping, supervision, and audit trails. Most of those expectations are friendlier to self-hosting than to API consumption.

This is the framing we use on engagements with broker-dealers, RIAs, banks, and asset managers.

The four binding constraints

1. Customer data residency. Customer financial information (account numbers, transaction history, holdings, communications) is subject to Reg S-P at the federal level plus state-specific privacy regimes. Sending it to a third-party API requires the API provider to be a service provider with appropriate contractual safeguards. OpenAI and Anthropic both offer this; it's not impossible. But the operational overhead — vendor due diligence, ongoing oversight, incident response coordination — is non-trivial.

2. SEC Rule 17a-4 / FINRA Rule 4511 — recordkeeping. Broker-dealers must retain specific communications and records in a non-rewritable, non-erasable format for prescribed periods. AI-generated communications to or about customers fall under this. The recordkeeping has to be examiner-ready.

3. SEC and FINRA examinations. When examiners arrive, they want to see records of how the AI made decisions that affected customer outcomes. Models, prompts, training data, and outputs all become potentially examinable.

4. Supervisory obligations. FINRA Rule 3110 requires supervisory systems reasonably designed to achieve compliance. AI tools used by registered representatives need supervisory oversight. The supervisor needs to be able to review what the AI did.

Each of these is solvable on the API path. They're easier on the on-prem path.

What examiners ask

A few questions we've watched examiners ask in 2025-2026 sweeps:

"Show me the policy governing your AI tools."
"Who authorized the deployment, and what risk assessment was conducted?"
"How do you supervise AI-assisted communications to customers?"
"What records do you retain about AI decisions, and how do they meet 17a-4?"
"If a customer complaint involves AI-generated content, how do you reconstruct what happened?"
"What's the vendor management around the AI service provider?"
"How do you train staff on appropriate AI use?"

The vendor-management question is the one that diverges most between API and on-prem. With OpenAI, the firm has to demonstrate ongoing oversight of OpenAI as a service provider. With on-prem, there's no third-party service provider for the inference layer.

What 17a-4 requires for AI records

The rule requires "non-rewritable, non-erasable" storage for prescribed records, retained for specified periods (typically 3-6 years).

AI-related records that may need this treatment:

Prompts sent to the AI when generating customer communications
Outputs from the AI used in customer-facing communications
Model version metadata at the time of generation
Supervisor review records (who approved, when, what changes)
Training data used for AI tuning (potentially, depending on examiner posture)

With OpenAI, the firm has its side of these records but may not have OpenAI's complete logs. The firm needs to demonstrate that its retention is complete despite the third-party involvement.

With on-prem, all records are in the firm's WORM storage from day one. No vendor-side gap.

Customer data residency in practice

Regulated firms generally land in one of three postures:

Maximum restriction. No customer data through any third-party AI. AI is allowed only for internal-only workflows (employee Q&A on policy documents, internal research). Compliance straightforward; AI utility limited.

Vendor-managed with contracts. OpenAI or Anthropic as a designated service provider with appropriate BAAs / DPAs / vendor management. Compliance demonstrable but operationally expensive — vendor due diligence, ongoing oversight, audit cooperation.

On-prem inference. Customer data never leaves the firm's environment. Compliance simpler — no vendor management on the inference path. Operational footprint shifts to internal IT.

The mid-market (regional broker-dealers, mid-sized RIAs, community banks) is moving toward on-prem because the vendor-management overhead is disproportionate to firm size. Larger institutions can afford the vendor management; small ones can't.

Supervisory architecture

Effective AI supervision in financial services looks like:

Pre-deployment review. Senior compliance reviews the AI tool's purpose, scope of use, training data, and expected outputs. Documented approval.

Workflow-level approval. Each workflow where AI is used (drafting client correspondence, summarizing market research, classifying customer inquiries) gets a separate approval with its own controls.

Real-time review. AI outputs intended for customer use route through human review before sending. The supervisor sees both the AI draft and any human edits.

Periodic sampling. Random sampling of AI interactions (e.g., 10% per week) for compliance review. Documented findings.

Adverse event review. Any customer complaint that touches AI gets escalated review.

These work with API-based AI; they work with on-prem AI; they don't work with shadow IT use of consumer AI tools (an actual major problem at smaller firms — registered reps using ChatGPT to draft client emails without firm oversight).

A side benefit of on-prem: it's easier to enforce. The firm's AI tools are the on-prem deployment. Consumer AI tools are blocked at the network layer. There's no defensible reason for a rep to be using ChatGPT when the firm has its own approved AI.

The GPU bill is the easy part

Hardware capex is the most visible cost but not the hard part:

$50K-150K for serious on-prem deployment
Power + cooling: $1-3K/year
Hardware refresh on a 3-year cycle

What's actually hard:

Building the supervisory architecture
Training staff on appropriate use
Documenting policies that examiners will accept
Maintaining the operational discipline over multi-year horizons

The hardware is line-item cost. The compliance program is the multi-year investment that justifies it.

What we ship on a financial-services engagement

A typical engagement scope:

AI Governance Policy — board-approved policy describing what AI is used for, under what controls, with what supervision.
Workflow-specific procedures — operational guides for each approved use case.
On-prem deployment — the actual infrastructure, with documented architecture, security posture, and operational runbooks.
Eval harness — both general accuracy and bias / fairness measurements appropriate to the financial services context.
Supervisory dashboards — supervisors can see what AI did, who reviewed it, what changes were made.
WORM-compliant retention — AI records stored to meet 17a-4 / 4511.
Examination readiness package — pre-prepared documentation for the questions examiners typically ask.

Engagement timeline: typically 16-24 weeks for serious financial-services deployments. About a third of the work is technical; two-thirds is governance, supervision, and documentation.

Where this lands

The honest summary for financial services in 2026:

API-based AI is acceptable for non-customer-facing internal workflows with appropriate vendor management.
On-prem AI is preferable for customer-facing or customer-data-touching workflows.
Compliance preparation matters more than technical capability. Examiners ask governance questions before technical questions.
The vendor management overhead at small firms argues for on-prem. Larger firms can afford it either way.

The trajectory is toward more, not less, regulatory attention on AI in financial services. SEC's AI rulemaking is in progress. State regulators are watching. Firms building compliance posture now will be ready when the rules tighten; firms hoping the regulators stay quiet will be unprepared.

For most of the mid-market, we recommend on-prem for the inference layer with a documented compliance program around it. The GPU bill is the easy part of that recommendation; the program is the hard part. Both are buildable.