Three-year TCO for a self-hosted LLM stack vs API spend

The buy-vs-self-host decision usually arrives in a budget meeting with one of two framings: "API is too expensive; we need to self-host" or "self-hosting is too complicated; we should just pay the API." Neither framing has the actual numbers.

This is what those numbers look like over three years at three usage tiers — small (10M tokens/month), medium (1B tokens/month), large (10B tokens/month). All on a 70B-class workload (Llama-3.3 or Qwen 2.5 72B serving general chat / RAG / customer support).

The line items

API side:

Per-token cost (GPT-4o or comparable): ~$0.30/M tokens blended
Volume discounts above 10B/mo (-15% typical)
No fixed costs

Self-host side:

Hardware capex (workstation or server with GPU)
Power + cooling
Network bandwidth (typically negligible for LAN; egress costs apply if API-style for external consumers)
Ops headcount (the bottleneck)
Eval harness maintenance (continuous)
Replacement / depreciation
One-time engineering setup

Small tier — 10M tokens/month

API spend: $3/month × 36 months = $108 over 3 years.

Self-host an H100 build:

Capex: $50,000
Power + cooling (mostly idle): ~$20/mo × 36 = $720
Ops: 0.05 FTE × 3 years × $150K loaded = $22,500
Resale (year 3): -$15,000
Total: $58,220 over 3 years.

API wins by ~500×. At this volume, self-hosting is structurally irrational unless other factors (compliance, latency, customization) dominate.

Medium tier — 1B tokens/month

API spend: $300/mo × 36 = $10,800.

Self-host H100 build:

Same fixed costs as above: ~$58,220.
Now meaningfully closer but still 5.4× the API cost.

API still wins. The crossover hasn't arrived.

A smaller 4090 setup ($3K capex, $300/mo ops):

Capex: $3,000
Power: $5/mo × 36 = $180
Ops: same 0.05 FTE = $22,500
Total: $25,680 — but 4090 capacity ceiling is ~300M tokens/mo for 7B model, much less for 70B. Won't cover this tier.

The honest answer at 1B: stay on API unless something other than cost is driving the discussion.

Large tier — 10B tokens/month

API spend: $3,000/mo × 36 × 0.85 (volume discount) = $91,800.

Self-host H100 build at saturation:

$58,220 over 3 years.
Covers ~1B/mo per server; need ~10 servers for 10B/mo at sustained load → $582K. Worse than API.

Self-host the actual workload: probably one or two H100s sustaining 70-80% load with cloud bursting for peaks. Real cost: ~$80-120K over 3 years depending on burst frequency.

Crossover. At this volume, self-hosting roughly matches API spend. The qualitative factors (control, latency, customization, data residency) become decisive.

Extra-large tier — 100B tokens/month

API spend: $30,000/mo × 36 = $1.08M.

Self-host: need 8-15 servers, dedicated ops team (2 FTEs), data center contract or major cloud commit.

Hardware: $400-800K
Ops: $1.2M (2 FTEs × $200K loaded × 3 yr)
Power/cooling: $100K
Total: $1.7-2.1M over 3 years.

Self-hosting loses again at the largest tier — because the team you need to run that scale eats the per-token savings. Companies that genuinely operate at 100B/mo (Bytedance, Mistral, Cohere) have AI engineering teams of 50-200 people; the math works at that scale because the team already exists.

For a mid-market or enterprise client without that team, this tier should stay on API or use a managed compute provider (Together AI, Fireworks, Modal).

The shape of the curve

Plotted, the API line slopes up linearly with volume. The self-host line has a floor (capex + ops, ~$60K/3yr for one server) and grows in steps as you add servers.

Crossover sits around 8-15B tokens/month for general workloads. Below: API. Above: self-host wins if you can operate it, loses if you can't because the ops team eats the savings.

What the analysis misses

Two factors that change the calculus:

1. Custom calibration and fine-tuning are only possible on weights you own. If the workload is highly specific (custom domain, narrow output format), a custom-calibrated 7B on-prem can match a 70B API call. The relevant volume comparison is then your-custom-7B-on-prem vs GPT-4o-API — different cost basis, often dramatically in self-host's favor at lower volumes.

2. Hidden API costs at scale. Above 1B tokens/mo, you start needing dedicated account management, rate-limit increases, contract negotiation. Real cost is higher than list price would suggest. Self-host has no equivalent overhead.

Both factors push the crossover lower than naive math suggests. We've shipped engagements where on-prem made sense at 500M tokens/mo because the workload was narrow enough that a custom 7B beat a generic 70B API for the specific application.

Three-year horizons aren't fixed

Three years is the right amortization horizon for hardware, but model capability changes faster. The 70B-class model you serve in 2026 might be a 32B-class model in 2028 doing the same work. If on-prem hardware was sized for 70B and the workload demand drops, you over-provisioned.

Conversely, the next generation of models (B200, B300) will deliver more throughput per dollar than current H100s. Investing in H100 capex in 2026 means missing the B200 generation. Cloud rental avoids the depreciation risk; you rent the latest hardware as it lands.

This is the durable argument for cloud over on-prem: someone else absorbs the depreciation. Worth ~$10-15K per server in implicit value.

The honest summary

Volume	Honest recommendation
Up to 1B tokens/mo	API. Cost math doesn't move; qualitative factors might.
1B-10B tokens/mo	Depends on qualitative factors. Cost is close.
10B+ tokens/mo	Self-host wins on cost if you can operate it.
Custom calibration matters	Self-host wins at lower volumes than the generic math suggests.
Team capacity uncertain	API. Don't try to operate what you can't sustain.

We've run the analysis honestly for 30+ engagements. About 15% end up on-prem, about 60% stay on API, and 25% are hybrid (sensitive data on-prem, generic traffic to API). The hybrid case is more common than the pure self-host case in real deployments.