"$0 per decision when self-hosted" is true in the same way that a car is free after you buy it. Here is the rest of the arithmetic.
Where the latency numbers come from
The widely quoted ~33 ms is a median single-question latency measured on an NVIDIA T4. Batch throughput amortises this further — reported figures land around 7.2 ms per question.
Three caveats before you quote those numbers internally:
- Median is not p95. Your users experience the tail.
- Hardware matters. A T4 is a modest, widely available card; consumer GPUs and newer data-centre cards will differ.
- First call is cold. Model loading dominates the first request unless you preload.
Memory footprint
Loading the English encoder, the multilingual encoder and the router together takes roughly 2 GB. That means:
- A small always-on instance is viable
- Edge deployment is realistic if you load only the checkpoint you need
- You can avoid GPU entirely for low-volume, latency-tolerant workloads
The three cost lines
1. Idle capacity
The dominant cost for most teams. A decision model that receives traffic in bursts still needs a process that stays up. Compare the monthly price of an always-on small instance against the per-token cost of a hosted API at your actual volume — the crossover point is usually lower than people expect.
2. Engineering time
Fine-tuning, temperature fitting, evaluation harness, model updates, dashboards. This is real and is usually the largest line item in year one.
3. Egress and storage
Small, but non-zero if you keep training data and evaluation sets in cloud storage.
When self-hosting genuinely wins
- Volume is high and steady. Cost per decision approaches zero as volume grows, whereas API billing scales linearly.
- Data cannot leave your network. Compliance requirements often settle this on their own.
- You need the probability, not just the label. Running your own calibration gives you control a hosted API will not.
When it does not
- Traffic is low or spiky — you pay for idle capacity.
- You have no labelled data and no plan to get any.
- Nobody on the team wants to own a model in production.
The honest framing: self-hosting trades money for control and latency. If you do not need the control, you are buying work rather than saving it.