Why AI costs explode in production
Most AI pilots die on unit economics — not model quality. A pilot uses the most capable frontier model for every request because it's the safest choice while you're proving the idea. Then usage grows, the cost curve compounds, and the price-per-correct-answer stays flat. Margin evaporates before you notice the trend line.
The teams whose AI features actually reach scale build four disciplines directly into the architecture — not into a spreadsheet someone reads on Monday. Here they are, in order of easiest-to-highest ROI.
Discipline 1 — Model routing
Model routing sends each request to the cheapest model that can answer it correctly. A cheap model (Haiku, GPT-4.1 Nano, Gemini Flash) handles routine classification and extraction. A mid-tier model (Sonnet, GPT-4.1) handles ambiguous cases. A frontier model (Opus, GPT-5, Gemini Pro) handles the small fraction that truly need it.
Well-tuned routing typically cuts model spend 60–90% — not by taking a quality hit, but by refusing to overpay for answers a smaller model already gets right.
How to build it
- Classify by complexity upfront. A tiny router model or a rules-based classifier picks the target tier for each request in <50ms.
- Fall through on low confidence. If the small model's answer is uncertain (log-prob, self-consistency, or a confidence score), auto-escalate to the mid tier — then to frontier if still uncertain.
- Measure per-route quality. Track accuracy per model tier weekly. Move traffic down the ladder as smaller models improve.
Our production AI trading platform routes ~78% of requests to a cheap model tier and ~5% to frontier. Total model spend is ~1/6th what a naive "always frontier" design would cost — with equivalent output quality on the tasks that matter.
Discipline 2 — Prompt caching
Prompt caching (offered by Anthropic, OpenAI, Google, and Amazon Bedrock) lets you re-use tokens across requests — the system prompt, few-shot examples, retrieved documents. Cached tokens cost 75–90% less and cut latency roughly in half.
Any system with a long system prompt or repeated context wins the moment you turn it on. If your app has a stable persona, a policy document, or an RAG context that reappears across requests, caching is often the single biggest cost lever.
How to build it
- Structure prompts stable-first. Put unchanging content (system prompt, policy, examples) at the top — that's what the cache holds. Put dynamic content (user turn, current retrieval) at the bottom.
- Mind the TTL. Provider caches expire (usually 5 minutes to 1 hour). Warm the cache proactively if your workload is bursty.
- Track cache hit rate. Anything below 60% hit rate on stable prompts means your prefix isn't as stable as you think.
Discipline 3 — Request batching
Batch APIs (Anthropic Batch, OpenAI Batch, Bedrock Batch Inference) run the same requests at 50% off list price in exchange for higher latency (usually returned within 24 hours). Any workload that isn't user-facing in real time — nightly enrichment, evals, backfills, offline classification — belongs on batch.
How to build it
- Split your workload. User-facing goes on real-time APIs. Everything else — content generation, categorization, evals, backfills — goes on batch.
- Queue and reconcile. Write a small batch orchestrator: enqueue → submit → poll → hydrate results back into your DB. Reuse it across every batchable workload.
- Measure the split. Track what % of requests are on batch vs real-time. Target >40% batch at scale.
Discipline 4 — Per-account cost ceilings
A per-account cost ceiling is a hard usage cap applied per user, tenant, or workspace. It exists so one runaway account cannot destroy your margin — whether from abuse, a bug, or a legitimate power user who suddenly 100xes their usage.
Every production AI platform needs one. It belongs in the architecture, not in a billing report you read on Monday. Without it, one bad day erases a month of margin.
How to build it
- Track cost per account in real time. Log the cost of every model call against the account making it. Real-time, not batched.
- Set soft and hard caps. Soft cap: warn the user and flag ops (e.g. at 80% of the plan's fair-use ceiling). Hard cap: refuse further requests or degrade to a cheaper model.
- Publish the limits. Users tolerate limits they know about. They rage at limits they discover mid-workflow.
When to self-host
Self-hosting a fine-tuned open-weights model (Llama 3.1, Mistral, DeepSeek, or a fine-tune) makes economic sense once your steady-state model spend passes roughly $15k–$30k/month AND the workload is stable enough to justify GPU utilization. Below that, API pricing wins on flexibility and total cost of ownership. Above that, self-hosting often cuts unit cost 3–5x.
Self-host too early and you're paying GPU time to sit idle. Self-host too late and you've overpaid for API tokens for a year. Watch the steady-state trend, not the pilot spike.
- Most AI projects fail on unit economics, not model quality. Build cost discipline into the architecture from day one.
- Model routing is the highest-ROI lever — 60–90% spend reduction with no quality hit.
- Prompt caching is free money on any system with stable context. Turn it on immediately.
- Batch anything not user-facing — nightly enrichment, evals, backfills — for a flat 50% discount.
- Per-account cost ceilings are non-negotiable in production. One runaway account should not be able to destroy your margin.
- Self-host only after you have stable, ~$15k+/month steady-state spend. Not before.
Related questions
Why do AI projects usually fail on cost, not model quality?
Most pilots use the most capable frontier model for every request — treating a $0.06 answer the same as a $0.001 answer. In production the cost curve compounds with usage while price-per-correct-answer stays flat, so unit economics fail before quality ever does.
What is model routing in AI cost optimization?
Model routing sends each request to the cheapest model that can answer it correctly. A cheap model handles routine questions; a mid-tier model handles ambiguous ones; a frontier model handles the small fraction that truly need it. Well-tuned routing typically cuts model spend 60–90%.
How much does prompt caching save on AI costs?
Prompt caching reduces the cost of repeated context by 75–90% on cached tokens and cuts latency in half. Any system with a long system prompt or repeated context wins immediately.
What is a per-account cost ceiling?
A per-account cost ceiling is a hard usage cap applied per user, tenant, or workspace, so one runaway account cannot destroy your margin. Every production AI platform needs one; it belongs in the architecture, not in a billing report.
When should you self-host models instead of using an API?
Self-host once your steady-state model spend passes roughly $15k–$30k/month AND the workload is stable enough to justify GPU utilization. Below that, API pricing wins on flexibility and total cost of ownership. Above that, self-hosting a fine-tuned open-weights model often cuts unit cost 3–5x.