LLM Cost Optimization in Production: Token Economics for Real Products

LLM bills have a property that surprises finance teams: they grow faster than usage. More users means more requests, but maturing features also mean longer prompts, bigger contexts, and chattier retries, so cost per request climbs while request count climbs. Most products can cut 40-70% of their LLM spend without hurting quality, because most spend is structural waste. This guide covers where it hides and the engineering that removes it.

You cannot optimize what you have not attributed

Step zero is the measurement layer: every LLM call logged with tenant, user, feature, model, token counts, and latency, the attribution pattern from our LLM integration patterns. With it, cost questions become queries: which feature burns the most, which tenant, which prompt version. Without it, optimization is guessing. Two derived metrics matter most: cost per completed task (not per request: a feature that retries three times per success is three times its apparent price) and context utilization: how many prompt tokens actually influenced the output.

Prompt caching: the free 50%

Every major provider now offers prompt caching: repeated prefix tokens (system prompts, tool definitions, static context) get cached at 10-25% of the normal input price. The engineering is ordering discipline: static content first, variable content last, so the prefix stays byte-identical across calls. A chat feature with a 3,000-token system prompt and tool schema, called thousands of times daily, recovers most of that spend for a refactor that takes an afternoon. The failure mode is a prompt template that interpolates a timestamp or user name into the header and silently breaks the cache on every call.

Model routing: most requests do not need the big model

Production traffic is a mix: 70-90% routine requests a small model handles at a tenth the price, and a hard tail that needs the frontier model. Routing patterns in order of sophistication:

• Static routing by task: classification, extraction, and reformatting go to the small model by design; synthesis and judgment go to the large one. This alone captures most of the win.
• Escalation on failure: try small, validate the output (schema check, confidence field, grounding check), escalate to large on failure. Works when validation is cheap and reliable.
• Learned routing: a classifier predicts difficulty per request. Rarely worth it before the first two are exhausted.

Route by task, then measure quality per route with the eval set, per our evaluation framework. Downgrades without regression gates are how cost optimization becomes a quality incident.

Context is the silent budget killer

Input tokens usually dominate spend, and most of them are wasted. The offenders and fixes:

• Unbounded chat history: cap it. Summarize older turns into a rolling digest; keep the last few verbatim. A 40-turn support conversation does not need 40 turns of context to answer turn 41.
• Over-retrieval in RAG: stuffing 20 chunks "to be safe" costs on every call and measurably hurts answer quality. Tune top-k with reranking, per our production RAG architecture guide: retrieve wide, rerank, pass 3-5.
• Redundant tool schemas: agents carrying 30 tool definitions per call when the task uses 4. Scope toolsets per workflow, which is also better agent design, per our agents guide.
• Verbose output formats: you pay for output tokens too. Structured outputs with terse schemas beat "explain your reasoning" fields nobody parses.

Batching and asynchrony

Every provider sells a batch tier at roughly half price for non-interactive work with a latency SLA of hours. Anything without a user waiting (nightly enrichment, document backfills, evaluation runs, report generation) belongs there, riding the same async job architecture from our integration patterns post. The design work is separating interactive from batchable at the queue level, so moving a workload to the batch tier is a config change, not a refactor.

Guardrails so optimization survives contact with the roadmap

• Budgets and alerts per feature and per tenant: a runaway loop or an abusive integration should page someone before finance notices, the quota pattern from our integration post.
• Regression gates on every downgrade: model swaps and prompt trims run the eval set before rollout, with quality thresholds that block the deploy.
• Price-aware architecture reviews: new features estimate cost per task at design time. The cheapest tokens are the calls you never make: a cache lookup, a rules engine, or a database query that answers before the model is invoked.

Cost optimization is not penny-pinching; it is what makes AI features economically shippable at scale, and it is usually the difference between "pilot" and "rollout" in a B2B contract. If your LLM bill is growing faster than your usage, our AI engineering team can audit the spend and build the routing, caching, and measurement layers described here.