LLM Cost Optimization: Where the Money Actually Goes

Teams usually discover their LLM feature is expensive in one of two ways: someone looks at the monthly spend, or someone looks at the cost per request and multiplies it by the growth forecast.

The instinct is to switch to a cheaper model. That is rarely the best first move, because the invoice is a poor guide to where the cost is coming from.

Where the tokens actually go

In a typical production system, a surprisingly small share of billed tokens is the user's question. The rest is overhead that accumulated one reasonable decision at a time:

• A system prompt repeated on every single call. Often a thousand tokens or more, sent identically, thousands of times.
• Full conversation history. Convenient to send, but it grows linearly and is re-billed on every turn.
• Oversized retrieved context. Retrieving ten documents when the answer was in one, because more context felt safer.
• Retry loops. A validation step that fails and resends is double-billing you at exactly the moment you have the least margin.
• Evaluation runs. Continuous evaluation across a full test set can quietly match or exceed production spend.
• Agent steps. Every tool call round-trip carries the whole accumulated context again.

None of these is unreasonable on its own. Together, they often dwarf the cost of the actual answer.

Seven levers, in order of return

1. Cut the context you send. Trimming retrieved context is usually the single largest and cheapest win. Measure retrieval recall at different context sizes rather than assuming more is better.

2. Route by difficulty. Send easy requests to a small model and escalate only when needed. This does more than save money: it usually improves latency for the majority of traffic.

3. Cache aggressively. Repeated system prompts, repeated user queries, and repeated tool results are all cacheable. Semantic caching handles paraphrases; exact-match caching handles the rest. Decide deliberately what may be cached, because for personalised or permissioned data the answer is often nothing.

4. Cap output length. Output tokens are usually priced higher than input tokens. A maximum length that is realistic for the task, enforced rather than suggested, prevents a rambling response from costing more than the question.

5. Batch offline work. Anything that does not need an immediate answer should not be paying real-time prices.

6. Review your evaluation suite. Evaluation that no longer changes any decision is cost without benefit. Keep the tests that catch regressions; retire the rest.

7. Revisit the model choice quarterly. The price-performance frontier moves fast. A choice that was right six months ago may no longer be, and re-testing is cheap compared with the alternative of assuming.

Do not optimise what you cannot attribute

Before tuning anything, make sure you can answer: what is the cost per request, per feature, per customer? Without attribution, optimisation is guesswork and the savings show up as a smaller invoice with no explanation, which means you cannot tell whether quality dropped too.

Log token counts and cost alongside quality metrics from day one. A cost-per-request dashboard that sits next to a quality dashboard turns this from an argument into a decision.

The trap: cutting cost by cutting quality

Every one of the levers above can be pulled too far. Trimming context without measuring recall degrades answers. Routing everything to the small model breaks the hard cases. Caching something permissioned is a security incident, not a saving.

The method is the same as for performance work generally: establish a quality baseline first, change one thing, measure both. Cost reduction that nobody notices is the goal. Cost reduction that users notice is a regression with a smaller invoice.