Controlling LLM costs: gateways, budgets, caching and routing

Short answer: You control LLM costs by sending every model call through one gateway that knows which team and task it belongs to, then cutting tokens at the source: cache the repeated parts of prompts, route routine work to smaller models, and cap spend with per-team budgets. Track cost per successful task rather than cost per token, because that is the only number that shows whether a saving is real.

Why do LLM bills grow faster than expected?

Model providers charge by the token. A token is a piece of a word, and you pay for the tokens you send (input) and the tokens the model writes back (output).

Two facts make bills climb quickly.

First, output is expensive. In one major provider's current price list, output tokens cost five times as much as input tokens on every model listed. A model that "thinks out loud" or writes long answers costs far more than one that answers briefly.

Second, model prices are very far apart. In that same list, the most capable models charge $10 per million input tokens, while the smallest charges $0.10 for prompts up to 100,000 tokens. That is a 100-fold gap. The FrugalGPT researchers made the same point in 2023: fees for popular model APIs can differ by two orders of magnitude.

Then there are the hidden multipliers that rarely show up in a pilot:

• Resent context. Every call sends the full system prompt again. A 4,000-token instruction block is paid for on every single request.
• Retrieval stuffing. Retrieval-augmented generation (RAG) adds document chunks to the prompt. Ten chunks "to be safe" can double the input size.
• Agent loops. An agent that takes five steps usually resends the growing conversation at each step, so input grows with every step.
• Retries and failures. Timeouts, malformed outputs and validation retries are all billed.
• Wrong model by default. Teams pick the strongest model during a prototype and never revisit the choice.

A simple worked example shows how this adds up. Say a support-reply task has 6,000 input tokens (instructions plus retrieved documents) and 500 output tokens, and an agent takes three steps, resending the context each time. That is 18,000 input tokens and 1,500 output tokens. If output costs five times input, the output alone is worth 7,500 input tokens, for a total of 25,500 "input-equivalent" tokens for one reply. Most of that is repeated context, which is exactly what caching and tighter retrieval attack.

What does an LLM gateway do for cost control?

An LLM gateway is one service that sits between your applications and the model providers. Applications call the gateway; the gateway calls the model. Nothing calls a provider directly.

That sounds like plumbing, but it is what makes every other cost control possible. You cannot set a budget, cache a prompt or switch a model for traffic you cannot see.

Here is what a gateway contributes to cost control, and what to watch for:

How does prompt caching cut LLM costs?

Most prompts start with the same long block: system instructions, tool definitions, policy text, a product catalogue. Prompt caching lets the provider keep that unchanged opening part (the "prefix") ready, so you pay much less when it is reused.

The numbers are published:

• One major provider charges cache reads at 0.1 times the base input price. Writing to the cache costs 1.25 times base input for a 5-minute cache, or 2 times for a 1-hour cache. The default cache lives for 5 minutes and is refreshed at no extra cost each time it is used.
• Another major provider turns prompt caching on by default for supported models, requires at least 1,024 tokens of prompt on its recent models, and discounts cached input by up to 95%. A hit needs the whole prefix to match exactly.

The break-even is quick. With a 1.25x write and a 0.1x read, sending the same prefix twice within five minutes costs 1.35 units instead of 2. Every further reuse costs a tenth of the normal price.

To get cache hits:

1. Put stable content first: instructions, tool definitions, reference text.
2. Put changing content last: the user's question, today's date, retrieved chunks.
3. Do not insert timestamps or request IDs at the top of the prompt. One changed character early on breaks the match.
4. Check the provider's response fields that report cached tokens, and track your hit rate.

Two other kinds of caching are worth knowing:

• Response caching stores the whole answer for an identical request. It works well for repeated lookups, such as classifying the same product description.
• Semantic caching returns a stored answer for a similar question. It saves more but can return a stale or slightly wrong answer. Use it only where that is acceptable, and always scope the cache by tenant and user permissions so one person's answer never reaches someone who should not see it.

Finally, not everything needs an instant answer. Both major providers above offer a batch mode at a 50% discount for requests that can wait; one documents completion within 24 hours. Nightly summaries, document tagging and evaluation runs are good fits.

When should you route to a small model instead of a large one?

Most business traffic is routine: classify this email, pull the invoice number out, write a two-line reply. Small models handle much of this well. The large model should be kept for work that actually needs it, such as multi-step reasoning or long, messy documents.

There are three common routing patterns:

• Static routing by task type. Classification always goes to the small model; contract review always goes to the large one. Simple and predictable.
• A router. A lightweight classifier looks at each request and predicts whether the small model will do.
• A cascade. Try the small model first; if its answer fails a check (a schema check, a confidence score, a validator), escalate to the large model.

The research is encouraging, with caveats. FrugalGPT, a cascade approach, reported matching the best individual model "with up to 98% cost reduction" on its benchmarks. The RouteLLM team reported cost reductions of over 85% on MT Bench, 45% on MMLU and 35% on GSM8K compared with using only the strong model, while still achieving 95% of its performance.

Notice the spread: 35% to 85% depending on the task. Savings depend on how much of your traffic is genuinely easy. The only way to know is to test on your own data.

A safe way to introduce routing:

1. Collect 100 to 300 real requests for one task, with answers your team agrees are correct.
2. Run them through the small and large models and score both.
3. Write down the quality bar the business will accept, such as "correct category on 95% of cases."
4. Route to the small model only the task types that clear the bar.
5. Keep sampling live outputs each week, because models and inputs drift.

How should you set LLM budgets and quotas per team?

Budgets work when someone owns them. Give each team or product feature a monthly budget with a named owner, and make the spend visible to them before it becomes a finance problem.

Useful limits, from widest to narrowest:

• Monthly budget per team or feature, with alerts at, say, 50%, 80% and 100%.
• Rate limits per user or API key, to stop a script from draining the month in an hour.
• Maximum tokens per request, both input and output.
• Maximum steps per agent run, so a looping agent stops instead of burning money.

Decide in advance what happens at the limit. Blocking everything can stop a customer-facing service. Common softer options are: fall back to a smaller model, move non-urgent work to the batch queue, or shorten the retrieved context. Keep hard blocks for internal and experimental use.

Many companies start with showback (each team sees its spend) before chargeback (each team pays from its own budget). Showback alone often changes behaviour.

How do you measure LLM cost per task?

Cost per token tells you about the provider's price. Cost per task tells you about your system.

Define the task in business terms: a resolved support ticket, a processed invoice, a summarised contract. Then:

Cost per successful task = all model spend tagged to that task type ÷ number of tasks that succeeded.

Include retries, failed attempts and every agent step in the numerator. Count only successful tasks in the denominator. This is what exposes a cheap model that fails often and quietly costs more than a pricier one that gets it right first time.

Review these per task type each month. When cost per successful task rises, the token metrics usually tell you why.

A 30-day plan to get LLM costs under control

1. Week 1: Centralise. Route all model calls through one gateway. Revoke provider keys held in individual apps and scripts.
2. Week 1: Tag. Require team, feature and task ID on every call.
3. Week 2: Baseline. Measure cost per successful task for your top five task types.
4. Week 2: Restructure prompts. Move stable content to the front and turn on caching. Trim retrieved chunks to what answers actually use.
5. Week 3: Test routing. Build a small test set for your highest-volume task and compare small and large models.
6. Week 3: Batch. Move work that can wait into the batch queue.
7. Week 4: Set budgets. Agree owners, monthly budgets, alerts and the behaviour at the limit.
8. Ongoing: Review monthly. Track cost per task and quality side by side.

How CloudOEngine helps

CloudOEngine is one platform and console for your data, pipelines and AI runtime. The AI runtime includes an LLM gateway, agents with tool access, and RAG, so model calls from your applications and agents go through one place rather than being scattered across scripts and services.

Because data, pipelines and AI share one permission model and one audit trail, you can see who called what and on whose behalf. Role-based access decides who can use which capability, and per-tenant encrypted secrets keep provider credentials out of application code.

For agents, every step and tool call is traced and inspectable. That matters for cost: loops, oversized context and unnecessary tool calls are visible in the trace instead of hiding inside a monthly invoice. When an agent's action needs a person's sign-off, Autopilot approval rules let a human approve it in the app or through an email approve link before it runs.

Governance is included rather than sold as an add-on: tenant isolation, role-based access, audit logs and encrypted secrets come with the platform. CloudOEngine is built in Bengaluru, India.

Book a demo · See the AI runtime

Frequently asked questions

What is the fastest way to reduce LLM costs?

Turn on prompt caching for the long, unchanging part of your prompts and move routine tasks to a smaller model. Both cut spend without rewriting your application, as long as requests already go through one place you can change.

How much does prompt caching save?

It depends on how often the same prompt prefix is reused. One major provider bills cache reads at 0.1 times the normal input price, and another says cached input is discounted by up to 95%, so a long system prompt that is reused often becomes much cheaper.

Do smaller models give worse answers?

On hard tasks, often yes; on routine tasks such as classification, extraction and short replies, often not. Research on routing found large cost cuts while keeping most of the quality of the strongest model, but you should check this on your own test set before switching.

What is an LLM gateway?

An LLM gateway is a single service that every application calls instead of calling model providers directly. It holds the provider keys, tags each request with a team and task, logs token use, and is the natural place to add caching, routing and spending limits.

Should we measure cost per token or cost per task?

Measure cost per successful task, such as cost per resolved ticket. Cost per token hides retries, agent loops and failed attempts, which is where much of the waste sits.

Sources

1. Prompt caching (Claude API documentation)
2. Pricing (Claude API documentation)
3. Prompt caching (OpenAI API documentation)
4. Batch API (OpenAI API documentation)
5. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (Chen, Zaharia, Zou, 2023)
6. RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing (LMSYS, 2024)

#llm cost #llm gateway #prompt caching #model routing #finops