Posted By 
LLM Cost Optimization on a LiteLLM Gateway: What One Week of Real Traffic Showed
Value Added Tech is a consulting and engineering team. We configure, implement and develop automation and AI systems for clients, mostly on n8n, Make.com, HeadlessOps, HubSpot, Notion and Slack. Most of that work is now done with AI agents. Our people run OpenCode, an open-source terminal agent, and every request goes through one self-hosted LiteLLM gateway to the models: Claude Sonnet, Opus and Haiku, GPT Terra, Gemini and others. The agents reach our tools through MCP servers.
In early November 2026 that gateway cost about $990 in nine days. This article explains why, what we changed on November 6 and 7, and what the logs show a week later. Every number has its date window. UTC dates throughout.
- Average cost per request
- $0.067 to $0.012
- all models, before and after the changes
- Cost per request
- -82%
- all models, before and after the changes
- Total weekly spend
- -36%
- $878.79 to $561.13
The problem: what the logs contained
We started from real spend logs, 15,900 requests over nine days to November 6.
- Half of every request was tool definitions. Tool schemas were 49.5% of request bytes on Sonnet. A request carried 178 tools and about 230 KB of MCP schemas.
- The cache hit rate was 95%, which hid the cost. The tool block sits at the start of the prompt, so it was cached. It was still billed at the cache-read rate on every turn, about 26% of Sonnet cost.
- Subagents were expensive to start. They were 22% of sessions and 45% of cost. Each start wrote about 117,000 tokens to the cache with the full toolset, about $0.44 per spawn on Sonnet.
- One price was wrong. A GLM model alias had no cache-read price, so its cached tokens, 98% of its traffic, were billed at the full input rate. Logged spend for it was about 4.5 times too high.
- of Sonnet request bytes were tool schemas
- 49.5%
- 178 tools, about 230 KB per request
- cache hit rate
- 95%
- tool block re-read on every turn
- of cost came from subagents
- 45%
- 22% of sessions
- GLM cost overstated
- 4.5x
- missing cache-read price
What we changed
We shipped 13 changes. These are the ones that touch cost.
- MCP servers off by default. Servers that are rarely needed are disabled in the base configuration and switched on per project. We also fixed a loop where a local override kept turning them back on.
- One catalog MCP instead of many. A single MCP server with three tools (search, describe, call) replaced about 250 direct tool schemas. The model looks a tool up when it needs it.
- Subagents forced onto Haiku. Subagents now run on Claude Haiku, the cheapest model we use. Haiku traffic ramps up from November 10 in the logs.
- Auto-compaction. We set it at 100,000 tokens and later moved the threshold to 185,000 tokens.
- Pricing fixes. We added the missing cache-read price and recomputed 452 rows (logged $60.17, corrected $13.12). We then checked every model with real traffic. This corrected the ledger. It is not part of any saving below.
- Big tool outputs go to a file. A large HeadlessOps run log is written to disk and only a short confirmation enters the conversation, instead of a 1 MB payload.
- Cheaper models were tried and mostly did not help. More on this below.
The rest was reliability and security work: SSO limited to the company domain, secret redaction in logs and gateway errors, and session ids in request metadata so spend can be traced per session.
Results: one week before, one week after
The pre window is October 31 to November 6. The post window is November 8 to 14. November 7, when the changes landed, is in neither. Raw request logs are kept for seven days, so pre-window figures come from the gateway’s daily spend table.
Average cost per request, all models
Our headline metric is the average cost of one request, across every model and every user.
The numbers behind this figure
| Day | Average cost per request, all models (cents) |
|---|---|
| Oct 31 | 5.05 |
| Nov 1 | 8.69 |
| Nov 2 | 6.55 |
| Nov 3 | 5.44 |
| Nov 4 | 7.53 |
| Nov 5 | 12.35 |
| Nov 6 | 4.11 |
| Nov 7 | 4.49 |
| Nov 8 | 3.36 |
| Nov 9 | 3.55 |
| Nov 10 | 2.09 |
| Nov 11 | 2.76 |
| Nov 12 | 1.33 |
| Nov 13 | 1.05 |
| Nov 14 | 1.58 |
| All models | Oct 31 to Nov 6 | Nov 8 to Nov 14 | Nov 13 to Nov 14 |
|---|---|---|---|
| Requests | 13,037 | 31,923 | 16,954 |
| Spend | $878.79 | $561.13 | $200.49 |
| Average cost per request | $0.0674 | $0.0176 | $0.0118 |
| Average cost per request, excluding the retired Fable 5.1 | $0.0609 | $0.0176 | $0.0118 |
Requests rose from about 13,000 to about 32,000 a week because we scaled our usage on purpose. With smaller prompts and cheaper subagents, each unit of work costs less, so we gave the agents more of it.
Where the cost went down
Cost per request by model, November 8 to 14, in cents:
The numbers behind this figure
| Model | cents per request |
|---|---|
| Claude Haiku 5.5 | 0.38 |
| GPT Terra | 0.55 |
| Gemini Flash 3.8 | 1.45 |
| Gemini Argon | 2.56 |
| Claude Sonnet 5.5 | 3.85 |
| GLM 5.3 (Nov 8 only) | 3.86 |
| Claude Opus 5.5 | 9.57 |
| Claude Fable 5.2 | 14.43 |
| Model | Requests | Spend | Cost per request |
|---|---|---|---|
| Claude Sonnet 5.5 | 9,425 | $363.27 | $0.0385 |
| Claude Opus 5.5 | 553 | $52.90 | $0.0957 |
| Claude Haiku 5.5 | 10,743 | $40.95 | $0.0038 |
| GLM 5.3 (Nov 8 only) | 884 | $34.08 | $0.0386 |
| GPT Terra | 4,913 | $26.80 | $0.0055 |
| Claude Fable 5.2 | 58 | $8.37 | $0.1443 |
| Gemini Argon | 255 | $6.54 | $0.0256 |
| Gemini Flash 3.8 | 351 | $5.09 | $0.0145 |
Sonnet, the same model against itself
Part of the drop in cost per request is the model mix: Haiku subagents and Terra now take a large share. For a like-for-like number, Sonnet alone, with flat volume:
| Sonnet | Oct 31 to Nov 6 | Nov 8 to Nov 14 | Change |
|---|---|---|---|
| Requests | 9,293 | 9,425 | +1% |
| Spend | $684.21 | $363.27 | -47% |
| Cost per request | $0.0736 | $0.0385 | -48% |
| Prompt tokens per request | 284,600 | 86,000 | -70% |
| Average tools per request | 178 | 28 | -84% |
| Cache-read share of prompt | 95.4% | 90.8% | -4.6 pts |
The numbers behind this figure
| Day | Sonnet prompt tokens per request (thousand tokens) |
|---|---|
| Oct 31 | 255.8 |
| Nov 1 | 415.8 |
| Nov 2 | 306.9 |
| Nov 3 | 238.7 |
| Nov 4 | 321.6 |
| Nov 5 | 337.3 |
| Nov 6 | 167 |
| Nov 7 | 135.6 |
| Nov 8 | 99.9 |
| Nov 9 | 108.2 |
| Nov 10 | 86.7 |
| Nov 11 | 92.1 |
| Nov 12 | 82.3 |
| Nov 13 | 69.6 |
| Nov 14 | 72.1 |
The measures went out together, so we do not split the effect between them.
Where the money goes now
Across all models, a request is mostly cache reads. Over November 1 to 8 the average mix of tokens was 94.8% cache reads, 3.6% cache writes, 1.1% fresh input and 0.5% output. A cache write costs 25 times a cache read on Sonnet, so as prompts shrink and the cache-read share falls (Sonnet 95.4% to 90.8%, Opus 94.3% to 88.9%), writes become the biggest cost line. For Sonnet they are now about half of spend. The next saving has to come from fewer rewrites of the prefix.
What the models cost, at our prices
These are the prices we are billed through the gateway, per million tokens, as read from the gateway model table on November 8.
| Model | Input | Output | Cache read | Cache write 5 min | Cache write 1 hour |
|---|---|---|---|---|---|
| Claude Sonnet 5.5 | $2.00 | $10.00 | $0.10 | $2.50 | $4.00 |
| Claude Opus 5.5 | $4.00 | $20.00 | $0.20 | $5.00 | $8.00 |
| Claude Haiku 5.5 (under 100K prompt) | $0.10 | $0.50 | $0.01 | $0.125 | $0.20 |
| GPT Terra | $1.00 | $5.00 | $0.05 | $1.25 | n/a |
| Gemini Flash 3.8 | $0.75 | $3.75 | $0.075 | n/a | n/a |
| Gemini Argon | $2.00 | $10.00 | $0.10 | n/a | n/a |
| Kimi K3.1 Pro | $2.00 | $10.00 | $0.20 | $2.00 | n/a |
| GLM 5.3 | $1.40 | $4.40 | $0.26 | n/a | n/a |
List input price is the wrong thing to compare. Our measured Sonnet mix, November 1 to 8, was 0.11% fresh input, 94.72% cache reads, 4.64% cache writes and 0.53% output. When 95% of the tokens are cache reads, the read price and the write price decide the bill.
| Token type | Share of Sonnet tokens |
|---|---|
| Cache read | 94.72% |
| Cache write | 4.64% |
| Output | 0.53% |
| Fresh input | 0.11% |
We applied the prices above to that same mix. The result is the cost of one million of our real tokens on each model.
The numbers behind this figure
| Model | USD per 1M tokens |
|---|---|
| Claude Haiku 5.5 | 0.02 |
| Gemini Flash 3.8 | 0.13 |
| GPT Terra | 0.13 |
| Gemini Argon | 0.24 |
| Claude Sonnet 5.5 | 0.27 |
| GLM 5.3 | 0.34 |
| Kimi K3.1 Pro | 0.34 |
| Claude Opus 5.5 | 0.53 |
This is why the cheaper open-weight models we tried did not help. Kimi and GLM have a lower or equal list input price, but 95% of our tokens are cache reads and their read prices are not lower than Sonnet’s ($0.20 and $0.26 against $0.10 per million). The input price barely applies. Terra and Flash were cheaper, and we read that next to public benchmark scores (Artificial Analysis via OpenRouter, November 8) before choosing models. On those, Terra scored 38.2 at default effort against 30.8 for Sonnet, at half the cost.
The result is specific to these models, these prices and our workload. It says nothing about open-weight or Chinese models in general. It also uses the 5-minute write price. At the 1-hour price, Sonnet’s cost on this mix rises by about 10% and the gaps narrow.
Two more things that did not pay off
Context compression with the sqz plugin. sqz compresses tool output before it enters the context. We measured under 1% savings in request bytes and made no claim about spend. It also added overhead, so we removed it from the organization default on November 6. We have not audited each individual machine.
Automatic routing. We evaluated LiteLLM’s automatic router for our multi-agent work and kept explicit model selection. We could not find the written evaluation when preparing this article, so we do not state its failure mode. One general point holds without it: a prompt cache belongs to one model, so a router that changes model between turns starts a new cache each time.
The cache TTL: a setting that never reached the agent
Claude caching lasts 5 minutes by default, or 1 hour at a higher write price. We set a 1-hour TTL on the gateway around November 7 and verified it with raw API probes.
Real traffic did not use it. In the post window 99.0% of Sonnet cache-write tokens were 5-minute writes. The gateway only adds its own breakpoint when the client sends none, and OpenCode sends its own without a TTL. Not one of 38,764 OpenCode requests had a 1-hour write. For the other clients (3,059 requests, $23) the 1-hour premium cost $4.61 and saved $0.10, a net $4.51 extra. Our modeled estimate for OpenCode on 1 hour was $5.50 to $50.70 extra on one key’s $347 of spend, because only 2% to 3% of requests follow a pause of 5 to 60 minutes.
An earlier estimate of $265 per month in savings from the 1-hour TTL is not supported, and we no longer use it. Pick a TTL from your own idle-gap data.
What is next
- Trim old tool results in batches. Every 30 to 40 messages, never every turn, because each rewrite of the prefix is a cache write at 25 times the read price. We estimate 8% to 12% on long sessions.
- Set the gateway TTL to 5 minutes for the clients that do use it, about $4.50 a week.
- Run a one-week A/B test on one key to isolate the single measures, since the first week gave us only the combined effect.
Takeaways
- Count the bytes in your request first. Half of ours was tool definitions.
- Check cache pricing on every model row. One missing price made a model look 4.5 times too expensive.
- Compare one model against itself. Fleet spend moves with model mix and with how much work you add.
- Compare models on your own token mix. At a 95% read share the read and write prices matter more than the input price.
- Treat a setting as unproven until real traffic confirms it.
For related work, see how we cut a monthly Make.com bill with AI automation and our multi-agent SEO content system.
