High Token Consumption — Full Curriculum

Personal study reference, not customer-facing. A practical playbook for finding where Workers AI / AI Gateway tokens go and cutting them without cutting quality — updated and re-verified against current docs (original was dated July 2026; several facts, a deprecated model, and a major new feature — Auto Router — have changed since). Every fact below is sourced from live Cloudflare docs (full list at the bottom).

Module 1How to use this & why AI Gateway matters

Six root causes, covered in the order they usually drive real dollar problems. For each: how to spot it in AI Gateway, how to fix it, and code to drop in.

#	Root cause	Typical token waste	Fix effort
1	No response cache	20–60% of tokens	10 minutes
2	Huge system prompt resent every request	30–70% of input tokens (agent workloads)	1 hour
3	No max_tokens ceiling	10–40% of output tokens	15 minutes
4	Wrong model for the job	Cost multiplier of 10–50×	1 day (eval work)
5	Chatty conversation history	Grows linearly with turns, unbounded	2 hours
6	Retries & duplicate requests	5–20% of all tokens	10 minutes

Module 2Cause 1 — No response cache

The single fastest cost win. If any part of your app asks the model the same question more than once, every duplicate is 100% wasted spend.

Fix — dashboard toggle

1. AI → AI Gateway → your gateway → Settings
2. Toggle Cache Responses on
3. Set TTL — 3600s (1hr) is a good start. Range is 60s to 1 month. Note: a newly auto-created gateway defaults to caching OFF — don't assume it's on by default.

Fix — per-request in code

const resp = await env.AI.run(
  "@cf/meta/llama-3.1-8b-instruct-fast",
  { messages: [{ role: "user", content: prompt }] },
  {
    gateway: {
      id: "your-gateway-id",
      skipCache: false,
      cacheTtl: 3600,
      cacheKey: "optional-custom-key",   // not in the original playbook
      collectLog: true,                 // not in the original playbook
    },
  }
);

Module 3Cause 2 — Huge system prompt resent every request

The silent killer for agentic workloads. An 8,000-token system prompt sent 500 times a day is 4M input tokens before the user says a word.

Fix — enable prefix caching with session affinity

const response = await env.AI.run(
  "@cf/moonshotai/kimi-k2.6",
  {
    messages: [
      { role: "system", content: LONG_SYSTEM_PROMPT },  // static, front-loaded
      { role: "user",   content: userQuery },           // dynamic, at the end
    ],
  },
  { extraHeaders: { "x-session-affinity": "ses_" + sessionId } }
);

• Do — place static content (system prompts, tool definitions) at the beginning
• Do — put user-specific/dynamic content at the end
• Don't — put timestamps at the top of your system prompt; a single changing token invalidates every downstream cached token

Module 4Cause 3 — No max_tokens ceiling

Without max_tokens, the model generates until it decides to stop. On expensive models, a runaway response can cost 40× what the actual answer needed.

const resp = await env.AI.run(
  "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
  { messages: [...], max_tokens: 500, temperature: 0.3 }
);

Module 5Cause 4 — Wrong model for the job

Most workloads don't need a 200B-parameter model. Prices differ by 10–50× across the catalog.

Current verified Workers AI pricing CF Docs · Oct 2026

Model	Input $/M	Output $/M	Cached $/M	Good for
llama-3.3-70b-instruct-fp8-fast	$0.293	$2.253	—	General reasoning, RAG, mid-complexity
glm-4.7-flash	$0.060	$0.400	—	Fast classification, routing, tool-calling
gpt-oss-20b	$0.200	$0.300	—	Cheap classification/routing alternative
glm-5.3-flash	$0.150	$0.500	$0.030	New — approaches larger-model coding/agentic quality at a fraction of the cost
glm-5.2	$1.400	$4.400	$0.260	Long-context reasoning, complex agents
glm-5.3	$1.400	$4.400	$0.260	New, 1M context agentic coding
kimi-k2.6	$0.950	$4.000	$0.160	New frontier model, replaces deprecated kimi-k2.5
kimi-k2.7-code	$0.950	$4.000	$0.190	Code generation, tool-use agents (now has concrete pricing — previously listed as "varies")
nemotron-3-120b-a12b	$0.500	$1.500	—	New, hybrid Mamba-transformer, built for multi-agent

Fix — model tiering pattern (route by task complexity)

// 1. Cheap classifier call
const route = await env.AI.run(
  "@cf/zai-org/glm-4.7-flash",
  {
    messages: [
      { role: "system", content: "Classify this request as one of: SIMPLE, COMPLEX, CODE. Respond with only the word." },
      { role: "user", content: userQuery },
    ],
    max_tokens: 5,
  }
);

// 2. Route to the right-sized model
const model = route.response.includes("CODE")    ? "@cf/moonshotai/kimi-k2.7-code"
            : route.response.includes("COMPLEX")  ? "@cf/meta/llama-3.3-70b-instruct-fp8-fast"
            :                                        "@cf/zai-org/glm-4.7-flash";

const answer = await env.AI.run(model, { messages: [...] });

Module 6Cause 5 — Chatty conversation history

Naive chat implementations send the entire conversation history every turn. Input tokens grow linearly, cost grows quadratically. Users don't feel it until the bill arrives.

Fix — cap the window with a rolling summary

const RECENT_TURNS = 6;
const older = history.slice(0, -RECENT_TURNS);
const recent = history.slice(-RECENT_TURNS);

let summary = "";
if (older.length > 0) {
  summary = await summarizeOnce(older);  // cheap model, ~300 tokens out
}

const messages = [
  { role: "system", content: SYSTEM_PROMPT },
  ...(summary ? [{ role: "system", content: "Prior context: " + summary }] : []),
  ...recent,
  { role: "user", content: userQuery },
];

Module 7Cause 6 — Silent retries & duplicates

Client retries on timeout, frontend double-clicks, queue redelivery — all bill twice. 5–20% of AI spend on undisciplined codebases is duplicates.

Fix — rate limiting as the seatbelt

AI Gateway Settings → Rate Limiting → set a per-token or per-IP cap. Won't fix bad code, but stops a runaway loop from becoming a runaway bill.

Fix — centralize retries at the gateway

AI Gateway Settings → Retry Requests → configure backoff (current defaults: up to 5 retries, 0.1–60s delay, Constant/Linear/Exponential backoff). Then remove the retry loop from client code — one place to reason about retries beats a dozen scattered implementations.

Module 8Settings map & what's new since July

Which setting fixes which cause

Setting	Fixes cause	Recommended default
Collect Logs	All 6 — diagnosis foundation	ON
Cache Responses	#1, #6	ON, TTL 3600s (off by default on new gateways)
Rate Limiting	#6 — blast-radius cap	Start at 100 req/min per token, tune
Spend Limits	All — hard budget stop	Set to 120% of expected monthly; scope by cf.user_id or custom metadata
Retry Requests	#6 — centralize retries	2 retries, exponential backoff
Authenticated Gateway	Stops unauthorized traffic from spending your tokens	ON

Metadata & per-user limits — a key name correction

await env.AI.run(
  "@cf/mistral/mistral-7b-instruct-v0.1",
  { prompt: "..." },
  { gateway: { id: "gateway_id", metadata: { team: "AI", user_id: 12345 } } }
);

What's genuinely new since July 2026

Feature	Why it matters for cost
Auto Router (cloudflare/auto) New	Built-in difficulty-aware routing to the cheapest capable model, with session affinity to preserve prompt-cache hits — see Module 5
User Insights — "Potential Savings" view New	Directly flags which requests could use a cheaper/faster model without quality loss — free, zero setup, probably the single most relevant new cost feature
Workers AI + AI Gateway billing unification New	Workers AI can now bill from prepaid AI Gateway credits; frontier models get 50 RPM (vs. 20 RPM standard) when paid via credits — changes capacity planning math
Custom Costs — cache-read/write rates New	Needed if you've negotiated cache pricing directly with a provider
Web Search API Beta	Lets agents ground responses instead of burning tokens on hallucination-correction loops

Module 9Field notes — Dynamic & Fallback Routing (Beta)

Hands-on notes from Fallback and Dynamic Routing. Live demo: cf-demo-app.dustinburke23nc.workers.dev/ai-gateway.

What worked (as of original testing)

Bugs flagged at the time — retest live before relying on these

• Budget Limit node reportedly accepted only whole-dollar integer values, not fractional amounts — made it impractical for a live click-demo at Workers AI prices.
• Configuring a fallback model to fire when a Rate Limit node's threshold was exceeded reportedly kept reverting on save — hitting the limit blocked the user instead of rerouting.
• With Cache Responses ON, identical prompts reportedly bypassed the Rate Limit node entirely (served from cache, counter never incremented).

How to position this today

Customer question	Honest answer
"Can we set a hard dollar cap?"	Yes — use Spend Limits (now GA) as the real hard cap on the platform. Use Dynamic Routing's Rate Limit node for request-count guardrails specifically.
"Can we downgrade to a cheaper model when a rate limit hits?"	Verify live — this had a known UI bug as of mid-2026. If still broken, position it as "we block the abuser at the gateway and save the LLM cost" instead, which is arguably the better cost story anyway.
"Does caching still work alongside routing?"	Yes, and that's desirable — cache hits cost nothing. Confirm live whether cached requests still bypass rate-limit counters before promising exact behavior.
"Is this production-ready?"	Fallback Routing: yes. Dynamic Routing: treat as mature but verify current UI behavior before a live customer demo, given known July-era rough edges.

Module 10Quick-start checklist & honest caveats

Honest caveats — what this playbook won't do

• Semantic cache doesn't exist yet. Two differently-phrased questions are two cache entries. Still roadmap-only.
• Model tiering requires eval work. Don't ship a model swap without a labelled test set — or lean on Auto Router instead of hand-rolling it.
• Prefix caching only helps supported models. Check the model page on developers.cloudflare.com/workers-ai/models per-model — glm-4.7-flash notably does not have a cached-token tier.
• Rate limits and spend limits cause user-visible errors when they trip. Set high initially, monitor, tighten later.
• These techniques are additive, not multiplicative. Cause 1 saving 30% + Cause 3 saving 20% combines to ~44% (0.7 × 0.8), not 50%. Model with real numbers.

Self-testQuiz

Click a question to reveal the answer. No peeking before you commit to one.

1. What's the single fastest cost win covered in this playbook, and what's its real ceiling?

Turning on response caching (Cause 1). Its ceiling: caching is exact-match only — two prompts differing by a single character are two separate cache entries. Semantic caching is still roadmap-only, not shipped.

2. True or false: a newly auto-created AI Gateway has response caching enabled by default.

False — caching defaults to OFF on a new gateway. Don't assume it's already saving money without checking Settings.

3. Where does the actual cached-token count appear in the `usage` response object — and what's the common mistake?

It's nested at usage.prompt_tokens_details.cached_tokens, not a top-level usage.cached_tokens field. Checking the wrong path will always read as zero/undefined even when caching is working.

4. Which model mentioned in the original July 2026 version of this playbook has since been deprecated, and what replaced it?

kimi-k2.5, deprecated May 30, 2026 — replaced by kimi-k2.6 and kimi-k2.7-code.

5. Does `glm-4.7-flash` support prefix-caching discounts like the larger GLM models do?

No — it has no published cached-token pricing tier, unlike glm-5.2/glm-5.3/kimi-k2.6/kimi-k2.7-code. Don't claim prefix-caching savings for it.

6. What is Auto Router, and why wasn't it in the original playbook?

A built-in cloudflare/auto model string that automatically classifies task difficulty and routes to the cheapest capable model, with session affinity to preserve prompt-cache hits. It's a feature that shipped after the original July 2026 playbook was written — it replaces the need to hand-roll a classifier-based model-tiering pattern.

7. Is "Spend Limits" still a Beta feature?

No — it launched as Beta in June 2026 but no longer carries a Beta label on the current Features page; it's treated as GA.

8. If a gateway sits behind Cloudflare Access, how does per-user metadata get attached for rate limiting / spend limits — and what's the gotcha?

The authenticated user ID is automatically attached as the reserved key cf.user_id — you don't pass it yourself. The gotcha: keys starting with cf. are reserved, so a customer-supplied metadata field named cf.user_id would be silently stripped.

9. What's the actual mechanism for enabling Workers AI prefix caching on a request?

Send the x-session-affinity header with a stable session/agent identifier via extraHeaders in the env.AI.run() call options — not a dashboard toggle.

10. Name two per-request options in the `gateway: {}` object beyond `id`, `skipCache`, and `cacheTtl`.

Any two of: cacheKey (custom cache key), collectLog (boolean), metadata (object, max 5 entries).

11. What's the current correct name for the per-request header that used to be called `cf-skip-cache`?

cf-aig-skip-cache (and cf-cache-ttl → cf-aig-cache-ttl). Old names still function but are deprecated.

12. A customer asks for a hard dollar cap on their AI spend. Which feature is the right answer, and which feature is NOT the right tool for that specific job?

Spend Limits is the real hard budget cap. Dynamic Routing's Rate Limit node is for request-count guardrails, not dollar caps — don't conflate the two.

13. What were the three specific Dynamic/Fallback Routing bugs flagged in the original playbook, and what's the correct way to describe their current status?

(1) Budget Limit node only accepted whole-dollar integers, (2) a Rate Limit node's fallback-model branch kept reverting on save, (3) cached responses bypassed the Rate Limit counter entirely. Correct current framing: these can't be confirmed fixed or still-broken from docs alone — the beta UI has matured significantly since, so retest live before citing them as current in a customer conversation.

14. Why does capping conversation history (Cause 5) compound well with system-prompt prefix caching (Cause 2)?

If the system prompt stays stable and only recent turns are appended at the end, prefix caching can still match the shared, unchanged prefix — saving both on raw prompt token count (fewer history tokens) and on cached-tier pricing for what remains.

15. What's the real quantitative mistake in assuming cost-saving techniques stack additively?

They're multiplicative on the remaining cost, not additive on the percentage. A 30% reduction plus a 20% reduction compounds to about 44% combined (0.7 × 0.8 = 0.56, i.e. 44% saved), not 50%.

16. What new free, zero-setup AI Gateway feature directly flags which requests could use a cheaper model without quality loss?

The User Insights "Potential Savings" view — arguably the most directly relevant new cost-optimization feature to ship since the original playbook.

17. What changed about Workers AI frontier-model rate limits when billed through AI Gateway prepaid credits?

Frontier models get 50 RPM when paid via AI Gateway credits, versus the standard 20 RPM — a direct result of the Workers AI + AI Gateway billing unification. Relevant for capacity planning on high-throughput agentic workloads.