High Token Consumption

Fix — Dashboard toggle (whole gateway)

1. Cloudflare Dashboard → AI → AI Gateway → your gateway → Settings
2. Toggle Cache Responses on
3. Set default TTL — 3600 seconds (1 hour) is a good starting point. Range is 60s to 1 month. CF Docs

Fix — Per-request in code (fine-grained control)

// Workers AI binding — cache this response for 1 hour
const resp = await env.AI.run(
  "@cf/meta/llama-3.1-8b-instruct-fast",
  { messages: [{ role: "user", content: prompt }] },
  {
    gateway: {
      id: "your-gateway-id",
      skipCache: false,
      cacheTtl: 3600,   // seconds
    },
  }
);

Fix — Enable prefix caching with session affinity

Send the x-session-affinity header with a stable identifier so consecutive requests route to the same model instance and can hit the cached prefix.

// Workers AI binding
const response = await env.AI.run(
  "@cf/moonshotai/kimi-k2.5",
  {
    messages: [
      { role: "system", content: LONG_SYSTEM_PROMPT },  // static, front-loaded
      { role: "user",   content: userQuery },           // dynamic, at the end
    ],
  },
  {
    extraHeaders: {
      "x-session-affinity": "ses_" + sessionId,
    },
  }
);

Fix — Structure your prompt for maximum cache hit rate CF Docs

• Do — place static content (system prompts, tool definitions, shared instructions) at the beginning
• Do — put user-specific and dynamic content at the end
• Do — reuse tool definitions unchanged across requests in the same session
• Don't — put timestamps at the top of your system prompt. A single changing token invalidates every downstream token. If you need time context, add it to the user message.

Fix — Always set a ceiling

const resp = await env.AI.run(
  "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
  {
    messages: [...],
    max_tokens: 500,       // hard ceiling
    temperature: 0.3,      // lower = more focused, often shorter
  }
);

Fix — Model tiering pattern (route by task complexity)

Use a cheap classifier to decide which model handles the actual request:

// 1. Cheap classifier call — llama-3.1-8b-fast, ~200 tokens
const route = await env.AI.run(
  "@cf/meta/llama-3.1-8b-instruct-fast",
  {
    messages: [{
      role: "system",
      content: "Classify this request as one of: SIMPLE, COMPLEX, CODE. Respond with only the word."
    }, {
      role: "user", content: userQuery
    }],
    max_tokens: 5,
  }
);

// 2. Route to the right-sized model
const model = route.response.includes("CODE")   ? "@cf/moonshotai/kimi-k2.7-code"
             : route.response.includes("COMPLEX") ? "@cf/meta/llama-3.3-70b-instruct-fp8-fast"
             :                                       "@cf/meta/llama-3.1-8b-instruct-fast";

const answer = await env.AI.run(model, { messages: [...] });

Fix — Cap the window with rolling summary

// Keep last N turns verbatim; summarize older content once
const RECENT_TURNS = 6;
const older = history.slice(0, -RECENT_TURNS);
const recent = history.slice(-RECENT_TURNS);

let summary = "";
if (older.length > 0) {
  // Summarize in one cheap call, cache the result on user session
  summary = await summarizeOnce(older);  // llama-3.1-8b-fast, 300 tokens out
}

const messages = [
  { role: "system", content: SYSTEM_PROMPT },
  ...(summary ? [{ role: "system", content: "Prior context: " + summary }] : []),
  ...recent,
  { role: "user", content: userQuery },
];

Fix — Use AI Gateway rate limiting to cap the blast radius

AI Gateway Settings → Rate Limit Requests → set a per-token or per-IP cap. This is your seatbelt. It won't fix bad code, but it stops a runaway loop from becoming a runaway bill.

Fix — Retry policy in Gateway, not client

AI Gateway Settings → Retry Requests → configure exponential backoff at the gateway. Then remove the retry loop in your client code. One place to reason about retries beats a dozen scattered implementations.