Token Budgets: Why Cost, Latency, and Context Quality Move Together

From Token to Transformer - 02 / 10

"The prompt is only a few pages. Why is the request still slow?"

Pages, words, and characters are not the units the model processes. A language model receives and generates token sequences. In practice, tokens become a measure of length, computation, capacity, and often price. Calling them a different kind of word count misses most of their engineering impact.

One request contains two different clocks

Inference latency is useful to split into two phases.

Prefill is the work required to read the existing input and build the internal state for that context. Longer inputs generally make this phase heavier. Standard full self-attention compares many pairs of positions, so compute and memory pressure rise quickly with sequence length. Caching, sparse patterns, and optimized kernels can reduce the practical cost, but they do not make long inputs free.

Decode is the autoregressive phase. The model emits one token, appends it to the sequence, and predicts the next. Generating 2,000 tokens takes a fundamentally different amount of serial work than generating 20. A KV cache avoids recomputing the entire history at every step, but generation is still sequential.

When a system is slow, first determine whether time is being spent reading the prompt or producing the answer. A single vague token limit cannot diagnose both.

The bill includes more than the user's words

Many model APIs price input and output tokens separately, and some distinguish cached input. The metered sequence can include:

system instructions and tool schemas;
conversation history;
documents inserted by retrieval;
the current user message;
generated output;
templates and special tokens hidden by the interface.

A one-line user request may carry tens of thousands of tokens of history and tools. Measuring only the visible text often ignores the dominant cost.

The context window is shared capacity

A context window is not a maximum amount of source material. It is the total token budget for everything used in the request. Conversation history, retrieved passages, code, instructions, and output allowance all compete for the same space.

Near the limit, a system must truncate, summarize, split the task, or reject the request. Silent truncation is especially risky: the answer can remain fluent even after an important constraint or piece of evidence has disappeared.

This is the main way token count affects quality. One additional token does not make the model less capable. Low-value material displaces facts, constraints, and output space that could change the answer.

Batching creates a hidden cost

Inference servers combine requests into batches to use hardware efficiently. Sequences with different lengths often need padding or more sophisticated packing.

If a 200-token request is naively batched with an 8,000-token request, much of the short request's allocated work can be padding. Production systems use length buckets, continuous batching, and packed attention to reduce this waste.

Token efficiency is therefore both a content problem and a scheduling problem. The same requests can produce different throughput under a different batching policy.

Optimize information density, not brevity

"Make every prompt short" is not a sound rule. Aggressive compression can remove edge cases and force the model to guess. A better objective is to make each part of the context earn its place:

Keep facts and constraints that can change the result.
Remove repeated background and ceremonial wording.
Retrieve evidence for the current question, not every vaguely related passage.
Convert long histories into verifiable state rather than replaying them forever.
Set output length to match the task instead of asking for maximum detail by default.

"Analyze this comprehensively and in as much detail as possible" increases output without defining quality. "Compare three options by implementation cost and failure risk, in no more than 120 words each" defines both the task and its budget.

A practical way to allocate context

Split the available window into four buckets:

text
fixed overhead: system prompt, tools, templates
task input: question, files, retrieved evidence
working margin: space needed for intermediate reasoning and state
output budget: maximum useful response length

Measure the fixed overhead first, then set task-specific limits. At minimum, log input tokens, output tokens, time to first token, total latency, and truncation reason. Without these measurements, token optimization becomes guesswork.

Takeaway

Tokens connect language behavior to real resources. More input makes prefill heavier, more output extends serial decoding, and a crowded context leaves less room for decisive evidence. When a provider charges per token, the same sequence also becomes a financial cost.

Good token management is not about making everything terse. It is about reserving limited context for the information most likely to change the result.