How hidden reasoning tokens turn answer quality into a latency, cost, and capacity tradeoff.

In this example, an application programming interface (API) returns a 140-word answer. The usage record gives 1,800 generated output tokens.

A large language model (LLM) with reasoning capability may use most of those tokens for internal work before it writes the visible answer. A short answer can follow a long request.

Near a model name, low, medium and high can look like writing preferences. They guide the quantity or type of work that a trained model does at inference time. That work still runs through Transformer layers, graphics processing unit (GPU) kernels, memory and the serving scheduler.

Sebastian Raschka’s overview shows how supervised fine-tuning (SFT), reinforcement learning (RL), token costs, chat templates or hard budgets can make effort modes. I start with a model that already supports the control and examine its effects on inference.

Token count measures only part of that behavior. More tokens might explore a better approach, repeat a failed one, use a tool or get stuck. An effort mode changes model behavior. It does not set the number of steps or guarantee quality.

Reasoning effort guides the quantity of model work for a request’s answer.

The choice can change accuracy. It also changes the first useful output’s time, the request’s active time, billed tokens, context left and capacity for other requests.

1. Count generated work beyond the visible answer

A reasoning request can generate more than the text shown to the user.

A request can involve several token categories:

• Input tokens encode the prompt, system instructions, tool definitions, and conversation state supplied to the model.
• Reasoning tokens represent model-generated intermediate work. Different models and APIs may have different representations. The tokens may be hidden, summarized, shown as a thinking trace, or represented as opaque reasoning items.
• Visible output tokens make the answer shown to the user.
• Protocol and tool tokens encode function calls, structured arguments, tool results, and control messages across one or more model turns.

Providers account for these categories differently. Use these categories to compare visible content with generated work.

OpenAI’s reasoning guide says that reasoning tokens are in usage, occupy context during generation and are billed as output tokens. Their readable chain-of-thought is hidden. Open-weight models may show traces between tokens such as <think> and </think>. Autoregressive generation is necessary in the two cases.

For diagnosis, use these categories for generated tokens:

These terms are not necessarily separate API fields. One agent task can also have several model calls. Tool results can become new input, with reasoning between calls.

In autoregressive generation, step $t+1$ uses the selections from step $t$. Batching shares a GPU step across requests. Speculative decoding can verify several proposed tokens together. The dependency in each request stays.

If higher effort generates more reasoning tokens, these tokens add decode work even when the final answer stays concise.

2. Compare learned modes with forced stopping

An effort label has an effect only if the model learned what the label means.

The string “Reasoning effort: high” can control one post-trained model. For a different model, it can be ordinary prompt text. A model must learn the mode for the signal to have its intended effect.

The table gives three mechanisms for the quantity of reasoning. They come from four reports: gpt-oss, Qwen3, Inkling and s1.

Qwen3, Section 4.3, uses thinking examples together with non-thinking examples. Its controller can insert a stop-thinking instruction at a threshold so the model answers from the accumulated trace. That is different from an arbitrary API output cap that may stop without a useful answer.

In Inkling, RL uses different effort signals and costs for each token. Higher token cost favors economical successful trajectories. gpt-oss instead gives an accessible example of a model interface with trained effort modes.

A learned low mode can use more tokens on a difficult item. A hard cap only sets a limit on generation. s1 uses budget forcing at runtime. It ends thinking at a maximum budget. To extend thinking, it suppresses the end-of-thinking token and appends “Wait”. Its ablations compare budget forcing with other length controls, such as token-count conditioning and rejection sampling. In those tests, budget forcing gave the best control and scaling. Excessive continuation can loop, and context stays finite.

Training changes the reasoning paths that a model tends to select and the time when it stops. Runtime controls determine which paths may continue. Token use and quality still depend on the task.

3. Model, effort, limits and verbosity

These API controls have effects on cost or response length in different ways:

Model choice selects the machine. Effort guides its approach to the task. A token limit puts a limit on total generation. Verbosity changes the visible answer’s detail.

These controls can change independently. A high-effort model can give a short answer after substantial internal work. A low-effort model can give a long visible report because the requested format has much detail. A request can also reach its hard output limit during reasoning and give no useful visible answer at all.

The API syntax is specific to the provider. In OpenAI's Responses API, a request can set effort as follows:

const response = await client.responses.create({
  model: "gpt-5.6",
  reasoning: { effort: "medium" },
  input: "Review this rollout plan for failure modes."
});

console.log({
  answer: response.output_text,
  reasoningTokens:
    response.usage.output_tokens_details.reasoning_tokens
});

The example uses the gpt-5.6 alias. As of July 2026, this alias routed to GPT-5.6 Sol (snapshot gpt-5.6-sol). Its model page documents medium as the default effort. Supported labels and defaults depend on the model.

Examine usage and the label together. Two medium-effort requests may use different quantities of reasoning as the model adapts to each task.

4. Examine internal generation across the stack

You can use the same four views from the earlier chapters to examine internal generation.

In the token view, each reasoning token is an autoregressive decision. Subsequent steps use it, even when the application cannot read it.

In the compute view, these steps run Transformer blocks and kernels. Architecture, batching, speculation and provider implementation determine their execution cost.

In the memory view, working context and cached state are necessary for generation. OpenAI documents context use by reasoning tokens. If a self-hosted model emits a normal-token thinking trace, it uses the sequence’s key/value (KV) cache path. A hosted provider may compress, discard or persist internal state differently. Its undocumented allocation stays unknown.

In the request view, reasoning, tool calls/results and answer generation keep work active. Longer requests can keep cache longer and use more decode iterations. They can also widen completion-time distributions.

A tool-free request can use the series introduction’s latency breakdown:

For an agent, add tool execution and the model turns around it:

The equations give causes of latency. Real systems overlap serialization, scheduling, networking, tool work and GPU execution. Some models interleave visible output with reasoning. The equations still show that a short answer can come from a long request.

Measure first useful output separately

Traditional streaming metrics assume the first token is useful output. A reasoning model can make that assumption incorrect in several ways.

An API may hide the reasoning trace, so the first token of the visible answer arrives only after substantial model work. A different system may stream a short preamble, do more reasoning, call a tool, and then continue. An open-weight model may stream its thinking trace even though the product UI subsequently collapses it.

For production measurement, record these different times:

• Time to first event
• Time to first visible token
• Time to the first useful content in the answer
• Time to final answer
• Time spent waiting on tools

A first event can make the UI feel responsive while total task latency stays unchanged. Measure that presentation benefit. Do not treat it as cheaper reasoning.

Schedule long requests alongside short ones

A small group of requests that generate tens of thousands of tokens can have effects on cache, batches and admission. This is possible even when most low-effort requests complete quickly. Capacity depends on that distribution and its average.

A continuous-batching scheduler can put long and interactive requests in the same batches, isolate queues or use asynchronous capacity. Each has a tradeoff between utilization, fairness and latency. High-effort traffic becomes a workload class at scale.

Calculate cache size over time

Use the hypothetical model from the synthesis chapter: self-hosted full attention. It uses 128 KiB of KV for each retained position and a 4096-token prompt. Keep its illustrative decode interval of 40 ms for each generated position. The model does not include prefill, pauses, tools, prefix sharing or compression.

Generating 512 retained positions keeps the request decoding for 20.48 seconds and ends with 576 MiB of KV cache that belongs to this request alone. The decode time for 4096 generated positions is 163.84 seconds. The final cache size is 1 GiB. The final answer could be equally short in the two cases. The retained reasoning positions make the difference.

A useful quantity is cache occupancy integrated over time. With constant interval $\Delta t$, prompt length $S_0$, generated length $N_g$, and cache bytes for each position $q_{\mathrm{KV}}$,

This continuous-growth approximation gives the byte-seconds. The requests use 10.88 and 122.88 GiB·s: about 11.3 times the area for eight times the generation. Retained prompt history and a growing suffix give the relation between token cost and capacity over time.

For a stable decode workload of admitted requests, Little's law uses the admission rate $\lambda$ (requests/s) and mean active time $E[W]$. The mean number of active requests is $\bar R=\lambda E[W]$. In the same way, the mean size of the private decode cache is $\lambda E[A_{\mathrm{KV}}]$. These are averages. They do not guarantee capacity during bursts or make an overloaded queue stable. Observed traces are necessary for variable decode time and cache retention.

This model is for an ordinary self-hosted sequence. Hosted reasoning uses generation resources, but unknown cache compression, discard or retention prevents calculation of the same curve from API usage alone.

5. Compare accepted results with resource cost

The papers ask different questions about quality gained from more work. Compare their mechanisms and limits before you treat their benchmark scores as one scaling curve.

Think Deep measures where intermediate-layer token predictions converge toward the final-layer distribution. That is an internal prediction statistic, not a direct semantic grade of a written trace. Its 48-sample, 50%-retention experiments include tests of early selection on mathematical/scientific benchmarks. Appendix analysis also cautions against comparing the ratio across models or effort modes. Projecting intermediate states and scoring candidates also has a runtime cost that a deployment must include. A closed API with only usage data cannot implement that measurement directly.

I distinguish buying more work from allocating work better. A longer attempt extends one history. Parallel attempts make several histories. Early selection and verification add their own computation. Include the cost of each part. Compare results with the same task acceptance rule.

A long failed attempt may reflect uncertainty or repetition. Token count measures cost, not value. The value is in the result: a corrected error, a passed test or a correct answer.

For production, express the objective for each completed task:

Here $e$ is the effort setting, and the coefficients represent the product's actual priorities. An offline code review can tolerate minutes if it catches a dangerous migration bug. A voice interface cannot tolerate that delay. A customer-support classification should not spend high-effort tokens proving an obvious label. A research agent may justify them when a weak answer would waste hours of human work.

Token price alone is also incomplete. A smaller model at higher effort may compete with a larger model at lower effort for some tasks. The smaller model can still be slower if it generates a much longer trace. A larger model may solve the same item with a smaller number of tokens. It may have a higher price for each token. Which combination wins is empirical.

Optimize cost for each accepted result with a latency objective.

6. Select an effort level with task evaluations

I would select a small effort-routing policy from representative task evaluations, or evals. I would not give each request medium or high effort by default.

Divide the workload into classes that have different failure costs:

These are evaluation starting points. A simple-looking request can hide a hard domain constraint, and a long prompt can ask for a trivial lookup. Prompt length alone is a poor router.

For each workload class, run the same tasks across adjacent effort levels. Record these measurements:

• Task success or grader score
• Visible answer correctness and completeness
• Reasoning and total generated tokens
• Time to first useful output and total wall time
• Tool-call count, retries, and tool failures
• Incomplete or timed-out responses
• Cost for each accepted result

Select the lowest effort on the quality/latency/cost Pareto frontier. A setting is on that frontier when no other setting is better on one objective without being worse on another. If high has twice the cost of medium with no improvement in accepted outcomes, its larger budget bought no gain.

Escalation can help when task difficulty is not easy to predict. A system can try a cheaper pass. It can find a failed test, missing evidence, a low verifier score or explicit uncertainty. It can then retry at higher effort. That decreases computation when the first pass works.

It also has a cost. A failed low-effort attempt uses tokens and wall time before the expensive attempt starts. For predictably difficult tasks or strict deadlines, a higher starting level that is appropriate can have a lower cost than a sequence of modes.

Select effort with outcome signals: verifier results, failed tests, unresolved constraints, tool state and remaining latency budget. Message length or words such as “analyze” are weak substitutes.

Record effort in traces and dashboards. Latency or cost can increase after a model update. The label shows which of these three caused the change:

• Requests became slower at the same effort.
• The router sent more traffic to higher modes.
• The model used more reasoning tokens within a mode.

Without the label, all three look like one rising average.

Use effort as part of serving policy

Effort starts as a model input and changes the generated work that kernels execute. Context and cache may stay live longer. The engine shares capacity with shorter traffic. The product pays for generation and waiting.

Select effort for the task, failure cost, deadline, model and surrounding workload.

I would pay for higher effort when evaluations show an improvement worth its cost.

Key takeaways

• Reasoning effort selects behavior learned during post-training. Token use still adapts for each task, and untrained models read the label as plain prompt text.
• Model choice, reasoning effort, hard output limits, and visible-answer verbosity control different parts of a request.
• Hidden reasoning still causes generated work, context pressure, active request time, and billed output tokens.
• Higher effort can give better results on hard tasks, but gains are task-dependent and may saturate. Judge quality by accepted results, not token length.
• I recommend that production systems select effort from evals. I recommend that systems measure quality, latency, token use, failures and cost together for each accepted task.

Series navigation

• Series index: The Life of a Token Across the LLM Stack
• Part 0: LLM Inference Is a Full-Stack Systems Problem
• Part 1: The Life of a Token Inside a Transformer
• Part 2: Why Transformer Performance Is Mostly a Matmul Problem
• Part 3: The Life of a Request Inside vLLM
• Part 4: A Token's Journey Across the LLM Stack
• Part 5: Reasoning Effort Is an Inference-Time Systems Knob (current)

@article{ji2026reasoningeffortinferencesystems,
  title = {Reasoning Effort Is an Inference-Time Systems Knob},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {July},
  url = {https://fusheng-ji.github.io/blog/posts/reasoning-effort-inference-systems/}
}

References

• Sebastian Raschka, Controlling Reasoning Effort in LLMs.
• OpenAI, Reasoning models.
• OpenAI, GPT-5.6 Sol model documentation.
• OpenAI, Introducing gpt-oss.
• Qwen Team, Qwen3 Technical Report.
• Thinking Machines Lab, Inkling: Our Open-Weights Model.
• DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.
• Chen et al., Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens.
• Muennighoff et al., s1: Simple test-time scaling.

Note: This blog was drafted and polished with assistance from ChatGPT, based on my reading of the cited reasoning-model reports and API documentation.

Updated 08 November 2026