Reasoning Effort Is an Inference-Time Systems Knob

Series The Life of a Token Across the LLM Stack Part 5 · 6 articles

Wenbo Ji 22 July 2026 16 min read

In brief

Reasoning effort is a request-level resource policy. It changes how much internal generation a trained reasoning model may use, which can improve difficult-task quality while increasing token usage, latency, context pressure, and serving cost.

Reasoning effort, model choice, output verbosity, and a hard token limit control different parts of an inference request.
More reasoning tokens mean more generated work, but a higher effort label is neither a fixed token budget nor a quality guarantee.
Production effort routing should be chosen from task-level evaluations that measure quality, latency, token use, and cost together.

One input token splitting into low-, medium-, and high-effort reasoning paths of increasing length before reconverging into one output token. Figure 1. The same request can take a short or long path through inference before producing a visible answer. Generated with GPT Image 2.

How hidden reasoning tokens turn answer quality into a latency, cost, and capacity tradeoff.

Suppose an API returns a 140-word answer. The usage record says the request generated 1,800 output tokens.

Nothing is necessarily wrong. A reasoning model may have spent most of those tokens on internal work before producing the text the user can see. The answer is short. The request was not.

That distinction is easy to lose when reasoning effort appears as a small menu beside the model name. low, medium, and high look like writing preferences. They are closer to resource policies. The setting guides a trained model toward a different amount or style of inference-time work, and that work has to pass through the same Transformer layers, GPU kernels, memory hierarchy, and serving scheduler as every visible token.

Sebastian Raschka's overview of reasoning effort explains how model developers can create those modes through supervised fine-tuning, reinforcement learning, token costs, chat templates, or hard budgets. This post starts one layer later. Once the model supports an effort control, what does that control do to an inference system?

The useful answer is not simply "higher effort produces more tokens." More tokens are part of the mechanism, but token count alone does not tell us whether the model explored better alternatives, repeated itself, used a tool well, or got stuck. Effort changes a distribution of behavior, not a guaranteed number of steps or a guaranteed quality gain.

Reasoning effort is a request-level decision about how much inference-time work may be worth spending on an answer.

That decision belongs in a full-stack model because it affects more than benchmark accuracy. It can change when the first useful output appears, how long a request remains active, how many generated tokens are billed, how much context headroom remains, and how other requests share the same serving capacity.

1. The Visible Answer Is Not the Whole Decode

For an ordinary non-reasoning request, it is tempting to picture generation as one clean sequence: prompt tokens go in, visible answer tokens come out. Reasoning models make that picture incomplete.

A request can involve several token categories:

Input tokens describe the prompt, system instructions, tool definitions, and conversation state presented to the model.
Reasoning tokens represent model-generated intermediate work. Depending on the model and API, they may be hidden, summarized, exposed as a thinking trace, or represented as opaque reasoning items.
Visible output tokens form the answer shown to the user.
Protocol and tool tokens encode function calls, structured arguments, tool results, and control messages across one or more model turns.

Provider accounting differs, so these categories should not be treated as a universal wire format. The important systems distinction is between what the user sees and what the model generated to get there.

OpenAI's reasoning guide makes the hidden work explicit. Reasoning tokens are not returned as readable chain-of-thought, but they appear in the response usage object, occupy context-window space during generation, and are billed as output tokens. Open-weight models may instead emit a visible trace between formatting tokens such as <think> and </think>. The interface differs; both forms require autoregressive generation.

A useful accounting identity is:

N_{\mathrm{generated}} = N_{\mathrm{reasoning}} + N_{\mathrm{visible}} + N_{\mathrm{control}}

This is a diagnostic decomposition, not a promise that every provider exposes all three fields. Tool-heavy agents complicate it further because one user task can contain several model calls, tool outputs that return as new input, and reasoning between calls.

The generated-token count matters because autoregressive decoding is sequential at the output boundary. The model cannot generate step $t+1$ until step $t$ has been selected. Batching lets many requests share a GPU step, and speculative decoding may verify several proposed tokens together, but neither removes the dependency chain inside one request.

If a higher effort setting leads a request to generate more reasoning tokens, the system has more decode work to perform before the task is done. A concise final answer does not erase that work.

A pipeline showing an effort setting entering a learned policy, which adaptively generates reasoning, tool, and final-answer tokens before usage and quality are measured. — Figure 2. An effort label selects learned behavior; it does not directly set a fixed number of reasoning tokens.

2. Effort Selects Learned Behavior

An effort label works only when the model has learned what the label means.

That sounds obvious, but it rules out a common interpretation: adding "Reasoning effort: high" to an arbitrary model does not install a new reasoning mode. The same string can act as a control signal in one post-trained model and as ordinary prompt text in another.

Public model reports show several ways to teach the behavior.

OpenAI's gpt-oss release exposes low, medium, and high effort in one open-weight model family. Its chat format passes an effort signal to a model that was post-trained for reasoning and tool use. The signal is useful because training created a policy that responds to it; the words alone are not the feature.

The Qwen3 technical report describes a Thinking Mode Fusion stage. Training mixes examples with extended thinking and examples that proceed directly to an answer, then reinforces the unified model's ability to follow both modes. Qwen3 also supports an inference-time thinking budget, which is a separate mechanism: the learned mode influences behavior, while the budget constrains how long thinking may continue.

Thinking Machines Lab's Inkling offers an especially clear example of continuous conditioning. During reinforcement learning, the system varies an effort value in the system message and changes the per-token cost associated with different samples. Higher token cost rewards economical solutions; lower token cost gives the policy more room to continue. At inference time, the value selects behavior learned under those training conditions.

These examples do not prove that every proprietary model uses the same recipe. They show the shared shape of the problem:

Give the policy an effort or mode signal.
Expose it to behavior at different reasoning lengths or costs.
Reward correct answers while shaping how much work is spent getting there.
Teach the model to finish usefully when the selected mode is short.

Supervised fine-tuning can associate a label with short or long target traces. Reinforcement learning can change length penalties or token costs by mode. A hard inference budget can stop generation near a threshold. Model developers can combine all three.

The distinction between a learned mode and a hard budget is important. A learned low setting can still reason adaptively and may spend more tokens on a difficult input than on a trivial one. A hard cap does not express task difficulty; it stops generation when the budget is gone. One changes the policy. The other defines a boundary.

Three mechanism cards comparing supervised mode fusion, effort-conditioned reinforcement learning, and a hard inference budget. — Figure 3. SFT and conditioned reinforcement learning teach effort-sensitive behavior; a hard budget instead stops generation at runtime.

3. Four Knobs That Should Not Be Collapsed

Reasoning APIs now expose several controls that sound related because they all affect cost or response length. They operate at different layers.

Control	What changes	What it does not guarantee
Model choice	Weights, architecture, capability, price, and often speed	A larger model is not always cheaper at lower effort or better on every task
Reasoning effort	The learned policy for allocating internal work	A fixed reasoning-token count or monotonic quality improvement
Hard output limit	Maximum generated tokens allowed for the request	A graceful transition from unfinished reasoning to a correct answer
Verbosity	Desired detail in the visible answer	Less internal reasoning or lower total inference cost

Model choice changes the machine doing the work. Effort changes how one selected model approaches the request. A token limit is a safety rail around total generation. Verbosity shapes the answer presented to the user.

Those controls can move independently. A high-effort model can return a short answer after substantial internal work. A low-effort model can produce a long visible report because the requested output format is verbose. A request can also hit its hard output limit during reasoning and return no useful visible answer at all.

The concrete API spelling is provider-specific. In OpenAI's Responses API, a minimal request can set effort like this:

const response = await client.responses.create({
  model: "gpt-5.6",
  reasoning: { effort: "medium" },
  input: "Review this rollout plan for failure modes."
});

console.log({
  answer: response.output_text,
  reasoningTokens:
    response.usage.output_tokens_details.reasoning_tokens
});

The supported labels are model-dependent. Current OpenAI documentation lists values that can include none, minimal, low, medium, high, xhigh, and max, but not every model accepts every value. The stable design lesson is to discover capabilities from the selected model rather than building product logic around a universal enum.

The usage field is more valuable than the label alone. Two medium-effort requests may use different amounts of reasoning because the model adapts to the task. The setting is an instruction to a policy, not a reservation for an exact token count.

One inference request showing model choice selecting weights, reasoning effort entering the policy, a hard output limit bounding generation, and verbosity shaping the visible answer. — Figure 4. Model choice, reasoning effort, a hard output limit, and verbosity act at different points in the request.

4. Hidden Tokens Re-enter the Full Stack

Once reasoning tokens are treated as generated work, the earlier layers in this series reappear.

At the token lifecycle, each reasoning token is another autoregressive decision. It becomes input to later steps in the same reasoning trajectory, even when its readable content is not exposed to the application.

At the compute lifecycle, those steps run Transformer blocks and their kernels. The exact execution path depends on the architecture, batching, speculative decoding, and provider implementation, but additional generated steps do not happen for free.

At the memory lifecycle, active generation needs working context and cached state. OpenAI documents that its reasoning tokens consume context-window space. For a self-hosted model that emits a thinking trace as normal tokens, those positions participate in the sequence and KV-cache path like other generated positions. A hosted provider may manage, compress, discard, or persist internal state differently; without documentation, its allocator details should remain an unknown rather than a confident diagram.

At the request lifecycle, the request stays alive while the model reasons, calls tools, receives results, and produces the final answer. Longer active lifetimes change how the scheduler sees the workload. They can extend cache residency, increase the number of decode iterations, and widen the distribution of completion times.

A request lifecycle with separate lanes for user-visible phases, model work, tools, and resource residency across queue, prefill, reasoning, tool calls, and the final answer. — Figure 5. Reasoning effort changes the length and shape of the active request between prefill and completion.

A causal latency model for a tool-free request is:

T_{\mathrm{request}} \approx T_{\mathrm{queue}} + T_{\mathrm{prefill}} + T_{\mathrm{reasoning}} + T_{\mathrm{visible}} + T_{\mathrm{stream}}

For an agent, add tool execution and the model turns around it:

T_{\mathrm{agent}} \approx T_{\mathrm{queue}} + \sum_{k=1}^{K} \left( T_{\mathrm{model},k} + T_{\mathrm{tool},k} \right)

These are causal ledgers, not stopwatch identities. Modern systems overlap serialization, scheduling, network transfer, tool work, and GPU execution. Some models can interleave visible output with reasoning instead of completing one hidden block first. The equations are still useful because they prevent a short final answer from being mistaken for a short request.

Time to first token becomes ambiguous

Traditional streaming metrics assume the first token is useful output. A reasoning model can break that assumption in several ways.

An API may hide the reasoning trace, so the first visible answer token arrives only after substantial model work. Another system may stream a short preamble, perform more reasoning, call a tool, and then continue. An open-weight model may stream its thinking trace even though the product UI later collapses it.

Production measurement should therefore distinguish:

time to first event,
time to first visible token,
time to first useful answer content,
time to final answer,
and time spent waiting on tools.

Optimizing the first of these can make a UI feel alive without reducing total task latency. That may be worthwhile, but it is a presentation improvement, not evidence that reasoning became cheaper.

More effort changes the workload mix

Serving capacity depends on distributions, not only averages. If most low-effort requests finish after a short decode while a small group of high-effort requests runs for tens of thousands of generated tokens, the long tail affects cache occupancy, batch composition, and admission policy.

A scheduler can keep those long requests in the same continuous batches as interactive traffic, isolate them in a different queue, or send them to asynchronous capacity. Each choice trades utilization, fairness, and latency. High effort is therefore not only a model setting. At scale, it becomes a workload class.

5. Quality per Token Is the Actual Objective

It is easy to turn reasoning effort into a one-dimensional story: more thinking, better answer. The evidence is less tidy.

Many reasoning benchmarks improve as models receive more test-time compute, especially on problems where search, verification, or backtracking helps. The gains also tend to depend on the task and flatten as effort rises. A larger token budget can be wasted on a simple retrieval task, while a hard mathematical or coding problem may benefit from another attempt or a longer verification path.

Longer is not a reliable quality metric by itself. A failed trajectory may be long because the model is uncertain or repeating an unproductive approach. The 2026 paper Think Deep, Not Just Long studies this problem directly and reports that raw response length is a weaker signal than measures designed to capture meaningful internal revision. Whether that particular measure becomes standard is less important than the warning: token count is a cost measurement, not a semantic evaluation.

The production objective should be expressed per completed task:

\mathrm{utility}(e) = \mathrm{quality}(e) - \lambda_c\,\mathrm{cost}(e) - \lambda_l\,\mathrm{latency}(e) - \lambda_f\,\mathrm{failure}(e)

Here $e$ is the effort setting, and the coefficients represent the product's actual priorities. An offline code review can tolerate minutes if it catches a dangerous migration bug. A voice interface cannot. A customer-support classification should not spend high-effort tokens proving an obvious label. A research agent may justify them when a weak answer would waste hours of human work.

Token price alone is also incomplete. A smaller model at higher effort may compete with a larger model at lower effort for some tasks. The smaller model can still be slower if it generates a much longer trace. A larger model may solve the same item with fewer tokens but charge more per token. Which combination wins is empirical.

The unit worth optimizing is not cost per output token. It is cost per accepted result under a latency objective.

6. Route Effort With Evals, Not Intuition

The safest default is not "always medium" or "always high." It is a small routing policy backed by representative tasks.

Start by dividing the workload into classes that have different failure costs:

Workload class	Starting point	What to verify
Extraction, formatting, simple classification	`none` or `low`	Schema validity, latency, false positives
Bounded drafting or direct code edits	`low`	Instruction following, test pass rate, retries
Multi-step analysis and tool use	`medium`	Task completion, tool errors, reasoning tokens
Difficult debugging, planning, or review	`high`	Defect discovery, evidence quality, tail latency
Rare quality-first asynchronous tasks	Highest supported mode	Marginal gain over `high`, total cost, timeouts

These are evaluation starting points, not universal product guidance. A simple-looking request can hide a hard domain constraint, and a long prompt can ask for a trivial lookup. Prompt length alone is a poor router.

For each workload class, run the same tasks across adjacent effort levels and record:

task success or grader score,
visible answer correctness and completeness,
reasoning and total generated tokens,
time to first useful output and total wall time,
tool-call count, retries, and tool failures,
incomplete or timed-out responses,
and cost per accepted result.

An evaluation loop that groups workloads, tests adjacent effort levels, measures quality, latency, tokens, cost, and failures, then deploys the lowest passing effort or escalates one level after verification failure. — Figure 6. Production routing evaluates adjacent effort levels and escalates only when tests, evidence, or a verifier show that the cheaper path failed.

Then choose the lowest effort that stays on the quality-latency-cost Pareto frontier. If high costs twice as much as medium and changes no accepted outcomes, it is not a quality setting. It is unused budget.

Escalation can help when task difficulty is hard to predict. A system can try a cheaper pass, detect a failed test, missing evidence, low verifier score, or explicit uncertainty, then retry at higher effort. That saves compute when the first pass works.

It also has a cost. A failed low-effort attempt consumes tokens and wall time before the expensive attempt begins. For tasks that are predictably difficult or have strict deadlines, starting at the appropriate higher level can be cheaper than cascading through several modes.

Automatic routing should therefore use signals tied to the task outcome: verifier results, test failures, unresolved constraints, tool state, and remaining latency budget. Vague prompt features such as message length or the presence of words like "analyze" are easy to implement and easy to fool.

Finally, keep the selected effort in traces and dashboards. If latency or cost regresses after a model update, the team needs to know whether requests became slower at the same effort, whether the router shifted traffic toward higher modes, or whether the model began using more reasoning tokens within a mode. Without that label, three different changes collapse into one rising average.

Reasoning Effort Is Serving Policy

The setting begins as an input to the model, but its consequences spread across the system.

The Transformer performs more or less generated work. Kernels execute that work. Context and cached state remain live while the request progresses. The serving engine decides how that request shares capacity with shorter traffic. The product pays for the generated tokens and the time the user waits.

This is why the best effort setting cannot be read from a model card. It depends on the task, the cost of a wrong answer, the latency budget, the selected model, and the rest of the serving workload.

Higher effort is useful when it buys a measured improvement. Otherwise it is simply a longer route to the same answer.

Key Takeaways

Reasoning effort selects behavior learned during post-training; it is not a fixed token reservation or a generic prompt trick.
Model choice, reasoning effort, hard output limits, and visible-answer verbosity control different parts of a request.
Hidden reasoning still creates generated work, context pressure, active request time, and billed output tokens.
Higher effort can improve hard tasks, but gains are task-dependent and may saturate; token length alone is not a quality signal.
Production systems should route effort from evals and measure quality, latency, token use, failures, and cost per accepted task together.

Citation

Please cite this article as:

Ji, Wenbo. “Reasoning Effort Is an Inference-Time Systems Knob”. fusheng-ji.github.io (July 2026). https://fusheng-ji.github.io/blog/posts/reasoning-effort-inference-systems/

Or use the BibTeX entry:

@article{ji2026reasoningeffortinferencesystems,
  title = {Reasoning Effort Is an Inference-Time Systems Knob},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {July},
  url = {https://fusheng-ji.github.io/blog/posts/reasoning-effort-inference-systems/}
}

References

Sebastian Raschka, Controlling Reasoning Effort in LLMs.
OpenAI, Reasoning models.
OpenAI, Model guidance.
OpenAI, Introducing gpt-oss.
Qwen Team, Qwen3 Technical Report.
Thinking Machines Lab, Inkling: Our Open-Weights Model.
DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.
Chen et al., Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens.

Note: This blog was drafted and polished with assistance from ChatGPT, based on my reading of the cited reasoning-model reports and API documentation. The hero illustration was generated with GPT Image 2; the technical diagrams are original SVGs.

#llm-inference