Illustration: one orange thread goes through a small loom and joins several black-outline threads in a woven strip to show contextual transformation. — A token's vector gains meaning from the context around it.

The Life of a Token Inside a Transformer

Part 1 of 5 The Life of a Token Across the LLM Stack

Wenbo Ji 31 May 2026 15 min read

In brief

A Transformer maps token IDs to vectors. Each layer has attention and an MLP. Attention reads permitted context positions. The MLP updates each token's vector without cross-position reads. The KV cache keeps previous keys and values for decode, which does not calculate them again.

Attention uses information from permitted positions. The MLP and RMSNorm update each token without cross-position reads.
RoPE encodes position through query/key rotations. A longer input alone is not sufficient for context extension, which needs methods such as YaRN.
KV-cache size is approximately 2 x layers x K/V heads x head dimension x tokens x bytes. GQA and long prompts change serving memory.

A token becomes a context-aware representation before the model calculates logits.

The easiest way to misread a language model is to imagine that it reads text as a person does. Before the Transformer receives the prompt, the text becomes token IDs. An embedding lookup maps those IDs to an S x H matrix of vectors. The model updates each vector, uses information from permitted positions and calculates next-token scores. These scores are logits.

This post follows those vectors through one forward pass. This path shows the computational work, the causes of latency and the reason for a key/value (KV) cache.

A multilayer perceptron (MLP) transforms the vector for each position. It does not read other positions. Attention lets a position read information from other positions permitted by the mask. Attention reads previous positions. Each generated token adds computation and also leaves keys and values that the service must keep.

1. Map text to token IDs and vectors

Tokenization comes first. A tokenizer maps text to integer IDs from a finite vocabulary. Its pieces may be words, word fragments, punctuation, bytes, control tokens or other learned units.

The model receives this sequence, not the original string:

\mathrm{text} \rightarrow \mathrm{token\ IDs}\ x_{1:S} \rightarrow X = E[x_{1:S}] \in \mathbb{R}^{S \times H}

Each ID selects one row of a learned embedding table. A model with hidden size 4096 uses a 4096-dimensional vector for each token. Other models use different widths. The ID only selects the row. The Transformer uses the vector for its calculations.

Special tokens identify turns, tool calls, fill-in-the-middle regions, generation endings and other controls. Prompt formatting and tokenization make this structured sequence before the first layer runs.

Position information is also necessary. Plain self-attention is permutation-equivariant. A permutation of the inputs gives the same permutation of the outputs unless position information or relative-position biases break that symmetry. LLMs often use rotary position embeddings (RoPE). Long-context variants may add extensions such as YaRN. Long context also increases attention work, cache storage and serving pressure.

At the tensor level, S token IDs become an S x H activation matrix. Positional information and masks specify the permitted interactions between positions.

This shape connects model choices to systems costs. Longer prompts increase S. Wider models increase H. More layers do the work again. Each change adds attention work, MLP work, activation movement and KV-cache memory.

Tokenization also has surprising effects. Rare words, code, URLs, numbers or unusual punctuation can give many tokens from a string that looks short. Two prompts with similar character counts can have different costs for prompt processing, or prefill. For this reason, production systems count tokens, not characters. Models calculate with tokens, caches keep tokens, and schedulers budget in tokens.

2. Per-token work and attention

A decoder-only Transformer has a stack of similar blocks. Their exact design varies with the model family. One question still explains inference cost: which operations combine positions, and which transform each position without cross-position reads?

Fig. 1 shows a representative pre-normalized decoder. Many decoders use RMSNorm, RoPE, grouped-query attention and gated MLPs. Other decoders use LayerNorm, a different residual order, sparse or local attention, mixture-of-experts layers or different positional encodings. The distinction between per-position and cross-position work helps examine all these variants.

A block has this structure:

hidden state
  -> normalization
  -> attention
  -> residual add
  -> normalization
  -> MLP
  -> residual add
  -> next hidden state

Fig. 1. A pre-normalized decoder block has two residual sublayers: attention and a gated MLP. Dashed residual edges supply each sublayer's unchanged input. Full interactive figure

Root mean square normalization (RMSNorm) rescales each vector by its root mean square. This keeps hidden-state magnitudes in a range where deep computation stays stable. RMSNorm does not combine different positions. With 1,000 tokens, it rescales 1,000 vectors independently. Elementwise operations around the MLP also use one position at a time.

The MLP also updates each token. One gated variant is GeGLU, which gates with a Gaussian error linear unit (GELU). Other model families use Swish-gated linear units (SwiGLU) or different activations:

\mathrm{MLP}(x) = W_2\left(\mathrm{GELU}(W_0x) \odot W_1x\right)

The activation and gate affect model quality. For cost, the MLP is mostly matrix multiplication. It expands the hidden vector into a wider space, applies a nonlinearity and gate, then projects the vector to its previous dimension.

In many dense decoder blocks, the MLP is one of the largest arithmetic costs. It has no attention matrix and no rule for access to other token positions. Width causes its cost: the expansion and projection of each vector require wide matrix multiplications (matmuls).

Attention is where tokens exchange information.

3. Attention across permitted positions

Self-attention first calculates queries, keys and values:

Q = XW_q,\qquad K = XW_k,\qquad V = XW_v

Attention then scores permitted query/key pairs and calculates a weighted mix of values:

\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)V

Fig. 2. Q and K give scaled scores. Masking and softmax give attention weights from these scores. V supplies the values on a separate branch. Full interactive figure

The query for a position describes the information to find. Each key describes the information available from a position. Each value contains the information to combine. The query/key score determines the contribution of each permitted value to the output. Softmax converts the permitted scores to weights with a sum of one.

Multi-head attention (MHA) runs several of these comparisons in parallel subspaces, called heads. For inference, the question is the number of separate key/value heads necessary for those query heads. Multi-query attention (MQA) and grouped-query attention (GQA) give different answers:

Attention design	Query heads	K/V heads	State kept for each cached token
MHA	$H_q$	$H_q$	separate K/V for every query head
MQA	$H_q$	1	one shared K/V set
GQA	$H_q$	$H_{\mathrm{kv}}$ between 1 and $H_q$	one K/V set per query group

If layers, head dimension and dtype stay fixed, cache bytes scale with $H_{\mathrm{kv}}$ . Fewer stored heads leave space for more history or more requests. They also change the model. For this reason, this is a quality decision and a memory decision.

The GQA paper converts MHA checkpoints with an average of K/V projections in each group. It then trains the model briefly ("uptraining"). Its experiments on group count and conversion method show that shared K/V heads change the model. A memory allocator cannot add this feature after training. The reported quality/speed tradeoff comes from T5 encoder-decoder experiments.

Other decoder models need their own checks. The serving engine must use the attention layout used to train the checkpoint.

GQA and paged KV memory solve different problems. Paged KV memory keeps the cache in fixed-size blocks, as vLLM's PagedAttention does. GQA decreases the quantity of history state that the model calculates. Paging changes the engine's allocation of that state. These two methods can help the same request.

Sequence length has its largest effect in attention. During prefill, each prompt position attends to its permitted history. Attention work increases with the prompt. During decode, the model calculates only the newest query, key and value. It still reads all the previous keys and values. This changes much of decode attention's cost from arithmetic to memory access.

The model projects the attention output to the hidden size and adds it through a residual connection. The vector then contains information from permitted previous positions.

This distinction helps explain inference. An MLP changes the features of one position without cross-position reads. Attention supplies information from the prompt.

4. Permitted context with masks

The mask specifies the positions that a token is permitted to read. A causal mask lets token i read token j only when j <= i. It prevents access to future positions. The model is trained and used to predict the next token from previous tokens.

With packed training data, segmentation masks may also prevent attention between tokens from different documents. Without these masks, packing could leak information between unrelated examples.

The mask specifies context, not the adjacency of tokens in memory. Two packed documents can share one tensor while their masks prevent cross-document attention. Their hidden states then exchange no information.

Long-context models may use more structured masks. Some layers use local attention. A token in those layers reads only a recent block. Other layers use global attention across the full sequence. A position may not have access to all previous tokens in each layer, but global layers give periodic long-range access. This combination changes cost and the information available at each position.

Within the mask, the attention weights determine the contribution from each permitted state.

5. Encode position and extend context

Attention compares queries with keys. Without position information, the model would have trouble with the distinction between the same token early and late in a prompt. Positional encoding supplies that sequence information.

RoPE rotates query and key coordinates as a function of position. For a two-dimensional coordinate pair with rotation $R_p$ at position $p$ , the dot product obeys

(R_p q)^\top(R_j k)=q^\top R_{j-p}k.

The relative distance between the two positions is part of the attention score. Different coordinate pairs rotate at different frequencies. These frequencies give the model several scales to distinguish positions. The model learned these rotations only for distances in its training data. An increase in permitted input length alone does not teach the model to use longer distances.

Context extension maps new distances to the positional range that the model learned. Uniform position interpolation compresses each frequency in the same way. YaRN, especially Sections 3.2–3.4, treats frequency bands differently and adds attention scaling. Its evaluations examine perplexity and retrieval. A larger window still needs tests on the task that will use it.

Position also affects the cache. If the positional transformation changes as the sequence grows, keys with the previous rotation convention cannot simply be read with the new convention. YaRN's dynamic-scaling discussion identifies this implementation issue. A deployment needs a consistent convention or a cache format that can apply the current transformation correctly.

Long context is more than a longer input array. It increases attention work, KV-cache memory and the prefill time before the first output token. It also changes batching. One long prompt can use far more compute and memory than a batch of short prompts. It can change the benefit from prefix caching, chunked prefill and scheduling policies. Streaming video generators have similar resource demands when they reuse a rolling KV cache across generated frames.

A large context window does not make all tokens equally cheap or equally relevant. Distant tokens still occupy cache and affect attention, even when the answer depends on a small part of the prompt. At the model level, long context is a capability. At the serving level, it is a resource cost.

6. Project hidden states to logits and sample

After the last block, each position has a hidden state. For next-token prediction, the model usually uses the hidden state at the last position. It projects that state to vocabulary-sized logits:

\mathrm{logits}_t = h_t W_{\mathrm{vocab}}

If the vocabulary has 128,000 entries, this projection gives 128,000 scores. Sampling logic selects a token according to temperature, top-p, top-k, greedy decoding, grammar constraints or other decoding rules.

The system appends the selected ID to the sequence. This ID becomes input to the next step. This loop is autoregressive generation. It lets each new token depend on everything generated before that step. The cost is one more forward pass for each generated token.

Sampling rules do not change the Transformer block. They change the appended token and the generation stop condition. For this reason, decoding settings also affect serving cost. A request that stops after ten tokens releases its cache quickly. A request that generates a thousand tokens increases its cache and returns to the scheduler many more times.

7. KV-cache contents and size

During prefill, the model calculates keys and values for each prompt token. During decode, each new token must attend to all previous tokens. The calculation of keys and values for the full history at each step would waste much work.

In a standard causal decode pass, previous keys and values do not change when the system appends a new token. The system can calculate them one time and keep them. This statement assumes the same model parameters, positional convention and prefix. A change to previous context creates a different cache-validity problem.

Those stored keys and values are the KV cache.

For a single request, the sequence is:

prompt tokens -> compute K/V -> store cache
new token -> compute new K/V -> append cache
next token -> read old K/V + append new K/V

A serving system makes this more difficult. Each active request holds its own cache. Longer prompts and longer generations keep more entries. Concurrent users compete for the same GPU memory. Prefix caching can use entries again when prompts share the same opening tokens. vLLM's PagedAttention keeps KV cache in fixed-size blocks, which reduces fragmentation compared with large contiguous allocations.

Prefill and decode use the cache differently. Prefill is the expensive setup step. It writes keys and values for each prompt position. Each decode step writes keys and values for the newest position and reads those of the full history. For an unchanged causal prefix, the previous entries stay valid after the system appends a new suffix token.

Parameter count alone does not specify serving cost. Two models with similar weight sizes can operate differently with long-context traffic. Layers, K/V heads, head dimension and dtype set the bytes kept for each token. Active sequence length sets the number of tokens for storage. The architecture fixes the KV-cache bytes for each token before the scheduler receives a request.

For an approximate memory estimate, each cached token keeps both keys and values across layers:

\mathrm{KVBytes} \approx 2 \cdot L \cdot H_{\mathrm{kv}} \cdot d_{\mathrm{head}} \cdot S \cdot b

Here $L$ is the number of layers, and $H_{\mathrm{kv}}$ is the number of key/value heads. The head dimension is $d_{\mathrm{head}}$ . Active sequence length is $S$ . The scalar size in bytes is $b$ .

Formula diagram: KV-cache bytes are approximately 2 × L × Hkv × dhead × S × b for one private (unshared) full-attention sequence. — Fig. 3. KV-cache size increases with layers, K/V heads, head dimension, cached tokens and dtype bytes. The factor two counts K and V. This is a capacity estimate for one private (unshared) full-attention sequence.

Transformer architecture connects to serving architecture through the cache. Attention requires keys and values from previous tokens. The serving engine selects their memory locations, addresses and the next request to receive memory. The bytes for each cached token help determine the number of concurrent requests.

The token-to-logits loop

Fig. 4. Tokenization gives IDs, embeddings give vectors, and Transformer blocks add context. Vocabulary projection and sampling give a new token ID. Each step embeds the newly appended ID, and no single token travels unchanged through the loop. Full interactive figure

The Transformer view gives systems constraints. Prefill processes a full prompt at one time. Decode adds one token for each step and reads cached history. These phases use different resources, as the series explains for prefill and decode across the full serving stack.

The KV cache exists because autoregressive attention can use previous keys and values again. Long context increases compute and memory pressure. Attention and MLP projections are mostly matmuls. Before the serving engine, the next chapter examines the matrix multiplications behind these projections and the effect of their shapes on GPU use.

Key takeaways

A Transformer calculates with token-indexed vectors, not raw language strings.
MLPs update each token vector. Attention lets tokens read from permitted positions.
Masks and positional encodings determine the permitted context and its computational cost.
The KV cache prevents repeated calculation of previous keys and values during decode.
The cache that makes decode practical also becomes a serving-system memory constraint.

Citation

Use this citation:

Ji, Wenbo. “The Life of a Token Inside a Transformer”. fusheng-ji.github.io (May 2026). https://fusheng-ji.github.io/blog/posts/life-of-a-token-transformer/

Or use the BibTeX entry:

@article{ji2026lifeofatokentransformer,
  title = {The Life of a Token Inside a Transformer},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {May},
  url = {https://fusheng-ji.github.io/blog/posts/life-of-a-token-transformer/}
}

References

Aleksa Gordic, Inside the Transformer: The Life of a Token.
Vaswani et al., Attention Is All You Need.
Zhang and Sennrich, Root Mean Square Layer Normalization.
Shazeer, GLU Variants Improve Transformer.
Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding.
Peng et al., YaRN: Efficient Context Window Extension of Large Language Models.
Shazeer, Fast Transformer Decoding: One Write-Head is All You Need.
Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.

Note: This blog was drafted and polished with assistance from ChatGPT. Its sources include Aleksa Gordic's tutorials and the cited primary papers.

LLM Inference

Updated 08 November 2026