How Transformer computation maps to GPU matrix multiplication kernels.

The same Transformer equation can run quickly or slowly. Its speed depends on the matrix shape that reaches the GPU. A prefill batch with thousands of prompt tokens gives the kernel a large, regular matrix. A decode step with a small active batch gives it only a few rows. That small matrix is harder to keep efficient, and memory movement starts to dominate.

Most of the heavy arithmetic in a Transformer is matrix multiplication, or matmul. The query, key and value projections are matmuls. The feed-forward multilayer perceptron (MLP) and final vocabulary projection also have matmuls.

This article examines how a GPU runs those matmuls: tiling, shared memory, registers, tensor cores and pipelines. A worked Roofline estimate shows whether the limit is from memory or compute. Four questions at the end help readers diagnose a slow operation.

A line, for example Q = XWq, states only the math. The kernel must also load data from memory and divide the work between thread blocks and warps. It uses tiles again, supplies tensor cores and accumulates partial results.

"Mostly matmul" is a narrow claim. Dense Transformers spend a large share of their floating-point operations (FLOPs) in projections and MLPs. Layer normalization, activations, attention scores, softmax, sampling, communication and memory allocation are also necessary for inference. Attention kernels have their own structure. When theoretical FLOPs do not turn into token throughput, the data path for matmul is one of the first places to look.

How Transformer operations become matmuls

Start with the Q/K/V and MLP equations from the Transformer article:

X holds the hidden states for a batch of tokens. The learned W matrices give queries, keys and values from those states.

The MLP also has linear projections:

Some notations put the weights on the other side. The operation stays the same: multiplication of activations and weights, an elementwise function, then another multiplication.

The final vocabulary projection is one more matmul:

With a large vocabulary, this projection can be costly, especially when few decode tokens are active. A Transformer has routing operations and feature transformations. Most of the expensive feature transformations are matmuls.

For an activation matrix of shape M x K and a weight matrix of shape K x N, the output has shape M x N:

For this model, K and N do not change. The requests give M, the number of token rows. In prefill, M can be large because the prompt's tokens run together. In decode, M is frequently near the number of active requests. Each request adds one new token for each step.

A server keeps the same weights in memory for each batch. Each batch has new activations, sequence lengths and KV-cache reads. A large prefill matmul can keep tensor cores busy. A decode matmul with few rows (small M) gives less parallel work. Weight loading, kernel launch overhead and cache reads then use a larger share of the time.

The equation stays the same, but the GPU gets a different job. Thus, one benchmark cannot summarize inference.

Give the GPU sufficient parallel work

Each output element of a matmul is a dot product. A large output matrix holds many independent dot products, and the GPU can run them in parallel.

NVIDIA GPUs have streaming multiprocessors (SMs). A programmer writes a kernel as thread blocks, and the GPU schedules those blocks across SMs. Each SM runs warps, groups of threads. Recent NVIDIA GPUs also have tensor cores.

These units make matrix multiply-accumulate (MMA) operations faster. Supported input formats include 16-bit floating point (fp16), bfloat16 (bf16) and 8-bit floating point (fp8).

A GPU does two jobs. It moves and stores data, and it does arithmetic on that data. Model equations show only the arithmetic. Performance problems frequently come from the data movement.

Measure work for each byte moved

Fast GPU code puts data in the correct place at the correct time. Global memory is large but slow to reach. The worked example below assumes high-bandwidth memory (HBM). The L2 cache is nearer to the threads. L1 cache and shared memory are nearer, and registers are nearest. Tensor cores can multiply values only after the values arrive.

A naive matmul rereads the same inputs from global memory for each output element. The arithmetic is simple, but the repeated loads make memory the bottleneck. If the kernel uses loaded values again, it avoids that traffic.

Arithmetic intensity is the number of FLOPs that a kernel does for each byte that it moves. The Roofline model uses it to give a throughput bound. At low intensity, memory bandwidth gives the speed limit, and the kernel is memory-bound. At high intensity, peak compute gives the speed limit, and the kernel is compute-bound. The crossover, the ridge point, depends on the hardware and the kernel.

Two matmuls with the same FLOP count can have different performance. Wide dimensions let the kernel use tiles again. Thin or irregular matrices expose memory and scheduling costs. Decode attention reads the key/value (KV) cache of each active sequence again. Thus, its memory traffic can dominate even though its equation looks small.

Weight-only quantization shows the same trade-off. Fewer bits for each weight decrease memory traffic and can keep more weights in on-chip cache. The gain appears only if the kernel uses the compressed format efficiently. Dequantization cost or unsupported shapes can erase it.

A tile load into fast memory pays off when many multiply-adds use the tile before eviction. High-performance matmul is mostly about this movement and reuse, more than about individual arithmetic instructions.

Use local tiles again before the next load

Tiling divides a large matmul into small pieces of reusable local work. Suppose the kernel computes:

C = A x B

The kernel divides A, B and C into tiles. A thread block loads one tile of A and one tile of B into shared memory. Its threads multiply smaller fragments and keep partial results in registers.

The kernel then advances along the reduction dimension K. It loads the next tiles and adds to the same output tile. This continues until the output tile is complete.

If the kernel uses each loaded value once, that load is expensive. If the value supplies many output elements before eviction, those elements share the load cost. Shared memory and registers let the kernel use the loaded values again. The path is:

Without reuse, an operation with many FLOPs can wait on memory. Thus, kernels select sizes at several levels: thread-block tiles, warp tiles and tensor-core instruction fragments.

Four constraints affect those selections:

• Occupancy: the number of active warps that an SM has available to hide waiting.
• Register pressure and spilling: more live values use more registers. Values that do not fit spill into slower memory.
• Bank conflicts: shared-memory accesses that compete for the same bank delay useful work.
• Coalescing: when adjacent threads read adjacent addresses, the hardware combines those accesses into fewer, efficient memory transactions.

Larger tiles make reuse better but use more shared memory and registers. If a kernel uses more registers, occupancy can decrease or spilling can occur. Higher occupancy hides latency only if the kernel supplies the tensor cores. No single tile size is best.

Thus, prefill's many rows and decode's small batches make different selections necessary. In decode, weight loading and KV-cache reads become more visible. A rough model compares arithmetic time with data-movement time:

Arithmetic intensity (written AI in the equations) is the ratio of the two quantities:

Worked example: when does a projection become compute-bound?

Use the hypothetical dense grouped-query attention (GQA) model from the synthesis article. It has 32 layers, hidden width 4096, 32 query heads, 8 K/V heads and head dimension 128. The tensors use bf16.

One query projection has $A\in\mathbb R^{M\times4096}$ and $B\in\mathbb R^{4096\times4096}$. Here $M$ is the number of token rows in this execution, not the sequence length of each request. Count a multiply-add as two FLOPs.

For an optimistic bound, read each input once. Write the output once:

Here $K=N=4096$, $b=2$ bytes for each scalar, $F$ is FLOPs and $D$ is bytes. The weights alone use 32 MiB. The number of rows that share them sets the intensity.

Assume illustrative ceilings of 200 TFLOP/s for compute and 1 TB/s for HBM bandwidth, in decimal units. The crossover is then 200 FLOPs/byte. The optimistic minimum time is $\max(F/P,D/B)$, where $P$ is the compute ceiling and $B$ is the bandwidth ceiling:

More rows use more total time but share each weight load across more tokens. A 256-token prefill chunk crosses this projection's ridge point. A 16-request decode batch stays below it. Thus, a serving policy can change performance with the same model weights.

This is a lower bound for one projection. Cache reuse, repeated tile loads, attention, normalization, launch overhead, communication and below-peak utilization all change measured time. A larger $M$ also makes runnable work and free memory necessary. A wait for more requests can break the latency target before it makes the matmul faster.

Separate attention traffic from GPU work scheduling

FlashAttention keeps attention score tiles on chip. It keeps a running softmax maximum and sum. Thus, it can combine tiles with no write of the full score and probability matrices to HBM. Exact dense attention does quadratic arithmetic. The speedup comes from the intermediate traffic and storage it avoids.

FlashAttention-2, Sections 3–4, targets the work that stays. It decreases non-matmul operations, adds parallel execution for query rows and decreases shared-memory communication between warps. Its A100 kernel measurements depend on sequence length and head dimension. They do not predict the performance of small online decode batches. The first method removes avoidable memory input/output (IO). The second makes the GPU utilization of the remaining work better. The two methods keep the K/V history necessary for the model. GQA and cache management address that separate cost. Thus, the methods complement each other.

Match tensor-core instructions to layout and precision

Tensor cores do matrix operations on small fragments, not single scalar multiply-adds. Hopper GPUs, for example the H100, show one NVIDIA design. It has fourth-generation Tensor Cores, FP8 support, the Tensor Memory Accelerator (TMA) and asynchronous execution features.

NVIDIA's programming stack has layers. CUDA is the familiar high-level interface. Parallel Thread Execution (PTX) is a lower-level virtual instruction set. SASS is the native machine code for one specific architecture. The layers explain why the same matmul becomes different instructions on different GPU generations and compilers.

Tensor cores reach a large share of their peak only when the kernel gives them the correct work. That makes supported data types, compatible layouts, staged fragments and sufficient independent work to hide latency necessary. Memory movement must keep pace too. Otherwise the operation uses only a fraction of the advertised throughput.

Overlap tile movement with computation

A kernel that loads, waits, computes and stores in separate blocking steps leaves its arithmetic units idle. On architectures with this capability, a kernel loads the next tile while the tensor cores do arithmetic on the current one.

A common pattern is producer/consumer scheduling. Some work moves tiles into shared memory. Other work uses the tiles that are available. A few buffers form a small pipeline. While the kernel uses one tile, the next one loads.

On Hopper, TMA is one documented method to move large blocks between global memory and shared memory.

Transfers in flight hide memory latency during arithmetic. The pipeline is fragile. Bank conflicts, register spills, partly filled tensor-core tiles and uncoalesced accesses can all stall it. Power and clock behavior also affect sustained throughput.

Three consequences follow for inference. First, peak FLOP/s differs from delivered tokens for each second, because tensor cores help only when kernels supply their inputs. Second, online inference changes kernel shapes. It mixes prefill and decode, short and long prompts, and changes of batch size. Third, serving choices change the shapes and memory paths that reach kernels. The vLLM article examines chunked prefill, continuous batching and paged attention in vLLM.

The kernel view shows the hardware cost of model and serving decisions.

Follow the data path through GPU memory

The arrows show where data moves, not one strict serial order. High-performance kernels use a pipeline of loads, compute and stores that overlap.

Diagnose a slow Transformer operation

When a Transformer operation is slow, start with four concrete questions.

Does the shape give the GPU sufficient work? Small decode batches may leave compute idle because there is too little parallel work. Large prefill batches use compute better, but they use more memory and can delay other requests.

Is arithmetic or memory movement the limit? MLP matmuls are frequently compute-heavy. Decode attention may use much of its time for KV-cache reads. The vocabulary projection can matter with small batches and large vocabularies.

Are layout and precision applicable to the hardware? Particular fragment shapes and data types are necessary for tensor cores. A model in bf16, fp16, fp8 or quantized weights depends on kernels that use those formats correctly.

Which shapes does the serving layer give? The scheduler decides how much work enters each kernel call and how prefill and decode interleave. Larger batches can increase utilization but add waiting time. Chunked prefill makes long prompts less disruptive but changes the sequence of kernel shapes. Continuous batching keeps decode flowing, but the active batch may be small or irregular.

A matmul kernel cannot tell a short chat request from a long document summary. It sees shapes, pointers, layouts and data types. The serving engine turns irregular user traffic into those shapes.

Compare GPU utilization with the workload's targets: time to first token (TTFT), inter-token latency, throughput and tail latency. High utilization may come from large prefill work that delays streamed tokens. Low utilization may be fine for sparse, latency-sensitive traffic. The same kernel can look excellent or mediocre with different batches from the serving layer.

Key takeaways

• Most dense Transformer arithmetic is in matmuls in projections, MLPs and frequently the vocabulary head.
• Delivered throughput depends on data movement, tile reuse, and tensor-core-friendly shapes.
• Shared memory, registers, and asynchronous pipelines make expensive loads reusable.
• Prefill and decode give different kernel shapes. Thus, the same equation can hit different bottlenecks.
• Serving policy can change the batches that strong kernels receive. This can make those kernels look better or worse.

Series navigation

• Series index: The Life of a Token Across the LLM Stack
• Part 0: LLM Inference Is a Full-Stack Systems Problem
• Part 1: The Life of a Token Inside a Transformer
• Part 2: Why Transformer Performance Is Mostly a Matmul Problem (current)
• Part 3: The Life of a Request Inside vLLM
• Part 4: A Token's Journey Across the LLM Stack
• Part 5: Reasoning Effort Is an Inference-Time Systems Knob

@article{ji2026transformerperformancematmulgpu,
  title = {Why Transformer Performance Is Mostly a Matmul Problem},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {June},
  url = {https://fusheng-ji.github.io/blog/posts/transformer-performance-matmul-gpu/}
}

References

• Aleksa Gordic, Inside NVIDIA GPUs: Anatomy of high performance matmul kernels.
• Aleksa Gordic, Inside the Transformer: The Life of a Token.
• Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.
• Dao, FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning.
• Williams, Waterman, and Patterson, Roofline: An Insightful Visual Performance Model for Multicore Architectures.
• NVIDIA, NVIDIA Hopper Architecture In-Depth.
• NVIDIA, CUTLASS Documentation.
• NVIDIA, CUDA Programming Guide.
• NVIDIA, Parallel Thread Execution ISA.

Note: I drafted and polished this blog with assistance from ChatGPT. Its sources are Aleksa Gordic's tutorials and the cited primary papers. The numerical workload is hypothetical and its timing calculation is a resource bound.

Updated 08 November 2026