What problem does Tensor Parallelism solve?

It splits very large layer math and weights across GPUs.

What is the one-line intuition for Tensor Parallelism?

It changes where work or state lives so the system can use hardware more efficiently.

Weight shards, activation shards or replicas, and partial GEMM outputs.

Column parallel computes output parts; row parallel sums partial products.

It is sensitive to fast interconnect because communication occurs inside Transformer layers.

Higher TP degree reduces per-rank weight and compute but increases coordination and may reduce GEMM efficiency.

TP can become communication-bound at layer boundaries or inefficient when shards are too small.

Using TP across slow nodes can erase compute gains.

Track collective time inside blocks, GEMM occupancy, and tokens/s.

TP reduces per-rank layer memory and compute but adds frequent collectives.

Keep TP inside a high-bandwidth node, then combine with DP or PP across nodes.

TP splits tensors within a layer; PP splits layers across pipeline stages.