Lab 07 - 3D Parallelism Plan

Overview

Lab 07: 3D Parallelism Plan

Annotated code reading lab. Running code is optional.

Related handbook section

3D Parallelism Plan

This lab maps directly to the handbook section. Read the related handbook section first, then use the lab page and starter file to connect the concept to concrete variables, shapes, APIs, and interview-ready explanations.

Open handbook section Open starter file

Concept Goal

3D Parallelism Plan

Read a 3D plan as mapping DP/FSDP, TP and PP groups onto hardware topology.

Mental Model

Mechanism to keep in mind

`tp_size` is commonly mapped to the fastest link domain when possible.
`pp_size` sets number of layer stages.
`dp_size` is the outer replica/shard axis.

Annotated Code Preview

Starter preview

Excerpt from code/lab-07-3d-parallelism-plan/parallelism_plan.yaml. The linked starter file is the source of truth.

Open starter file

# 3D Parallelism Plan
# Annotated reading material. This file is the source of truth for the lab preview.
# Running it is not required; use it to explain the configuration contract.

world_size: 64
node_shape: 8_nodes_x_8_gpus
parallelism:
  tp_size: 8   # keep within one node when possible
  pp_size: 4
  dp_size: 2
reading_note: map each collective to the link it crosses

# Reading focus:
# Read a 3D plan as mapping DP/FSDP, TP and PP groups onto hardware topology.
#
# What to notice:
# - Parallel degrees multiply to world size.
# - Topology placement matters as much as degree numbers.
# - Collective frequency determines link sensitivity.

Line-by-line Explanation

What each block is doing

Setup / contract: `tp_size` is commonly mapped to the fastest link domain when possible.
Main transition: `pp_size` sets number of layer stages.
Interview hook: `dp_size` is the outer replica/shard axis.

What to Notice

Reading checkpoints

Parallel degrees multiply to world size.
Topology placement matters as much as degree numbers.
Collective frequency determines link sensitivity.

Common Misunderstandings

What this lab prevents

Do not give degrees without rank placement.
Do not put high-frequency TP collectives on slow links if avoidable.

Interview Explanation

How to say it out loud

Read a 3D plan as mapping DP/FSDP, TP and PP groups onto hardware topology. Then explain the code by naming the state being transformed, the axis or shape that matters, and the tradeoff that would appear in a real system.

External intuition notes

Additional intuition

Use official docs and papers for API behavior and factual claims; use blogs only to improve the mental picture.
If support matrices, performance behavior or backend choices are version-sensitive, check current docs before repeating them.
A strong interview answer names the state object, the shape or axis it changes, and the tradeoff it creates.

InfraLens