Lab 08: Topology Communication
Annotated code reading lab. Running code is optional.
Topology Communication
This lab maps directly to the handbook section. Read the related handbook section first, then use the lab page and starter file to connect the concept to concrete variables, shapes, APIs, and interview-ready explanations.
Topology Communication
Read communication plans by asking which collective crosses which link.
Mechanism to keep in mind
- `all_reduce` is common in DDP/TP.
- `all_gather` and `reduce_scatter` appear in common FSDP-style execution.
- `all_to_all` can appear in expert/sequence routing, but exact collectives are implementation-dependent.
Annotated Code Preview
Open starter fileStarter preview
Excerpt from code/lab-08-topology-communication/topology_notes.md. The linked starter file is the source of truth.
Distributed training scales beyond one device by partitioning data, model state, or computation across ranks. The key questions are what is replicated, what is sharded, which collective runs on the critical path, and how optimizer semantics stay consistent.What each block is doing
- Setup / contract
- `all_reduce` is common in DDP/TP.
- Main transition
- `all_gather` and `reduce_scatter` appear in common FSDP-style execution, with timing controlled by configuration and version.
- Interview hook
- `all_to_all` can appear in expert/sequence routing, but exact collectives are implementation-dependent.
Reading checkpoints
- Fast links should carry frequent large collectives.
- Bucket size and overlap affect timelines.
- NCCL behavior should be checked in current docs.
What this lab prevents
- Do not optimize FLOPs while ignoring communication.
- Do not assume all GPU pairs have equal bandwidth.
How to say it out loud
Read communication plans by asking which collective crosses which link. Then explain the code by naming the state being transformed, the axis or shape that matters, and the tradeoff that would appear in a real system.
Additional intuition
- Use official docs and papers for API behavior and factual claims; use blogs only to improve the mental picture.
- If support matrices, performance behavior or backend choices are version-sensitive, check current docs before repeating them.
- A strong interview answer names the state object, the shape or axis it changes, and the tradeoff it creates.