Lab 04: Pipeline Schedule Reading
Annotated code reading lab. Running code is optional.
Pipeline Schedule Reading
This lab maps directly to the handbook section. Read the related handbook section first, then use the lab page and starter file to connect the concept to concrete variables, shapes, APIs, and interview-ready explanations.
Pipeline Schedule Reading
Read a pipeline schedule by tracking microbatches through stages.
Mechanism to keep in mind
- `stage` owns consecutive layers.
- `microbatch` keeps stages busy.
- `bubble` is idle time before/after steady state.
Annotated Code Preview
Open starter fileStarter preview
Excerpt from code/lab-04-pipeline-schedule-reading/pipeline_schedule.md. The linked starter file is the source of truth.
Read this system as a pipeline graph: inputs become conditions, loaders instantiate model components, the scheduler loop updates latents, and offload or control paths change memory and latency.What each block is doing
- Setup / contract
- `stage` owns consecutive layers.
- Main transition
- `microbatch` keeps stages busy.
- Interview hook
- `bubble` is idle time before/after steady state.
Reading checkpoints
- PP cuts depth, not tensor width.
- More microbatches reduce bubble but affect memory.
- Activations cross stage boundaries.
What this lab prevents
- Pipeline parallelism does not make a single sample skip stages.
- Throughput and latency are different questions.
How to say it out loud
Read a pipeline schedule by tracking microbatches through stages. Then explain the code by naming the state being transformed, the axis or shape that matters, and the tradeoff that would appear in a real system.
Additional intuition
- Use official docs and papers for API behavior and factual claims; use blogs only to improve the mental picture.
- If support matrices, performance behavior or backend choices are version-sensitive, check current docs before repeating them.
- A strong interview answer names the state object, the shape or axis it changes, and the tradeoff it creates.