InfraLens

Attention Backend Comparison Reading

This lab maps directly to the handbook section. Read the related handbook section first, then use the lab page and starter file to connect the concept to concrete variables, shapes, APIs, and interview-ready explanations.

Mechanism to keep in mind

• `math` names the formula.
• `kernel` names how the formula is executed.
• `runtime` names serving-time cache/request management.

Reading checkpoints

• FlashAttention is exact attention with lower HBM traffic for supported layouts/backends.
• SDPA is an API that may choose different kernels.
• Transformers attention backend option names and defaults are version-sensitive and model-specific.
• PagedAttention is about KV cache blocks in serving.

What this lab prevents

• Do not call FlashAttention an approximation.
• Do not confuse kernel backend with serving scheduler.

How to say it out loud

Compare math-level attention with kernel/runtime choices: SDPA, FlashAttention and PagedAttention. Then explain the code by naming the state being transformed, the axis or shape that matters, and the tradeoff that would appear in a real system.

Additional intuition

• Use official docs and papers for API behavior and factual claims; use blogs only to improve the mental picture.
• If support matrices, performance behavior or backend choices are version-sensitive, check current docs before repeating them.
• A strong interview answer names the state object, the shape or axis it changes, and the tradeoff it creates.

Official, paper and practical references

• Attention Is All You Need
• PyTorch scaled_dot_product_attention
• Hugging Face KV cache strategies
• Hugging Face attention interface
• FlashAttention paper