Lab 05 - RoPE Mental Model

Overview

Lab 05: RoPE Mental Model

Annotated code reading lab. Running code is optional.

Related handbook section

RoPE Mental Model

This lab maps directly to the handbook section. Read the related handbook section first, then use the lab page and starter file to connect the concept to concrete variables, shapes, APIs, and interview-ready explanations.

Open handbook section Open starter file

Concept Goal

RoPE Mental Model

Read RoPE as a Q/K rotation before attention, not as an added embedding vector.

Mental Model

Mechanism to keep in mind

`pair` groups adjacent dimensions into a 2D plane.
`angle` depends on position and frequency.
Only Q/K need the rotation because they create attention scores.

Annotated Code Preview

Starter preview

Excerpt from code/lab-05-rope-mental-model/rope_reading.py. The linked starter file is the source of truth.

Open starter file

# RoPE Mental Model
# Annotated reading material. Running this file is optional.
# Source-of-truth focus: Read RoPE as a Q/K rotation before attention, not as an added embedding vector.

def rotate_pair(x0, x1, cos, sin):
    # A 2D rotation: this is the mental model for one RoPE pair.
    return x0 * cos - x1 * sin, x0 * sin + x1 * cos

position = 7
frequency = 0.01
angle = position * frequency
q_pair_after_rope = rotate_pair("q_even", "q_odd", "cos(angle)", "sin(angle)")

# What to explain while reading:
# - pair groups adjacent dimensions into a 2D plane.
# - angle depends on position and frequency.
# - Only Q/K need the rotation because they create attention scores.
#
# Common traps:
# - RoPE is not simply sinusoidal addition.
# - Changing RoPE scaling can change model behavior.

Line-by-line Explanation

What each block is doing

Setup / contract: `pair` groups adjacent dimensions into a 2D plane.
Main transition: `angle` depends on position and frequency.
Interview hook: Only Q/K need the rotation because they create attention scores.

What to Notice

Reading checkpoints

RoPE affects relative-position behavior through dot products.
The value tensor is normally not rotated.
Long-context behavior is version/model dependent; check current docs.

Common Misunderstandings

What this lab prevents

RoPE is not simply sinusoidal addition.
Changing RoPE scaling can change model behavior.

Interview Explanation

How to say it out loud

Read RoPE as a Q/K rotation before attention, not as an added embedding vector. Then explain the code by naming the state being transformed, the axis or shape that matters, and the tradeoff that would appear in a real system.

External intuition notes

Additional intuition

Use official docs and papers for API behavior and factual claims; use blogs only to improve the mental picture.
If support matrices, performance behavior or backend choices are version-sensitive, check current docs before repeating them.
A strong interview answer names the state object, the shape or axis it changes, and the tradeoff it creates.

InfraLens