BLOG ILLUSTRATION / EXECUTION STORY

Residual paths keep each sublayer input

A representative pre-normalized decoder applies attention and an MLP. Each residual path keeps its sublayer input unchanged.

Residual paths keep each sublayer inputA representative pre-normalized decoder applies attention and an MLP. Each residual path keeps its sublayer input unchanged. Input state Normalize for each token Attention mix positions Residual add input + attention Normalize for each token Gated MLP per-position transform Residual add state + MLP unchanged input skip
Two pre-normalized residual sublayers use attention and a gated MLP. Dashed residual edges keep each sublayer input unchanged. This is a representative decoder, not a universal architecture. The author made this explanatory diagram. The playback is an example. Animation time is not measured runtime. Open full-size SVG