Lab 06 - FFN / SwiGLU Parameter Count

Overview

Lab 06: FFN / SwiGLU Parameter Count

Annotated code reading lab. Running code is optional.

Related handbook section

FFN / SwiGLU Parameter Count

This lab maps directly to the handbook section. Read the related handbook section first, then use the lab page and starter file to connect the concept to concrete variables, shapes, APIs, and interview-ready explanations.

Open handbook section Open starter file

Concept Goal

FFN / SwiGLU Parameter Count

Use code-like formulas to see why FFN/SwiGLU often dominates Transformer parameter count.

Mental Model

Mechanism to keep in mind

`up` and `gate` are separate projections in SwiGLU.
`down` returns to hidden size.
The multiplier is an approximation for quick interview math.

Annotated Code Preview

Starter preview

Excerpt from code/lab-06-ffn-swiglu-parameter-count/ffn_swiglu_params.py. The linked starter file is the source of truth.

Open starter file

# FFN / SwiGLU Parameter Count
# Annotated reading material. Running this file is optional.
# Source-of-truth focus: Use code-like formulas to see why FFN/SwiGLU often dominates Transformer parameter count.

hidden = 4096
ffn_hidden = 11008
up = hidden * ffn_hidden
gate = hidden * ffn_hidden
down = ffn_hidden * hidden
swiglu_params = up + gate + down

# What to explain while reading:
# - up and gate are separate projections in SwiGLU.
# - down returns to hidden size.
# - The multiplier is an approximation for quick interview math.
#
# Common traps:
# - Do not describe FFN as sequence mixing.
# - Do not ignore FFN when estimating memory.

Line-by-line Explanation

What each block is doing

Setup / contract: `up` and `gate` are separate projections in SwiGLU.
Main transition: `down` returns to hidden size.
Interview hook: The multiplier is an approximation for quick interview math.

What to Notice

Reading checkpoints

Attention is not always the largest parameter bucket.
SwiGLU uses gating, so count both value and gate projections.
Bias/norm are usually small in rough estimates.

Common Misunderstandings

What this lab prevents

Do not describe FFN as sequence mixing.
Do not ignore FFN when estimating memory.

Interview Explanation

How to say it out loud

Use code-like formulas to see why FFN/SwiGLU often dominates Transformer parameter count. Then explain the code by naming the state being transformed, the axis or shape that matters, and the tradeoff that would appear in a real system.

External intuition notes

Additional intuition

Use official docs and papers for API behavior and factual claims; use blogs only to improve the mental picture.
If support matrices, performance behavior or backend choices are version-sensitive, check current docs before repeating them.
A strong interview answer names the state object, the shape or axis it changes, and the tradeoff it creates.

InfraLens