BLOG ILLUSTRATION / EXECUTION STORY

Representations before a new token

Examine representations from formatted text to logits. Then add the new sampled ID at the end of the sequence. Embed this ID for the next position.

Representations before a new tokenFormatted text becomes IDs, embeddings, contextual states and vocabulary logits. Sampling adds a new ID at the end for embedding at the next position. It does not tokenize that ID as text again. There is not one unchanged token at all stages. Text format + tokenize IDs → vectors embedding lookup Transformer contextual vectors Prediction state last input position Vocabulary logits scores for next token New token ID sample from logits add ID at the end → next input
IDs, embeddings, contextual hidden states and logits are different representations. Sampling adds the next token to the end of the sequence and starts the next request step. Author-created explanatory diagram. The playback is an example. Animation time is not measured runtime. Open full-size SVG