| 1 | This is figure one from the paper — the Transformer architecture. Read it from the bottom up. Inputs and outputs both come in as embeddings, with a sinusoidal positional encoding added. The encoder on the left is a stack of six identical layers; each layer has multi-head self-attention followed by a position-wise feed-forward network, with Add and Norm wrapping each sub-layer. The decoder on the right is also six layers, but inserts a third sub-layer that attends back to the encoder output, and its self-attention is masked so it cannot peek at future tokens. The whole thing ends with a linear projection and softmax to produce token probabilities. The skeleton is the same as prior seq2seq models — what is new is what fills the boxes. |