| 1 | To appreciate the move, it helps to look at where sequence modeling was sitting in 2017. Recurrent networks — RNNs, LSTMs, GRUs — were the default, but they compute step by step and resist parallelization. Convolutional alternatives like ByteNet and ConvS2S could parallelize, but signals between distant positions still had to travel through many layers, growing linearly or logarithmically with distance. Self-attention does something different: every position relates to every other position in one constant-time hop. Visually you can think of it as moving from a chain to a local field to a fully connected mesh — and that mesh is what the Transformer leans into. |