| 1 | A quick word on how this was trained. The base model trains in twelve hours on eight P100 GPUs in one machine; the big model trains in three and a half days on the same hardware. Batches were grouped by similar sequence length to keep about twenty-five thousand source and twenty-five thousand target tokens per batch. The optimizer is Adam with beta two pushed up to 0.98, and a custom learning rate schedule that warms up linearly for four thousand steps then decays as one over the square root of the step count. Regularization is dropout at 0.1 applied to every sub-layer output and to the sum of embedding and positional encoding, plus label smoothing of 0.1. |