Attention Is All You Need: The Transformer Revolution

Attention Is All You Need

Introducing the Transformer: A Novel Neural Network Architecture for Sequence Transduction

---

Photo by Mirella Callage on Unsplash

• Background and Motivation
• Transformer Model Architecture
• Key Components: Self-Attention and Positional Encoding
• Training Regime
• Experimental Results
• Model Variations
• Broader Impact and Conclusion

---

Photo by Teemu Paananen on Unsplash

1

Background and Motivation

Limitations of Recurrent and Convolutional Models in Sequence Transduction

---

Photo by Conny Schneider on Unsplash

• Recurrent models (RNN/LSTM/GRU): Sequential nature precludes parallelization within training examples, especially for long sequences
• Convolutional models: Operations grow linearly or logarithmically with distance between positions, difficult to learn long-range dependencies
• Previous attention mechanisms used in conjunction with recurrent networks

2

Transformer Model Architecture

Encoder-Decoder Structure Based Solely on Attention Mechanisms

---

Photo by Umberto on Unsplash

• Encoder: N=6 identical layers with multi-head self-attention and position-wise feed-forward network
• Decoder: N=6 layers with masked multi-head self-attention, encoder-decoder attention, and feed-forward
• Residual connections around each sub-layer, followed by layer normalization; d_model=512

• Each encoder layer: Multi-head self-attention sub-layer + Position-wise fully connected feed-forward network
• Decoder adds: Encoder-decoder attention sub-layer (queries from decoder, keys/values from encoder)
• Masked self-attention in decoder prevents attending to future positions
• All sub-layers produce d_model=512 outputs, with residual connections: LayerNorm(x + Sublayer(x))

3

Attention Mechanisms

Scaled Dot-Product and Multi-Head Attention

---

Photo by Shane Rounce on Unsplash

• Attention(Q, K, V) = softmax(QK^T / sqrt(dk)) V
• Dot-products scaled by 1/sqrt(dk) to prevent vanishing gradients
• Faster and more space-efficient than additive attention
• Inputs: Q, K dimension dk, V dimension dv

• Linear projections to h=8 heads: dk = dv = d_model / h = 64
• Parallel attention on projected Q, K, V; concatenate and project outputs
• Jointly attend to information from different subspaces at different positions
• Used in encoder self-attention, decoder self-attention, and encoder-decoder attention

• FFN(x) = max(0, xW1 + b1)W2 + b2; dff=2048, applied identically to each position
• Positional Encoding: Sinusoidal functions added to input embeddings
• PE(pos,2i) = sin(pos / 10000^{2i/dmodel}), PE(pos,2i+1) = cos(...)
• Allows extrapolation to longer sequences; learned embeddings also work similarly

4

Training and Results

State-of-the-Art Performance on Machine Translation

---

Photo by Deng Xiang on Unsplash

• 28.4: EN→DE BLEU
• 41.8: EN→FR BLEU
• 27.3: EN→DE Base
• 3.5: Training Days

• O(1) sequential operations and path length vs. O(n) for RNNs
• O(n^2 · d) complexity per layer, faster than RNNs when n < d
• Constant operations between any two positions, unlike CNNs' log(n)
• Interpretable: Attention heads learn syntactic/semantic tasks (see appendix visualizations)

> We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

— Vaswani et al., 2017

Transformer: First transduction model relying entirely on self-attention Achieves new SOTA on translation tasks with faster training Generalizes to parsing; foundation for modern LLMs, ViTs, and more

Future: Extend to images, audio, video; less sequential generation https://github.com/tensorflow/tensor2tensor

---

Photo by Vinh Nguyen on Unsplash