Attention Is All You Need: the architecture behind modern language models

TL;DR

The paper proposes the Transformer, a sequence-to-sequence model that drops recurrence and convolution entirely and relies only on attention. Because every position can attend to every other position in a single step, the model trains with far more parallelism than a recurrent network. On the two machine-translation benchmarks the authors used, it matched or beat the best published systems at a fraction of the training cost.

The problem

In 2017, the dominant translation systems were encoder–decoder networks built from recurrent layers (LSTMs or GRUs) or from convolutions, usually with an attention mechanism connecting encoder and decoder. In the paper's own comparison (Table 2), the best earlier ensembles came from a convolutional model (ConvS2S), just ahead of Google's recurrent GNMT; the best earlier single model was a mixture-of-experts model (MoE). Both designs have a structural cost. In a recurrent network, computing the hidden state at position $t$ needs the state at $t-1$; that sequential dependency limits parallelism within a training example, which matters more as sequences get longer. In a convolutional one, the number of operations needed to relate two positions grows with the distance between them, which the authors argue makes long-range dependencies harder to learn.

The authors asked whether attention — previously an add-on to recurrent models — could do the whole job.

The method

Scaled dot-product attention

Each token is projected into a query, a key and a value. A token's output is a weighted sum of all values, where the weights come from how well its query matches every key:

The division by $\sqrt{d_k}$ keeps dot products from growing with the key dimension, which would otherwise push the softmax into regions with very small gradients.

Multi-head attention

Instead of one attention function, the model runs several in parallel ("heads"), each with its own learned projections, then concatenates the results. Different heads can specialise in different relationships between positions. The base model uses 8 heads.

The rest of the block

Each layer combines attention with a position-wise feed-forward network, residual connections and layer normalisation. The encoder and decoder are each a stack of these layers (6 in the base model). The decoder's self-attention is masked so a position cannot attend to later positions — preserving the left-to-right generation needed for translation.

Positional information

Attention on its own is order-agnostic: shuffling the input would shuffle the output identically. To give the model a sense of order, the authors add positional encodings — fixed sinusoids of different frequencies — to the input embeddings.

What the paper reports

The headline results, as reported by the authors in the abstract:

The authors also applied the architecture to English constituency parsing to show it generalises beyond translation.

These are the authors' own measurements under their own setup. BLEU scores are sensitive to tokenisation and evaluation details, so they are best compared within the same paper rather than across papers.

Why it mattered

The architecture's parallelism made it practical to train much larger models on much more data. Later model families adopted it as their backbone — for example BERT (Devlin et al., 2018 (external site)) used the encoder side, and GPT-3 (Brown et al., 2020 (external site)) used a decoder-only variant.

Limitations and open questions

• Quadratic cost in sequence length. Every position attends to every other, so compute and memory for attention grow with the square of the sequence length. Much later work focuses on reducing this cost.
• Narrow evaluation. The paper's evidence is two translation benchmarks plus parsing. The broader claims people make about Transformers come from later work, not from this paper.
• Positional encoding choices. The paper uses fixed sinusoidal encodings and reports that learned embeddings performed similarly; how best to represent position remained an active research question.

Reading notes for practitioners

If you are building on Transformer-based models today, the ideas from this paper that most often matter in practice are: attention cost grows quadratically with context length; masking determines what a model can "see"; and positional representation limits how well a model handles sequences longer than those it was trained on.