PaperNeurIPS 2017 · 2017
Attention Is All You Need
3pages built on it
2more pages cite it
1narrative
6registry papers cite it
In the Atlas
ConceptPrimary source of
Attention Mechanism
Computes each output element as a learned, input-dependent weighted average of value vectors, letting every element aggregate information from any other regardless of distance.
ConceptPrimary source of
Positional Encoding
Mechanisms that inject token order into permutation-invariant attention: absolute encodings added to embeddings, learned tables, and rotary schemes that rotate queries and keys so position enters the score.
ConceptPrimary source of
Transformer
Sequence-to-sequence architecture assembled entirely from attention and position-wise feedforward sublayers, each wrapped in a residual connection and layer normalisation — no recurrence, no convolution.
Cited as a reference by 2 pages, grouped by domain
Representation learning 2
Appears in 1 narrative
Lineage
Papers in the Atlas registry only · one dot per paperBuilt upon by · 6
Builds on · 1
2014Neural Machine Translation by Jointly Learning to Align and TranslateBahdanau, Cho, BengioBuilt upon by · 6
2020Transformers are RNNs: Fast Autoregressive Transformers with Linear AttentionKatharopoulos, Vyas et al.2021Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsLiu, Lin et al.2021Learning Transferable Visual Models From Natural Language SupervisionRadford, Kim et al.2021RoFormer: Enhanced Transformer with Rotary Position EmbeddingSu, Lu et al.2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessDao, Fu et al.2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head CheckpointsAinslie, Lee-Thorp et al.