Training data-efficient image transformers & distillation through attention | VitaVision
PaperICML 2021 (arXiv 2020) · 2020

Training data-efficient image transformers & distillation through attention

Hugo TouvronMatthieu CordMatthijs DouzeFrancisco MassaAlexandre SablayrollesHervé Jégou
arXiv 2012.12877DOIView in graph
1page built on it
3more pages cite it
1narrative
3registry papers cite it

In the Atlas

ModelPrimary source of
DeiT

Data-efficient image transformers: ViT's architecture unchanged, made ImageNet-1k-competitive by a heavy augmentation/regularization recipe and distillation through attention — a dedicated distillation token supervised by a convnet teacher's hard decisions.

Cited as a reference by 3 pages, grouped by domain

Representation learning 3
DINOKnowledge DistillationSwin Transformer

Appears in 1 narrative

NarrativeFoundation Models for Vision

Lineage

Papers in the Atlas registry only · one dot per paper
Built upon by · 3
’15’16’17’18’19’20’21Emerging Properties in Self-Supervised Vision Transformers (2021)Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021)Vision Transformers for Dense Prediction (2021)Distilling the Knowledge in a Neural Network (2015)Deep Residual Learning for Image Recognition (2016)An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020)