Distilling the Knowledge in a Neural Network | VitaVision
PaperNeurIPS 2014 Deep Learning Workshop (arXiv 2015) · 2015

Distilling the Knowledge in a Neural Network

Geoffrey E. HintonOriol VinyalsJ. Dean
arXiv 1503.02531DOIView in graph
1page built on it
1more page cites it
1narrative
1registry paper cites it

In the Atlas

ConceptPrimary source of
Knowledge Distillation

Training a student network to match a teacher's softened output distribution — the temperature softmax, the T² gradient scaling, hard-label distillation, and self-distillation with a momentum teacher.

Cited as a reference by 1 page, grouped by domain

Representation learning 1
DeiT

Appears in 1 narrative

NarrativeFoundation Models for Vision

Lineage

Papers in the Atlas registry only · one dot per paper
Built upon by · 1
’12’13’14’15’16’17’18’19’20Training data-efficient image transformers & distillation through attention (2020)ImageNet Classification with Deep Convolutional Neural Networks (2012)