Deep Learning and Distillation

Hinton's later work helped show how multilayer neural networks could learn useful hierarchies of features. He also co-developed knowledge distillation, in which a smaller model learns from the output behavior of a larger model or ensemble.