Definition
Distillation was popularised in a 2015 paper by Hinton and colleagues. A large model produces outputs for many examples, and a smaller model is trained to reproduce them. The student learns the teacher's behaviour on the task without needing its size.
Many companies use distillation to move a working prototype built on a frontier model onto a cheaper model for production. Provider terms of service often restrict using one company's models to train competing models, so check the licence before distilling.