What Is Model Distillation?

Model Distillation — a technique in which a smaller "student" model is trained to reproduce the behaviour of a larger "teacher" model, transferring much of its capability into a more efficient form.

Distillation makes models cheaper and faster to run, which is why distilled models are common in production. It carries governance and legal nuance: distilling from a third-party model may breach its terms of use or intellectual-property rights, and the student can inherit the teacher's biases and weaknesses while being harder to trace back to them.

Source: Machine-learning literature