When model distillation is worth the extra training work

Two kinds of distillation with different requirements

Sequence-level distillation is the simple kind. The teacher answers a set of prompts, you keep the answers that pass your checks, and the student is fine-tuned on them as if they were human labels. OpenAI documented this flow for its own hosted fine-tuning, a platform that is now closed to new users and is being wound down for eligible existing customers. The same flow works with an open-weight student trained in your own environment and any teacher you can call, a hosted API included.

Token-level distillation trains the student on the teacher's probability for each token rather than only its final text. Hinton and colleagues wrote in 2015 that such soft targets, when they have high entropy, carry much more information per training case than hard labels. Generalized Knowledge Distillation, proposed by Agarwal and colleagues, also lets the student generate its own outputs and has the teacher score them, which addresses the gap between what a student sees in training and what it produces at inference. In Hugging Face TRL, the experimental GKDTrainer loads the teacher as a model next to the student, so this route needs a teacher whose weights you can run.

When the extra work pays off

• The teacher already passes your acceptance criteria on the task. Treat this as the entry criterion: filtering, human corrections and extra task data can sometimes lift a student above its teacher, but a teacher you would deploy gives a known baseline to start from.
• Volume is high enough for the student's lower serving cost to recover the cost of generating data and training.
• Human labels are scarce or slow, while the teacher's outputs can be checked cheaply by rules or a sampled review.
• Latency limits or data location rules make calling the teacher in production impractical.

If you already have thousands of verified outcomes, train on those directly. A teacher adds cost and its own errors, and real labels are the better reference.

Budget for the teacher as well as the student

Generating the training set is inference you pay for. A hypothetical budget, with the assumptions stated: 60,000 prompts of 1,500 input tokens and 250 output tokens come to 90 million input and 15 million output tokens. Multiply by the teacher's current per-token prices, add the generations you discard after filtering, and add expert review of a sample.

On-policy methods add a second bill. The teacher scores the student's outputs throughout training, so it needs GPU capacity during every training run, not only before it. And the whole cycle repeats when the task changes: a new category or policy means new teacher outputs, new filtering and a new student.

What quality you can lose

The student learns the distribution you showed it, so its weak spots are predictable.

• Inputs outside the training prompts, such as new templates, languages or request types
• Rare categories the teacher saw too few times to pass on
• Teacher errors that your filter did not catch, reproduced as faithfully as correct answers
• General abilities outside the task, such as following unusual instructions, which a narrow training set does not maintain

The GKD authors also note that a small student may lack the capacity to imitate the teacher fully, which is why their method allows other loss functions. If a gap persists after you add data, a larger student or a narrower task is the usual way forward.

Evaluate against the truth, not only against the teacher

Agreement with the teacher is easy to measure and misleading on its own, because it rewards copying the teacher's mistakes. Score teacher and student on a held-out set labelled by people, and add a probe set of inputs the task does not cover, to check that the student flags them instead of answering confidently.

A hypothetical result for a contract-clause classifier tested on 800 clauses labelled by lawyers:

• All clauses: teacher right on 736, student on 728, and the two agree on 95 percent of clauses
• 60 rare clause types: teacher right on 53, student on 48
• 100 out-of-scope documents: teacher flags 90, student flags 52

The headline gap is one percentage point, but the student is weaker on rare clauses and misses 48 of the 100 documents it should not classify, so it cannot be trusted to decide which inputs to hand back. A hybrid design needs a separate routing gate, such as a document-type check or an out-of-scope classifier, validated on its own labelled set before it sends rare types and unfamiliar documents to the teacher. Measure how many out-of-scope cases still slip through the gate to the student, and how much traffic the teacher still receives, before counting the saving.