Knowledge Distillation: Interview Questions and Answers

Computing the gradient of $\mathrm{KL}(p_S \| p_T)$ requires an expectation under the student’s distribution, not the teacher’s. MiniLLM uses policy gradient (REINFORCE with a teacher-provided baseline) to obtain unbiased gradient estimates from on-policy student samples — more expensive than forward KD, which requires only a single teacher forward pass. Gu et al. (2024) evaluated distillation from GPT-2-XL into GPT-2 variants (120M to 1.5B parameters) and reported consistent improvements on Dolly, Self-Instruct, and S-NI open-ended generation benchmarks over forward-KL baselines.

Q12 [Advanced] Compare black-box and white-box LLM distillation

Q: What information access does each paradigm assume, and how does this shape the methods, quality, and legal trade-offs involved?

A: White-box distillation assumes access to the teacher’s internal states: probability distributions over the vocabulary at each decoding step, hidden activations, or attention patterns. This enables richer transfer signals — token-level soft targets (as in DistilBERT and TinyBERT), intermediate layer supervision, or reverse-KL objectives (MiniLLM). The full token distribution carries far more information per training example than the single sampled token.

Black-box distillation assumes only API-level access: the student observes input-output text pairs but not probabilities or activations. The student is fine-tuned on teacher-generated completions — effectively supervised fine-tuning on teacher outputs. Projects like Stanford Alpaca and Vicuna demonstrated this paradigm: LLaMA-7B fine-tuned on GPT-3.5-turbo or GPT-4 completions acquires substantial instruction-following capability without access to teacher internals.

The central limitation of black-box distillation is information loss: the student trains on the teacher’s mode (greedy or sampled completion) rather than its full conditional distribution. For tasks with multiple valid continuations, the teacher’s probability mass over alternatives carries signal that is entirely discarded. Black-box distillation also raises legal concerns: many providers explicitly prohibit using API outputs to train competing models.

Hybrid approaches partially bridge the gap. Speculative decoding (Leviathan et al., 2023) trains a small draft model to approximate teacher token-by-token, using the teacher to verify drafts in parallel. GKD (Agarwal et al., 2024) enables on-policy distillation by providing access to a subset of teacher probability mass at training time, combining the tractability of white-box methods with the deployment flexibility of black-box ones.

Q13 [Advanced] Distinguish sequence-level from token-level knowledge distillation

Q: What information does sequence-level distillation capture that token-level distillation misses, and what are the computational trade-offs?

A: Token-level distillation trains the student to match the teacher’s conditional $p_T(x_t | x_{

The limitation is exposure bias: the student always sees teacher-forced prefixes at train time but generates autoregressively at inference, where early errors are compounded through subsequent steps. A student that perfectly matches each conditional under teacher forcing may still diverge severely in free-running generation.

Sequence-level KD (Kim & Rush, 2016) addresses this by distilling the sequence distribution directly. The teacher generates complete sequences via beam search, and the student is trained to maximize $\log p_S(\hat{x})$ on this teacher-generated corpus. Because the training data consists of teacher outputs rather than teacher-forced prefixes, the student trains on sequences it will actually produce at inference — exposure bias is mitigated by construction.

The trade-off is dataset cost: generating a sequence-level corpus requires running the teacher decoder over the full training set, which is expensive for large vocabularies and long sequences. Token-level distillation computes soft targets in a single forward pass. Online methods like MiniLLM (Gu et al., 2024) combine sequence-level thinking with continuous adaptation by sampling from the student during training, avoiding the one-time corpus cost while preserving on-policy distribution alignment.

Vision Model Distillation

Q14 [Basic] Explain DeiT’s distillation token mechanism

Q: How does DeiT incorporate teacher supervision into ViT training, and what type of teacher works best?

A: DeiT (Touvron et al., 2021) trains Vision Transformers on ImageNet-1k without the massive external datasets required by the original ViT. The central mechanism is a distillation token: a learnable vector appended to the input patch sequence alongside the class token. Like the class token, the distillation token interacts with all patch tokens through self-attention and is projected to logits at the output — but it is supervised by the teacher’s prediction rather than the ground-truth label:

where $y_T$ is the teacher’s top-1 prediction (hard distillation) or softened distribution (soft distillation). The two tokens interact through attention throughout the forward pass but receive independent supervision signals.

Touvron et al. (2021) found that CNN teachers (specifically RegNetY-160) outperform transformer teachers for DeiT. This is attributed to inductive bias complementarity: a CNN teacher encodes translational equivariance that encourages the ViT student to develop local sensitivity, beneficial for ImageNet despite not being architecturally mandated. DeiT-B with a RegNetY-160 teacher achieves 85.2% top-1 accuracy on ImageNet, training only on ImageNet data — a result previously achievable only with JFT-pretrained ViTs.

Q15 [Advanced] Analyze the capacity gap problem and teacher assistant distillation

Q: When does a larger teacher produce a worse student, and how does the teacher assistant strategy address the root cause?

A: The capacity gap problem arises when teacher and student architectures differ substantially in size. Despite the intuition that a stronger teacher should always produce a better student, Mirzadeh et al. (2020) showed empirically on CIFAR-10/100 and ImageNet that student accuracy peaks at an intermediate teacher size and then declines as the teacher grows beyond that optimum. A very large teacher’s decision surface may be too complex for a small student to approximate — the distillation gradient becomes poorly conditioned and the student converges to a suboptimal solution.

The Teacher Assistant (TA) strategy inserts intermediate-capacity networks between teacher and student: $T \to TA \to S$. Each step compresses by a smaller factor, keeping the capacity gap manageable at each stage. For a teacher with $N_T$ parameters and a student with $N_S$, a single TA with $N_{TA} \approx \sqrt{N_T \cdot N_S}$ (geometric mean) equalizes the two compression ratios.

Mirzadeh et al. (2020) demonstrated consistent gains across architectures: for a WRN-40-2 teacher and WRN-16-1 student on CIFAR-100, interposing a WRN-40-1 TA raised the student’s top-1 accuracy by approximately 1.2 points. The improvement persists across multiple layers of assistants, though returns diminish.

The main practical cost is a multiplied training budget: the TA itself must be distilled from the teacher before the final student can be trained, effectively chaining $k$ distillation runs for $k-1$ assistants. This makes TA most justified when the primary bottleneck is distillation quality rather than training time, and when the teacher-student capacity ratio substantially exceeds $10\times$.

Q16 [Advanced] Identify challenges in distilling dense prediction models

Q: How do object detection and segmentation tasks complicate knowledge transfer compared to image classification, and what strategies address these complications?

A: Classification distillation transfers a fixed-length logit vector per image. Dense prediction models produce spatially-structured outputs — bounding boxes and class scores over feature pyramid levels for detection, or pixel-wise segmentation maps — and the intermediate feature maps driving localization quality are as important as the final classification heads. Naively applying soft logit matching (Hinton et al., 2015) to detection heads ignores the backbone features that encode spatial context critical to localization.

A core complication is foreground-background imbalance in feature-space distillation. A naive FitNets-style loss applied to backbone spatial features forces the student to replicate activations at all $H \times W$ positions, the vast majority of which correspond to background. This floods the gradient with low-information signal and dilutes supervision from the handful of foreground locations the student most needs to learn. Wang et al. (2019) showed that weighting distillation losses by predicted foreground probability — applying larger penalties in regions the teacher identifies as object-containing — significantly improves transfer for two-stage detectors.

For multi-scale architectures, distillation losses applied at the feature pyramid network (FPN) outputs, rather than backbone features alone, transfer multi-scale context more effectively because FPN features directly feed the detection heads. Matching across architectures with different backbone strides requires careful spatial alignment of teacher and student feature maps.

For dense segmentation, matching $H \times W$ pixel-wise activations is memory-intensive. Selective distillation at semantically significant spatial locations — identified via the teacher’s own confidence or attention patterns — reduces memory cost while focusing supervision on the positions most responsible for accurate boundary delineation.

Q17 [Advanced] Explain DINO as a self-supervised self-distillation framework

Q: How does DINO apply distillation without labels to learn visual representations, and what makes its attention maps emergently interpretable?

A: DINO (Caron et al., 2021) — Distillation with No labels — trains a student ViT to match the outputs of a momentum teacher: an exponential moving average (EMA) of the student weights that is never directly updated by gradient descent:

where $m$ increases from 0.996 to 1 during training. The teacher thus provides a temporally stable target slightly ahead of the student in representation quality, avoiding the instability of symmetric mutual learning.

The training objective is a cross-entropy between the student’s output over strongly-augmented views of an image and the teacher’s output over a weakly-augmented view. A centering operation — subtracting a running mean from teacher outputs — prevents dimensional collapse without requiring negative pairs, contrastive loss, or explicit regularization. No labels are used at any point.

A remarkable emergent property is that DINO’s last-layer self-attention heads segment foreground objects with high accuracy despite receiving no segmentation supervision. Each head attends to a different semantically coherent region, and combining them recovers object masks comparable to supervised segmentation baselines on some benchmarks. Caron et al. (2021) reported that a frozen ViT-S/8 trained with DINO achieves 74.5% top-1 accuracy on ImageNet via $k$-NN classification.

From a distillation perspective, DINO demonstrates that the student-teacher asymmetry — strong augmentation for the student, weak augmentation for the teacher — combined with EMA stabilization constitutes a self-supervised signal powerful enough to learn transferable visual representations without any external teacher or label.

Frontiers and Open Problems

Q18 [Basic] Contrast online and offline knowledge distillation

Q: How does online distillation remove the dependency on a pre-trained teacher, and what are the trade-offs relative to offline approaches?

A: In offline distillation, the pipeline is sequential: train a large teacher to convergence, freeze it, then distill into a student. The teacher’s predictions are static throughout student training and represent a fixed quality ceiling.

Online distillation trains teacher and student (or multiple peer networks) simultaneously, removing the requirement for a pre-trained teacher. Deep Mutual Learning (DML; Zhang et al., 2018) is the canonical example: two networks of the same architecture are co-trained, each minimizing cross-entropy on labels plus KL divergence toward the other’s current predictions. Neither network is frozen; both improve through mutual supervision throughout training.

The advantage of online distillation is that no large pre-trained model is needed, which lowers the total training budget. The disadvantage is that early in training both networks are weak, so the soft targets they exchange may carry little useful information or may even be misleading. Offline distillation from a fully converged, high-capacity teacher provides a stable and strong training signal from the first iteration.

Hybrid approaches partially reconcile these trade-offs by building a slowly-moving teacher as an EMA of student checkpoints — the teacher is updated throughout training, but stably enough to provide high-quality soft targets. This is the mechanism DINO (Caron et al., 2021) uses in the self-supervised setting and is closely related to mean-teacher approaches in semi-supervised learning.

Q19 [Advanced] Explain why Born Again Networks improve over same-capacity teachers

Q: What does the success of Born Again Networks reveal about the role of soft targets in knowledge distillation?

A: Born Again Networks (BAN; Furlanello et al., 2018) challenge the standard justification for knowledge distillation. The conventional account holds that a larger, more accurate teacher transfers superior knowledge the student could not acquire from labels alone. BANs refute this by showing that a student of identical architecture and capacity surpasses its teacher when trained with KD from that teacher — a result that cannot be explained by superior teacher knowledge, since the student has the same representational capacity.

The improvement persists across generations: a BAN generation-1 student outperforms the original teacher, and an ensemble of $k$ BAN generations improves further (Furlanello et al., 2018), though gains diminish with each generation and eventually plateau.

Several mechanisms have been proposed to explain the improvement. First, soft targets provide implicit label smoothing: rather than hard 0/1 labels, the student trains against a distribution that penalizes overconfident predictions and yields lower-variance gradient estimates near the decision boundary. Second, the inter-class structure in soft targets encodes dark knowledge — learned inter-class similarities — that constrains the student toward generalization-favoring parameter regions absent from one-hot training. Third, the soft-label objective may distort the loss landscape in ways that favor flatter loss basins with better test generalization, even when the teacher’s test accuracy matches what the student’s capacity would permit under standard training.

The practical implication is that self-distillation is a competitive regularizer even without a larger teacher, often matching or outperforming dropout and label smoothing in specific architectures. It also motivates iterative distillation schedules for further squeezing performance from a fixed-capacity network.

Q20 [Advanced] Analyze data-free knowledge distillation and generative strategies

Q: Why is data-free distillation fundamentally harder than standard KD, and how have generative adversarial approaches addressed the core challenge?

A: Standard knowledge distillation requires data to run through the teacher and obtain soft targets. Data-free KD must perform distillation without access to any real training data — a setting motivated by privacy constraints, licensing restrictions, or scenarios where the original training set is unavailable after deployment.

The core difficulty is that without inputs, the teacher’s knowledge is inaccessible: its soft targets, intermediate activations, and attention maps are all conditioned on data. The student cannot compute any distillation gradient without first synthesizing inputs that activate the teacher meaningfully.

Early methods (Lopes et al., 2017) stored activation statistics — per-layer mean activations and covariance matrices — during teacher training, then reconstructed synthetic training images from these metadata at distillation time. This avoids storing raw data but requires metadata collection upfront.

Generative adversarial approaches avoid this by training a generator $G$ jointly with the student. DAFL (Fang et al., 2019) minimizes a combined objective:

where $\mathcal{L}_\text{one-hot}$ rewards generating images the teacher classifies with high confidence (encouraging class-discriminative synthesis), and $\mathcal{L}_\text{ie}$ rewards information entropy across classes (preventing the generator from collapsing to a single easy class). The student trains on teacher outputs for the generated inputs.

The persistent challenge is generator mode collapse: without diversity constraints, $G$ tends to produce a small set of synthetic images that maximize teacher confidence, covering only a subset of the teacher’s decision surface. Recent approaches use batch normalization statistics from the teacher as a proxy for the real data distribution — these statistics are accessible even without the training data — to regularize the generator toward producing activations that match the feature distribution the teacher was trained on, substantially improving coverage and downstream distillation quality.

Quick Reference

Resources

• Hinton et al., Distilling the Knowledge in a Neural Network (2015)
• Romero et al., FitNets: Hints for Thin Deep Nets (2015)
• Zagoruyko & Komodakis, Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer (2017)
• Park et al., Relational Knowledge Distillation (2019)
• Tian et al., Contrastive Representation Distillation (2020)
• Sanh et al., DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019)
• Jiao et al., TinyBERT: Distilling BERT for Natural Language Understanding (2020)
• Gu et al., MiniLLM: Knowledge Distillation of Large Language Models (2024)
• Agarwal et al., GKD: Generalized Knowledge Distillation for Auto-regressive Sequence Models (2024)
• Kim & Rush, Sequence-Level Knowledge Distillation (2016)
• Touvron et al., Training data-efficient image transformers & distillation through attention (2021)
• Mirzadeh et al., Improved Knowledge Distillation via Teacher Assistant (2020)
• Wang et al., Distilling Object Detectors with Fine-Grained Feature Imbalance (2019)
• Caron et al., Emerging Properties in Self-Supervised Vision Transformers (2021)
• Zhang et al., Deep Mutual Learning (2018)
• Furlanello et al., Born Again Neural Networks (2018)
• Fang et al., Data-Free Adversarial Distillation (2019)
• Lopes et al., Data-Free Knowledge Distillation for Deep Neural Networks (2017)
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding (2023)