Data Free Distillation

• Just as “data free” implies, there is no training data. Instead, the data is newly or synthetically generated.
• Specifically, in (Chen et al., 2019a; Ye et al., 2020; Micaelli and Storkey, 2019; Yoo et al., 2019; Hu et al., 2020), the transfer data is generated by a GAN. In the proposed data-free knowledge distillation method (Lopes et al., 2017), the transfer data to train the student network is reconstructed by using the layer ac- tivations or layer spectral activations of the teacher net- work.
• Yin et al. (2020) proposed DeepInversion, which uses knowledge distillation to generate synthesized images for data-free knowledge transfer. Nayak et al. (2019) proposed zero-shot knowledge distillation that does not use existing data.
• The transfer data is pro- duced by modelling the softmax space using the pa- rameters of the teacher network. In fact, the target data in (Micaelli and Storkey, 2019; Nayak et al., 2019) is generated by using the information from the fea- ture representations of teacher networks.
• distilling knowl- edge from a teacher model into a student neural network (Kimura et al., 2018; Shen et al., 2021).
• data distillation, which is similar to data-free distillation (Radosavovic et al., 2018; Liu et al., 2019d; Zhang et al., 2020d). In data distillation, new training annotations of unlabeled data generated from the teacher model are employed to train a student model.