Adversarial Distillation

• Furthermore, an effective intermediate supervision, i.e., the squeezed knowledge, was used by Shu et al. (2019) to mitigate the capacity gap between the teacher and the student.
• In the third category, adversarial knowledge dis- tillation is carried out in an online manner, i.e., the teacher and the student are jointly optimized in each it- eration (Wang et al., 2018e; Chung et al., 2020)