Distillation Loss

• $l(p,softmax(z))+T^{2}l(softmax(\frac{r}{T}),softmax(\frac{z}{T}))$
• Negative Cross Entropy + other
• p is the true Probability Distributions
• z,r are outputs of the student and teacher model
• T is the temperature to make Softmax smoother