Label Smoothing

• Dense layer is generally the last one and combined with soft max leads to a Probability distribution
• Assume true label to be y, then a truth Probability distribution would be $p_i=1$ If i=y and 0 otherwise
• During training, minimize negative Cross Entropy loss to make these Distributions similar
• We know, $l(p,q)=-log{p}_y=-z_y+log({\Sigma }_{i=1}^{K}exp(z_i))$
• Where the optimal solution is $z_y^{\ast }=\inf$
  • The output scores are encouraged to be distinctive which leads to overfitting
  • Leads to
• Instead \cases{1-\epsilon& if i=1\\\frac{\epsilon}{(K-1)} & \text{otherwise}}
• The optimal Solution is
  • $log((K-1)(1-ϵ)/ϵ)+\alpha$ if $i=y$
  • $\alpha$ otherwise
    • Any real number
    • Finite output from the last layer that generalizes well
• If $ϵ=0$ , $log((k-1)\frac{1-ϵ}{ϵ})$ is $\infty$
• As $ϵ$ increases, the gap decreases
• If $ϵ=\frac{K-1}{K}$, all optimizal $z_i^{\ast }$ are identical