Chapter 9 - Regularization

Skeptic : Mixup Skeptic: How does this tie into Optimizers Skeptic : Label Smoothing : Mentioned later Batch Normalization - The idea is to normalize the inputs of each layer in such a way that, they have a mean activation output zero and a unit standard deviation. Skeptic : AdamW

• Regularization

Explicit Regularization

• Regularization Term
• From Maximum Likelihood,
  • 
• 
• from Negative Log Likelihood

Lp Regularization

Gradient Descent

SGD

Improving Performance

• Early Stopping tricks
• Ensembling models
• Dropout
• Adversarial Learning
• Bayesian
• Transfer Learning
• Multi Task Learning
• augmentation
• Label Smoothing
• Self Supervised