Batch Normalization

• bias=False for Linear/Conv2D for input and True for outputdeeplearning
• Normalizesarchitecture
• Input Distributions change per layer → Make sure they stay similar
• Reduces co variate shift because now the network must adapt per layer
• During testing : use stats saved during training
• Simplifies learning dynamics
  • Can use larger learning rate
  • Higher order interactions are suppressed because the mean and std are independant of the activations which makes training easier
• Cant work with small batches. Not great with RNN
• 
• ${\mu }_j\leftarrow \frac{1}{m}{\Sigma }_{i=1}^m{x}_{ij}$
• ${\sigma }_j^2\leftarrow \frac{1}{m}{\Sigma }_{i=1}^m(x_{ij}-{\mu }_j{)}^2$
• ${\hat{x}}_{ij}\leftarrow \frac{x_{ij}-{\mu }_j}{\sqrt{{\sigma }_j^2+ϵ}}$
• ${\hat{x}}_{ij}\leftarrow \gamma {\hat{x}}_{ij}+\beta$

Why

• Stable forward prop
• Higher learning rates
• Regularization