Layer Normalization

• For RNNs etc
• Mean and variance calculated independantly for each element of the batch by aggregating over the Features dimensions.
• (Compared to Batch Normalization)

$$\begin{array}{rl}& \\ & {\mu }_{\mathcal{B}}\leftarrow \frac{1}{m}{\Sigma }_{i=1}^m{x}_i\\ & {\sigma }_{\mathcal{B}}^2\leftarrow \frac{1}{m}{\Sigma }_{i=1}^m(x_i-{\mu }_{\mathcal{B}}{)}^2\\ & {\hat{x}}_i\leftarrow \frac{x_i-{\mu }_{\mathcal{B}}}{\sqrt{{\sigma }_{\mathcal{B}}^2+ϵ}}\\ & y_i=\gamma {\hat{x}}_i+\beta \end{array}$$

Problem

• From [Visualizing the Loss Landscape of Neural Nets](Visualizing the Loss Landscape of Neural Nets.md),
•