Backprop

• Gradient $\nabla l(\theta )=[\frac{\partial l}{\partial {\theta }_1}(\theta ),\dots ,\frac{\partial L}{\partial {\theta }_L}(\theta )]$
  • partial derivs of the loss wrt weights Forward pass
    • Store result of operation in u
  • Backward Pass
    • Traverse the graph backwards
      • Chain Rule : $\frac{dl}{d{\theta }_i}={\Sigma }_{k\in parents(l)}\frac{\partial l}{\partial u_k}\frac{\partial u_k}{\partial {\theta }_i}$
      • \begin{align} &\frac{d\hat y}{d\mathbf{W_1}}\\ &= \frac{\partial \hat y}{\partial u_2} \frac{\partial u_2}{\partial h_1} \frac{\partial h_1}{\partial u_1} \frac{\partial u_1}{\partial \mathbf{W_1}} \\ &= \frac{\partial \sigma (u2)}{\partial u_2} \frac{\partial \mathbf{W}^T_2 h_1}{\partial h_1} \frac{\partial \sigma (u1)}{\partial u_1} \frac{\partial \mathbf{W}^T_1 x}{\partial \mathbf{W}_1} \end{align}
      • Collecting all the $\partial \sigma (u_i)$ wrt params →architecture exponentially decreases wrt depth of the network : Vanishing
        • Solved by Activation Functions