Long Short Term Memory (LSTM)

• 
• Smaller chance of exploding or vanishingarchitecture
• Better ability to model long term dependencies
• Gated connections
• Gates that learn to forget some aspects, and remember others better
• Splitting state into parts → output pred and Feature Learning
• At the end of the day, these could not handle too long sequences. Therefore → Transformer

The Math

• Gates
  • Forget $g_f=\sigma (W_{hf}{h}_{t-1}+W_{xf}{x}_t+b_f)$
    • How much of the previous cell state is used
  • Input $g_i=\sigma (W_{hi}{h}_{t-1}+W_{xi}{x}_t+b_i)$
    • How proposal is added to the state
  • Output $g_o=\sigma (W_{ho}{h}_{t-1}+W_{xo}{x}_t+b_o)$
    • Component wise products
• Hidden state
  • $C_t$ to model cross timestep dependencies
    • Cell state proposal : $\hat{C}=tanh(W_{hc}{h}_{t-1}+W_{xc}{x}_t+b_c)$
    • Final cell state : $C_t=g_f\cdot C_{t-1}+g_i\cdot \hat{C}$
  • $h_t$ to predict output
    • $h_t=g_o\cdot {\sigma }_y(C_t)$