Gated Recurrent Unit (GRU)

• 
• Simplified [LSTM)](Long Short Term Memory (LSTM|Long Short Term Memory (LSTM|LSTM)](LSTM)](Long Short Term Memory (LSTM|Long Short Term Memory (LSTM|LSTM).md).md)
• It has an input and forget gate, no output gate
• Faster than LSTM in training, but does not perform well in many tasks
• Tries to forget what is not important

The Math

• Two gates, Sigmoid
  • Reset : $g_r=\sigma (W_{hr}{h}_{t-1}+W_{xr}{x}_t+b_r)$
  • Update : $g_u=\sigma (W_{hu}{h}_{t-1}+W_{xu}{x}_t+b_u)$
• Hidden state proposal
  • ${\hat{h}}_t=tanh(W_{xh}{x}_t+W_{hh}{g}_r\cdot h_{t-1}+b_h)$
• Final hidden state
  • Linear Interpolation between last hidden state and proposal
  • $h_t=(1-g_u)\cdot h_{t-1}+g_u\cdot {\hat{h}}_t$