Attention Alignment

• If there are sequences $x,y$
  • Encoder is any Recurrent with a forward state ${\overrightarrow{h}}^T$ and ${\overleftarrow{h}}^T$ for backward
  • Concat them represents the preceding and following word annotations
    • $h_i=[{\overrightarrow{h}}_i^T;{\overleftarrow{h}}_i^T]$, $i=1,\dots ,n$
    • Decoder has hidden state $s_t=f(s_{t-1},y_{t-1},c_t)$ for the output word at position t for $t=1,\dots ,m$
      • Context vector $c_t$ is a sum of hidden states of the input seq, weighted by alignment scores
      • $c_t={\Sigma }_{i=1}^n{\alpha }_{t,i}{h}_i$
      • How well the two words are aligned is given by
      • ${\alpha }_{t,i}=align(y_t,x_i)$
      • Taking Softmax
        • $\frac{exp(score(s_{t-1},h_i))}{{\Sigma }_{i^{\prime }-1}^{n}exp(score(s_{t-1},h_i^{\prime }))}$
• $f_{att}(h_i,s_j)=v_a^{T}tanh(W_a[h_i;s_j])$
• $v_a$ and $W_a$ are the learned Attention params
• $h$ is the hidden state for the encoder
• $s$ is the hidden state for the decoder
• Matrix of alignment
  • 
  • Final scores calculated with a Softmax