Scaled Dot Product Attention

• Vaswani et al., 2017
• Q is query, K is key V is value. Same dims
• $q_i=W_q{x}_i$ , $k_i=W_k{x}_i$ , $v_i=W_v{x}_i$
  • $w_{ij}^{\prime }=q_i^T{k}_j$
  • $y_i={\Sigma }_j{w}_{ij}{v}_j$
• Softmax is sensitive to large values. Which sucks for thearchitecture
• The avg value of the dot product grows with Embedding dimension k. So scale back.
  • $\sqrt{k}$ . Vector in ${\mathbb{R}}^k$ with all values as c
  • Euclidean length is $\sqrt{kc}$
• $Attention(Q,K,V)=softmax(\frac{Q{K}^T}{\sqrt{d_k}})V$
• Generalization of Soft Attention
• 
• Attention Alignment score ${\alpha }_{t,i}=\frac{s_t^T{h}_i}{\sqrt{n}}$