Embedding

• More complex than 1 hot
• Lookup table is an example.
  • $token\_embedding(i)=gather(W,i)$
• Say vocabulary is (the cat walks)
  • Embedding vector v that will be learnt
  • Values like : $v_{the}$, $v_{cat}$, $v_{walks}$