CBOW
- Continous implementation of Bag of words
- tries to predict the current target word (the center word) based on the source context words (surrounding words)
- “the quick brown fox jumps over the lazy dog”, this can be pairs of (context_window, target_word) where if we consider a context window of size 2, we have examples like ([quick, fox], brown), ([the, brown], quick), ([the, dog], lazy) and so on
- context window
- several times faster to train than the skip-gram, slightly better accuracy for the frequent words.
- CBOW is prone to overfit frequent words because they appear several time along with the same context.
- tends to find the probability of a word occurring in a context
- it generalizes over all the different contexts in which a word can be used
- also a 1-hidden-layer neural network
- The synthetic training task now uses the average of multiple input context words, rather than a single word as in skip-gram, to predict the center word.
- Again, the projection weights that turn one-hot words into averageable vectors, of the same width as the hidden layer, are interpreted as the word embeddings.