- VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
- Self Supervised
- based on maximizing the agreement between Embedding vectors from different views of the same image
- rivial solution is obtained when the encoder outputs constant vectors
- Mode Collapse is often avoided through implicit biases
- explicitly avoids the collapse problem with a simple Regularization term on the variance of the embeddings along each dimension individually
- triple objective: learning invariance to different views with a invariance term, avoiding collapse of the representations with a variance preservation term, and maximizing the information content of the representation with a Covariance Regularization term
- Bias Vs Variance
- combines the variance term with a decorrelation mechanism based on redundancy reduction and Covariance Regularization
- does not require the Embedding branches to be identical or even similar