toc: true title: DeiT
tags: [‘temp’]
DeiT
- paper
- blog
- Conv free Transformer, Vision Transformer
- does not require very large amount of data id:: 62a8a66a-941e-4a6d-918a-bb49cd496b15
- Knowledge Distillation
- teacher-student strategy specific to transformers
- Distillation Token
- ConvNet as teacher through Attention id:: 62a8a6b2-abf4-4869-934e-c75d05884304
- ImageNet
- #+BEGIN_CAUTION Heh. Didnt they say no convs? #+END_CAUTION