Dynamic Sparsity

• train intrinsically sparse neural networks from scratch using only a small proportion of parameters and FLOPs
• Dynamic Sparsity enables training sparse models from scratch, hence the training and inference FLOPs and memory requirements are only a small fraction of the dense models.
• models built with dynamic Sparsity can be trained from scratch to match their dense counterparts without involving any pre-training or dense training