Cosine Learning Rate Decay

• Instead of Learning Rate Warmup and then decay
• ${\eta }_{\mathrm{t}}=\frac{1}{2}\left(1+\cos \left(\frac{\mathrm{t}\pi }{\mathrm{T}}\right)\right)\eta$
• Rate decreases slowly at first, then almost linear in the middle and slows down again in the end
•