Learning Rate Decay

Learning Rate Decay deeplearning

• Scale of loss landscape changes
• Reduce step size near optima
• Factor ${\alpha }_{i+1}=d\cdot {\alpha }_i$
• Cosine Learning Rate Decay

…