NADAM

• extension of Adam that uses the Nesterov momentum technique to accelerate the optimization convergence further
• combines the momentum of Nesterov’s method with the adaptive learning rates of Adam
• $m_t={\beta }_1\ast m_{(t-1)}+(1-{\beta }_1)\ast g_t$ $v_t={\beta }_2\ast v_{(t-1)}+(1-{\beta }_2)\ast g_{t}2$ $\widehat{m_t}=\frac{m_t}{1-{\beta }_{1}t}$ $\widehat{v_t}=\frac{v_t}{1-{\beta }_{2}t}$ $\text{parameter}=\text{parameter}-learning\_rate\ast \frac{\widehat{m_t}+(1-{\beta }_1)\ast g_t}{\sqrt{\widehat{v_t}}+ϵ}$
• Where beta1 and beta2 are two hyperparameters, m_t and v_t are moving averages of the gradients, g_t is the gradient at time t, and learning_rate; epsilon is the same as before.