AdaDelta

• $RMS[\Delta \theta {]}_t=\sqrt{E[\Delta {\theta }^2{]}_t+ϵ}$

& \Delta \theta_{t}= -\frac{RMS[\Delta \theta]_{t-1}}{RMS[g]_{t}}g_{t}\\ & \theta_{t+1}= \theta_{t}+ \Delta \theta_{t} \end{align}$$