Sigmoid

• $\sigma (x)=\frac{1}{1+exp(-x)}$
• $\frac{d\sigma }{dx}(x)=\sigma (x)(1-\sigma (x))$
  • max : 0.25
• Logistic
• Xavier/Glorot init
• RNN : Hidden
• Bernoulli Distribution over a binary variable
• 
•