GELU

• Paper
• Smoother Relu
• $x\Phi (x)$ where $\Phi (x)$ is the Normal Distribution CDF
• Weights inputs by percentile, rather than by sign like Relu
• $GELU(x)=xP(X\le x)=x\Phi (x)=x.\frac{1}{2}\left[1+erf\left(\frac{x}{\sqrt{2}}\right)\right]$
• If $X\sim \mathcal{N}(0,1)$
• Used in GPT3, Transformer, Vision Transformer, BERT
•