Orthogonal Initialization

• simple approach to solving the problem Vanishingexploding gradients
• when applying repeated matrix multiplications, the eigenvalues are what dictate the growth or death of the result
• eigenvalues of orthogonal matrices have an absolute value of 1
• no matter how many matrix multiplications the result doesen’t explode nor vanishes