Term

• Penalty term
• Cost function that penalizes model params $\theta$ with a high degree of flexibility
• $h_{opt}=argmi{n}_{h\in \mathcal{H}}\frac{1}{N}{\Sigma }_{i=1}^{N}L(h(x_i),y_i)+{\alpha }^{2}R({\theta }_h)$
• ${\alpha }^2$ determines how much regularizer affects the model
  • Larger : soft models
  • Increasing → Down regulating flexibility
  • 0 = overfitting and unregularized risk
  • $\infty$ does not care about training data at all. Only cares about minimal penalty
    • Dead model

Types

• Lp Regularization for p =2
  • Soft models
  • Squared sum of model params