Maximum Likelihood

Effect

• Consider a model $f[x,\varphi ]$ that computes an output from input $x$.
• Consider the model computes a conditional probability distribution $Pr(Y|x)$ , Y is output
• This encourages each output $y_i$ to have high probability under $Pr(y_i|x_i)$ computed from input $x_i$
• 

Computing a distribution over inputs

• Choose a parametric distribution $Pr(y|\theta )$ defined on output domain y.
• use the network to compute one or more of the parameters $\theta$ of this distribution
• For example, suppose the prediction domain is the set of real numbers, so y ∈ R. Here, we might choose the univariate normal distribution, which is defined on R. This distribution is defined by the mean μ and variance σ2, so θ = {μ,σ^2}. The machine learning model might predict the mean μ, and the variance σ^2 could be treated as an unknown constant.

Maximum likelihood criterion

Log likelihood criterion

Minimizing Log Likelihood Loss

• Negative Log Likelihood

Inference

• When we perform inference, we often want a point estimate rather than a distribution, so we return the maximum of the distribution
•