Bayesian Prior

• Use prior knowledge as beliefs (param vectors $\theta$). Cast in the form of a Probability distribution over the space $\Theta$ .
  • Weak knowledge most times
  • For a K parametric PDF $p_x$ , $\Theta \in {\mathbb{R}}^K$ .
  • Not connected to Random variable(RVS).
  • Does not model outcomes. Instead has “beliefs” about true distribution $P_{X_i}$
  • Each $\theta \in {\mathbb{R}}^K$ corresponds to one specific PDF $p_X(\theta )$ → single candidate distribution ${\hat{P}}_X$ for values $x_i$ (In frequentist, it models single data points)
  • Since this is a distribution over Distributions, it is a hyperdistribution
  • N dim PDF $p_{\otimes }{x}_i:{\mathbb{R}}^N\to {\mathbb{R}}^{\ge 0}$ for the distribution of $RV{\otimes }_i{X}_i$
    • $p_{{\otimes }_i}{x}_i((x_i,\dots ,x_N))=p_{x_1},\dots ,p_{x_N}(x_N)={\Pi }_i{p}_X(x_i)$
    • $p_{{\otimes }_i}x(D|\theta )$ → PDF values on a data sample D $p_{{\otimes }_i}{x}_i((x_i,\dots ,x_N))=p_{{\otimes }_i}(\theta )(D)$
  • When $\theta$ is fixed then $p_{{\otimes }_i}x(D|\theta )$ is a function of data vectors D. For each sample, it describes how probable this distribution is assuming the true distribution of X is $p_X(\theta )$
  • When D is fixed, then it is a function of $\theta$. But this does not really measure anything.
    • Integral over $\theta$ is not 1
    • It is a function of $\theta$ and so it is a likelihood function. MLE
    • If given data D → it can show which models are more likely than others.
    • Higher values of $p_{{\otimes }_i}x(D|\theta )$ are better