Linear Regression

• Minimization problem $(w,b)=argmi{n}_{w^{\ast },b^{\ast }}{\Sigma }_{i-1}^N(w^{\ast }{x}_i+b^{\ast }-y_i{)}^2$
  • Affine Function
  • $w=argmi{n}_{w^{\ast }}||({\Sigma }_{j=1}^n{w}_j^{\ast }{\varphi }_j)-y|{|}^2$
    • w* is just $(w_1^{\ast }\dots w_n^{\ast })$
    • $u_j$ of U forms orthonormal basis of $\mathcal{F}$
• X : nxN matrix , y : N dim vector
• Solution : $[w,b]\in {\mathbb{R}}^{n+1}$
• $w^{\prime }=(X{X}^{\prime }{)}^{-1}Xy$
  • If y is has vector data too (size k)
    • $W^{\prime }=(X{X}^{\prime }{)}^{-1}XY$
    • Y : N x k matrix
• 
• ${\varphi }_1,{\varphi }_2\dots$ form a subspace $\mathcal{F}$ with dim = n
  • linearly independant vectors. If not, drop as many as possible
• The optimal solution y_opt is the projection of y on that subspace and has the smallest distance from y
  • $y_{opt}=w_1{\varphi }_1+w_2{\varphi }_2$
• $({\Sigma }_{j=1}^n{w}_j^{\ast }{\varphi }_j)$ is a vector on $\mathcal{F}$
• Ridge Regression
• Window Based Regression

General Defination

• Training data : $(x_i,y_i{)}_{i=1,..,N}$ and $\in {\mathbb{R}}^k$

• Search space H

  • Candidate functions $h:{\mathbb{R}}^n\to {\mathbb{R}}^k$

• Loss function $L:{\mathbb{R}}^k\times {\mathbb{R}}^k\to {\mathbb{R}}^{n\ge 0}$

  • Quadratic Loss

• Solution : $h_{opt}=argmi{n}_{h\in \mathcal{H}}{\Sigma }_{i=1}^{N}L(h(x_i),y_i)$

  • $\mathcal{H}$ is all linear functions from ${\mathbb{R}}^n$ to ${\mathbb{R}}^k$

• [Left psuedo inverse](Left psuedo inverse.md)