Chapter 16 - Normalizing Flows

• Warning : If I explain every equation fully, we won’t get lunch. So I will share this document later.
• 

Intro

• [[Articles/Scalar/CycleGANscalarGAN|GANs]] → samples should be as close to real data as possible → hard to identify distribution
• Normalizing flow → take an input distribution → convert it to a more complicated one using a NN → learn a probability model
• 

Algebra

• Jacobian Matrix
• bijective function, surjection

What is a Normalizing Flow

• Density estimation is hard
  • we need to run backprop in deep learning models, the posterior $p(z|x$) should be easy to calculate the derivative of. So, we use a gaussian distribution when we can. (Of course this is not really nice always)
• NF models offer better and more powerful distribution approximation
• Transforms a simple distribution into a complex one by applying a sequence of invertible transformation functions
  • Flowing through a chain of transformations, we repeatedly substitute the variable for the new one according to the change of variables theorem and eventually obtain a probability distribution of the final target variable.
  • 
• only works with integer data

Uses of Normalizing Flows

• 
• Approximating density models

Steps for a Normalizing Flow

• Consider a continuous random variable $z\sim p_{\theta (z)}=\mathcal{N}(z;0,I)$ (simple distribution like Spherical gaussian)
• Transform this distribution using a composition of functions to a more complicated one. $x=f_{\theta }(z)$
• each $f_i$ is invertible (bijective function), so all of the transformations are also invertible $x=f_{\theta }(z)=f_k\circ \dots \circ f_2\circ f_1(z)$
  • 
• So what is the PDF of x?
  • Maybe, $p_{\theta (x)}=p_{\theta }(f_{\theta }^{-1}(x))$
  • but this is not really always true, and depends on the behavior of the function f
  • why?
    • 
    • z = f(x) is where the point in x moves to z
    • the det is the mass that gets pushed in the neighborhood
  • forward and backward mapping
    • 
• From the Change of variable theorem and Jacobian Matrix, we get $p_{\theta }(x)=p_{\theta }(f^{-1}(\mathbf{x}))\left|\det (\frac{\partial f^{-1}(x)}{\partial \mathbf{x}})\right|$
  • Assume $f^{-1}(x)=z$
  • $p_{\theta }(x)=p_{\theta }(z)\left|\det (\frac{\partial z}{\partial \mathbf{x}})\right|$
  • A valid pdf must always integrate to 1
• So what is a normalizing flow?
  • A sequence of bijective transformations is a normalizing flow. After z is initially sampled, it flows through these steps.
    • Density function is re-normalized at each step to ensure it remains valid.
• forward mapping for deep neural networks - -
• Loss function
  • Let us parameterize the likelihood function as a flow
    • We have, $p_{\theta (x)}=p_{\theta }(z)\left|\det (\frac{\partial f^{-1}}{\partial \mathbf{x}})\right|$
    • Applying log on both sides, $\log p_{\theta (x)}=\log p_{\theta }(z)+\log \left|\det (\frac{\partial f^{-1}}{\partial \mathbf{x}})\right|$
  • Overall transformation f is a composition of a sequence of functions
    • replace the log determinant with the sum of the log determinants of the intermediate jacobians
    • $\log p_{\theta (x)}=\log p_{\theta }(z)+{\Sigma }_{i=1}^K\log \left|\det (\frac{\partial f^{-1}}{\partial \mathbf{x}})\right|$
    • Using this, we can now use Maximum Likelihood to train our model since we can calculate the log marginal likelihood exactly
    • Exact posterior inference (unique z for a given x using ($z=f^{-1}(x$))
  • This determinant is now very expensive to compute (of course it is :P), so we ensure that the Jacobian is triangular.
    • determinant of a triangular matrix = product of it’s diagonal
    • to ensure triangular : the paper NICE - non linear independant components estimation uses coupling flows
  • Using coupling flows, we now get $\log p_{\theta (x)}={\Sigma }_{i=1}^D[\log (p_{\theta }(f^{-1}(x{)}_i))-log(|S_{ii}|)]$
• Now it is annoying to only have latent space = dimensionality of the data (need this for a bijection)
  • Density estimation using real NVP

Other Detailed Information (if time permits)

• Properties Required by Network Layers for Normalizing Flows
• How do we handle different types of data - Types of Normalizing flows

Vs Others

• vs. VAE
  • can only get the lower bound on log-likelihood (ELBO loss)
  • approximate posterior $q_{\theta }(z|x)$
• GAN
  • no log likelihood, only min-max evaluation
  • no latent variable inference

References + Resources for the Math

• Ari seff
• lilian weng
• hans van gorp
• 4 hour cvpr tutorial