Phase Transition Model Zoo

• konstantin schurholt
• transfer learning, model averaging, scaling models does not always work

Phase Transition in Physical Systems

• 
• between phases is an interpolation
• order params with some control params
  • eg : temp and pressure in water
• are there phases in NN?

Regimes in NN

• Double Descent
• Gradient Descent
• Scaling loss : kaplan et al.20
• transitions between regimes can be sharp or smooth
• Understanding deep learning still requires rethinking generalization

Statistical Mechanics

• eg : Hopfield networks , Perceptron, tree parity machine

Control Params in NN

• temperature and load are relative

Temp

• 

Load

• 

Emperical Phases in NN Performance

• 
• taxonomizing local vs global
  • https://proceedings.neurips.cc/paper/2021/hash/9b72e31dac81715466cd580a448cf823-Abstract.html
• expected phases emerge on the load temperature landscape
• load : complexity/capactity
• temperature : noisiness of training
• 5 phases, phase IV-B is the best
• ~5k models trained from scratch
• hessian : minima
  • flatness of the minima
    • flatter minima generalizes better : depending on where you are
  • 
• mode connectivity
  • 
  • models trained with same config + different seed
  • if so, find a path where the loss is the same
  • 
• 

Phase Transition in Model Zoo

• phase transitions are a general phenomenon in NNs
• 
• all of these requires datasets of models that cover phase transitions
• since temp/load are basically have the same effects, didnt train every possibility
• 
  • Vision Transformer are very different
    • big optimal phase
    • for CIFAR, bad performance without data augmentation
      • (could also be because the dataset has a lot of noise)
• 

Applications

• research on transitions
  • might identify root causes for performance
  • 
  • 
  •