Statistical Mechanics of Neural Networks

Description

Preface Contents About the Author Acronyms 1 Introduction References 2 Spin Glass Models and Cavity Method 2.1 Multi-spin Interaction Models 2.2 Cavity Method 2.3 From Cavity Method to Message Passing Algorithms References 3 Variational Mean-Field Theory and Belief Propagation 3.1 Variational Method 3.2 Variational Free Energy 3.2.1 Mean-Field Approximation 3.2.2 Bethe Approximation 3.2.3 From the Bethe to Naive Mean-Field Approximation 3.3 Mean-Field Inverse Ising Problem References 4 Monte Carlo Simulation Methods 4.1 Monte Carlo Method 4.2 Importance Sampling 4.3 Markov Chain Sampling 4.4 Monte Carlo Simulations in Statistical Physics 4.4.1 Metropolis Algorithm 4.4.2 Parallel Tempering Monte Carlo References 5 High-Temperature Expansion 5.1 Statistical Physics Setting 5.2 High-Temperature Expansion 5.3 Properties of the TAP Equation References 6 Nishimori Line 6.1 Model Setting 6.2 Exact Result for Internal Energy 6.3 Proof of No RSB Effects on the Nishimori Line References 7 Random Energy Model 7.1 Model Setting 7.2 Phase Diagram References 8 Statistical Mechanical Theory of Hopfield Model 8.1 Hopfield Model 8.2 Replica Method 8.2.1 Replica-Symmetric Ansätz 8.2.2 Zero-Temperature Limit 8.3 Phase Diagram 8.4 Hopfield Model with Arbitrary Hebbian Length 8.4.1 Computation of the Disorder-Averaged Free Energy 8.4.2 Derivation of Saddle-Point Equations 8.4.3 Computation Transformation to Solve the SDE 8.4.4 Zero-Temperature Limit References 9 Replica Symmetry and Replica Symmetry Breaking 9.1 Generalized Free Energy and Complexity of States 9.2 Applications to Constraint Satisfaction Problems 9.3 More Steps of Replica Symmetry Breaking References 10 Statistical Mechanics of Restricted Boltzmann Machine 10.1 Boltzmann Machine 10.2 Restricted Boltzmann Machine 10.3 Free Energy Calculation 10.4 Thermodynamic Quantities Related to Learning 10.5 Stability Analysis 10.6 Variational Mean-Field Theory for Training Binary RBMs 10.6.1 RBMs with Binary Weights 10.6.2 Variational Principle 10.6.3 Experiments References 11 Simplest Model of Unsupervised Learning with Binary Synapses 11.1 Model Setting 11.2 Derivation of sMP and AMP Equations 11.3 Replica Computation 11.3.1 Explicit form of langleZn rangle 11.3.2 Estimation of langleZnrangle Under Replica Symmetry Ansätz 11.3.3 Derivation of Free Energy and Saddle-Point Equations 11.4 Phase Transitions 11.5 Measuring the Temperature of Dataset References 12 Inherent-Symmetry Breaking in Unsupervised Learning 12.1 Model Setting 12.1.1 Cavity Approximation 12.1.2 Replica Computation 12.1.3 Stability Analysis 12.2 Phase Diagram 12.3 Hyper-Parameters Inference References 13 Mean-Field Theory of Ising Perceptron 13.1 Ising Perceptron model 13.2 Message-Passing-Based Learning 13.3 Replica Analysis 13.3.1 Replica Symmetry 13.3.2 Replica Symmetry Breaking 13.4 Further Theory Development References 14 Mean-Field Model of Multi-layered Perceptron 14.1 Random Active Path Model 14.1.1 Results from Cavity Method 14.1.2 An Infinite Depth Analysis 14.2 Mean-Field Training Algorithms 14.3 Spike and Slab Model 14.3.1 Ensemble Perspective 14.3.2 Training Equations References 15 Mean-Field Theory of Dimension Reduction 15.1 Mean-Field Model 15.2 Linear Dimensionality and Correlation Strength 15.2.1 Iteration Equations for Correlation Strength 15.2.2 Mechanism of Dimension Reduction 15.3 Dimension Reduction with Correlated Synapses 15.3.1 Model Setting 15.3.2 Mean-Field Calculation 15.3.3 Numerical Results Compared with Theory References 16 Chaos Theory of Random Recurrent Neural Networks 16.1 Spiking and Rate Models 16.2 Dynamical Mean-Field Theory 16.2.1 Dynamical Mean-Field Equation 16.2.2 Regimes of Network Dynamics 16.3 Lyapunov Exponent and Chaos 16.4 Excitation-Inhibition Balance Theory 16.5 Training Recurrent Neural Networks 16.5.1 Force-Training 16.5.2 Backpropagation Through Time References 17 Statistical Mechanics of Random Matrices 17.1 Spectral Density 17.2 Replica Method and Semi-circle Law 17.3 Cavity Approach and Marchenko–Pastur Law 17.4 Spectral Densities of Random Asymmetric Matrices References 18 Perspectives References