Normalization Techniques in Deep Learning

Description

​This book presents and surveys normalization techniques with a deep analysis in training deep neural networks.  In addition, the author provides technical details in designing new normalization methods and network architectures tailored to specific tasks.  Normalization methods can improve the training stability, optimization efficiency, and generalization ability of deep neural networks (DNNs) and have become basic components in most state-of-the-art DNN architectures.  The author provides guidelines for elaborating, understanding, and applying normalization methods.  This book is ideal for readers working on the development of novel deep learning algorithms and/or their applications to solve practical problems in computer vision and machine learning tasks.  The book also serves as a resource researchers, engineers, and students who are new to the field and need to understand and train DNNs. Preface Reference Acknowledgements References Contents About the Author 1 Introduction [DELETE] 1.1 Denotations and Definitions 1.1.1 Optimization Objective 1.1.2 Neural Networks 1.1.3 Training DNNs 1.1.4 Normalization 2 Motivation and Overview of Normalization in DNNs [DELETE] 2.1 Theory of Normalizing Input 2.2 Towards Normalizing Activations 2.2.1 Proximal Back-Propagation Framework 2.2.2 K-FAC Approximation 2.2.3 Highlights of Motivation 3 A General View of Normalizing Activations [DELETE] 3.1 Normalizing Activations by Population Statistics 3.2 Local Statistics in a Sample 3.3 Batch Normalization 4 A Framework for Normalizing Activations as Functions [DELETE] 4.1 Normalization Area Partitioning 4.2 Normalization Operation 4.2.1 Beyond Standardization Towards Whitening 4.2.2 Variations of Standardization 4.2.3 Reduced Standardization 4.3 Normalization Representation Recovery 5 Multi-mode and Combinational Normalization [DELETE] 5.1 Multiple Modes 5.2 Combination 6 BN for More Robust Estimation [DELETE] 6.1 Normalization as Functions Combining Population Statistics 6.2 Robust Inference Methods for BN 7 Normalizing Weights [DELETE] 7.1 Constraints on Weights 7.2 Training with Constraints 8 Normalizing Gradients [DELETE] 9 Analysis of Normalization [DELETE] 9.1 Scale Invariance in Stabilizing Training 9.1.1 Auto-Tuning on Learning Rate 9.2 Improved Conditioning in Optimization 9.3 Stochasticity for Generalization 9.3.1 Theoretical Model for Stochasticity 9.3.2 Empirical Analyses for Stochasticity 9.4 Effects on Representation 9.4.1 Constraint on Feature Representation 9.4.2 Effect on Representational Capacity of Model 10 Normalization in Task-Specific Applications [DELETE] 10.1 Domain Adaptation 10.1.1 Domain Generalization 10.1.2 Robust Deep Learning Under Covariate Shift 10.1.3 Learning Universal Representations 10.2 Image Style Transfer 10.2.1 Image Translation 10.3 Training GANs 10.4 Efficient Deep Models 11 Summary and Discussion A Appendix A.1 Back-Propagation Through Eigenvalue Decomposition A.2 Derivation of Constraint Number of Normalization Methods A.3 Proofs of Theorems