Statistical Machine Learning: A Unified Framework (Chapman & Hall/CRC Texts in Statistical Science)

Description

The recent rapid growth in the variety and complexity of new machine learning architectures requires the development of improved methods for designing, analyzing, evaluating, and communicating machine learning technologies. Statistical Machine Learning: A Unified Framework provides students, engineers, and scientists with tools from mathematical statistics and nonlinear optimization theory to become experts in the field of machine learning. In particular, the material in this text directly supports the mathematical analysis and design of old, new, and not-yet-invented nonlinear high-dimensional machine learning algorithms. Features: Unified empirical risk minimization framework supports rigorous mathematical analyses of widely used supervised, unsupervised, and reinforcement machine learning algorithms Matrix calculus methods for supporting machine learning analysis and design applications Explicit conditions for ensuring convergence of adaptive, batch, minibatch, MCEM, and MCMC learning algorithms that minimize both unimodal and multimodal objective functions Explicit conditions for characterizing asymptotic properties of M-estimators and model selection criteria such as AIC and BIC in the presence of possible model misspecification This advanced text is suitable for graduate students or highly motivated undergraduate students in statistics, computer science, electrical engineering, and applied mathematics. The text is self-contained and only assumes knowledge of lower-division linear algebra and upper-division probability theory. Students, professional engineers, and multidisciplinary scientists possessing these minimal prerequisites will find this text challenging yet accessible. About the Author: Richard M. Golden (Ph.D., M.S.E.E., B.S.E.E.) is Professor of Cognitive Science and Participating Faculty Member in Electrical Engineering at the University of Texas at Dallas. Dr. Golden has published articles and given talks at scientific conferences on a wide range of topics in the fields of both statistics and machine learning over the past three decades. His long-term research interests include identifying conditions for the convergence of deterministic and stochastic machine learning algorithms and investigating estimation and inference in the presence of possibly misspecified probability models. Cover Half Title Series Page Title Page Copyright Page Contents Symbols Preface Part I: Inference and Learning Machines 1. A Statistical Machine Learning Framework 1.1 Statistical Machine Learning: A Brief Overview 1.2 Machine Learning Environments 1.2.1 Feature Vectors 1.2.1.1 Feature Vector Representations 1.2.1.2 The Curse of Feature Vector Dimensionality 1.2.1.3 Feature Vector Engineering Methods 1.2.2 Stationary Statistical Environments 1.2.3 Strategies for Teaching Machine Learning Algorithms 1.2.4 Prior Knowledge 1.2.4.1 Feature Maps Specify Relevant Statistical Regularities 1.2.4.2 Similar Inputs Predict Similar Responses 1.2.4.3 Only a Small Number of Model Parameters Are Relevant 1.2.4.4 Different Feature Detectors Share Parameters 1.3 Empirical Risk Minimization Framework 1.3.1 ANN Graphical Notation 1.3.2 Risk Functions 1.3.3 Regularization Terms 1.3.4 Optimization Methods 1.3.4.1 Batch Gradient Descent Learning 1.3.4.2 Adaptive Gradient Descent Learning 1.4 Theory-Based System Analysis and Design 1.4.1 Stage 1: System Specification 1.4.2 Stage 2: Theoretical Analyses 1.4.3 Stage 3: Physical Implementation 1.4.4 Stage 4: System Behavior Evaluation 1.5 Supervised Learning Machines 1.5.1 Discrepancy Functions 1.5.2 Basis Functions and Hidden Units 1.5.3 Recurrent Neural Networks (RNNs) 1.6 Unsupervised Learning Machines 1.7 Reinforcement Learning Machines 1.7.1 Reinforcement Learning Overview 1.7.2 Value Function Reinforcement Passive Learning 1.7.3 Policy Gradient Reinforcement Reactive Learning 1.8 Further Readings 2. Set Theory for Concept Modeling 2.1 Set Theory and Logic 2.2 Relations 2.2.1 Types of Relations 2.2.2 Directed Graphs 2.2.3 Undirected Graphs 2.3 Functions 2.4 Metric Spaces 2.5 Further Readings 3. Formal Machine Learning Algorithms 3.1 Environment Models 3.1.1 Time Models 3.1.2 Event Environments 3.2 Machine Models 3.2.1 Dynamical Systems 3.2.2 Iterated Maps 3.2.3 Vector Fields 3.3 Intelligent Machine Models 3.4 Further Readings Part II: Deterministic Learning Machines 4. Linear Algebra for Machine Learning 4.1 Matrix Notation and Operators 4.2 Linear Subspace Projection Theorems 4.3 Linear System Solution Theorems 4.4 Further Readings 5. Matrix Calculus for Machine Learning 5.1 Convergence and Continuity 5.1.1 Deterministic Convergence 5.1.2 Continuous Functions 5.2 Vector Derivatives 5.2.1 Vector Derivative Definitions 5.2.2 Theorems for Computing Matrix Derivatives 5.2.3 Useful Derivative Calculations for Deep Learning 5.2.4 Gradient Backpropagation for Deep Learning 5.3 Objective Function Analysis 5.3.1 Taylor Series Expansions 5.3.2 Gradient Descent Type Algorithms 5.3.3 Critical Point Classification 5.3.3.1 Identifying Critical Points 5.3.3.2 Identifying Flat Regions 5.3.3.3 Identifying Local Minimizers 5.3.3.4 Identifying Saddlepoints 5.3.3.5 Identifying Global Minimizers 5.3.4 Lagrange Multipliers 5.4 Further Readings 6. Convergence of Time-Invariant Dynamical Systems 6.1 Dynamical System Existence Theorems 6.2 Invariant Sets 6.3 Lyapunov Convergence Theorems 6.3.1 Lyapunov Functions 6.3.2 Invariant Set Theorems 6.3.2.1 Finite State Space Convergence Analyses 6.3.2.2 Continuous State Space Convergence Analyses 6.4 Further Readings 7. Batch Learning Algorithm Convergence 7.1 Search Direction and Stepsize Choices 7.1.1 Search Direction Selection 7.1.2 Stepsize Selection 7.2 Descent Algorithm Convergence Analysis 7.3 Descent Strategies 7.3.1 Gradient and Steepest Descent 7.3.2 Newton-Type Descent 7.3.2.1 Newton-Raphson Algorithm 7.3.2.2 Levenberg-Marquardt Algorithm 7.3.3 L-BFGS and Conjugate Gradient Descent Methods 7.4 Further Readings Part III: Stochastic Learning Machines 8. Random Vectors and Random Functions 8.1 Probability Spaces 8.1.1 Sigma-Fields 8.1.2 Measures 8.2 Random Vectors 8.2.1 Measurable Functions 8.2.2 Discrete, Continuous, and Mixed Random Vectors 8.3 Existence of the Radon-Nikodým Density (Optional Reading) 8.3.1 Lebesgue Integral 8.3.2 The Radon-Nikod ym Probability Density Function 8.3.3 Vector Support Specification Measures 8.4 Expectation Operations 8.4.1 Random Functions 8.4.2 Expectations of Random Functions 8.4.3 Conditional Expectation and Independence 8.5 Concentration Inequalities 8.6 Further Readings 9. Stochastic Sequences 9.1 Types of Stochastic Sequences 9.2 Partially Observable Stochastic Sequences 9.3 Stochastic Convergence 9.3.1 Convergence with Probability One 9.3.2 Convergence in Mean Square 9.3.3 Convergence in Probability 9.3.4 Convergence in Distribution 9.3.5 Stochastic Convergence Relationships 9.4 Combining and Transforming Stochastic Sequences 9.5 Further Readings 10. Probability Models of Data Generation 10.1 Learnability of Probability Models 10.1.1 Correctly Specified and Misspecified Models 10.1.2 Smooth Parametric Probability Models 10.1.3 Local Probability Models 10.1.4 Missing-Data Probability Models 10.2 Gibbs Probability Models 10.3 Bayesian Networks 10.3.1 Factoring a Chain 10.3.2 Bayesian Network Factorization 10.4 Markov Random Fields 10.4.1 The Markov Random Field Concept 10.4.2 MRF Interpretation of Gibbs Distributions 10.5 Further Readings 11. Monte Carlo Markov Chain Algorithm Convergence 11.1 Monte Carlo Markov Chain (MCMC) Algorithms 11.1.1 Countably Infinite First-Order Chains on Finite State Spaces 11.1.2 Convergence Analysis of Monte Carlo Markov Chains 11.1.3 Hybrid MCMC Algorithms 11.1.4 Finding Global Minimizers and Computing Expectations 11.1.5 Assessing and Improving MCMC Convergence Performance 11.1.5.1 Assessing Convergence for Expectation Approximation 11.1.5.2 MCMC Convergence Challenges and Heuristics 11.2 MCMC Metropolis-Hastings (MH) Algorithms 11.2.1 Metropolis-Hastings Algorithm Definition 11.2.2 Convergence Analysis of Metropolis-Hastings Algorithms 11.2.3 Important Special Cases of the Metropolis-Hastings Algorithm 11.2.4 Metropolis-Hastings Machine Learning Applications 11.3 Further Readings 12. Adaptive Learning Algorithm Convergence 12.1 Stochastic Approximation (SA) Theory 12.1.1 Passive versus Reactive Statistical Environments 12.1.1.1 Passive Learning Environments 12.1.1.2 Reactive Learning Environments 12.1.2 Average Downward Descent 12.1.3 Annealing Schedule 12.1.4 The Main Stochastic Approximation Theorem 12.1.5 Assessing Stochastic Approximation Algorithm Convergence 12.2 Learning in Passive Statistical Environments Using SA 12.2.1 Implementing Different Optimization Strategies 12.2.2 Improving Generalization Performance 12.3 Learning in Reactive Statistical Environments Using SA 12.3.1 Policy Gradient Reinforcement Learning 12.3.2 Stochastic Approximation Expectation Maximization 12.3.3 Markov Random Field Learning (Contrastive Divergence) 12.3.4 Generative Adversarial Network (GAN) Learning 12.4 Further Readings Part IV: Generalization Performance 13. Statistical Learning Objective Function Design 13.1 Empirical Risk Function 13.2 Maximum Likelihood (ML) Estimation Methods 13.2.1 ML Estimation: Probability Theory Interpretation 13.2.2 ML Estimation: Information Theory Interpretation 13.2.2.1 Entropy 13.2.2.2 Cross-Entropy Minimization: ML Estimation 13.2.3 Properties of Cross-Entropy Global Minimizers 13.2.4 Pseudolikelihood Empirical Risk Function 13.2.5 Missing-Data Likelihood Empirical Risk Function 13.3 Maximum a Posteriori (MAP) Estimation Methods 13.3.1 Parameter Priors and Hyperparameters 13.3.2 Maximum a Posteriori (MAP) Risk Function 13.3.3 Bayes Risk Interpretation of MAP Estimation 13.4 Further Readings 14. Simulation Methods for Evaluating Generalization 14.1 Sampling Distribution Concepts 14.1.1 K-Fold Cross-Validation 14.1.2 Sampling Distribution Estimation with Unlimited Data 14.2 Bootstrap Methods for Sampling Distribution Simulation 14.2.1 Bootstrap Approximation of Sampling Distribution 14.2.2 Monte Carlo Bootstrap Sampling Distribution Estimation 14.3 Further Readings 15. Analytic Formulas for Evaluating Generalization 15.1 Assumptions for Asymptotic Analysis 15.2 Theoretical Sampling Distribution Analysis 15.3 Con dence Regions 15.4 Hypothesis Testing for Model Comparison Decisions 15.4.1 Classical Hypothesis Testing 15.4.2 Bayesian Hypothesis Testing 15.5 Further Readings 16. Model Selection and Evaluation 16.1 Cross-Validation Risk Model Selection Criteria 16.2 Bayesian Model Selection Criteria 16.2.1 Bayesian Model Selection Problem 16.2.2 Laplace Approximation for Multidimensional Integration 16.2.3 Bayesian Information Criteria 16.3 Model Misspecification Detection Model Selection Criteria 16.3.1 Nested Models Method for Assessing Model Misspecification 16.3.2 Information Matrix Discrepancy Model Selection Criteria 16.4 Further Readings References Algorithm Index Subject Index