Multivariate Data Analysis on Matrix Manifolds: (with Manopt)

Description

This graduate-level textbook aims to give a unified presentation and solution of several commonly used techniques for multivariate data analysis (MDA). Unlike similar texts, it treats the MDA problems as optimization problems on matrix manifolds defined by the MDA model parameters, allowing them to be solved using (free) optimization software Manopt. The book includes numerous in-text examples as well as Manopt codes and software guides, which can be applied directly or used as templates for solving similar and new problems. The first two chapters provide an overview and essential background for studying MDA, giving basic information and notations. Next, it considers several sets of matrices routinely used in MDA as parameter spaces, along with their basic topological properties. A brief introduction to matrix (Riemannian) manifolds and optimization methods on them with Manopt complete the MDA prerequisite. The remaining chapters study individual MDA techniques in depth. The number of exercises complement the main text with additional information and occasionally involve open and/or challenging research questions. Suitable fields include computational statistics, data analysis, data mining and data science, as well as theoretical computer science, machine learning and optimization. It is assumed that the readers have some familiarity with MDA and some experience with matrix analysis, computing, and optimization.  Preface Contents 1 Introduction 2 Matrix analysis and differentiation 2.1 Matrix algebra 2.2 Vector spaces, bases, and linear maps 2.3 Metric, normed and inner product spaces 2.4 Euclidean spaces of matrices and their norms 2.5 Matrix differentials and gradients 2.6 Conclusion 2.7 Exercises 3 Matrix manifolds in MDA 3.1 Several useful matrix sets 3.1.1 mathcalGL(n) 3.1.2 mathcalO(n) 3.1.3 mathcalO(n,p) 3.1.4 mathcalO0(n,p) 3.1.5 mathcalOB(n) 3.1.6 mathcalOB(n,p) 3.1.7 mathcalG(n,p) 3.2 Differentiable manifolds 3.3 Examples of matrix manifolds in MDA 3.3.1 mathcalGL(n) 3.3.2 mathcalO(n) 3.3.3 mathcalO(n,p) 3.3.4 mathcalO0(n,p) 3.3.5 mathcalOB(n) 3.3.6 mathcalG(n,p) 3.4 Tangent spaces 3.4.1 mathcalGL(n) 3.4.2 mathcalO(n) 3.4.3 mathcalO(n,p) 3.4.4 mathcalO0(n,p) 3.4.5 mathcalOB(n) 3.4.6 mathcalOB(n,p) 3.4.7 mathcalG(n,p) 3.5 Optimization on matrix manifolds 3.5.1 Dynamical systems 3.5.2 Iterative schemes 3.5.3 Do it yourself! 3.6 Optimization with Manopt 3.6.1 Matrix manifolds in Manopt 3.6.2 Solvers 3.6.3 Portability 3.7 Conclusion 3.8 Exercises 4 Principal component analysis (PCA) 4.1 Introduction 4.2 Definition and main properties 4.3 Correspondence analysis (CA) 4.4 PCA interpretation 4.5 Simple structure rotation in PCA (and FA) 4.5.1 Simple structure concept 4.5.2 Simple structure criteria and rotation methods 4.5.3 Case study: PCA interpretation via rotation methods 4.5.4 Rotation to independent components 4.6 True simple structure: sparse loadings 4.6.1 SPARSIMAX: rotation-like sparse loadings 4.6.2 Know-how for applying SPARSIMAX 4.6.3 Manopt code for SPARSIMAX 4.7 Sparse PCA 4.7.1 Sparse components: genesis, history and present times 4.7.2 Taxonomy of PCA subject to ell1 constraint (LASSO) 4.8 Function-constrained sparse components 4.8.1 Orthonormal sparse component loadings 4.8.2 Uncorrelated sparse components 4.8.3 Numerical Example 4.8.4 Manopt code for weakly correlated sparse components with nearly orthonormal loadings 4.9 New generation dimension reduction 4.9.1 Centroid method 4.9.2 Randomized SVD 4.9.3 CUR approximations 4.9.4 The Nyström Method 4.10 Conclusion 4.11 Exercises 5 Factor analysis (FA) 5.1 Introduction 5.2 Fundamental equations of EFA 5.2.1 Population EFA definition 5.2.2 Sample EFA definition 5.3 EFA parameters estimation 5.3.1 Classical EFA estimation 5.3.2 EFA estimation on manifolds 5.4 ML exploratory factor analysis 5.4.1 Gradients 5.4.2 Optimality conditions 5.5 LS and GLS exploratory factor analysis 5.5.1 Gradients 5.5.2 Optimality conditions 5.6 Manopt codes for classical … 5.6.1 Standard case: Ψ2 Op 5.6.2 Avoiding Heywood cases: Ψ2 > Op 5.7 EFA as a low-rank-plus-sparse matrix … 5.8 Sparse EFA 5.8.1 Introduction 5.8.2 Sparse factor loadings with penalized EFA 5.8.3 Implementing sparseness 5.8.4 Numerical examples 5.9 Comparison to other methods 5.10 Conclusion 5.11 Exercises 6 Procrustes analysis (PA) 6.1 Introduction 6.2 Orthonormal PA 6.2.1 Orthogonal Penrose regression (OPR) 6.2.2 Projected Hessian 6.2.3 Orthonormal Penrose regression (OnPR) 6.2.4 Ordinary orthonormal PA 6.3 Oblique PA 6.3.1 Basic formulations and solutions 6.4 Robust PA 6.4.1 Some history remarks 6.4.2 Robust OnPR 6.4.3 Robust oblique PA 6.4.4 PA with M-estimator 6.5 Multi-mode PA 6.5.1 PCA and one-mode PCA 6.5.2 Multi-mode PCA and related PA problems 6.5.3 Global minima on calO(p) 6.6 Some other PA problems 6.6.1 Average of rotated matrices: generalized PA 6.6.2 Mean rotation 6.7 PA application to EFA of large data 6.7.1 The classical case n > p 6.7.2 The modern case p ggn 6.7.3 EFA and RPCA when pggn 6.7.4 Semi-sparse PCA (well-defined EFA) 6.8 Conclusion 6.9 Exercises 7 Linear discriminant analysis (LDA) 7.1 Introduction 7.2 LDA of vertical data (n>p) 7.2.1 Standard canonical variates (CVs) 7.2.2 Orthogonal canonical variates (OCVs) 7.3 Sparse CVs and sparse OCVs 7.4 LDA of horizontal data (p > n) 7.4.1 LDA through GSVD 7.4.2 LDA and pattern recognition 7.4.3 Null space LDA (NLDA) 7.4.4 LDA with CPC, PLS and MDS 7.4.5 Sparse LDA with diagonal W 7.4.6 Function-constrained sparse LDA 7.4.7 Sparse LDA based on minimization of the classification error 7.4.8 Sparse LDA through optimal scoring (SLDA) 7.4.9 Multiclass sparse discriminant analysis 7.4.10 Sparse LDA through GEVD 7.4.11 Sparse LDA without sparse-inducing penalty 7.5 Conclusion 7.6 Exercises 8 Cannonical correlation analysis (CCA) 8.1 Introduction 8.2 Classical CCA Formulation and Solution 8.3 Alternative CCA Definitions 8.4 Singular Scatter Matrices C11 and/or C22 8.5 Sparse CCA 8.5.1 Sparse CCA Through Sparse GEVD 8.5.2 LS Approach to Sparse CCA 8.6 CCA Relation to LDA and PLS 8.6.1 CCA and LDA 8.6.2 CCA and PLS 8.7 More Than Two Groups of Variables 8.7.1 CCA Generalizations 8.7.2 CCA Based on CPC 8.8 Conclusion 8.9 Exercises 9 Common principal components (CPC) 9.1 Introduction 9.2 CPC estimation problems 9.3 ML- and LS-CPC 9.3.1 Gradients and optimality conditions 9.3.2 Example: Fisher's Iris data 9.3.3 Appendix: MATLAB code for FG algorithm 9.4 New procedures for CPC estimation 9.4.1 Classic numerical solutions of ML- and LS-CPC 9.4.2 Direct calculation of individual eigenvalues/variances 9.4.3 CPC for known individual variances 9.5 CPC for dimension reduction 9.6 Proportional covariance matrices 9.6.1 ML and LS proportional principal components 9.6.2 Dimension reduction with PPC 9.7 Some relations between CPC and ICA 9.7.1 ICA formulations 9.7.2 ICA by contrast functions 9.7.3 ICA methods based on diagonalization 9.8 Conclusion 9.9 Exercises 10 Metric multidimensional scaling (MDS) and related methods 10.1 Introduction 10.2 Proximity measures 10.3 Metric MDS 10.3.1 Basic identities and classic solution 10.3.2 MDS that fits distances directly 10.3.3 Some related/adjacent MDS problems 10.4 INDSCAL – Individual Differences Scaling 10.4.1 The classical INDSCAL solution and some problems 10.4.2 Orthonormality-constrained INDSCAL 10.5 DINDSCAL – Direct INDSCAL 10.5.1 DINDSCAL model 10.5.2 DINDSCAL solution 10.5.3 Manopt code for DINDSCAL 10.6 DEDICOM 10.6.1 Introduction 10.6.2 Alternating DEDICOM 10.6.3 Simultaneous DEDICOM 10.7 GIPSCAL 10.7.1 GIPSCAL model 10.7.2 GISPSCAL solution 10.7.3 Three-way GIPSCAL 10.8 Tensor data analysis 10.8.1 Basic notations and definitions 10.8.2 CANDECOMP/PARAFAC (CP) 10.8.3 Three-mode PCA (TUCKER3) 10.8.4 Multi-mode PCA 10.8.5 Higher order SVD (HOSVD) 10.9 Conclusion 10.10 Exercises 11 Data analysis on simplexes 11.1 Archetypal analysis (AA) 11.1.1 Introduction 11.1.2 Definition of the AA problem 11.1.3 AA solution on multinomial manifold 11.1.4 AA solution on oblique manifolds 11.1.5 AA as interior point flows 11.1.6 Conclusion 11.1.7 Exercises 11.2 Analysis of compositional data (CoDa) 11.2.1 Introduction 11.2.2 Definition and main properties 11.2.3 Geometric clr structure of the data simplex 11.2.4 PCA and sparse PCA for compositions 11.2.5 Case study 11.2.6 Manopt code for sparse PCA of CoDa 11.2.7 Conclusion 11.2.8 Exercises Bibliography