Machine Learning for Asset Managers

Description

Successful investment strategies are specific implementations of general theories. An investment strategy that lacks a theoretical justification is likely to be false. Hence, an asset manager should concentrate her efforts on developing a theory rather than on backtesting potential trading rules. The purpose of this Element is to introduce machine learning (ML) tools that can help asset managers discover economic and financial theories. ML is not a black box, and it does not necessarily overfit. ML tools complement rather than replace the classical statistical methods. Some of ML's strengths include (1) a focus on out-of-sample predictability over variance adjudication; (2) the use of computational methods to avoid relying on (potentially unrealistic) assumptions; (3) the ability to “learn” complex specifications, including nonlinear, hierarchical, and noncontinuous interaction effects in a high-dimensional space; and (4) the ability to disentangle the variable search from the specification search, robust to multicollinearity and other substitution effects. Cover Title page Copyright page Machine Learning for Asset Managers Contents 1 Introduction 1.1 Motivation 1.2 Theory Matters 1.2.1 Lesson 1: You Need a Theory 1.2.2 Lesson 2: ML Helps Discover Theories 1.3 How Scientists Use ML 1.4 Two Types of Overfitting 1.4.1 Train Set Overfitting 1.4.2 Test Set Overfitting 1.5 Outline 1.6 Audience 1.7 Five Popular Misconceptions about Financial ML 1.7.1 ML Is the Holy Grail versus ML Is Useless 1.7.2 ML Is a Black Box 1.7.3 Finance Has Insufficient Data for ML 1.7.4 The Signal-to-Noise Ratio Is Too Low in Finance 1.7.5 The Risk of Overfitting Is Too High in Finance 1.8 The Future of Financial Research 1.9 Frequently Asked Questions In Simple Terms, What Is ML? How Is ML Different from Econometric Regressions? How Is ML Different from Big Data? How Is the Asset Management Industry Using ML? And Quantitative Investors Specifically? What Are Some of the Ways That ML Can Be Applied to Investor Portfolios? What Are the Risks? Is There Anything That Investors Should Be Aware of or Look Out For? How Do You Expect ML to Impact the Asset Management Industry in the Next Decade? How Do You Expect ML to Impact Financial Academia in the Next Decade? Isn’t Financial ML All about Price Prediction? Why Don’t You Discuss a Wide Range of ML Algorithms? Why Don’t You Discuss a Specific Investment Strategy, Like Many Other Books Do? 1.10 Conclusions 1.11 Exercises 2 Denoising and Detoning 2.1 Motivation 2.2 The Marcenko–Pastur Theorem 2.3 Random Matrix with Signal 2.4 Fitting the Marcenko–Pastur Distribution 2.5 Denoising 2.5.1 Constant Residual Eigenvalue Method 2.5.2 Targeted Shrinkage 2.6 Detoning 2.7 Experimental Results 2.7.1 Minimum Variance Portfolio 2.7.2 Maximum Sharpe Ratio Portfolio 2.8 Conclusions 2.9 Exercises 3 Distance Metrics 3.1 Motivation 3.2 A Correlation-Based Metric 3.3 Marginal and Joint Entropy 3.4 Conditional Entropy 3.5 Kullback–Leibler Divergence 3.6 Cross-Entropy 3.7 Mutual Information 3.8 Variation of Information 3.9 Discretization 3.10 Distance between Two Partitions 3.11 Experimental Results 3.11.1 No Relationship 3.11.2 Linear Relationship 3.11.3 Nonlinear Relationship 3.12 Conclusions 3.13 Exercises 4 Optimal Clustering 4.1 Motivation 4.2 Proximity Matrix 4.3 Types of Clustering 4.4 Number of Clusters 4.4.1 Observations Matrix 4.4.2 Base Clustering 4.4.3 Higher-Level Clustering 4.5 Experimental Results 4.5.1 Generation of Random Block Correlation Matrices 4.5.2 Number of Clusters 4.6 Conclusions 4.7 Exercises 5 Financial Labels 5.1 Motivation 5.2 Fixed-Horizon Method 5.3 Triple-Barrier Method 5.4 Trend-Scanning Method 5.5 Meta-labeling 5.5.1 Bet Sizing by Expected Sharpe Ratio 5.5.2 Ensemble Bet Sizing 5.6 Experimental Results 5.7 Conclusions 5.8 Exercises 6 Feature Importance Analysis 6.1 Motivation 6.2 p-Values 6.2.1 A Few Caveats of p-Values 6.2.2 A Numerical Example 6.3 Feature Importance 6.3.1 Mean-Decrease Impurity 6.3.2 Mean-Decrease Accuracy 6.4 Probability-Weighted Accuracy 6.5 Substitution Effects 6.5.1 Orthogonalization 6.5.2 Cluster Feature Importance Step 1: Features Clustering Step 2: Clustered Importance Clustered MDI Clustered MDA 6.6 Experimental Results 6.7 Conclusions 6.8 Exercises 7 Portfolio Construction 7.1 Motivation 7.2 Convex Portfolio Optimization 7.3 The Condition Number 7.4 Markowitz’s Curse 7.5 Signal as a Source of Covariance Instability 7.6 The Nested Clustered Optimization Algorithm 7.6.1 Correlation Clustering 7.6.2 Intracluster Weights 7.6.3 Intercluster Weights 7.7 Experimental Results 7.7.1 Minimum Variance Portfolio 7.7.2 Maximum Sharpe Ratio Portfolio 7.8 Conclusions 7.9 Exercises 8 Testing Set Overfitting 8.1 Motivation 8.2 Precision and Recall 8.3 Precision and Recall under Multiple Testing 8.4 The Sharpe Ratio 8.5 The “False Strategy” Theorem Theorem 8.6 Experimental Results 8.7 The Deflated Sharpe Ratio 8.7.1 Effective Number of Trials 8.7.2 Variance across Trials 8.8 Familywise Error Rate 8.8.1 Šidàk’s Correction 8.8.2 Type I Errors under Multiple Testing 8.8.3 Type II Errors under Multiple Testing 8.8.4 The Interaction between Type I and Type II Errors 8.9 Conclusions 8.10 Exercises Appendix A: Testing on Synthetic Data Appendix B: Proof of the “False Strategy” Theorem Bibliography References Acknowledgments