Finding Ghosts in Your Data: Anomaly Detection Techniques with Examples in Python

Description

Discover key information buried in the noise of data by learning a variety of anomaly detection techniques and using the Python programming language to build a robust service for anomaly detection against a variety of data types. The book starts with an overview of what anomalies and outliers are and uses the Gestalt school of psychology to explain just why it is that humans are naturally great at detecting anomalies. From there, you will move into technical definitions of anomalies, moving beyond "I know it when I see it" to defining things in a way that computers can understand. The core of the book involves building a robust, deployable anomaly detection service in Python. You will start with a simple anomaly detection service, which will expand over the course of the book to include a variety of valuable anomaly detection techniques, covering descriptive statistics, clustering, and time series scenarios. Finally, you will compare your anomaly detection service head-to-head with a publicly available cloud offering and see how they perform. The anomaly detection techniques and examples in this book combine psychology, statistics, mathematics, and Python programming in a way that is easily accessible to software developers. They give you an understanding of what anomalies are and why you are naturally a gifted anomaly detector. Then, they help you to translate your human techniques into algorithms that can be used to program computers to automate the process. You’ll develop your own anomaly detection service, extend it using a variety of techniques such as including clustering techniques for multivariate analysis and time series techniques for observing data over time, and compare your service head-on against a commercial service. What You Will LearnUnderstand the intuition behind anomaliesConvert your intuition into technical descriptions of anomalous dataDetect anomalies using statistical tools, such as distributions, variance and standard deviation, robust statistics, and interquartile rangeApply state-of-the-art anomaly detection techniques in the realms of clustering and time series analysisWork with common Python packages for outlier detection and time series analysis, such as scikit-learn, PyOD, and tslearnDevelop a project from the ground up which finds anomalies in data, starting with simple arrays of numeric data and expanding to include multivariate inputs and even time series data Who This Book Is For For software developers with at least some familiarity with the Python programming language, and who would like to understand the science and some of the statistics behind anomaly detection techniques. Readers are not required to have any formal knowledge of statistics as the book introduces relevant concepts along the way. Table of Contents About the Author About the Technical Reviewer Introduction Chapter 1: The Importance of  Anomalies and Anomaly Detection Defining Anomalies Outlier Noise vs. Anomalies Diagnosing an Example What If We’re Wrong? Anomalies in the Wild Finance Medicine Sports Analytics A $23 Million Mistake A Persistent Anomaly Web Analytics And Many More Classes of Anomaly Detection Statistical Anomaly Detection Clustering Anomaly Detection Model-Based Anomaly Detection Building an Anomaly Detector Key Goals How Do Humans Handle Anomalies? Known Unknowns Conclusion Chapter 2: Humans Are Pattern Matchers A Primer on the Gestalt School Key Findings of the Gestalt School Emergence Reification Invariance Multistability Principles Implied in the Key Findings Meaningfulness Conciseness Closure Similarity Good Continuation Figure and Ground Proximity Connectedness Common Region Symmetry Common Fate Synchrony Helping People Find Anomalies Use Color As a Signal Limit Nonmeaningful Information Enable “Connecting the Dots” Conclusion Chapter 3: Formalizing Anomaly Detection The Importance of Formalization “I’ll Know It When I See It” Isn’t Enough Human Fallibility Marginal Outliers The Limits of Visualization The First Formal Tool: Univariate Analysis Distributions and Histograms The Normal Distribution Mean, Variance, and Standard Deviation Additional Distributions Log-Normal Uniform Cauchy Robustness and the Mean The Susceptibility of Outliers The Median and “Robust” Statistics Beyond the Median: Calculating Percentiles Control Charts Conclusion Chapter 4: Laying Out the Framework Tools of the Trade Choosing a Programming Language Making Plumbing Choices Reducing Architectural Variables Developing an Initial Framework Battlespace Preparation Framing the API Input and Output Signatures Defining a Common Signature Defining an Outlier Sensitivity and Fraction of Anomalies Single Solution Combined Arms Framing the Solution Containerizing the Solution Conclusion Chapter 5: Building a Test Suite Tools of the Trade Unit Test Library Integration Testing Writing Testable Code Keep Methods Separated Emphasize Use Cases Functional or Clean: Your Choice Creating the Initial Tests Unit Tests Integration Tests Conclusion Chapter 6: Implementing the First Methods A Motivating Example Ensembling As a Technique Sequential Ensembling Independent Ensembling Choosing Between Sequential and Independent Ensembling Implementing the First Checks Standard Deviations from the Mean Median Absolute Deviations from the Median Distance from the Interquartile Range Completing the run_tests() Function Building a Scoreboard Weighting Results Determining Outliers Updating Tests Updating Unit Tests Updating Integration Tests Conclusion Chapter 7: Extending the Ensemble Adding New Tests Checking for Normality Approaching Normality A Framework for New Tests Grubbs’ Test for Outliers Generalized ESD Test for Outliers Dixon’s Q Test Calling the Tests Updating Tests Updating Unit Tests Updating Integration Tests Multi-peaked Data A Hidden Assumption The Solution: A Sneak Peek Conclusion Untitled Chapter 8: Visualize the Results Building a Plan What Do We Want to Show? How Do We Want to Show It? Developing a Visualization App Getting Started with Streamlit Building the Initial Screen Displaying Results and Details Conclusion Chapter 9: Clustering and Anomalies What Is Clustering? Common Cluster Terminology K-Means Clustering K-Nearest Neighbors When Clustering Makes Sense Gaussian Mixture Modeling Implementing a Univariate Version Updating Tests Common Problems with Clusters Choosing the Correct Number of Clusters Clustering Is Nondeterministic Alternative Approaches Tree-Based Approaches The Problem with Trees Conclusion Chapter 10: Connectivity-Based Outlier Factor (COF) Distance or Density? Local Outlier Factor Connectivity-Based Outlier Factor Introducing Multivariate Support Laying the Groundwork Implementing COF Test and Website Updates Unit Test Updates Integration Test Updates Website Updates Conclusion Chapter 11: Local Correlation Integral (LOCI) Local Correlation Integral Discovering the Neighborhood Multi-granularity Deviation Factor (MDEF) Multivariate Algorithm Ensembles Ensemble Types COF Combinations Incorporating LOCI Test and Website Updates Unit Test Updates Website Updates Conclusion Chapter 12: Copula-Based Outlier Detection (COPOD) Copula-Based Outlier Detection What’s a Copula? Intuition Behind COPOD Implementing COPOD Test and Website Updates Unit Test Updates Integration Test Updates Website Updates Conclusion Chapter 13: Time and Anomalies What Is Time Series? Time Series Changes Our Thinking Autocorrelation Smooth Movement The Nature of Change Data Requirements Time Series Modeling (Weighted) Moving Average Exponential Smoothing Autoregressive Models What Constitutes an Outlier? Local Outlier Behavioral Changes over Time Local Non-outlier in a Global Change Differences from Peer Groups Common Classes of Technique Conclusion Untitled Chapter 14: Change Point Detection What Is Change Point Detection? Benefits of Change Point Detection Change Point Detection with ruptures Dynamic Programming PELT Implementing Change Point Detection Test and Website Updates Unit Tests Integration Tests Website Updates Avenues of Further Improvement Conclusion Chapter 15: An Introduction to Multi-series Anomaly Detection What Is Multi-series Time Series? Key Aspects of Multi-series Time Series What Needs to Change? What’s the Difference? Leading and Lagging Factors Available Processes Cross-Euclidean Distance Cross-Correlation Coefficient SameTrend (STREND) Common Problems Conclusion Chapter 16: Standard Deviation of Differences (DIFFSTD) What Is DIFFSTD? Calculating DIFFSTD Key Assumptions Writing DIFFSTD Series Processing Segmentation Comparing the Norm Determining Outliers Test and Website Updates Unit Tests Integration Tests Website Updates Conclusion Chapter 17: Symbolic Aggregate Approximation (SAX) What Is SAX? Motifs and Discords Subsequences and Matches Discretizing the Data Implementing SAX Segmentation and Blocking Making SAX Multi-series Scoring Outliers Test and Website Updates Unit and Integration Tests Website Updates Conclusion Chapter 18: Configuring Azure Cognitive Services Anomaly Detector Gathering Market Intelligence Amazon Web Services: SageMaker Microsoft Azure: Cognitive Services Google Cloud: AI Services Configuring Azure Cognitive Services Set Up an Account Using the Demo Application Conclusion Chapter 19: Performing a Bake-Off Preparing the Comparison Supervised vs. Unsupervised Learning Choosing Datasets Scoring Results Performing the Bake-Off Accessing Cognitive Services via Python Accessing Our API via Python Dataset Comparisons Lessons Learned Making a Better Anomaly Detector Increasing Robustness Extending the Ensembles Training Parameter Values Conclusion Untitled Appendix Index