Cleaning Data for Effective Data Science: Doing the other 80% of the work with Python, R, and command-line tools

Description

A comprehensive guide for data scientists to master effective data cleaning tools and techniques Key FeaturesMaster data cleaning techniques in a language-agnostic mannerLearn from intriguing hands-on examples from numerous domains, such as biology, weather data, demographics, physics, time series, and image processingWork with detailed, commented, well-tested code samples in Python and RBook Description It is something of a truism in data science, data analysis, or machine learning that most of the effort needed to achieve your actual purpose lies in cleaning your data. Written in David’s signature friendly and humorous style, this book discusses in detail the essential steps performed in every production data science or data analysis pipeline and prepares you for data visualization and modeling results. The book dives into the practical application of tools and techniques needed for data ingestion, anomaly detection, value imputation, and feature engineering. It also offers long-form exercises at the end of each chapter to practice the skills acquired. You will begin by looking at data ingestion of data formats such as JSON, CSV, SQL RDBMSes, HDF5, NoSQL databases, files in image formats, and binary serialized data structures. Further, the book provides numerous example data sets and data files, which are available for download and independent exploration. Moving on from formats, you will impute missing values, detect unreliable data and statistical anomalies, and generate synthetic features that are necessary for successful data analysis and visualization goals. By the end of this book, you will have acquired a firm understanding of the data cleaning process necessary to perform real-world data science and machine learning tasks. What you will learnIdentify problem data pertaining to individual data pointsDetect problem data in the systematic “shape” of the dataRemediate data integrity and hygiene problemsPrepare data for analytic and machine learning tasksImpute values into missing or unreliable dataGenerate synthetic features that are more amenable to data science, data analysis, or visualization goals.Who This Book Is For This book is designed to benefit software developers, data scientists, aspiring data scientists, and students who are interested in data analysis or scientific computing. Basic familiarity with statistics, general concepts in machine learning, knowledge of a programming language (Python or R), and some exposure to data science are helpful. A glossary, references, and friendly asides should help bring all readers up to speed. The text will also be helpful to intermediate and advanced data scientists who want to improve their rigor in data hygiene and wish for a refresher on data preparation issues. Cover Copyright Contributors Table of Contents Preface Part I - Data Ingestion Chapter 1: Tabular Formats Tidying Up CSV Sanity Checks The Good, the Bad, and the Textual Data The Bad The Good Spreadsheets Considered Harmful SQL RDBMS Massaging Data Types Repeating in R Where SQL Goes Wrong (and How to Notice It) Other Formats HDF5 and NetCDF-4 Tools and Libraries SQLite Apache Parquet Data Frames Spark/Scala Pandas and Derived Wrappers Vaex Data Frames in R (Tidyverse) Data Frames in R (data.table) Bash for Fun Exercises Tidy Data from Excel Tidy Data from SQL Denouement Chapter 2: Hierarchical Formats JSON What JSON Looks Like NaN Handling and Data Types JSON Lines GeoJSON Tidy Geography JSON Schema XML User Records Keyhole Markup Language Configuration Files INI and Flat Custom Formats TOML Yet Another Markup Language NoSQL Databases Document-Oriented Databases Missing Fields Denormalization and Its Discontents Key/Value Stores Exercises Exploring Filled Area Create a Relational Model Denouement Chapter 3: Repurposing Data Sources Web Scraping HTML Tables Non-Tabular Data Command-Line Scraping Portable Document Format Image Formats Pixel Statistics Channel Manipulation Metadata Binary Serialized Data Structures Custom Text Formats A Structured Log Character Encodings Exercises Enhancing the NPY Parser Scraping Web Traffic Denouement Part II - The Vicissitudes of Error Chapter 4: Anomaly Detection Missing Data SQL Hierarchical Formats Sentinels Miscoded Data Fixed Bounds Outliers Z-Score Interquartile Range Multivariate Outliers Exercises A Famous Experiment Misspelled Words Denouement Chapter 5: Data Quality Missing Data Biasing Trends Understanding Bias Detecting Bias Comparison to Baselines Benford’s Law Class Imbalance Normalization and Scaling Applying a Machine Learning Model Scaling Techniques Factor and Sample Weighting Cyclicity and Autocorrelation Domain Knowledge Trends Discovered Cycles Bespoke Validation Collation Validation Transcription Validation Exercises Data Characterization Oversampled Polls Denouement Part III - Rectification and Creation Chapter 6: Value Imputation Typical-Value Imputation Typical Tabular Data Locality Imputation Trend Imputation Types of Trends A Larger Coarse Time Series Understanding the Data Removing Unusable Data Imputing Consistency Interpolation Non-Temporal Trends Sampling Undersampling Oversampling Exercises Alternate Trend Imputation Balancing Multiple Features Denouement Chapter 7: Feature Engineering Date/Time Fields Creating Datetimes Imposing Regularity Duplicated Timestamps Adding Timestamps String Fields Fuzzy Matching Explicit Categories String Vectors Decompositions Rotation and Whitening Dimensionality Reduction Visualization Quantization and Binarization One-Hot Encoding Polynomial Features Generating Synthetic Features Feature Selection Exercises Intermittent Occurrences Characterizing Levels Denouement Part IV - Ancillary Matters Closure What You Know What You Don’t Know (Yet) Glossary Other Books You May Enjoy Index