Dark Data: Why What You Don’t Know Matters

Description

A practical guide to making good decisions in a world of missing data In the era of big data, it is easy to imagine that we have all the information we need to make good decisions. But in fact the data we have are never complete, and may be only the tip of the iceberg. Just as much of the universe is composed of dark matter, invisible to us but nonetheless present, the universe of information is full of dark data that we overlook at our peril. In Dark Data, data expert David Hand takes us on a fascinating and enlightening journey into the world of the data we don't see. Dark Data explores the many ways in which we can be blind to missing data and how that can lead us to conclusions and actions that are mistaken, dangerous, or even disastrous. Examining a wealth of real-life examples, from the Challenger shuttle explosion to complex financial frauds, Hand gives us a practical taxonomy of the types of dark data that exist and the situations in which they can arise, so that we can learn to recognize and control for them. In doing so, he teaches us not only to be alert to the problems presented by the things we don’t know, but also shows how dark data can be used to our advantage, leading to greater understanding and better decisions. Today, we all make decisions using data. Dark Data shows us all how to reduce the risk of making bad ones. Cover Contents Preface Part 1: Dark Data: Their Origins and Consequences Chapter 1: Dark Data: What We Don’t See Shapes Our World The Ghost of Data So You Think You Have All the Data? Nothing Happened, So We Ignored It The Power of Dark Data All around Us Chapter 2: Discovering Dark Data: What We Collect and What We Don’t Dark Data on All Sides Data Exhaust, Selection, and Self-Selection From the Few to the Many Experimental Data Beware Human Frailties Chapter 3: Definitions and Dark Data: What Do You Want to Know? Different Definitions and Measuring the Wrong Thing You Can’t Measure Everything Screening Selection on the Basis of Past Performance Chapter 4: Unintentional Dark Data: Saying One Thing, Doing Another The Big Picture Summarizing Human Error Instrument Limitations Linking Data Sets Chapter 5: Strategic Dark Data: Gaming, Feedback, and Information Asymmetry Gaming Feedback Information Asymmetry Adverse Selection and Algorithms Chapter 6: Intentional Dark Data: Fraud and Deception Fraud Identity Theft and Internet Fraud Personal Financial Fraud Financial Market Fraud and Insider Trading Insurance Fraud And More Chapter 7: Science and Dark Data: The Nature of Discovery The Nature of Science If Only I’d Known That Tripping over Dark Data Dark Data and the Big Picture Hiding the Facts Retraction Provenance and Trustworthiness: Who Told You That? Part II: Illuminating and Using Dark Data Chapter 8: Dealing with Dark Data: Shining a Light Hope! Linking Observed and Missing Data Identifying the Missing Data Mechanism Working with the Data We Have Going Beyond the Data: What If You Die First? Going Beyond the Data: Imputation Iteration Wrong Number! Chapter 9: Benefiting from Dark Data: Reframing the Question Hiding Data Hiding Data from Ourselves: Randomized Controlled Trials What Might Have Been Replicated Data Imaginary Data: The Bayesian Prior Privacy and Confidentiality Preservation Collecting Data in the Dark Chapter 10: Classifying Dark Data: A Route through the Maze A Taxonomy of Dark Data Illumination Notes Index