The Four Generations of Entity Resolution

Description

This book organizes entity resolution (ER) into four generations based on the challenges posed by "the four Vs," Veracity, Volume, Variety, and Velocity. Entity resolution lies at the core of data integration and cleaning and, thus, a bulk of the research examines ways for improving its effectiveness and time efficiency. For each generation, we outline the corresponding ER workflow, discuss the state-of-the-art methods per workflow step, and present current research directions. The discussion of these methods takes into account a historical perspective, explaining the evolution of the methods over time along with their similarities and differences. The lecture also discusses the available ER tools and benchmark datasets that allow expert as well as novice users to make use of the available solutions. The initial ER methods primarily target Veracity in the context of structured (relational) data that are described by a schema of well-known quality and meaning. To achieve high effectiveness, they leverage schema, expert, and/or external knowledge. Part of these methods are extended to address Volume, processing large datasets through multi-core or massive parallelization approaches, such as the MapReduce paradigm. However, these early schema-based approaches are inapplicable to Web Data, which abound in voluminous, noisy, semi-structured, and highly heterogeneous information. To address the additional challenge of Variety, recent works on ER adopt a novel, loosely schema-aware functionality that emphasizes scalability and robustness to noise. Another line of present research focuses on the additional challenge of Velocity, aiming to process data collections of a continuously increasing volume. The latest works, though, take advantage of the significant breakthroughs in Deep Learning and Crowdsourcing, incorporating external knowledge to enhance the existing words to a significant extent. Preface Acknowledgments Entity Resolution: Past, Present, and Yet-to-Come Preliminaries Computational Cost Performance Evaluation Generation 1: Addressing Veracity Schema Alignment Blocking Local Blocking Methods Global Blocking Methods Hybrid Methods Discussion Matching Distance-Based Methods Probabilistic Methods Supervised Methods Active Learning Methods Unsupervised Methods Collective Methods Rule-Based Methods String Similarity Joins Clustering Generation 2: Also Addressing Volume Blocking Matching Clustering Generation 3: Also Addressing Variety Schema Refinement Block Building Block Processing Matching Context-Based Matching Context-Free Matching Clustering Parallelization Block Building Block Processing Matching Clustering Generation 4: Also Addressing Velocity Progressive Entity Resolution Prioritization Incremental Entity Resolution Leveraging External Knowledge Deep Learning Schema Matching Blocking Matching Crowdsourced Entity Resolution Resources for Entity Resolution ER Tools ER Datasets Possible Directions for Future Work Bibliography Authors' Biographies