An Introduction to Duplicate Detection

Description

Data Cleansing: Introduction and Motivation Data Quality Data Quality Dimensions Data Cleansing Causes for Duplicates Intra-Source Duplicates Inter-Source Duplicates Use Cases for Duplicate Detection Customer Relationship Management Scientific Databases Data Spaces and Linked Open Data Lecture Overview Problem Definition Formal Definition Complexity Analysis Data in Complex Relationships Data Model Challenges of Data with Complex Relationships Similarity Functions Token-based Similarity Jaccard Coefficient Cosine Similarity Using Token Frequency and Inverse Document Frequency Similarity Based on Tokenization Using q-grams Edit-based Similarity Edit Distance Measures Jaro and Jaro-Winkler Distance Hybrid Functions Extended Jaccard Similarity Monge-Elkan Measure Soft TF/IDF Measures for Data with Complex Relationships Other Similarity Measures Rule-based Record Comparison Equational Theory Duplicate Profiles Duplicate Detection Algorithms Pairwise Comparison Algorithms Blocking Sorted-Neighborhood Comparison Algorithms for Data with Complex Relationships Hierarchical Relationships Relationships Forming a Graph Clustering Algorithms Clustering Based on the Duplicate Pair Graph Clustering Adjusting to Data & Cluster Characteristics Evaluating Detection Success Precision and Recall Data Sets Real-World Data Sets Synthetic Data Sets Towards a Duplicate Detection Benchmark Conclusion and Outlook Bibliography Authors' Biographies