Mastering Data Mining with Python - Find Patterns Hidden in Your Data

Description

Learn how to create more powerful data mining applications with this comprehensive Python guide to advance data analytics techniques About This Book Dive deeper into data mining with Python - don't be complacent, sharpen your skills! From the most common elements of data mining to cutting-edge techniques, we've got you covered for any data-related challenge Become a more fluent and confident Python data-analyst, in full control of its extensive range of libraries Who This Book Is For This book is for data scientists who are already familiar with some basic data mining techniques such as SQL and machine learning, and who are comfortable with Python. If you are ready to learn some more advanced techniques in data mining in order to become a data mining expert, this is the book for you! What You Will Learn Explore techniques for finding frequent itemsets and association rules in large data sets Learn identification methods for entity matches across many different types of data Identify the basics of network mining and how to apply it to real-world data sets Discover methods for detecting the sentiment of text and for locating named entities in text Observe multiple techniques for automatically extracting summaries and generating topic models for text See how to use data mining to fix data anomalies and how to use machine learning to identify outliers in a data set In Detail Data mining is an integral part of the data science pipeline. It is the foundation of any successful data-driven strategy - without it, you'll never be able to uncover truly transformative insights. Since data is vital to just about every modern organization, it is worth taking the next step to unlock even greater value and more meaningful understanding. If you already know the fundamentals of data mining with Python, you are now ready to experiment with more interesting, advanced data analytics techniques using Python's easy-to-use interface and extensive range of libraries. In this book, you'll go deeper into many often overlooked areas of data mining, including association rule mining, entity matching, network mining, sentiment analysis, named entity recognition, text summarization, topic modeling, and anomaly detection. For each data mining technique, we'll review the state-of-the-art and current best practices before comparing a wide variety of strategies for solving each problem. We will then implement example solutions using real-world data from the domain of software engineering, and we will spend time learning how to understand and interpret the results we get. By the end of this book, you will have solid experience implementing some of the most interesting and relevant data mining techniques available today, and you will have achieved a greater fluency in the important field of Python data analytics. Style and approach This book will teach you the intricacies in applying data mining using real-world scenarios and will act as a very practical solution to your data mining needs." Cover Copyright Credits About the Author About the Reviewers www.PacktPub.com Table of Contents Preface Expanding Your Data Mining Toolbox What is data mining? How do we do data mining? The Fayyad et al. KDD process The Han et al. KDD process The CRISP-DM process The Six Steps process Which data mining methodology is the best? What are the techniques used in data mining? What techniques are we going to use in THIS book? How do we set up our data mining work environment? Summary Association Rule Mining What are frequent itemsets? The diapers and beer urban legend Frequent itemset mining basics Towards association rules Support Confidence Association rules An example with data Added value - fixing a flaw in the plan Methods for finding frequent itemsets A project – discovering association rules in software project tags Summary Entity Matching What is entity matching? Merging data Merging datasets vertically Merging datasets horizontally Techniques for matching Attribute-based similarity matching Be careful of pairwise comparisons Leverage rare values Methods for matching attributes Range-based or distance from target String edit distance Hamming distance Levenshtein distance Soundex Leveraging disjoint sets Context-based similarity matching Machine learning-based entity matching Evaluation of entity matching techniques Efficiency - how long does it take to do the matching? Effectiveness – how accurate are the matches that we generate? Usefulness - how practical is the matching procedure to use? Entity matching project Difficulties with matching software projects Two examples Matching on project names Matching on people names Matching on URLs Matching on topics and description keywords The dataset The code The results How many entity matches did we find? How good are the pairs we found? Summary Network Analysis What is a network? Measuring a network Degree of a network Diameter of a network Walks, paths, and trails in a network Components of a network Centrality of a network Closeness centrality Degree centrality Betweenness centrality Other measures of centrality Representing graph data Adjacency matrix Edge lists and adjacency lists Differences between graph data structures Importing data into a graph structure Adjacency list format Edge list format GEXF and GraphML GDF Python pickle JSON JSON node and link series JSON trees Pajek format A real project Exploring the data Generating the network files Understanding our data as a network Generating simple network metrics Playing with the parameters of a network Analyzing subgraphs Analyzing cliques and centrality in the subgraphs Looking for change over time Summary Sentiment Analysis What is sentiment analysis? The basics of sentiment analysis The structure of an opinion Document-level and sentence-level analysis Important features of opinions Sentiment analysis algorithms General-purpose data collections Hu and Liu's sentiment analysis lexicon SentiWordNet Vader sentiment Sentiment mining application Motivating the project Data preparation Data analysis of chat messages Data analysis of e-mail messages Summary Named Entity Recognition in Text Why look for named entities? Techniques for named entity recognition Tagging parts of speech Classes of named entities Building and evaluating NER systems NER and partial matches Handling partial matches Named entity recognition project A simple NER tool Apache Board meeting minutes Django IRC chat GnuIRC summaries LKML e-mails Summary Automatic Text Summarization What is automatic text summarization? Tools for text summarization Naive text summarization using NLTK Text summarization using Gensim Text summarization using Sumy Sumy's Luhn summarizer Sumy's TextRank summarizer Sumy's LSA summarizer Sumy's Edmundson summarizer Summary Topic Modeling What is topic modeling? Latent Dirichlet Allocation Gensim for topic modeling Understanding Gensim LDA topics Understanding Gensim LDA passes Applying a Gensim LDA model to new documents Serializing Gensim LDA objects Serializing a dictionary Serializing a corpus Serializing a model Gensim LDA for a larger project Summary Mining for Data Anomalies What are data anomalies? Missing data Locating missing data Zero values Fixing missing data Ignore the problem rows Fix the problem manually Use a fabricated value Use a central measure Use Last Observation Carried Forward Use a similar value Use the most likely value Data errors Truncated fields Data type and character set errors Logic or semantic errors Outliers Visual mining for outliers Statistical detection of outliers Summary Index