Probabilistic Topic Models: Foundation and Application

Description

This book introduces readers to the theoretical foundation and application of topic models. It provides readers with efficient means to learn about the technical principles underlying topic models. More concretely, it covers topics such as fundamental concepts, topic model structures, approximate inference algorithms, and a range of methods used to create high-quality topic models. In addition, this book illustrates the applications of topic models applied in real-world scenarios. Readers will be instructed on the means to select and apply suitable models for specific real-world tasks, providing this book with greater use for the industry. Finally, the book presents a catalog of the most important topic models from the literature over the past decades, which can be referenced and indexed by researchers and engineers in related fields. We hope this book can bridge the gap between academic research and industrial application and help topic models play an increasingly effective role in both academia and industry. This book offers a valuable reference guide for senior undergraduate students, graduate students, and researchers, covering the latest advances in topic models, and for industrial practitioners, sharing state-of-the-art solutions for topic-related applications. The book can also serve as a reference for job seekers preparing for interviews. Preface Contents 1 Basics 1.1 Linear Algebra 1.1.1 Vector 1.1.2 Matrix 1.1.3 Matrix Operations 1.1.3.1 Matrix Addition and Subtraction 1.1.3.2 Matrix Multiplication 1.1.3.3 Matrix Transposition 1.1.3.4 Matrix Inversion 1.1.4 Orthogonal Matrix 1.1.5 Eigenvalues and Eigenvectors 1.2 Probability Theory 1.2.1 Probability Distribution 1.2.2 Independence 1.2.3 Expected Value, Variance, and Standard Deviation 1.2.4 Common Probability Distributions 1.2.5 Exponential Family 1.2.6 Bayes' Theorem 1.2.7 Conjugate Distribution 1.2.8 Divergence 1.2.8.1 Kullback-Leibler Divergence 1.2.8.2 Jensen-Shannon Divergence 1.3 Bayesian Networks 1.3.1 Representation 1.3.2 Conditional Independence 1.3.3 Uncertain Reasoning 1.3.4 Parameter Learning 1.3.5 Structure Learning References 2 Topic Models 2.1 Basic Concepts 2.2 Latent Semantic Analysis 2.3 Probabilistic LSA 2.4 Latent Dirichlet Allocation 2.5 SentenceLDA 2.6 Topic over Time Model 2.7 Topical Word Embedding 2.8 Hierarchical Topic Models 2.8.1 Pachinko Allocation Model 2.8.2 Rephil References 3 Pre-processing of Training Data 3.1 Word Segmentation 3.1.1 Chinese Word Segmentation Tools 3.1.2 Word Segmentation Granularity 3.2 Normalization 3.3 Filtering 3.3.1 Stopword Filtering 3.3.2 Low-Frequency Word Filtering 3.3.3 Part-of-Speech-Based Filtering 3.4 Word Sorting References 4 Expectation Maximization 4.1 Basics 4.1.1 The First Method of E-Step 4.1.2 The Second Method of E-Step 4.1.3 M-Step 4.2 Convergence of the EM Algorithm 4.3 GEM Algorithm 4.4 Applications of the EM Algorithm 4.4.1 PLSA 4.4.2 PCLSA References 5 Markov Chain Monte Carlo Sampling 5.1 Markov Chain 5.2 Monte Carlo Method 5.3 Markov Chain Monte Carlo 5.4 Gibbs Sampling 5.4.1 Basic Concepts 5.4.2 Application of Gibbs Sampling in LDA 5.5 Metropolis–Hastings Sampling with Alias Method 5.5.1 Metropolis–Hastings Sampling 5.5.2 Alias Method 5.5.3 Application of Metropolis–Hastings Sampling in LDA References 6 Variational Inference 6.1 Mathematical Foundation 6.2 Evidence Lower Bound 6.3 Mean Field Variational Inference 6.4 Applying Mean Field Variational Inference to LDA 6.4.1 Joint Distribution 6.4.2 Variational Factorization 6.4.3 Evidence Lower Bound 6.4.4 Variational Optimization with Partial Derivatives 6.5 Comparison of Variational Inference and MCMC References 7 Distributed Training 7.1 Distributed Computing Architectures 7.1.1 MapReduce 7.1.2 ParameterServer 7.2 Distributed MCMC Sampling 7.2.1 Distributed MCMC Sampling with MapReduce 7.2.2 Distributed MCMC Sampling with ParameterServer 7.3 Distributed Variational Inference References 8 Parameter Setting 8.1 Hyperparameters 8.1.1 Hyperparameter Optimization Based on MCMC Sampling 8.1.2 Hyperparameter Optimization Based on Variational Inference 8.2 The Number of Topics 8.3 Advanced Metrics for Model Evaluation References 9 Topic Deduplication and Model Compression 9.1 Topic Deduplication 9.1.1 Precise Topic Deduplication 9.1.1.1 Topic Similarity Analysis 9.1.1.2 Topic Fusion 9.1.2 Fast Topic Deduplication 9.2 Model Compression 9.2.1 Topic-Dimension Compression 9.2.2 Word-Dimension Compression References 10 Applications 10.1 Semantic Representation 10.1.1 Text Classification 10.1.2 Text Clustering 10.1.3 Click-Through Rate Prediction 10.2 Semantic Matching 10.2.1 Short–Long Text Matching 10.2.1.1 Advertising Page Ranking 10.2.1.2 Keyword Extraction 10.2.2 Long–Long Text Matching 10.3 Semantic Visualization 10.3.1 Basic Visualization 10.3.2 Advanced Visualization 10.3.3 General-Purpose Visualization Tools 10.3.4 Other Applications References A Topic Models A.1 Common Topic Models A.2 Topic Models with Advanced Features of Documentsor Words A.3 Topic Models with Supervised Information A.4 Topic Models with Word Embedding A.5 Topic Models with Sentiment Information A.6 Topic Models with Hierarchical Structure A.7 Topic Models with Network Structure A.8 Topic Models with Time Information A.9 Topic Models with Geographic Information A.10 Topic Models with Bayesian Nonparametrics A.11 Distributed Training of Topic Models A.12 Visualization of Topic Models A.13 Applications in Recommendation System A.14 Applications in Information Retrieval and InformationExtraction A.15 Applications in Event Analysis References