AI-Powered Search Version 13

Description

AI powered Search MEAP V13 Copyright Welcome Brief contents Chapter 1: Introducing AI-powered search 1.1 Searching for User Intent 1.1.1 Search Engines 1.1.2 Recommendation Engines 1.1.3 The Information Retrieval Continuum 1.1.4 Semantic Search and Knowledge Graphs 1.1.5 Understanding the Dimensions of User Intent 1.2 Key Technologies for AI-powered search 1.2.1 Programming language: Python 1.2.2 Data Processing Framework: Spark (PySpark) 1.2.3 Delivery Mechanism: Docker Containers 1.2.4 Code Setup and Walkthroughs: Jupyter Notebooks 1.2.5 Choosing the right search engine technology 1.3 Target Audience for AI-Powered Search 1.3.1 Targeted Skillsets and Occupations 1.3.2 System Requirements for Running Code Examples 1.4 When to consider AI-powered search 1.5 How does AI-powered search work? 1.5.1 The Core Search Foundation 1.5.2 Reflected Intelligence through Feedback Loops 1.5.3 Curated vs. Black-box AI 1.5.4 Architecture for an AI-powered search engine 1.6 Summary Chapter 2: Working with natural language 2.1 The myth of unstructured data 2.1.1 Types of unstructured data 2.1.2 Data types in traditional structured databases 2.1.3 Joins, fuzzy joins, and entity resolution in unstructured data 2.2 The structure of natural language 2.3 Distributional semantics and word embeddings 2.4 Modeling domain-specific knowledge 2.5 Challenges in natural language understanding for search 2.5.1 The challenge of ambiguity (polysemy) 2.5.2 The challenge of understanding context 2.5.3 The challenge of personalization 2.5.4 Challenges interpreting queries vs. documents 2.5.5 Challenges interpreting query intent 2.6 The fuel powering AI-powered search 2.7 Summary Chapter 3: Ranking and content-based relevance 3.1 Scoring query and document vectors with cosine similarity 3.1.1 Mapping text to vectors 3.1.2 Calculating similarity between dense vector representations 3.1.3 Calculating similarity between sparse vector representations 3.1.4 Term Frequency (TF): measuring how well documents match a term 3.1.5 Inverse Document Frequency (IDF): measuring the importance of a term in the query 3.1.6 TF-IDF: a balanced weighting metric for text-based relevance 3.2 Controlling the relevance calculation 3.2.1 BM25: Lucene’s default text-similarity algorithm 3.2.2 Functions, functions, everywhere! 3.2.3 Choosing multiplicative vs. additive boosting for relevance functions 3.2.4 Differentiating matching (filtering) vs. ranking (scoring) of documents 3.2.5 Logical matching: weighting the relationships between terms in a query 3.2.6 Separating concerns: filtering vs. scoring 3.3 Implementing user and domain-specific relevance ranking 3.4 Summary Chapter 4: Crowdsourced relevance 4.1 Working with User Signals 4.1.1 Signals vs. Content 4.1.2 Setting up our product and signals datasets (RetroTech) 4.1.3 Exploring the signals data 4.1.4 Modeling users, sessions, and requests 4.2 Introduction to Reflected Intelligence 4.2.1 What is Reflected Intelligence? 4.2.2 Popularized Relevance through Signals Boosting 4.2.3 Personalized Relevance through Collaborative Filtering 4.2.4 Generalized Relevance through Learning to Rank 4.2.5 Other reflected intelligence models 4.2.6 Crowdsourcing from content 4.3 Summary Chapter 5: Knowledge graph learning 5.1 Working with knowledge graphs 5.2 Building a knowledge graph explicitly into your search engine 5.3 Automatic extraction of knowledge graphs from content 5.3.1 Extracting arbitrary relationships from text 5.3.2 Extracting hyponyms from text 5.4 Learning intent by traversing semantic knowledge graphs 5.4.1 What is a semantic knowledge graph? 5.4.2 Indexing the datasets 5.4.3 Structure of a semantic knowledge graph 5.4.4 Calculating edge weights to score relatedness of nodes 5.4.5 Using semantic knowledge graphs for query expansion 5.4.6 Using semantic knowledge graphs for content-based recommendations 5.4.7 Using semantic knowledge graphs to model arbitrary relationships 5.5 Using knowledge graphs for semantic search 5.6 Summary Chapter 6: Using context to learn domain-specific language 6.1 Classifying query intent 6.2 Query sense disambiguation 6.3 Learning related phrases from query signals 6.3.1 Mining query logs for related queries 6.3.2 Finding related queries through product interactions 6.4 Phrase detection from user signals 6.4.1 Treating queries as entities 6.4.2 Extracting entities from more complex queries 6.5 Misspellings and alternative representations 6.5.1 Learning spelling corrections from documents 6.5.2 Learning spelling corrections from user signals 6.6 Pulling it all together 6.7 Summary Chapter 8: Signals boosting models 8.1 Basic signals boosting 8.2 Normalizing Signals 8.3 Fighting Signal Spam 8.3.1 Using signal spam to manipulate search results 8.3.2 Combatting signal spam through user-based filtering 8.4 Combining multiple signal types 8.5 Time decays and short-lived signals 8.5.1 Handling time-sensitive documents 8.5.2 Handling time-sensitive signals 8.6 Index-time vs. Query-time boosting: balancing scale vs. flexibility 8.6.1 Tradeoffs when using query-time boosting 8.6.2 Implementing Index-time signals boosting 8.7 Summary Chapter 10: Learning to rank for generalizable search relevance 10.1 What is Learning to Rank? 10.1.1 Implementing learning to rank in the real-world 10.2 Step 1: A judgment list, starting with the training data 10.3 Step 2 - feature logging and engineering 10.3.1 Storing features in a modern search engine 10.3.2 Logging features from our Solr corpus 10.4 Step 3 - transforming LTR to a traditional machine learning problem 10.4.1 SVMRank: Transforming ranking to binary classification 10.4.2 Transforming our LTR training data to binary classification 10.5 Step 4—Training (and testing!) the model 10.5.1 Turning a separating hyperplane’s vector into a scoring function 10.5.2 Taking the model for a test drive 10.5.3 Validating the model 10.6 Steps 5 and 6 - upload a model and search 10.6.1 A note on LTR performance 10.7 Rinse and repeat 10.8 Summary Chapter 11: Building learning to rank training data from user clicks 11.1 (Re)creating judgment lists from signals 11.1.1 Generating implicit, probabilistic judgments from signals 11.1.2 Training an LTR model using probabilistic judgments 11.1.3 Click-through Rate: Your First Click Model 11.1.4 Common biases in judgments 11.2 Overcoming Position Bias: The Search Engine Returned ithigher, it must be better! 11.2.1 Defining Position Bias 11.2.2 Position bias in RetroTech data 11.2.3 A Click Model that Overcomes Position Bias: Simplified DynamicBayesian Network 11.3 Handling Confidence Bias: not upending your model from afew lucky clicks 11.3.1 The Low Confidence Problem in RetroTech Click Data 11.3.2 Using a Beta Prior to Model Confidence Probabilistically 11.4 Exploring your training data in an LTR System 11.5 Summary Chapter 12: Overcoming bias in learned relevance models 12.1 Our Automated LTR engine in a few lines of code 12.1.1 Turning clicks into training data (Chapter 11 in one line of code) 12.1.2 Model training & evaluation in a few function calls 12.2 A/B testing a new model 12.2.1 Taking a better model out for a test drive 12.2.2 Defining an A/B test in the context of automated LTR 12.2.3 Graduating the better model into an A/B test 12.2.4 When 'good' models go bad: what we can learn about a failed A/B test? 12.3 Overcoming Presentation Bias: Knowing When to Explore vs Exploit 12.3.1 Presentation bias in RetroTech training data 12.3.2 Beyond the ad-hoc: thoughtfully exploring with a Gaussian Process 12.3.3 Training and Analyzing a Gaussian Process 12.3.4 Examining the outcome of our explorations 12.4 Explore, exploit, gather, rinse, repeat: the full Automated LTR loop 12.5 Summary Chapter 13: Semantic search with dense vectors 13.1 Language Translation as an Analogy for Text Representation 13.1.1 Representation of Meaning through Text Embeddings 13.2 Search using Dense Vectors 13.2.1 A brief refresher on sparse vectors 13.2.2 A conceptual dense vector search engine 13.3 Getting Text Embeddings by using using a Transformer Encoder 13.3.1 What is a Transformer? 13.3.2 Openly available pre-trained transformer models 13.4 Applying Transformers to Search 13.4.1 Using the Outdoors StackExchange dataset 13.4.2 Fine-tuning and the Semantic Text Similarity Benchmark (STS-B) 13.4.3 Introducing SBERT, a transformer library built around similarity between sentences 13.5 Natural Language Autocomplete 13.5.1 Getting noun phrases and verb phrases for our nearest-neighbor vocabulary 13.5.2 Getting embeddings 13.5.3 Approximate Nearest-Neighbor search 13.5.4 Approximate Nearest-Neighbor index implementation 13.6 Semantic Search with large language model embeddings 13.6.1 Getting titles and their embeddings 13.6.2 Creating and searching the nearest-neighbor index Appendix A: Running the code examples A.1 Overall Structure of Code Examples A.2 Pulling the source code A.3 Building and running the code A.4 Working with Jupyter A.5 Working with Docker Notes