Data Analysis with LLMs: Text, tables, images and sound

Description

Speed up common data science tasks with AI assistants like ChatGPT and Large Language Models (LLMs) from Anthropic, Cohere, Open AI, Google, Hugging Face, and more! Data Analysis with LLMs teaches you to use the new generation of AI assistants and Large Language Models (LLMs) to aid and accelerate common data science tasks. Learn how to use LLMs to: • Analyze text, tables, images, and audio files • Extract information from multi-modal data lakes • Classify, cluster, transform, and query multimodal data • Build natural language query interfaces over structured data sources • Use LangChain to build complex data analysis pipelines • Prompt engineering and model configuration All practical, Data Analysis with LLMs takes you from your first prompts through advanced techniques like creating LLM-based agents for data analysis and fine-tuning existing models. You’ll learn how to extract data, build natural language query interfaces, and much more. About the technology Large Language Models (LLMs) can streamline and accelerate almost any data science task. Master the techniques in this book, and you’ll be able to analyze large amounts of text, tabular and graph data, images, videos, and more with clear natural language prompts and a few lines of Python code. About the book Data Analysis with LLMs shows you exactly how to integrate generative AI into your day-to-day work as a data scientist. In it, Cornell professor Immanuel Trummer guides you through a series of engaging projects that introduce OpenAI’s Python library, tools like LangChain and LlamaIndex, and LLMs from Anthropic, Cohere, and Hugging Face. As you go, you’ll use AI to query structured and unstructured data, analyze sound and images, and optimize the cost and quality of your data analysis process. What's inside • Classify, cluster, transform, and query multimodal data • Build natural language query interfaces over structured data sources • Create LLM-based agents for autonomous data analysis • Prompt engineering and model configuration About the reader For data scientists and data analysts who know the basics of Python. About the author Immanuel Trummer is an associate professor of computer science at Cornell University and a member of the Cornell Database Group. brief contents contents preface acknowledgments about this book Who should read this book? How this book is organized: A road map About the code liveBook discussion forum about the author about the cover illustration Part 1 Introducing language models 1 Analyzing data with large language models 1.1 What can language models do? 1.2 What you will learn 1.3 How to use language models 1.3.1 Prompting 1.3.2 Example prompt 1.3.3 Interfaces 1.4 Using language models for data analysis 1.4.1 Using language models directly on data 1.4.2 Data analysis via external tools 1.5 Minimizing costs 1.5.1 Picking the best model 1.5.2 Optimally configuring models 1.5.3 Prompt engineering 1.6 Advanced software frameworks and agents Summary 2 Chatting with ChatGPT 2.1 Accessing the web interface 2.2 Making introductions 2.3 Processing text with ChatGPT 2.4 Processing tables with ChatGPT 2.4.1 Processing tables in the web interface 2.4.2 Processing tables on your platform Summary References Part 2 Data analysis with language models 3 The OpenAI Python library 3.1 Prerequisites 3.2 Installing OpenAI's Python library 3.3 Listing available models 3.4 Chat completion 3.5 Customizing model behavior 3.5.1 Configuring termination conditions 3.5.2 Configuring output generation 3.5.3 Configuring randomization 3.5.4 Customization example 3.5.5 Further parameters Summary 4 Analyzing text data 4.1 Preliminaries 4.2 Classification 4.2.1 Overview 4.2.2 Creating prompts 4.2.3 Calling the model 4.2.4 End-to-end classification code 4.2.5 Classifying documents 4.2.6 Running the code 4.2.7 Trying out variants 4.3 Text extraction 4.3.1 Overview 4.3.2 Generating prompts 4.3.3 Postprocessing 4.3.4 End-to-end extraction code 4.3.5 Trying it out 4.4 Clustering 4.4.1 Overview 4.4.2 Calculating embeddings 4.4.3 Clustering vectors 4.4.4 End-to-end code for text clustering 4.4.5 Trying it out 4.4.6 Other use cases for embedding vectors Summary References 5 Analyzing structured data 5.1 Chapter outline 5.2 A natural language query interface for analyzing game sales 5.2.1 Setting up an SQLite database 5.2.2 SQL basics 5.2.3 Overview 5.2.4 Generating prompts for text-to-SQL translation 5.2.5 Complete code 5.2.6 Trying it out 5.3 A general natural language query interface 5.3.1 Executing queries 5.3.2 Extracting the database structure 5.3.3 Complete code 5.3.4 Trying it out 5.4 A natural language query interface for graph data 5.4.1 What is graph data? 5.4.2 Setting up a Neo4j database 5.4.3 The Cypher query language 5.4.4 Translating questions to Cypher queries 5.4.5 Generating prompts 5.4.6 Complete code 5.4.7 Trying it out Summary 6 Analyzing images and videos 6.1 Setup 6.2 Answering questions about images 6.2.1 Specifying multimodal input 6.2.2 Code discussion 6.2.3 Trying it out 6.3 Tagging people in images 6.3.1 Overview 6.3.2 Encoding locally stored images 6.3.3 Sending locally stored images to OpenAI 6.3.4 The end-to-end implementation 6.3.5 Trying it out 6.4 Generating titles for videos 6.4.1 Overview 6.4.2 Encoding video frames 6.4.3 The end-to-end implementation 6.4.4 Trying it out Summary 7 Analyzing audio data 7.1 Preliminaries 7.2 Transcribing audio files 7.2.1 Transcribing speech 7.2.2 End-to-end code 7.2.3 Trying it out 7.3 Querying relational data via voice 7.3.1 Preliminaries 7.3.2 Overview 7.3.3 Recording audio 7.3.4 End-to-end code 7.3.5 Trying it out 7.4 Speech-to-speech translation 7.4.1 Overview 7.4.2 Generating speech 7.4.3 End-to-end code 7.4.4 Trying it out Summary Part 3 Advanced topics 8 GPT alternatives 8.1 Anthropic 8.1.1 Chatting with Claude 8.1.2 Python library 8.2 Cohere 8.2.1 Chatting with Command R+ 8.2.2 Python library 8.3 Google 8.3.1 Chatting with Gemini 8.3.2 The Python library 8.4 Hugging Face 8.4.1 Web platform 8.4.2 Python library Summary References 9 Optimizing cost and quality 9.1 Example scenario 9.2 Untuned classifier 9.3 Model tuning 9.4 Model selection 9.5 Prompt engineering 9.6 Tunable classifier 9.7 Fine-tuning 9.8 Generating training data 9.9 Starting a fine-tuning job 9.10 Using the fine-tuned model Summary 10 Software frameworks 10.1 LangChain 10.2 Classifying reviews with LangChain 10.2.1 Overview 10.2.2 Creating a classification chain 10.2.3 Putting it together 10.2.4 Trying it out 10.3 Agents: Putting the large language model into the driver's seat 10.4 Building an agent for data analysis 10.4.1 Overview 10.4.2 Creating an agent with LangChain 10.4.3 Complete code for data-analysis agent 10.4.4 Trying it out 10.5 Adding custom tools 10.5.1 The currency converter 10.5.2 Trying it out 10.6 Indexing multimodal data with LlamaIndex 10.6.1 Overview 10.6.2 Installing LlamaIndex 10.6.3 Implementing a simple question-answering system 10.6.4 Trying it out 10.7 Concluding remarks Summary References index