An Intuitive Exploration of Artificial Intelligence: Theory and Applications of Deep Learning

Description

This book develops a conceptual understanding of Artificial Intelligence (AI), Deep Learning and Machine Learning in the truest sense of the word. It is an earnest endeavor to unravel what is happening at the algorithmic level, to grasp how applications are being built and to show the long adventurous road in the future. An Intuitive Exploration of Artificial Intelligence offers insightful details on how AI works and solves problems in computer vision, natural language understanding, speech understanding, reinforcement learning and synthesis of new content. From the classic problem of recognizing cats and dogs, to building autonomous vehicles, to translating text into another language, to automatically converting speech into text and back to speech, to generating neural art, to playing games, and the author’s own experience in building solutions in industry, this book is about explaining how exactly the myriad applications of AI flow out of its immense potential. The book is intended to serve as a textbook for graduate and senior-level undergraduate courses in AI. Moreover, since the book provides a strong geometrical intuition about advanced mathematical foundations of AI, practitioners and researchers will equally benefit from the book. Preface Contents Acronyms Author's Note Part I Foundations 1 AI Sculpture 1.1 Manifolds in High Dimensions 1.2 Sculpting Process 1.3 Notational Convention 1.4 Regression and Classification 1.4.1 Linear Regression and Logistic Regression 1.4.2 Regression Loss and Cross-Entropy Loss 1.4.3 Sculpting with Shades 1.5 Discriminative and Generative AI 1.6 Success of Discriminative Methods 1.7 Feature Engineering in Classical ML 1.8 Supervised and Unsupervised AI 1.9 Beyond Manifolds 1.10 Chapter Summary 2 Make Me Learn 2.1 Learnable Parameters 2.1.1 The Power of a Single Neuron 2.1.2 Neurons Working Together 2.2 Backpropagation of Gradients 2.2.1 Partial Derivatives 2.2.2 Forward and Backward Passes 2.3 Stochastic Gradient Descent 2.3.1 Handling Difficult Landscapes 2.3.2 Stabilization of Training 2.4 Chapter Summary 3 Images and Sequences 3.1 Convolutional Neural Networks 3.1.1 The Biology of the Visual Cortex 3.1.2 Pattern Matching 3.1.3 3-D Convolution 3.2 Recurrent Neural Networks 3.2.1 Neurons with States 3.2.2 The Power of Recurrence 3.2.3 Going Both Ways 3.2.4 Attention 3.3 Self-Attention 3.4 LSTM 3.5 Beyond Images and Sequences 3.6 Chapter Summary 4 Why AI Works 4.1 Convex Polytopes 4.2 Piecewise Linear Function 4.2.1 Subdivision of the Input Space 4.2.2 Piecewise Non-linear Function 4.2.3 Carving Out the Feature Spaces 4.3 Expressive Power of AI 4.4 Convolutional Neural Network 4.5 Recurrent Neural Network 4.6 Architectural Variations 4.7 Attention and Carving 4.8 Optimization Landscape 4.8.1 Graph-Induced Polynomial 4.8.2 Gradient of the Loss Function 4.8.3 Visualization 4.8.4 Critical Points 4.9 The Mathematics of Loss Landscapes 4.9.1 Random Polynomial Perspective 4.9.2 Random Matrix Perspective 4.9.3 Spin Glass Perspective 4.9.4 Computational Complexity Perspective 4.9.5 SGD and Critical Points 4.9.6 Confluence of Perspectives 4.10 Distributed Representation and Intrinsic Dimension 4.11 Chapter Summary 5 Novice to Maestro 5.1 How AI Learns to Sculpt 5.1.1 Training Data 5.1.2 Evaluation Metrics 5.1.3 Hyperparameter Search 5.1.4 Regularization 5.1.5 Bias and Variance 5.1.6 A Fairy Tale in the Land of ML 5.2 Learning Curves 5.3 From the Lab to the Dirty Field 5.4 System Design 5.5 Flavors of Learning 5.6 Ingenuity and Big Data in the Success of AI 5.7 Chapter Summary 6 Unleashing the Power of Generation 6.1 Creating Universes 6.2 To Recognize It, Learn to Draw It 6.3 General Definition 6.4 Generative Parameters 6.5 Generative AI Models 6.5.1 Restricted Boltzmann Machines 6.5.2 Autoencoders 6.5.3 Variational Autoencoder 6.5.4 Pixel Recursive Models 6.5.5 Generative Adversarial Networks 6.5.6 Wasserstein Generative Adversarial Networks 6.6 The Carving Process in Generative AI 6.7 Representation of Individual Signals 6.8 Chapter Summary 7 The Road Most Rewarded 7.1 Reinforcement Learning 7.2 Learning an Optimal Policy 7.3 Deep Q-Learning 7.4 Policy Gradient 7.4.1 Intuition 7.4.2 Mathematical Analysis 7.5 Let's Play and Explore 7.6 Chapter Summary 8 The Classical World 8.1 Maximum Likelihood Estimation 8.2 Uncertainty in Estimation 8.3 Linear Models 8.3.1 Linear Regression 8.3.2 The Geometry of Linear Regression 8.3.3 Regularization 8.3.4 Logistic Regression 8.4 Classical ML 8.4.1 k-Nearest Neighbors 8.4.2 Naive Bayes Classifier 8.4.3 FDA, LDA, and QDA 8.4.4 Support Vector Machines 8.4.5 Neural Networks 8.4.6 Decision Trees 8.4.7 Gaussian Process Regression 8.4.8 Unsupervised Methods 8.5 XGBoost 8.5.1 Relevance Ranking 8.6 Chapter Summary Part II Applications 9 To See Is to Believe 9.1 Image Classification 9.1.1 Motivating Examples 9.1.2 LeNet 9.1.3 Stacked Autoencoders 9.1.4 AlexNet 9.1.5 VGG 9.1.6 ResNet 9.1.7 Inception V3 9.1.8 Showroom of Models 9.1.9 Your Own Network 9.2 Object Detection as Classification 9.2.1 Sliding Window Method 9.2.2 Region Proposal Method 9.3 Regression on Images 9.3.1 Motivating Examples 9.3.2 Object Detection as Regression 9.3.2.1 Regression Output 9.3.2.2 Grid-Based Approach 9.4 Attention in Computer Vision 9.5 Semantic Segmentation 9.6 Image Similarity 9.7 Video Analysis 9.8 3-D Data 9.9 Self-Driving Vehicles 9.10 Present and Future 9.11 Protein Folding 9.12 Chapter Summary 10 Read, Read, Read 10.1 Natural Language Understanding 10.2 Embedding Words in a Semantic Space 10.3 Sequence to Sequence 10.3.1 Encoder-Decoder Architecture 10.3.2 Neural Machine Translation 10.4 Attention Mechanism 10.5 Self-Attention 10.6 Creativity in NLU Solutions 10.7 AI and Human Culture 10.8 Recommender Systems 10.9 Reward-Based Formulations 10.10 Chapter Summary 11 Lend Me Your Ear 11.1 Classical Speech Recognition 11.2 Spectrogram to Transcription 11.2.1 Alignment-Free Temporal Connections 11.2.2 End-to-End Solution 11.2.3 Don't Listen to Others 11.3 Speech Synthesis 11.4 Handwriting Recognition 11.5 Chapter Summary 12 Create Your Shire and Rivendell 12.1 From Neurons to Art 12.1.1 DeepDream 12.1.2 Style Transfer 12.2 Image Translation 12.3 DeepFake 12.4 Creative Applications 12.5 Chapter Summary 13 Math to Code to Petaflops 13.1 Software Frameworks 13.1.1 The Twentieth Century 13.1.2 The Twenty-First Century 13.1.3 AI Frameworks 13.2 Let's Crunch Numbers 13.2.1 Computing Hardware 13.2.2 GPU Machines 13.2.3 Cloud GPU Instances 13.2.4 Training Script 13.2.5 Deployment 13.3 Speeding Up Training 13.3.1 Data Parallelism 13.3.2 Delayed and Compressed SGD 13.4 Open Ecosystem and Efficient Hardware 13.5 Chapter Summary 14 AI and Business 14.1 Strategy 14.2 Organization 14.3 Execution 14.4 Evaluation 14.5 Startups 14.6 Chapter Summary Part III The Road Ahead 15 Keep Marching On 15.1 Robust AI 15.1.1 Adversarial Examples 15.1.2 Learning from Human Vision 15.1.3 Fusion of Evidence 15.1.4 Interpretable AI 15.2 AI Extraordinaire 15.3 Chapter Summary 16 Benevolent AI for All 16.1 Benefits of AI 16.2 AI in Medicine 16.3 Dangers of AI 16.4 AI-Human Conflict 16.5 Choices Ahead 16.6 Chapter Summary 17 Am I Looking at Myself? 17.1 Is It Computable or Non-computable? 17.2 Is Consciousness Everywhere? 17.3 Who Is the Storyteller? 17.4 Chapter Summary A Solutions Answer of Exercise 1 Answer of Exercise 2 Answer of Exercise 3 Answer of Exercise 4 Answer of Exercise 5 Answer of Exercise 6 Answer of Exercise 7 Answer of Exercise 9 Answer of Exercise 8 Answer of Exercise 10 Answer of Exercise 11 Answer of Exercise 12 Answer of Exercise 13 Answer of Exercise 14 Answer of Exercise 15 Answer of Exercise 16 Answer of Exercise 17 Answer of Exercise 18 Answer of Exercise 19 Answer of Exercise 20 Answer of Exercise 21 Answer of Exercise 22 Answer of Exercise 23 Answer of Exercise 24 Answer of Exercise 25 Answer of Exercise 26 Answer of Exercise 27 Answer of Exercise 28 Answer of Exercise 29 Answer of Exercise 30 Answer of Exercise 31 Answer of Exercise 32 Answer of Exercise 33 Answer of Exercise 34 Answer of Exercise 35 B Lab Exercises and Projects B.1 Exercises B.2 Exploration B.3 Debate and Discussion Further Reading Glossary References Index