Grokking Deep Learning

Description

Artificial Intelligence is the most exciting technology of the century, and Deep Learning is, quite literally, the “brain” behind the world’s smartest Artificial Intelligence systems out there. Grokking Deep Learning is the perfect place to begin the deep learning journey. Rather than just learning the “black box” API of some library or framework, readers will actually understand how to build these algorithms completely from scratch. Contents......Page 3 Preface......Page 11 About this Book......Page 12 Welcome to......Page 15 Why you should learn deep learning......Page 16 Why you should read this book......Page 17 What you need to get started......Page 19 Summary......Page 20 Fundamental Concepts......Page 21 What is deep learning?......Page 22 What is machine learning?......Page 23 Supervised machine learning......Page 24 Unsupervised machine learning......Page 25 Parametric vs. nonparametric learning......Page 26 Supervised parametric learning......Page 27 Unsupervised parametric learning......Page 29 Nonparametric learning......Page 30 Summary......Page 31 Forward Propagation......Page 32 Step 1: Predict......Page 33 A simple neural network making a prediction......Page 35 What is a neural network?......Page 36 What does this neural network do?......Page 37 Making a prediction with multiple inputs......Page 39 Multiple inputs: What does this neural network do?......Page 41 Multiple inputs: Complete runnable code......Page 46 Making a prediction with multiple outputs......Page 47 Predicting with multiple inputs and outputs......Page 49 Multiple inputs and outputs: How does it work?......Page 51 Predicting on predictions......Page 53 A quick primer on NumPy......Page 55 Summary......Page 57 Gradient Descent......Page 58 Compare......Page 59 Learn......Page 60 Compare: Does your network make good predictions?......Page 61 Why measure error?......Page 62 What’s the simplest form of neural learning?......Page 63 Hot and cold learning......Page 65 Characteristics of hot and cold learning......Page 66 Calculating both direction and amount from error......Page 67 One iteration of gradient descent......Page 69 Learning is just reducing error......Page 71 Let’s watch several steps of learning......Page 73 Why does this work? What is weight_delta, really?......Page 75 Tunnel vision on one concept......Page 77 A box with rods poking out of it......Page 78 Derivatives: Take two......Page 79 What you don’t really need to know......Page 80 How to use a derivative to learn......Page 81 Look familiar?......Page 82 Breaking gradient descent......Page 83 Visualizing the overcorrections......Page 84 Divergence......Page 85 Introducing alpha......Page 86 Alpha in code......Page 87 Memorizing......Page 88 Generalizing Gradient Descent......Page 89 Gradient descent learning with multiple inputs......Page 90 Gradient descent with multiple inputs explained......Page 92 Let’s watch several steps of learning......Page 96 Freezing one weight: What does it do?......Page 98 Gradient descent learning with multiple outputs......Page 100 Gradient descent with multiple inputs and outputs......Page 102 What do these weights learn?......Page 104 Visualizing weight values......Page 106 Visualizing dot products (weighted sums)......Page 107 Summary......Page 108 Intro to Backpropagation......Page 109 The streetlight problem......Page 110 Preparing the data......Page 112 Matrices and the matrix relationship......Page 113 Creating a matrix or two in Python......Page 116 Building a neural network......Page 117 Learning the whole dataset......Page 118 Full, batch, and stochastic gradient descent......Page 119 Neural networks learn correlation......Page 120 Up and down pressure......Page 121 Edge case: Overfittin......Page 123 Edge case: Confliting pressure......Page 124 Learning indirect correlation......Page 126 Creating correlation......Page 127 Stacking neural networks: A review......Page 128 Backpropagation: Long-distance error attribution......Page 129 Backpropagation: Why does this work?......Page 130 Linear vs. nonlinear......Page 131 Why the neural network still doesn’t work......Page 132 The secret to sometimes correlation......Page 133 A quick break......Page 134 Your first deep neual network......Page 135 Backpropagation in code......Page 136 One iteration of backpropagation......Page 138 Putting it all together......Page 140 Why do deep networks matter?......Page 141 Picture NNs......Page 142 It’s time to simplify......Page 143 Correlation summarization......Page 144 The previously overcomplicated visualization......Page 145 The simplified visualiztion......Page 146 Simplifying even further......Page 147 Let’s see this network predict......Page 148 Visualizing using letters instead of pictures......Page 149 Linking the variables......Page 150 Everything side by side......Page 151 The importance of visualization tools......Page 152 Intro to Regularization & Batching......Page 153 Three-layer network on MNIST......Page 154 Well, that was easy......Page 156 Memorization vs. generalization......Page 157 Overfitting in neual networks......Page 158 Where overfitting omes from......Page 159 The simplest regularization: Early stopping......Page 160 Industry standard regularization: Dropout......Page 161 Why dropout works: Ensembling works......Page 162 Dropout in code......Page 163 Dropout evaluated on MNIST......Page 165 Batch gradient descent......Page 166 Summary......Page 168 Activation Functions......Page 169 What is an activation function?......Page 170 Standard hidden-layer activation functions......Page 173 Standard output layer activation functions......Page 174 The core issue: Inputs have similarity......Page 176 softmax computation......Page 177 Activation installation instructions......Page 178 Multiplying delta by the slope......Page 180 Converting output to slope (derivative)......Page 181 Upgrading the MNIST network......Page 182 Intro to Convolutional NNs......Page 184 Reusing weights in multiple places......Page 185 The convolutional layer......Page 186 A simple implementation in NumPy......Page 188 Summary......Page 192 NNs that understand Language......Page 193 What does it mean to understand language?......Page 194 Natural language processing (NLP)......Page 195 Supervised NLP......Page 196 IMDB movie reviews dataset......Page 197 Capturing word correlation in input data......Page 198 Predicting movie reviews......Page 199 Intro to an embedding layer......Page 200 Interpreting the output......Page 202 Neural architecture......Page 203 Comparing word embeddings......Page 205 What is the meaning of a neuron?......Page 206 Filling in the blank......Page 207 Meaning is derived from loss......Page 209 King – Man + Woman ~= Queen......Page 212 Word analogies......Page 213 Summary......Page 214 Recurrent Layers for Variable-Length Data......Page 215 The challenge of arbitrary length......Page 216 Do comparisons really matter?......Page 217 The surprising power of averaged word vectors......Page 218 How is information stored in these embeddings?......Page 219 How does a neural network use embeddings?......Page 220 The limitations of bag-of-words vectors......Page 221 Using identity vectors to sum word embeddings......Page 222 Matrices that change absolutely nothing......Page 223 Learning the transition matrices......Page 224 Learning to create useful sentence vectors......Page 225 Forward propagation in Python......Page 226 How do you backpropagate into this?......Page 227 Let’s train it!......Page 228 Setting things up......Page 229 Forward propagation with arbitrary length......Page 230 Backpropagation with arbitrary length......Page 231 Weight update with arbitrary length......Page 232 Execution and output analysis......Page 233 Summary......Page 235 Deep Learning Framework......Page 236 What is a deep learning framework?......Page 237 Introduction to tensors......Page 238 Introduction to automatic gradient computation (autograd)......Page 239 A quick checkpoint......Page 241 Tensors that are used multiple times......Page 242 Upgrading autograd to support multiuse tensors......Page 243 How does addition backpropagation work?......Page 245 Adding support for negation......Page 246 Adding support for additional functions......Page 247 Using autograd to train a neural network......Page 251 Adding automatic optimization......Page 253 Adding support for layer types......Page 254 Layers that contain layers......Page 255 Loss-function layers......Page 256 How to learn a framework......Page 257 Nonlinearity layers......Page 258 The embedding layer......Page 260 Adding indexing to autograd......Page 261 The embedding layer (revisited)......Page 262 The cross-entropy layer......Page 263 The recurrent neural network layer......Page 265 Summary......Page 268 Long Short-Term Memory......Page 269 Character language modeling......Page 270 The need for truncated backpropagation......Page 271 Truncated backpropagation......Page 272 A sample of the output......Page 275 Vanishing and exploding gradients......Page 276 A toy example of RNN backpropagation......Page 277 Long short-term memory (LSTM) cells......Page 278 Some intuition about LSTM gates......Page 279 The long short-term memory layer......Page 280 Upgrading the character language model......Page 281 Training the LSTM character language model......Page 282 Tuning the LSTM character language model......Page 283 Summary......Page 284 Federated Learning......Page 285 The problem of privacy in deep learning......Page 286 Federated learning......Page 287 Learning to detect spam......Page 288 Let’s make it federated......Page 290 Hacking into federated learning......Page 291 Secure aggregation......Page 292 Homomorphic encryption......Page 293 Homomorphically encrypted federated learning......Page 294 Summary......Page 295 Where to go from here......Page 296 Index......Page 303