Applied Deep Learning with TensorFlow 2: Learn to Implement Advanced Deep Learning Techniques with Python

Description

Table of Contents About the Author About the Contributing Author About the Technical Reviewer Acknowledgments Foreword Introduction Chapter 1: Optimization and Neural Networks A Basic Understanding of Neural Networks The Problem of Learning A First Definition of Learning [Advanced Section] Assumption in the Formulation A Definition of Learning for Neural Networks Constrained vs. Unconstrained Optimization [Advanced Section] Reducing a Constrained Problem to an Unconstrained Optimization Problem Absolute and Local Minima of a Function Optimization Algorithms Line Search and Trust Region Steepest Descent The Gradient Descent Algorithm Choosing the Right Learning Rate Variations of GD Mini-Batch GD Stochastic GD How to Choose the Right Mini-Batch Size [Advanced Section] SGD and Fractals Exercises Conclusion Chapter 2: Hands-on with a Single Neuron A Short Overview of a Neuron’s Structure A Short Introduction to Matrix Notation An Overview of the Most Common Activation Functions Identity Function Sigmoid Function Tanh (Hyperbolic Tangent) Activation Function ReLU (Rectified Linear Unit) Activation Function Leaky ReLU The Swish Activation Function Other Activation Functions How to Implement a Neuron in Keras Python Implementation Tips: Loops and NumPy Linear Regression with a Single Neuron The Dataset for the Real-World Example Dataset Splitting Linear Regression Model Keras Implementation The Model’s Learning Phase Model’s Performance Evaluation on Unseen Data Logistic Regression with a Single Neuron The Dataset for the Classification Problem Dataset Splitting The Logistic Regression Model Keras Implementation The Model’s Learning Phase The Model’s Performance Evaluation Conclusion Exercises References Chapter 3: Feed-Forward Neural Networks A Short Review of Network’s Architecture and Matrix Notation Output of Neurons A Short Summary of Matrix Dimensions Example: Equations for a Network with Three Layers Hyper-Parameters in Fully Connected Networks A Short Review of the Softmax Activation Function for Multiclass Classifications A Brief Digression: Overfitting A Practical Example of Overfitting Basic Error Analysis Implementing a Feed-Forward Neural Network in Keras Multiclass Classification with Feed-Forward Neural Networks The Zalando Dataset for the Real-World Example Modifying Labels for the Softmax Function: One-Hot Encoding The Feed-Forward Network Model Keras Implementation Gradient Descent Variations Performances Comparing the Variations Examples of Wrong Predictions Weight Initialization Adding Many Layers Efficiently Advantages of Additional Hidden Layers Comparing Different Networks Tips for Choosing the Right Network Estimating the Memory Requirements of Models General Formula for the Memory Footprint Exercises References Chapter 4: Regularization Complex Networks and Overfitting What Is Regularization About Network Complexity ℓp Norm ℓ2 Regularization Theory of ℓ2 Regularization Keras Implementation ℓ1 Regularization Theory of ℓ1 Regularization and Keras Implementation Are the Weights Really Going to Zero? Dropout Early Stopping Additional Methods Exercises References Chapter 5: Advanced Optimizers Available Optimizers in Keras in TensorFlow 2.5 Advanced Optimizers Exponentially Weighted Averages Momentum RMSProp Adam Comparison of the Optimizers’ Performance Small Coding Digression Which Optimizer Should You Use? Chapter 6: Hyper-Parameter Tuning Black-Box Optimization Notes on Black-Box Functions The Problem of Hyper-Parameter Tuning Sample Black-Box Problem Grid Search Random Search Coarse to Fine Optimization Bayesian Optimization Nadaraya-Watson Regression Gaussian Process Stationary Process Prediction with Gaussian Processes Acquisition Function Upper Confidence Bound (UCB) Example Sampling on a Logarithmic Scale Hyper-Parameter Tuning with the Zalando Dataset A Quick Note about the Radial Basis Function Exercises References Chapter 7: Convolutional Neural Networks Kernels and Filters Convolution Examples of Convolution Pooling Padding Building Blocks of a CNN Convolutional Layers Pooling Layers Stacking Layers Together An Example of a CNN Conclusion Exercises References Chapter 8: A Brief Introduction to Recurrent Neural Networks Introduction to RNNs Notation The Basic Idea of RNNs Why the Name Recurrent Learning to Count Conclusion Further Readings Chapter 9: Autoencoders Introduction Regularization in Autoencoders Feed-Forward Autoencoders Activation Function of the Output Layer ReLU Sigmoid The Loss Function Mean Square Error Binary Cross-Entropy The Reconstruction Error Example: Reconstructing Handwritten Digits Autoencoder Applications Dimensionality Reduction Equivalence with PCA Classification Classification with Latent Features The Curse of Dimensionality: A Small Detour Anomaly Detection Model Stability: A Short Note Denoising Autoencoders Beyond FFA: Autoencoders with Convolutional Layers Implementation in Keras Exercises Further Readings Chapter 10: Metric Analysis Human-Level Performance and Bayes Error A Short Story About Human-Level Performance Human-Level Performance on MNIST Bias Metric Analysis Diagram Training Set Overfitting Test Set How to Split Your Dataset Unbalanced Class Distribution: What Can Happen Datasets with Different Distributions k-fold Cross Validation Manual Metric Analysis: An Example Exercises References Chapter 11: Generative Adversarial Networks (GANs) Introduction to GANs Training Algorithm for GANs A Practical Example with Keras and MNIST A Note on Training Conditional GANs Conclusion Appendix A: Introduction to Keras Some History Understanding the Sequential Model Understanding Keras Layers Setting the Activation Function Using Functional APIs Specifying Loss Functions and Metrics Putting It All Together and Training Modeling evaluate() and predict () Using Callback Functions Saving and Loading Models Saving Your Weights Manually Saving the Entire Model Conclusion Appendix B: Customizing Keras Customizing Callback Classes Example of a Custom Callback Class Custom Training Loops Calculating Gradients Custom Training Loop for a Neural Network Index