Natural Language Processing with TensorFlow: Teach language to machines using Python's deep learning library

Description

Natural Language Processing with TensorFlow Table of Contents Natural Language Processing with TensorFlow Why subscribe? PacktPub.com Contributors About the author About the reviewers Packt is searching for authors like you Preface Who this book is for What this book covers To get the most out of this book Download the example code files Download the color images Conventions used Get in touch Reviews 1. Introduction to Natural Language Processing What is Natural Language Processing? Tasks of Natural Language Processing The traditional approach to Natural Language Processing Understanding the traditional approach Example – generating football game summaries Drawbacks of the traditional approach The deep learning approach to Natural Language Processing History of deep learning The current state of deep learning and NLP Understanding a simple deep model – a Fully-Connected Neural Network The roadmap – beyond this chapter Introduction to the technical tools Description of the tools Installing Python and scikit-learn Installing Jupyter Notebook Installing TensorFlow Summary 2. Understanding TensorFlow What is TensorFlow? Getting started with TensorFlow TensorFlow client in detail TensorFlow architecture – what happens when you execute the client? Cafe Le TensorFlow – understanding TensorFlow with an analogy Inputs, variables, outputs, and operations Defining inputs in TensorFlow Feeding data with Python code Preloading and storing data as tensors Building an input pipeline Defining variables in TensorFlow Defining TensorFlow outputs Defining TensorFlow operations Comparison operations Mathematical operations Scatter and gather operations Neural network-related operations Nonlinear activations used by neural networks The convolution operation The pooling operation Defining loss Optimization of neural networks The control flow operations Reusing variables with scoping Implementing our first neural network Preparing the data Defining the TensorFlow graph Running the neural network Summary 3. Word2vec – Learning Word Embeddings What is a word representation or meaning? Classical approaches to learning word representation WordNet – using an external lexical knowledge base for learning word representations Tour of WordNet Problems with WordNet One-hot encoded representation The TF-IDF method Co-occurrence matrix Word2vec – a neural network-based approach to learning word representation Exercise: is queen = king – he + she? Designing a loss function for learning word embeddings The skip-gram algorithm From raw text to structured data Learning the word embeddings with a neural network Formulating a practical loss function Efficiently approximating the loss function Negative sampling of the softmax layer Hierarchical softmax Learning the hierarchy Optimizing the learning model Implementing skip-gram with TensorFlow The Continuous Bag-of-Words algorithm Implementing CBOW in TensorFlow Summary 4. Advanced Word2vec The original skip-gram algorithm Implementing the original skip-gram algorithm Comparing the original skip-gram with the improved skip-gram Comparing skip-gram with CBOW Performance comparison Which is the winner, skip-gram or CBOW? Extensions to the word embeddings algorithms Using the unigram distribution for negative sampling Implementing unigram-based negative sampling Subsampling – probabilistically ignoring the common words Implementing subsampling Comparing the CBOW and its extensions More recent algorithms extending skip-gram and CBOW A limitation of the skip-gram algorithm The structured skip-gram algorithm The loss function The continuous window model GloVe – Global Vectors representation Understanding GloVe Implementing GloVe Document classification with Word2vec Dataset Classifying documents with word embeddings Implementation – learning word embeddings Implementation – word embeddings to document embeddings Document clustering and t-SNE visualization of embedded documents Inspecting several outliers Implementation – clustering/classification of documents with K-means Summary 5. Sentence Classification with Convolutional Neural Networks Introducing Convolution Neural Networks CNN fundamentals The power of Convolution Neural Networks Understanding Convolution Neural Networks Convolution operation Standard convolution operation Convolving with stride Convolving with padding Transposed convolution Pooling operation Max pooling Max pooling with stride Average pooling Fully connected layers Putting everything together Exercise – image classification on MNIST with CNN About the data Implementing the CNN Analyzing the predictions produced with a CNN Using CNNs for sentence classification CNN structure Data transformation The convolution operation Pooling over time Implementation – sentence classification with CNNs Summary 6. Recurrent Neural Networks Understanding Recurrent Neural Networks The problem with feed-forward neural networks Modeling with Recurrent Neural Networks Technical description of a Recurrent Neural Network Backpropagation Through Time How backpropagation works Why we cannot use BP directly for RNNs Backpropagation Through Time – training RNNs Truncated BPTT – training RNNs efficiently Limitations of BPTT – vanishing and exploding gradients Applications of RNNs One-to-one RNNs One-to-many RNNs Many-to-one RNNs Many-to-many RNNs Generating text with RNNs Defining hyperparameters Unrolling the inputs over time for Truncated BPTT Defining the validation dataset Defining weights and biases Defining state persisting variables Calculating the hidden states and outputs with unrolled inputs Calculating the loss Resetting state at the beginning of a new segment of text Calculating validation output Calculating gradients and optimizing Outputting a freshly generated chunk of text Evaluating text results output from the RNN Perplexity – measuring the quality of the text result Recurrent Neural Networks with Context Features – RNNs with longer memory Technical description of the RNN-CF Implementing the RNN-CF Defining the RNN-CF hyperparameters Defining input and output placeholders Defining weights of the RNN-CF Variables and operations for maintaining hidden and context states Calculating output Calculating the loss Calculating validation output Computing test output Computing the gradients and optimizing Text generated with the RNN-CF Summary 7. Long Short-Term Memory Networks Understanding Long Short-Term Memory Networks What is an LSTM? LSTMs in more detail How LSTMs differ from standard RNNs How LSTMs solve the vanishing gradient problem Improving LSTMs Greedy sampling Beam search Using word vectors Bidirectional LSTMs (BiLSTM) Other variants of LSTMs Peephole connections Gated Recurrent Units Summary 8. Applications of LSTM – Generating Text Our data About the dataset Preprocessing data Implementing an LSTM Defining hyperparameters Defining parameters Defining an LSTM cell and its operations Defining inputs and labels Defining sequential calculations required to process sequential data Defining the optimizer Decaying learning rate over time Making predictions Calculating perplexity (loss) Resetting states Greedy sampling to break unimodality Generating new text Example generated text Comparing LSTMs to LSTMs with peephole connections and GRUs Standard LSTM Review Example generated text Gated Recurrent Units (GRUs) Review The code Example generated text LSTMs with peepholes Review The code Example generated text Training and validation perplexities over time Improving LSTMs – beam search Implementing beam search Examples generated with beam search Improving LSTMs – generating text with words instead of n-grams The curse of dimensionality Word2vec to the rescue Generating text with Word2vec Examples generated with LSTM-Word2vec and beam search Perplexity over time Using the TensorFlow RNN API Summary 9. Applications of LSTM – Image Caption Generation Getting to know the data ILSVRC ImageNet dataset The MS-COCO dataset The machine learning pipeline for image caption generation Extracting image features with CNNs Implementation – loading weights and inferencing with VGG- Building and updating variables Preprocessing inputs Inferring VGG-16 Extracting vectorized representations of images Predicting class probabilities with VGG-16 Learning word embeddings Preparing captions for feeding into LSTMs Generating data for LSTMs Defining the LSTM Evaluating the results quantitatively BLEU ROUGE METEOR CIDEr BLEU-4 over time for our model Captions generated for test images Using TensorFlow RNN API with pretrained GloVe word vectors Loading GloVe word vectors Cleaning data Using pretrained embeddings with TensorFlow RNN API Defining the pretrained embedding layer and the adaptation layer Defining the LSTM cell and softmax layer Defining inputs and outputs Processing images and text differently Defining the LSTM output calculation Defining the logits and predictions Defining the sequence loss Defining the optimizer Summary 10. Sequence-to-Sequence Learning – Neural Machine Translation Machine translation A brief historical tour of machine translation Rule-based translation Statistical Machine Translation (SMT) Neural Machine Translation (NMT) Understanding Neural Machine Translation Intuition behind NMT NMT architecture The embedding layer The encoder The context vector The decoder Preparing data for the NMT system At training time Reversing the source sentence At testing time Training the NMT Inference with NMT The BLEU score – evaluating the machine translation systems Modified precision Brevity penalty The final BLEU score Implementing an NMT from scratch – a German to English translator Introduction to data Preprocessing data Learning word embeddings Defining the encoder and the decoder Defining the end-to-end output calculation Some translation results Training an NMT jointly with word embeddings Maximizing matchings between the dataset vocabulary and the pretrained embeddings Defining the embeddings layer as a TensorFlow variable Improving NMTs Teacher forcing Deep LSTMs Attention Breaking the context vector bottleneck The attention mechanism in detail Implementing the attention mechanism Defining weights Computing attention Some translation results – NMT with attention Visualizing attention for source and target sentences Other applications of Seq2Seq models – chatbots Training a chatbot Evaluating chatbots – Turing test Summary 11. Current Trends and the Future of Natural Language Processing Current trends in NLP Word embeddings Region embedding Input representation Learning region embeddings Implementation – region embeddings Classification accuracy Probabilistic word embedding Ensemble embedding Topic embedding Neural Machine Translation (NMT) Improving the attention mechanism Hybrid MT models Penetration into other research fields Combining NLP with computer vision Visual Question Answering (VQA) Caption generation for images with attention Reinforcement learning Teaching agents to communicate using their own language Dialogue agents with reinforcement learning Generative Adversarial Networks for NLP Towards Artificial General Intelligence One Model to Learn Them All A joint many-task model – growing a neural network for multiple NLP tasks First level – word-based tasks Second level – syntactic tasks Third level – semantic-level tasks NLP for social media Detecting rumors in social media Detecting emotions in social media Analyzing political framing in tweets New tasks emerging Detecting sarcasm Language grounding Skimming text with LSTMs Newer machine learning models Phased LSTM Dilated Recurrent Neural Networks (DRNNs) Summary References A. Mathematical Foundations and Advanced TensorFlow Basic data structures Scalar Vectors Matrices Indexing of a matrix Special types of matrices Identity matrix Diagonal matrix Tensors Tensor/matrix operations Transpose Multiplication Element-wise multiplication Inverse Finding the matrix inverse – Singular Value Decomposition (SVD) Norms Determinant Probability Random variables Discrete random variables Continuous random variables The probability mass/density function Conditional probability Joint probability Marginal probability Bayes' rule Introduction to Keras Introduction to the TensorFlow seq2seq library Defining embeddings for the encoder and decoder Defining the encoder Defining the decoder Visualizing word embeddings with TensorBoard Starting TensorBoard Saving word embeddings and visualizing via TensorBoard Summary Index