Python: Real-World Data Science: A course in four modules

Description

Unleash the power of Python and its robust data science capabilities About This BookUnleash the power of Python 3 objectsLearn to use powerful Python libraries for effective data processing and analysisHarness the power of Python to analyze data and create insightful predictive modelsUnlock deeper insights into machine learning with this vital guide to cutting-edge predictive analyticsWho This Book Is ForEntry-level analysts who want to enter in the data science world will find this course very useful to get themselves acquainted with Python’s data science capabilities for doing real-world data analysis. What You Will LearnInstall and setup PythonImplement objects in Python by creating classes and defining methodsGet acquainted with NumPy to use it with arrays and array-oriented computing in data analysisCreate effective visualizations for presenting your data using MatplotlibProcess and analyze data using the time series capabilities of pandasInteract with different kind of database systems, such as file, disk format, Mongo, and RedisApply data mining concepts to real-world problemsCompute on big data, including real-time data from the InternetExplore how to use different machine learning models to ask different questions of your dataIn DetailThe Python: Real-World Data Science course will take you on a journey to become an efficient data science practitioner by thoroughly understanding the key concepts of Python. This learning path is divided into four modules and each module are a mini course in their own right, and as you complete each one, you’ll have gained key skills and be ready for the material in the next module. The course begins with getting your Python fundamentals nailed down. After getting familiar with Python core concepts, it’s time that you dive into the field of data science. In the second module, you'll learn how to perform data analysis using Python in a practical and example-driven way. The third module will teach you how to design and develop data mining applications using a variety of datasets, starting with basic classification and affinity analysis to more complex data types including text, images, and graphs. Machine learning and predictive analytics have become the most important approaches to uncover data gold mines. In the final module, we'll discuss the necessary details regarding machine learning concepts, offering intuitive yet informative explanations on how machine learning algorithms work, how to use them, and most importantly, how to avoid the common pitfalls. Style and approachThis course includes all the resources that will help you jump into the data science field with Python and learn how to make sense of data. The aim is to create a smooth learning path that will teach you how to get started with powerful Python libraries and perform various data science techniques in depth. Cover......Page 1 Meet Your Course Guide......Page 3 Table of Contents......Page 10 Introduction and First Steps – Take a Deep Breath......Page 32 A proper introduction......Page 33 Enter the Python......Page 35 Developer productivity......Page 36 Satisfaction and enjoyment......Page 37 Setting up the environment......Page 38 Python 2 versus Python 3 – the great debate......Page 39 What you need for this course......Page 40 Installing IPython......Page 41 Installing additional packages......Page 43 How you can run a Python program......Page 44 Running the Python interactive shell......Page 45 Running Python as a GUI application......Page 47 How is Python code organized......Page 48 How do we use modules and packages......Page 49 Names and namespaces......Page 52 Scopes......Page 54 Guidelines on how to write good code......Page 57 The Python culture......Page 58 A note on the IDEs......Page 60 Introducing object-oriented......Page 62 Objects and classes......Page 64 Specifying attributes and behaviors......Page 66 Data describes objects......Page 67 Behaviors are actions......Page 68 Hiding details and creating the public interface......Page 70 Composition......Page 72 Inheritance......Page 75 Inheritance provides abstraction......Page 77 Multiple inheritance......Page 78 Case study......Page 79 Creating Python classes......Page 90 Adding attributes......Page 92 Talking to yourself......Page 93 More arguments......Page 94 Initializing the object......Page 96 Explaining yourself......Page 98 Modules and packages......Page 100 Absolute imports......Page 103 Relative imports......Page 104 Organizing module contents......Page 106 Who can access my data?......Page 109 Third-party libraries......Page 111 Case study......Page 112 Basic inheritance......Page 124 Extending built-ins......Page 127 Overriding and super......Page 128 Multiple inheritance......Page 130 The diamond problem......Page 132 Different sets of arguments......Page 137 Polymorphism......Page 140 Using an abstract base class......Page 143 Creating an abstract base class......Page 144 Demystifying the magic......Page 146 Case study......Page 147 Expecting the Unexpected......Page 164 Raising exceptions......Page 165 Raising an exception......Page 166 The effects of an exception......Page 168 Handling exceptions......Page 169 The exception hierarchy......Page 175 Defining our own exceptions......Page 176 Case study......Page 181 Treat objects as objects......Page 194 Adding behavior to class data with properties......Page 198 Properties in detail......Page 201 Decorators – another way to create properties......Page 203 Deciding when to use properties......Page 205 Manager objects......Page 207 Removing duplicate code......Page 209 In practice......Page 211 Case study......Page 214 Empty objects......Page 226 Tuples and named tuples......Page 228 Named tuples......Page 230 Dictionaries......Page 231 Using defaultdict......Page 235 Counter......Page 237 Lists......Page 238 Sorting lists......Page 240 Sets......Page 244 Extending built-ins......Page 248 Queues......Page 253 FIFO queues......Page 254 LIFO queues......Page 256 Priority queues......Page 257 Case study......Page 259 Python built-in functions......Page 268 Reversed......Page 269 Enumerate......Page 271 File I/O......Page 272 Placing it in context......Page 274 An alternative to method overloading......Page 276 Default arguments......Page 277 Variable argument lists......Page 279 Unpacking arguments......Page 283 Functions are objects too......Page 284 Using functions as attributes......Page 288 Callable objects......Page 289 Case study......Page 290 Strings......Page 300 String manipulation......Page 301 String formatting......Page 303 Escaping braces......Page 304 Keyword arguments......Page 305 Container lookups......Page 306 Object lookups......Page 307 Making it look right......Page 308 Strings are Unicode......Page 310 Converting bytes to text......Page 311 Converting text to bytes......Page 312 Mutable byte strings......Page 314 Regular expressions......Page 315 Matching patterns......Page 316 Matching a selection of characters......Page 317 Escaping characters......Page 318 Matching multiple characters......Page 319 Grouping patterns together......Page 320 Getting information from regular expressions......Page 321 Serializing objects......Page 323 Customizing pickles......Page 325 Serializing web objects......Page 328 Case study......Page 331 Design patterns in brief......Page 340 Iterators......Page 341 The iterator protocol......Page 342 List comprehensions......Page 344 Set and dictionary comprehensions......Page 347 Generator expressions......Page 348 Generators......Page 350 Yield items from another iterable......Page 352 Coroutines......Page 355 Back to log parsing......Page 358 Closing coroutines and throwing exceptions......Page 360 The relationship between coroutines, generators, and functions......Page 361 Case study......Page 362 The decorator pattern......Page 372 A decorator example......Page 373 Decorators in Python......Page 376 The observer pattern......Page 378 An observer example......Page 379 The strategy pattern......Page 381 A strategy example......Page 382 The state pattern......Page 384 A state example......Page 385 The singleton pattern......Page 391 Singleton implementation......Page 392 A template example......Page 396 The adapter pattern......Page 402 The facade pattern......Page 406 The flyweight pattern......Page 408 The command pattern......Page 412 The abstract factory pattern......Page 417 The composite pattern......Page 422 Why test?......Page 430 Test-driven development......Page 432 Unit testing......Page 433 Assertion methods......Page 435 Reducing boilerplate and cleaning up......Page 437 Organizing and running tests......Page 438 Ignoring broken tests......Page 439 Testing with py.test......Page 441 One way to do setup and cleanup......Page 443 A completely different way to set up variables......Page 446 Skipping tests with py.test......Page 450 Imitating expensive objects......Page 451 How much testing is enough?......Page 455 Case study......Page 458 Implementing it......Page 459 Concurrency......Page 468 Threads......Page 469 Shared memory......Page 472 The global interpreter lock......Page 473 Multiprocessing......Page 474 Multiprocessing pools......Page 476 Queues......Page 479 Futures......Page 481 AsyncIO......Page 484 AsyncIO in action......Page 485 Reading an AsyncIO future......Page 486 AsyncIO for networking......Page 487 Using executors to wrap blocking code......Page 490 Executors......Page 492 Case study......Page 493 Introducing Data Analysis and Libraries......Page 506 Data analysis and processing......Page 507 An overview of the libraries in data analysis......Page 510 NumPy......Page 513 PyMongo......Page 514 The scikit-learn library......Page 515 NumPy Arrays and Vectorized Computation......Page 518 Data types......Page 519 Array creation......Page 521 Indexing and slicing......Page 523 Fancy indexing......Page 524 Numerical operations on arrays......Page 525 Array functions......Page 526 Data processing using arrays......Page 528 Saving an array......Page 530 Linear algebra with NumPy......Page 531 NumPy random numbers......Page 533 An overview of the pandas package......Page 538 Series......Page 539 The DataFrame......Page 541 Reindexing and altering labels......Page 545 Head and tail......Page 546 Binary operations......Page 547 Functional statistics......Page 548 Function application......Page 550 Sorting......Page 551 Indexing and selecting data......Page 553 Computational tools......Page 555 Working with missing data......Page 556 Hierarchical indexing......Page 559 The Panel data......Page 562 Data Visualization......Page 566 The matplotlib API primer......Page 567 Line properties......Page 570 Figures and subplots......Page 572 Scatter plots......Page 575 Bar plots......Page 576 Contour plots......Page 577 Histogram plots......Page 578 Legends and annotations......Page 579 Plotting functions with pandas......Page 583 Bokeh......Page 586 MayaVi......Page 587 Time series primer......Page 590 Working with date and time objects......Page 591 Resampling time series......Page 599 Downsampling time series data......Page 600 Upsampling time series data......Page 602 Timedeltas......Page 606 Time series plotting......Page 607 Reading data from text format......Page 614 Writing data to text format......Page 619 Interacting with data in binary format......Page 620 HDF5......Page 621 Interacting with data in MongoDB......Page 622 The simple value......Page 627 List......Page 628 Set......Page 629 Ordered set......Page 630 Data Analysis Application Examples......Page 634 Data munging......Page 635 Cleaning data......Page 637 Filtering......Page 640 Merging data......Page 643 Reshaping data......Page 647 Data aggregation......Page 648 Grouping data......Page 651 Getting Started with Data Mining......Page 660 Introducing data mining......Page 661 What is affinity analysis?......Page 662 Loading the dataset with NumPy......Page 663 Implementing a simple ranking of rules......Page 665 Ranking to find the best rules......Page 668 What is classification?......Page 671 Loading and preparing the dataset......Page 672 Implementing the OneR algorithm......Page 673 Testing the algorithm......Page 676 scikit-learn estimators......Page 680 Nearest neighbors......Page 681 Distance metrics......Page 682 Loading the dataset......Page 684 Moving towards a standard workflow......Page 686 Running the algorithm......Page 687 Setting parameters......Page 688 Preprocessing using pipelines......Page 691 Standard preprocessing......Page 692 Pipelines......Page 693 Loading the dataset......Page 698 Collecting the data......Page 699 Using pandas to load the dataset......Page 700 Cleaning up the dataset......Page 701 Extracting new features......Page 702 Decision trees......Page 704 Parameters in decision trees......Page 705 Using decision trees......Page 706 Putting it all together......Page 707 Random forests......Page 711 How do ensembles work?......Page 712 Applying Random forests......Page 713 Engineering new features......Page 715 Affinity analysis......Page 718 Algorithms for affinity analysis......Page 719 Choosing parameters......Page 720 Loading with pandas......Page 721 Sparse data formats......Page 722 The Apriori implementation......Page 723 The Apriori algorithm......Page 725 Implementation......Page 726 Extracting association rules......Page 729 Evaluation......Page 733 Extracting Features with Transformers......Page 738 Representing reality in models......Page 739 Common feature patterns......Page 741 Creating good features......Page 744 Feature selection......Page 745 Selecting the best individual features......Page 747 Feature creation......Page 751 Creating your own transformer......Page 753 The transformer API......Page 754 Implementation details......Page 755 Unit testing......Page 756 Putting it all together......Page 758 Social Media Insight Using Naive Bayes......Page 760 Disambiguation......Page 761 Downloading data from a social network......Page 762 Loading and classifying the dataset......Page 764 Creating a replicable dataset from Twitter......Page 769 Text transformers......Page 773 Bag-of-words......Page 774 N-grams......Page 775 Naive Bayes......Page 776 Bayes' theorem......Page 777 Naive Bayes algorithm......Page 778 How it works......Page 779 Application......Page 780 Extracting word counts......Page 781 Training the Naive Bayes classifier......Page 782 Putting it all together......Page 783 Evaluation using the F1-score......Page 784 Getting useful features from models......Page 785 Discovering Accounts to Follow Using Graph Mining......Page 790 Loading the dataset......Page 791 Classifying with an existing model......Page 792 Getting follower information from Twitter......Page 794 Building the network......Page 797 Creating a graph......Page 800 Creating a similarity graph......Page 802 Finding subgraphs......Page 806 Connected components......Page 807 Optimizing criteria......Page 810 Beating CAPTCHAs with Neural Networks......Page 818 Artificial neural networks......Page 819 An introduction to neural networks......Page 820 Creating the dataset......Page 822 Drawing basic CAPTCHAs......Page 823 Splitting the image into individual letters......Page 824 Creating a training dataset......Page 826 Adjusting our training dataset to our methodology......Page 828 Training and classifying......Page 829 Back propagation......Page 831 Predicting words......Page 832 Ranking mechanisms for words......Page 837 Putting it all together......Page 838 Authorship Attribution......Page 842 Applications and use cases......Page 843 Attributing authorship......Page 844 Getting the data......Page 846 Function words......Page 849 Counting function words......Page 850 Classifying with function words......Page 851 Support vector machines......Page 852 Kernels......Page 854 Character n-grams......Page 855 Extracting character n-grams......Page 856 Using the Enron dataset......Page 857 Creating a dataset loader......Page 858 Putting it all together......Page 863 Evaluation......Page 864 Obtaining news articles......Page 868 Using a Web API to get data......Page 869 Reddit as a data source......Page 872 Getting the data......Page 873 Extracting text from arbitrary websites......Page 875 Finding the stories in arbitrary websites......Page 876 Putting it all together......Page 877 Grouping news articles......Page 879 The k-means algorithm......Page 880 Evaluating the results......Page 883 Extracting topic information from clusters......Page 885 Using clustering algorithms as transformers......Page 886 Evidence accumulation......Page 887 How it works......Page 891 Implementation......Page 892 An introduction to online learning......Page 893 Implementation......Page 894 Classifying Objects in Images Using Deep Learning......Page 900 Application scenario and goals......Page 901 Use cases......Page 904 Deep neural networks......Page 905 Implementation......Page 906 An introduction to Theano......Page 907 An introduction to Lasagne......Page 908 Implementing neural networks with nolearn......Page 913 GPU optimization......Page 917 When to use GPUs for computation......Page 918 Running our code on a GPU......Page 919 Setting up the environment......Page 920 Getting the data......Page 923 Creating the neural network......Page 924 Putting it all together......Page 926 Working with Big Data......Page 930 Big data......Page 931 Application scenario and goals......Page 932 Intuition......Page 934 A word count example......Page 936 Hadoop MapReduce......Page 937 Application......Page 938 Getting the data......Page 939 Extracting the blog posts......Page 941 Training Naive Bayes......Page 944 Putting it all together......Page 947 Training on Amazon's EMR infrastructure......Page 952 Scikit-learn tutorials......Page 958 Comparing classifiers......Page 959 The Eclat algorithm......Page 960 More complex algorithms......Page 961 Local n-grams......Page 962 Mahotas......Page 963 More resources......Page 964 Giving Computers the Ability to Learn from Data......Page 970 The three different types of machine learning......Page 971 Classification for predicting class labels......Page 972 Regression for predicting continuous outcomes......Page 973 Discovering hidden structures with unsupervised learning......Page 975 Dimensionality reduction for data compression......Page 976 An introduction to the basic terminology and notations......Page 978 A roadmap for building machine learning systems......Page 980 Training and selecting a predictive model......Page 981 Using Python for machine learning......Page 982 Training Machine Learning Algorithms for Classification......Page 986 Artificial neurons – a brief glimpse into the early history of machine learning......Page 987 Implementing a perceptron learning algorithm in Python......Page 993 Training a perceptron model on the Iris dataset......Page 996 Adaptive linear neurons and the convergence of learning......Page 1002 Minimizing cost functions with gradient descent......Page 1003 Implementing an Adaptive Linear Neuron in Python......Page 1005 Large scale machine learning and stochastic gradient descent......Page 1011 Choosing a classification algorithm......Page 1018 Training a perceptron via scikit-learn......Page 1019 Logistic regression intuition and conditional probabilities......Page 1025 Learning the weights of the logistic cost function......Page 1029 Training a logistic regression model with scikit-learn......Page 1031 Tackling overfitting via regularization......Page 1034 Maximum margin classification with support vector machines......Page 1038 Maximum margin intuition......Page 1039 Dealing with the nonlinearly separable case using slack variables......Page 1040 Alternative implementations in scikit-learn......Page 1043 Solving nonlinear problems using a kernel SVM......Page 1044 Using the kernel trick to find separating hyperplanes in higher dimensional space......Page 1046 Decision tree learning......Page 1049 Maximizing information gain – getting the most bang for the buck......Page 1051 Building a decision tree......Page 1057 Combining weak to strong learners via random forests......Page 1059 K-nearest neighbors – a lazy learning algorithm......Page 1061 Dealing with missing data......Page 1068 Eliminating samples or features with missing values......Page 1070 Understanding the scikit-learn estimator API......Page 1071 Handling categorical data......Page 1073 Encoding class labels......Page 1074 Performing one-hot encoding on nominal features......Page 1076 Partitioning a dataset in training and test sets......Page 1077 Bringing features onto the same scale......Page 1079 Selecting meaningful features......Page 1081 Sparse solutions with L1 regularization......Page 1082 Sequential feature selection algorithms......Page 1088 Assessing feature importance with random forests......Page 1094 Compressing Data via Dimensionality Reduction......Page 1098 Unsupervised dimensionality reduction via principal component analysis......Page 1099 Total and explained variance......Page 1100 Feature transformation......Page 1104 Principal component analysis in scikit-learn......Page 1106 Supervised data compression via linear discriminant analysis......Page 1109 Computing the scatter matrices......Page 1111 Selecting linear discriminants for the new feature subspace......Page 1114 Projecting samples onto the new feature space......Page 1116 LDA via scikit-learn......Page 1117 Kernel functions and the kernel trick......Page 1119 Implementing a kernel principal component analysis in Python......Page 1125 Example 1 – separating half-moon shapes......Page 1126 Example 2 – separating concentric circles......Page 1130 Projecting new data points......Page 1133 Kernel principal component analysis in scikit-learn......Page 1137 Streamlining workflows with pipelines......Page 1140 Loading the Breast Cancer Wisconsin dataset......Page 1141 Combining transformers and estimators in a pipeline......Page 1142 The holdout method......Page 1144 K-fold cross-validation......Page 1146 Debugging algorithms with learning and validation curves......Page 1150 Diagnosing bias and variance problems with learning curves......Page 1151 Addressing overfitting and underfitting with validation curves......Page 1154 Fine-tuning machine learning models via grid search......Page 1156 Tuning hyperparameters via grid search......Page 1157 Algorithm selection with nested cross-validation......Page 1158 Looking at different performance evaluation metrics......Page 1160 Reading a confusion matrix......Page 1161 Optimizing the precision and recall of a classification model......Page 1162 Plotting a receiver operating characteristic......Page 1164 The scoring metrics for multiclass classification......Page 1168 Learning with ensembles......Page 1172 Implementing a simple majority vote classifier......Page 1176 Combining different algorithms for classification with majority vote......Page 1183 Evaluating and tuning the ensemble classifier......Page 1186 Bagging – building an ensemble of classifiers from bootstrap samples......Page 1192 Leveraging weak learners via adaptive boosting......Page 1197 Predicting Continuous Target Variables with Regression Analysis......Page 1208 Introducing a simple linear regression model......Page 1209 Exploring the Housing Dataset......Page 1210 Visualizing the important characteristics of a dataset......Page 1211 Solving regression for regression parameters with gradient descent......Page 1216 Estimating the coefficient of a regression model via scikit-learn......Page 1220 Fitting a robust regression model using RANSAC......Page 1222 Evaluating the performance of linear regression models......Page 1225 Using regularized methods for regression......Page 1228 Turning a linear regression model into a curve – polynomial regression......Page 1230 Modeling nonlinear relationships in the Housing Dataset......Page 1232 Dealing with nonlinear relationships using random forests......Page 1235 Decision tree regression......Page 1236 Random forest regression......Page 1238 Chapter 2: Object-oriented Design......Page 1244 Chapter 6: Interacting with Databases......Page 1245 Chapter 3: Predicting Sports Winners with Decision Trees......Page 1246 Chapter 8: Beating CAPTCHAs with Neural Networks......Page 1247 Chapter 1: Giving Computers the Ability to Learn from Data......Page 1248 Chapter 6: Learning Best Practices for Model Evaluation and Hyperparameter Tuning......Page 1249 Chapter 8: Predicting Continuous Target Variables with Regression Analysis......Page 1250