Python for Data Science for Dummies

Description

Unleash the power of Python for your data analysis projects with For Dummies! Python is the preferred programming language for data scientists and combines the best features of Matlab, Mathematica, and R into libraries specific to data analysis and visualization. Python for Data Science For Dummies shows you how to take advantage of Python programming to acquire, organize, process, and analyze large amounts of information and use basic statistics concepts to identify trends and patterns. You'll get familiar with the Python development environment, manipulate data, design compelling visualizations, and solve scientific computing challenges as you work your way through this user-friendly guide. Covers the fundamentals of Python data analysis programming and statistics to help you build a solid foundation in data science concepts like probability, random distributions, hypothesis testing, and regression models Explains objects, functions, modules, and libraries and their role in data analysis Walks you through some of the most widely-used libraries, including NumPy, SciPy, BeautifulSoup, Pandas, and MatPlobLib Whether you're new to data analysis or just new to Python, Python for Data Science For Dummies is your practical guide to getting a grip on data overload and doing interesting things with the oodles of information you uncover. Title Page Copyright Page Table of Contents Introduction About This Book Foolish Assumptions Icons Used in This Book Beyond the Book Where to Go from Here Part I Getting Started with Python for Data Science Chapter 1 Discovering the Match between Data Science and Python Defining the Sexiest Job of the 21st Century Considering the emergence of data science Outlining the core competencies of a data scientist Linking data science and big data Understanding the role of programming Creating the Data Science Pipeline Preparing the data Performing exploratory data analysis Learning from data Visualizing Obtaining insights and data products Understanding Python’s Role in Data Science Considering the shifting profile of data scientists Working with a multipurpose, simple, and efficient language Learning to Use Python Fast Loading data Training a model Viewing a result Chapter 2 Introducing Python’s Capabilities and Wonders Why Python? Grasping Python’s core philosophy Discovering present and future development goals Working with Python Getting a taste of the language Understanding the need for indentation Working at the command line or in the IDE Performing Rapid Prototyping and Experimentation Considering Speed of Execution Visualizing Power Using the Python Ecosystem for Data Science Accessing scientific tools using SciPy Performing fundamental scientific computing using NumPy Performing data analysis using pandas Implementing machine learning using Scikit‐learn Plotting the data using matplotlib Parsing HTML documents using Beautiful Soup Chapter 3 Setting Up Python for Data Science Considering the Off‐the‐Shelf Cross‐Platform Scientific Distributions Getting Continuum Analytics Anaconda Getting Enthought Canopy Express Getting pythonxy Getting WinPython Installing Anaconda on Windows Installing Anaconda on Linux Installing Anaconda on Mac OS X Downloading the Datasets and Example Code Using IPython Notebook Defining the code repository Understanding the datasets used in this book Chapter 4 Reviewing Basic Python Working with Numbers and Logic Performing variable assignments Doing arithmetic Comparing data using Boolean expressions Creating and Using Strings Interacting with Dates Creating and Using Functions Creating reusable functions Calling functions in a variety of ways Using Conditional and Loop Statements Making decisions using the if statement Choosing between multiple options using nested decisions Performing repetitive tasks using for Using the while statement Storing Data Using Sets, Lists, and Tuples Performing operations on sets Working with lists Creating and using Tuples Defining Useful Iterators Indexing Data Using Dictionaries Part II Getting Your Hands Dirty with Data Chapter 5 Working with Real Data Uploading, Streaming, and Sampling Data Uploading small amounts of data into memory Streaming large amounts of data into memory Sampling data Accessing Data in Structured Flat‐File Form Reading from a text file Reading CSV delimited format Reading Excel and other Microsoft Office files Sending Data in Unstructured File Form Managing Data from Relational Databases Interacting with Data from NoSQL Databases Accessing Data from the Web Chapter 6 Conditioning Your Data Juggling between NumPy and pandas Knowing when to use NumPy Knowing when to use pandas Validating Your Data Figuring out what’s in your data Removing duplicates Creating a data map and data plan Manipulating Categorical Variables Creating categorical variables Renaming levels Combining levels Dealing with Dates in Your Data Formatting date and time values Using the right time transformation Dealing with Missing Data Finding the missing data Encoding missingness Imputing missing data Slicing and Dicing: Filtering and Selecting Data Slicing rows Slicing columns Dicing Concatenating and Transforming Adding new cases and variables Removing data Sorting and shuffling Aggregating Data at Any Level Chapter 7 Shaping Data Working with HTML Pages Parsing XML and HTML Using XPath for data extraction Working with Raw Text Dealing with Unicode Stemming and removing stop words Introducing regular expressions Using the Bag of Words Model and Beyond Understanding the bag of words model Working with n‐grams Implementing TF‐IDF transformations Working with Graph Data Understanding the adjacency matrix Using NetworkX basics Chapter 8 Putting What You Know in Action Contextualizing Problems and Data Evaluating a data science problem Researching solutions Formulating a hypothesis Preparing your data Considering the Art of Feature Creation Defining feature creation Combining variables Understanding binning and discretization Using indicator variables Transforming distributions Performing Operations on Arrays Using vectorization Performing simple arithmetic on vectors and matrices Performing matrix vector multiplication Performing matrix multiplication Part III Visualizing the Invisible Chapter 9 Getting a Crash Course in MatPlotLib Starting with a Graph Defining the plot Drawing multiple lines and plots Saving your work Setting the Axis, Ticks, Grids Getting the axes Formatting the axes Adding grids Defining the Line Appearance Working with line styles Using colors Adding markers Using Labels, Annotations, and Legends Adding labels Annotating the chart Creating a legend Chapter 10 Visualizing the Data Choosing the Right Graph Showing parts of a whole with pie charts Creating comparisons with bar charts Showing distributions using histograms Depicting groups using box plots Seeing data patterns using scatterplots Creating Advanced Scatterplots Depicting groups Showing correlations Plotting Time Series Representing time on axes Plotting trends over time Plotting Geographical Data Visualizing Graphs Developing undirected graphs Developing directed graphs Chapter 11 Understanding the Tools Using the IPython Console Interacting with screen text Changing the window appearance Getting Python help Getting IPython help Using magic functions Discovering objects Using IPython Notebook Working with styles Restarting the kernel Restoring a checkpoint Performing Multimedia and Graphic Integration Embedding plots and other images Loading examples from online sites Obtaining online graphics and multimedia Part IV Wrangling Data Chapter 12 Stretching Python’s Capabilities Playing with Scikit‐learn Understanding classes in Scikit‐learn Defining applications for data science Performing the Hashing Trick Using hash functions Demonstrating the hashing trick Working with deterministic selection Considering Timing and Performance Benchmarking with timeit Working with the memory profiler Running in Parallel Performing multicore parallelism Demonstrating multiprocessing Chapter 13 Exploring Data Analysis The EDA Approach Defining Descriptive Statistics for Numeric Data Measuring central tendency Measuring variance and range Working with percentiles Defining measures of normality Counting for Categorical Data Understanding frequencies Creating contingency tables Creating Applied Visualization for EDA Inspecting boxplots Performing t‐tests after boxplots Observing parallel coordinates Graphing distributions Plotting scatterplots Understanding Correlation Using covariance and correlation Using nonparametric correlation Considering chi‐square for tables Modifying Data Distributions Using the normal distribution Creating a Z‐score standardization Transforming other notable distributions Chapter 14 Reducing Dimensionality Understanding SVD Looking for dimensionality reduction Using SVD to measure the invisible Performing Factor and Principal Component Analysis Considering the psychometric model Looking for hidden factors Using components, not factors Achieving dimensionality reduction Understanding Some Applications Recognizing faces with PCA Extracting Topics with NMF Recommending movies Chapter 15 Clustering Clustering with K‐means Understanding centroid‐based algorithms Creating an example with image data Looking for optimal solutions Clustering big data Performing Hierarchical Clustering Moving Beyond the Round-Shaped Clusters: DBScan Chapter 16 Detecting Outliers in Data Considering Detection of Outliers Finding more things that can go wrong Understanding anomalies and novel data Examining a Simple Univariate Method Leveraging on the Gaussian distribution Making assumptions and checking out Developing a Multivariate Approach Using principal component analysis Using cluster analysis Automating outliers detection with SVM Part V Learning from Data Chapter 17 Exploring Four Simple and Effective Algorithms Guessing the Number: Linear Regression Defining the family of linear models Using more variables Understanding limitations and problems Moving to Logistic Regression Applying logistic regression Considering when classes are more Making Things as Simple as Naïve Bayes Finding out that Naïve Bayes isn’t so naïve Predicting text classifications Learning Lazily with Nearest Neighbors Predicting after observing neighbors Choosing your k parameter wisely Chapter 18 Performing Cross‐Validation, Selection, and Optimization Pondering the Problem of Fitting a Model Understanding bias and variance Defining a strategy for picking models Dividing between training and test sets Cross‐Validating Using cross‐validation on k folds Sampling stratifications for complex data Selecting Variables Like a Pro Selecting by univariate measures Using a greedy search Pumping Up Your Hyperparameters Implementing a grid search Trying a randomized search Chapter 19 Increasing Complexity with Linear and Nonlinear Tricks Using Nonlinear Transformations Doing variable transformations Creating interactions between variables Regularizing Linear Models Relying on Ridge regression (L2) Using the Lasso (L1) Leveraging regularization Combining L1 & L2: Elasticnet Fighting with Big Data Chunk by Chunk Determining when there is too much data Implementing Stochastic Gradient Descent Understanding Support Vector Machines Relying on a computational method Fixing many new parameters Classifying with SVC Going nonlinear is easy Performing regression with SVR Creating a stochastic solution with SVM Chapter 20 Understanding the Power of the Many Starting with a Plain Decision Tree Understanding a decision tree Creating classification and regression trees Making Machine Learning Accessible Working with a Random Forest classifier Working with a Random Forest regressor Optimizing a Random Forest Boosting Predictions Knowing that many weak predictors win Creating a gradient boosting classifier Creating a gradient boosting regressor Using GBM hyper‐parameters Part VI The Part of Tens Chapter 21 Ten Essential Data Science Resource Collections Gaining Insights with Data Science Weekly Obtaining a Resource List at U Climb Higher Getting a Good Start with KDnuggets Accessing the Huge List of Resources on Data Science Central Obtaining the Facts of Open Source Data Science from Masters Locating Free Learning Resources with Quora Receiving Help with Advanced Topics at Conductrics Learning New Tricks from the Aspirational Data Scientist Finding Data Intelligence and Analytics Resources at AnalyticBridge Zeroing In on Developer Resources with Jonathan Bower Chapter 22 Ten Data Challenges You Should Take Meeting the Data Science London + Scikit‐learn Challenge Predicting Survival on the Titanic Finding a Kaggle Competition that Suits Your Needs Honing Your Overfit Strategies Trudging Through the MovieLens Dataset Getting Rid of Spam Emails Working with Handwritten Information Working with Pictures Analyzing Amazon.com Reviews Interacting with a Huge Graph Index EULA