Web Scraping with Python : Collecting Data from the Modern Web

Description

Preface What Is Web Scraping? Why Web Scraping? About This Book Conventions Used in This Book Using Code Examples Safari® Books Online How to Contact Us Acknowledgments I. Building Scrapers 1. Your First Web Scraper Connecting An Introduction to BeautifulSoup Installing BeautifulSoup Running BeautifulSoup Connecting Reliably 2. Advanced HTML Parsing You Don’t Always Need a Hammer Another Serving of BeautifulSoup find() and findAll() with BeautifulSoup Other BeautifulSoup Objects Navigating Trees Dealing with children and other descendants Dealing with siblings Dealing with your parents Regular Expressions Regular Expressions and BeautifulSoup Accessing Attributes Lambda Expressions Beyond BeautifulSoup 3. Starting to Crawl Traversing a Single Domain Crawling an Entire Site Collecting Data Across an Entire Site Crawling Across the Internet Crawling with Scrapy 4. Using APIs How APIs Work Common Conventions Methods Authentication Responses API Calls Echo Nest A Few Examples Twitter Getting Started A Few Examples Google APIs Getting Started A Few Examples Parsing JSON Bringing It All Back Home More About APIs 5. Storing Data Media Files Storing Data to CSV MySQL Installing MySQL Some Basic Commands Integrating with Python Database Techniques and Good Practice “Six Degrees” in MySQL Email 6. Reading Documents Document Encoding Text Text Encoding and the Global Internet A brief overview of encoding types Encodings in action CSV Reading CSV Files PDF Microsoft Word and .docx II. Advanced Scraping 7. Cleaning Your Dirty Data Cleaning in Code Data Normalization Cleaning After the Fact OpenRefine Installation Using OpenRefine Filtering Cleaning 8. Reading and Writing Natural Languages Summarizing Data Markov Models Six Degrees of Wikipedia: Conclusion Natural Language Toolkit Installation and Setup Statistical Analysis with NLTK Lexicographical Analysis with NLTK Additional Resources 9. Crawling Through Forms and Logins Python Requests Library Submitting a Basic Form Radio Buttons, Checkboxes, and Other Inputs Submitting Files and Images Handling Logins and Cookies HTTP Basic Access Authentication Other Form Problems 10. Scraping JavaScript A Brief Introduction to JavaScript Common JavaScript Libraries jQuery Google Analytics Google Maps Ajax and Dynamic HTML Executing JavaScript in Python with Selenium Handling Redirects 11. Image Processing and Text Recognition Overview of Libraries Pillow Tesseract Installing Tesseract NumPy Processing Well-Formatted Text Scraping Text from Images on Websites Reading CAPTCHAs and Training Tesseract Training Tesseract Retrieving CAPTCHAs and Submitting Solutions 12. Avoiding Scraping Traps A Note on Ethics Looking Like a Human Adjust Your Headers Handling Cookies Timing Is Everything Common Form Security Features Hidden Input Field Values Avoiding Honeypots The Human Checklist 13. Testing Your Website with Scrapers An Introduction to Testing What Are Unit Tests? Python unittest Testing Wikipedia Testing with Selenium Interacting with the Site Drag and drop Taking screenshots Unittest or Selenium? 14. Scraping Remotely Why Use Remote Servers? Avoiding IP Address Blocking Portability and Extensibility Tor PySocks Remote Hosting Running from a Website Hosting Account Running from the Cloud Additional Resources Moving Forward A. Python at a Glance Installation and “Hello, World!” B. The Internet at a Glance C. The Legalities and Ethics of Web Scraping Trademarks, Copyrights, Patents, Oh My! Copyright Law Trespass to Chattels The Computer Fraud and Abuse Act robots.txt and Terms of Service Three Web Scrapers eBay versus Bidder’s Edge and Trespass to Chattels United States v. Auernheimer and The Computer Fraud and Abuse Act Field v. Google: Copyright and robots.txt Index