Beginning Apache Spark Using Azure Databricks: Unleashing Large Cluster Analytics in the Cloud

Description

Analyze vast amounts of data in record time using Apache Spark with Databricks in the Cloud. Learn the fundamentals, and more, of running analytics on large clusters in Azure and AWS, using Apache Spark with Databricks on top. Discover how to squeeze the most value out of your data at a mere fraction of what classical analytics solutions cost, while at the same time getting the results you need, incrementally faster. This book explains how the confluence of these pivotal technologies gives you enormous power, and cheaply, when it comes to huge datasets. You will begin by learning how cloud infrastructure makes it possible to scale your code to large amounts of processing units, without having to pay for the machinery in advance. From there you will learn how Apache Spark, an open source framework, can enable all those CPUs for data analytics use. Finally, you will see how services such as Databricks provide the power of Apache Spark, without you having to know anything about configuring hardware or software. By removing the need for expensive experts and hardware, your resources can instead be allocated to actually finding business value in the data. This book guides you through some advanced topics such as analytics in the cloud, data lakes, data ingestion, architecture, machine learning, and tools, including Apache Spark, Apache Hadoop, Apache Hive, Python, and SQL. Valuable exercises help reinforce what you have learned. What You Will Learn Discover the value of big data analytics that leverage the power of the cloudGet started with Databricks using SQL and Python in either Microsoft Azure or AWSUnderstand the underlying technology, and how the cloud and Apache Spark fit into the bigger picture See how these tools are used in the real world Run basic analytics, including machine learning, on billions of rows at a fraction of a cost or free Who This Book Is For Data engineers, data scientists, and cloud architects who want or need to run advanced analytics in the cloud. It is assumed that the reader has data experience, but perhaps minimal exposure to Apache Spark and Azure Databricks. The book is also recommended for people who want to get started in the analytics field, as it provides a strong foundation. Table of Contents About the Author About the Technical Reviewer Introduction Chapter 1: Introduction to Large-Scale Data Analytics Analytics, the hype Analytics, the reality Large-scale analytics for fun and profit Data: Fueling analytics Free as in speech. And beer! Into the clouds Databricks: Analytics for the lazy ones How to analyze data Large-scale examples from the real world Telematics at Volvo Trucks Fraud detection at Visa Customer analytics at Target Targeted ads at Cambridge Analytica Summary Chapter 2: Spark and Databricks Apache Spark, the short overview Databricks: Managed Spark The far side of the Databricks moon Spark architecture Apache Spark processing Working with data Data processing Storing data Cool components on top of Core Summary Chapter 3: Getting Started with Databricks Cloud-only Community edition: No money? No problem Mostly good enough Getting started with the community edition Commercial editions: The ones you want Databricks on Amazon Web Services Azure Databricks Summary Chapter 4: Workspaces, Clusters, and Notebooks Getting around in the UI Clusters: Powering up the engines Data: Getting access to the fuel Notebooks: Where the work happens Summary Chapter 5: Getting Data into Databricks Databricks File System Navigating the file system The FileStore, a portal to your data Schemas, databases, and tables Hive Metastore The many types of source files Going binary Alternative transportation Importing from your computer Getting data from the Web Working with the shell Basic importing with Python Getting data with SQL Mounting a file system Mounting example Amazon S3 Mounting example Microsoft Blog Storage Getting rid of the mounts How to get data out of Databricks Summary Chapter 6: Querying Data Using SQL The Databricks flavor Getting started Picking up data Filtering data Joins and merges Ordering data Functions Windowing functions A view worth keeping Hierarchical data Creating data Manipulating data Delta Lake SQL UPDATE, DELETE, and MERGE Keeping Delta Lake in order Transaction logs Selecting metadata Gathering statistics Summary Chapter 7: The Power of Python Python: The language of choice A turbo-charged intro to Python Finding the data DataFrames: Where active data lives Getting some data Selecting data from DataFrames Chaining combo commands Working with multiple DataFrames Slamming data together Summary Chapter 8: ETL and Advanced Data Wrangling ETL: A recap An overview of the Spark UI Cleaning and transforming data Finding nulls Getting rid of nulls Filling nulls with values Removing duplicates Identifying and clearing out extreme values Taking care of columns Pivoting Explode When being lazy is good Caching data Data compression A short note about functions Lambda functions Storing and shuffling data Save modes Managed vs. unmanaged tables Handling partitions Summary Chapter 9: Connecting to and from Databricks Connecting to and from Databricks Getting ODBC and JDBC up and running Creating a token Preparing the cluster Let’s create a test table Setting up ODBC on Windows Setting up ODBC on OS X Connecting tools to Databricks Microsoft Excel on Windows Microsoft Power BI Desktop on Windows Tableau on OS X PyCharm (and more) via Databricks Connect Using RStudio Server Accessing external systems A quick recap of libraries Connecting to external systems Azure SQL Oracle MongoDB Summary Chapter 10: Running in Production General advice Assume the worst Write rerunnable code Document in the code Write clear, simple code Print relevant stuff Jobs Scheduling Running notebooks from notebooks Widgets Running jobs with parameters The command line interface Setting up the CLI Running CLI commands Creating and running jobs Accessing the Databricks File System Picking up notebooks Keeping secrets Secrets with privileges Revisiting cost Users, groups, and security options Users and groups Using SCIM provisioning Access Control Workspace Access Control Cluster, Pool, and Jobs Access Control Table Access Control Personal Access Tokens The rest Summary Chapter 11: Bits and Pieces MLlib Frequent Pattern Growth Creating some data Preparing the data Running the algorithm Parsing the results MLflow Running the code Checking the results Updating tables Create the original table Connect from Databricks Pulling the delta Verifying the formats Update the table A short note about Pandas Koalas, Pandas for Spark Playing around with Koalas The future of Koalas The art of presenting data Preparing data Using Matplotlib Building and showing the dashboard Adding a widget Adding a graph Schedule run REST API and Databricks What you can do What you can’t do Getting ready for APIs Example: Get cluster data Example: Set up and execute a job Example: Get the notebooks All the APIs and what they do Delta streaming Running a stream Checking and stopping the streams Running it faster Using checkpoints Index