Data Wrangling with JavaScript

Description

brief contents......Page 3 contents......Page 4 preface......Page 12 about this book......Page 14 1.1 Why data wrangling?......Page 18 1.2 What?s data wrangling?......Page 19 1.3 Why a book on JavaScript data wrangling?......Page 20 1.4 What will you get out of this book?......Page 21 1.5 Why use JavaScript for data wrangling?......Page 22 1.6 Is JavaScript appropriate for data analysis?......Page 23 1.8 Assembling your toolkit......Page 24 1.9 Establishing your data pipeline......Page 25 1.9.1 Setting the stage......Page 26 1.9.3 Planning......Page 27 1.9.4 Acquisition, storage, and retrieval......Page 30 1.9.5 Exploratory coding......Page 32 1.9.6 Clean and prepare......Page 35 1.9.7 Analysis......Page 36 1.9.8 Visualization......Page 37 1.9.9 Getting to production......Page 39 2 Getting started with Node.js......Page 42 2.1 Starting your toolkit......Page 43 2.3 Getting the code and data......Page 44 2.3.2 Downloading the code......Page 45 2.3.5 Running Node.js code......Page 46 2.3.7 Getting the data......Page 47 2.4 Installing Node.js......Page 48 2.4.1 Checking your Node.js version......Page 49 2.5.1 Creating a Node.js project......Page 50 2.5.2 Creating a command-line application......Page 53 2.5.3 Creating a code library......Page 55 2.5.4 Creating a simple web server......Page 57 2.6 Asynchronous coding......Page 62 2.6.1 Loading a single file......Page 63 2.6.2 Loading multiple files......Page 66 2.6.3 Error handling......Page 68 2.6.4 Asynchronous coding with promises......Page 69 2.6.5 Wrapping asynchronous operations in promises......Page 72 2.6.6 Async coding with ?async? and ?await?......Page 74 3 Acquisition, storage, and retrieval......Page 76 3.1 Building out your toolkit......Page 77 3.3 The core data representation......Page 78 3.3.1 The earthquakes website......Page 79 3.3.2 Data formats covered......Page 81 3.3.3 Power and flexibility......Page 82 3.4.1 Loading data from text files......Page 83 3.4.2 Loading data from a REST API......Page 86 3.4.3 Parsing JSON text data......Page 87 3.4.4 Parsing CSV text data......Page 91 3.4.6 Importing data from MongoDB......Page 95 3.4.7 Importing data from MySQL......Page 99 3.5.2 Exporting data to text files......Page 102 3.5.3 Exporting data to JSON text files......Page 104 3.5.4 Exporting data to CSV text files......Page 106 3.5.5 Exporting data to a database......Page 107 3.5.6 Exporting data to MongoDB......Page 108 3.5.7 Exporting data to MySQL......Page 109 3.7 Expanding the process......Page 112 4 Working with unusual data......Page 116 4.1 Getting the code and data......Page 117 4.2 Importing custom data from text files......Page 118 4.3.1 Identifying the data to scrape......Page 121 4.3.2 Scraping with Cheerio......Page 122 4.4 Working with binary data......Page 124 4.4.1 Unpacking a custom binary file......Page 125 4.4.2 Packing a custom binary file......Page 128 4.4.4 Converting JSON to BSON......Page 130 4.4.5 Deserializing a BSON file......Page 131 5 Exploratory coding......Page 132 5.2 Analyzing car accidents......Page 133 5.4 Iteration and your feedback loop......Page 134 5.5 A first pass at understanding your data......Page 135 5.7 Prototyping with Excel......Page 137 5.8 Exploratory coding with Node.js......Page 139 5.8.1 Using Nodemon......Page 140 5.8.2 Exploring your data......Page 142 5.8.3 Using Data-Forge......Page 145 5.8.4 Computing the trend column......Page 147 5.8.5 Outputting a new CSV file......Page 151 5.9 Exploratory coding in the browser......Page 152 5.10 Putting it all together......Page 158 6 Clean and prepare......Page 160 6.1 Expanding our toolkit......Page 161 6.5 Where does broken data come from?......Page 162 6.6 How does data cleanup fit into the pipeline?......Page 163 6.7 Identifying bad data......Page 164 6.9 Responses to bad data......Page 165 6.10 Techniques for fixing bad data......Page 166 6.11.1 Rewriting bad rows......Page 167 6.11.2 Filtering rows of data......Page 172 6.11.3 Filtering columns of data......Page 175 6.12.1 Aggregating rows of data......Page 176 6.12.2 Combining data from different files using globby......Page 178 6.12.3 Splitting data into separate files......Page 180 6.13 Building a data processing pipeline with Data-Forge......Page 182 7 Dealing with huge data files......Page 185 7.2 Fixing temperature data......Page 186 7.3 Getting the code and data......Page 187 7.4 When conventional data processing breaks down......Page 188 7.5.1 Incremental data processing......Page 189 7.5.2 Incremental core data representation......Page 190 7.5.3 Node.js file streams basics primer......Page 191 7.5.4 Transforming huge CSV files......Page 195 7.5.5 Transforming huge JSON files......Page 201 7.5.6 Mix and match......Page 207 8 Working with a mountain of data......Page 208 8.2 Dealing with a mountain of data......Page 209 8.4.2 Go back to small......Page 210 8.4.4 Prepare your data offline......Page 211 8.5 More Node.js limitations......Page 212 8.7 Working with large databases......Page 213 8.7.1 Database setup......Page 214 8.7.2 Opening a connection to the database......Page 215 8.7.3 Moving large files to your database......Page 216 8.7.4 Incremental processing with a database cursor......Page 218 8.7.5 Incremental processing with data windows......Page 219 8.7.7 Filtering using queries......Page 222 8.7.8 Discarding data with projection......Page 224 8.7.9 Sorting large data sets......Page 225 8.8.3 Processing data in parallel......Page 227 9 Practical data analysis......Page 234 9.1 Expanding your toolkit......Page 235 9.3 Getting the code and data......Page 236 9.4.1 Sum......Page 237 9.4.3 Standard deviation......Page 238 9.5 Group and summarize......Page 239 9.6 The frequency distribution of temperatures......Page 244 9.7.1 Yearly average temperature......Page 248 9.7.2 Rolling average......Page 250 9.7.4 Linear regression......Page 253 9.7.5 Comparing time series......Page 256 9.7.6 Stacking time series operations......Page 259 9.8.2 Types of correlation......Page 260 9.8.3 Determining the strength of the correlation......Page 261 9.8.4 Computing the correlation coefficient......Page 262 10 Browser-based visualization......Page 264 10.2 Getting the code and data......Page 265 10.3 Choosing a chart type......Page 266 10.4 Line chart for New York City temperature......Page 267 10.4.1 The most basic C3 line chart......Page 268 10.4.2 Adding real data......Page 270 10.4.3 Parsing the static CSV file......Page 271 10.4.4 Adding years as the X axis......Page 273 10.4.5 Creating a custom Node.js web server......Page 275 10.4.6 Adding another series to the chart......Page 278 10.4.7 Adding a second Y axis to the chart......Page 280 10.4.8 Rendering a time series chart......Page 281 10.5.1 Bar chart......Page 283 10.5.3 Pie chart......Page 284 10.5.4 Stacked bar chart......Page 286 10.5.5 Scatter plot chart......Page 287 10.6 Improving the look of our charts......Page 288 10.7 Moving forward with your own projects......Page 289 11 Server-side visualization......Page 291 11.1 Expanding your toolkit......Page 292 11.3 The headless browser......Page 293 11.4.2 Nightmare and Electron......Page 295 11.4.3 Our process: capturing visualizations with Nightmare......Page 296 11.4.4 Prepare a visualization to render......Page 297 11.4.5 Starting the web server......Page 299 11.4.6 Procedurally start and stop the web server......Page 300 11.4.7 Rendering the web page to an image......Page 301 11.4.8 Before we move on . . .......Page 302 11.4.9 Capturing the full visualization......Page 304 11.4.10 Feeding the chart with data......Page 306 11.4.11 Multipage reports......Page 309 11.4.12 Debugging code in the headless browser......Page 311 11.4.13 Making it work on a Linux server......Page 313 11.5.1 Web scraping......Page 314 11.5.2 Other uses......Page 315 12 Live data......Page 316 12.1 We need an early warning system......Page 317 12.3 Dealing with live data......Page 318 12.4 Building a system for monitoring air quality......Page 319 12.5 Set up for development......Page 321 12.6.1 HTTP POST for infrequent data submission......Page 322 12.6.2 Sockets for high-frequency data submission......Page 325 12.7 Refactor for configuration......Page 327 12.8 Data capture......Page 329 12.9 An event-based architecture......Page 331 12.10 Code restructure for event handling......Page 333 12.10.1 Triggering SMS alerts......Page 334 12.10.2 Automatically generating a daily report......Page 335 12.11 Live data processing......Page 338 12.12 Live visualization......Page 339 13 Advanced visualization with D3......Page 346 13.1 Advanced visualization......Page 347 13.3 Visualizing space junk......Page 348 13.4 What is D3?......Page 349 13.5 The D3 data pipeline......Page 350 13.6 Basic setup......Page 351 13.7.1 SVG circle......Page 352 13.7.3 SVG text......Page 354 13.7.4 SVG group......Page 355 13.8.1 Element state......Page 356 13.8.2 Selecting elements......Page 357 13.8.3 Manually adding elements to our visualization......Page 359 13.8.4 Scaling to fit......Page 361 13.8.5 Procedural generation the D3 way......Page 363 13.8.6 Loading a data file......Page 366 13.8.7 Color-coding the space junk......Page 368 13.8.8 Adding interactivity......Page 369 13.8.9 Adding a year-by-year launch animation......Page 370 14 Getting to production......Page 375 14.1 Production concerns......Page 376 14.2 Taking our early warning system to production......Page 377 14.3 Deployment......Page 378 14.4 Monitoring......Page 381 14.5.2 Practice defensive programming......Page 383 14.5.3 Data protection......Page 384 14.5.4 Testing and automation......Page 385 14.5.5 Handling unexpected errors......Page 389 14.5.6 Designing for process restart......Page 391 14.6 Security......Page 392 14.6.2 Privacy and confidentiality......Page 393 14.7.1 Measurement before optimization......Page 395 14.7.3 Horizontal scaling......Page 396 appendix A JavaScript cheat sheet......Page 399 appendix B Data-Forge cheat sheet......Page 403 appendix C Getting started with Vagrant......Page 405 index......Page 408