Applied ELK Stack: Data Insights and Business Metrics With Collective Capability of ElasticSearch, Logstash and Kibana

Description

Use the ELK (Elasticsearch, Logstash, and Kibana) stack to build systems that provide actionable insights and business metrics from data sources, including creating amazing visualizations and dashboards. Learn how to set up the ELK stack, build a data pipeline, and create customized plugins. Applied ELK Stack will teach you to configure the software, install tools, and build a data pipeline. You will learn the key features of Logstash and its role in the ELK stack, including creating Logstash plugins, which will enable you to use your own customized plugins. The importance of Elasticsearch and Kibana in the ELK stack is covered, along with various types of advanced data analysis, including charts, tables, and maps. The simple and powerful nature of ELK stack has contributed to its quick adoption. With this book you will learn: The need for log analytics, and current challenges How to perform real-time data analytics on streaming data, and turn them into actionable insights How to create indexing and delete data The different components of ELK (Elasticsearch, Logstash, and Kibana) stack Shipping, Filtering, and Parsing Events with Logstash How to build amazing visualizations and dashboards using Data Discovery, Visualization, and Dashboard with Kibana CHAPTER 1: Introduction to ELK Stack Log Analysis in Today's World The ELK Stack Elasticsearch Logstash Kibana ELK Data Pipeline ELK Stack Installation Installing Elasticsearch Running Elasticsearch Elasticsearch Configuration and Settings Environment Variables System Configuration Installing Elasticsearch Plugins Installing Logstash Running Logstash Logstash with Elasticsearch Output Configuring Logstash Installing Logstash Forwarder Extending Logstash Functionality Installing Kibana Kibana Configuration Kibana Interface Summary Sample Dataset Data Format Logstash Configuration Comments Configuring for Events Field References Conditionals Metadata Filtering Events Shipping Events Reloading Configuration File Multiline Event Configuration Analyzing Events Data Visualization Building a Line Chart Building a Bar Chart Building a Metric Building a Data Table Summary CHAPTER 3: Extending Logstash Plugin Management Download and Installation Plugin Installation Updating Plugin Uninstallation Plugin Structure Prerequisite Basic Structure Configuration Setup Execution Teardown Build Custom Plugin Plugin Packaging Summary Ubiquity of Data Elasticsearch Cluster Node Anatomy of a Document Metadata Information Index Type Id Shard Primary and Replica Shard Elasticsearch API Cluster Health and Configuration Index Management Specify Id Custom Id Auto-Generated Id Document Management Document Retrieval Partial Document Retrieval Document Existence Multiple Document Retrieval Document Updates Updating Documents Partially Partial Updates with Scripts Conflicting Updates Document Creation Document Deletion Bulk Operations Bulk Request Size Conflict Management Summary Search Your Way Simple Searches Search Without Parameter hits took shards timeout Multi-Index, Multi-Type Pagination Search Lite The _all Field Query Mashup Query DSL Query Clause Construction Working with Multiple Clauses Filter and Query Performance Concerns Key Filters and Queries term Filter terms Filter range Filter exists and missing Filters bool Filter match_all Query match Query multi_match Query bool Query Filter-Query Combination Filtering a Query Only Filter Filter via Query Query Validation Error Diagnostics Summary CHAPTER 6: Mapping and Analysis Data Mapping and Analysis Exact Values and Full Text Inverted Index Data Analysis Prepackaged Analyzers When to Use Analyzers You can Test Analyzers Assign Analyzer Data Mapping Simple Field Types Observe the Mapping Mapping Customization index analyzer Mapping Revision Mapping Test Complex Field Types Multi-value Fields Empty Fields Multi-level Objects Indexing Inner Objects Inner Object Arrays Summary Aggregation Basics Buckets Metrics The Two Together Fun with Aggregation Metrics to the Rescue Buckets within Buckets Multiple Metrics Data Visualization with Bar Charts Time Series Aggregations Multi-Tier Correlation Aggregation Scoping Global Bucket Aggregations with Query Filters Query with Filter Filter Bucket Post Filter Multivalue Bucket Sorting Intrinsic Sorts Metric Based Sorting Summary Introducing Kibana Kibana Features Kibana User Interface Discover Page Time Filter Query and Search Data Free Text Search Field Searches Range Searches New Search Saving Search Field Search Using Field List Summary CHAPTER 9: Kibana - Data Visualization Visualize Page Metrics and Buckets Aggregations Buckets Metrics Advanced Options Choosing Search Data Source Visualization Canvas Toolbar Aggregation Designer Preview Canvas Building Visualization Visualization Types Area Chart Data Table Line Chart Markdown Widget Metric Pie Chart Tile Map Heat Map Desaturate Map Tiles Vertical Bar Chart Summary Introduction to Dashboard Page Working with the Toolbar New Dashboard Option Adding Visualizations Search Bar Save Dashboard Load Saved Dashboard Sharing the Saved Dashboard Working with Dashboard Canvas Move Visualization Resize Visualization Edit Visualization Remove Visualization Embed Dashboard in Web Page Debug Panel Table Request Response Statistics Summary CHAPTER 11: Designing for Scale Elasticsearch Cluster for Scale Adding Nodes to Cluster Discovering Cluster Nodes Multicast Discovery Unicast Discovery Master Node Election Fault Detection Removal of Nodes from Cluster Decommissioning of Nodes Upgrading Elasticsearch Nodes Rolling Restart Quick Restart Cluster Information Scaling Options Over-Sharding Data Slicing Maximizing Throughput Aliases Working with Alias Benefits of Aliases Living with Aliases Creating Aliases Camouflaging Documents with Filters Routing Significance of Routing Routing Strategies Determining Shards Routing Configuration Routing in Combination with Aliases Summary CHAPTER 12: ELK Stack in Production Deployment Considerations Memory Disks Network Java Virtual Machine Data Management Request Grouping Bulk Indexing, Updating and Deleting Multisearch and Multiget APIs Elasticsearch Tuning Lucene Segment Optimization Thresholds for Refresh and Flush Merge Policies Store Throttling Cache Management Filter Caches Shard Query Cache JVM Heap and OS Cache Warmers for Caches Configuration Management Better than Defaults Index Templates Monitoring and Troubleshooting Health of the Cluster Detect Index Problem Examining Individual Nodes indices Section OS and Process Sections JVM Section Threadpool Section F5 and Network Sections Circuit Breaker Cluster Statistics Index Statistics Pending Tasks Logging Slowlog Rolling Restarts Backup and Restore Cluster Backup Creating the Repository Snapshot of All Open Indices Snapshot of Particular Index Listing Snapshot of Information Snapshot Deletion Monitoring Snapshot Progress Cancelling Snapshot Restore from Snapshot Monitor Restore Operations Cancelling Restore Summary