Software Telemetry: Reliable logging and monitoring

Description

Software Telemetry shows you how to efficiently collect, store, and analyze system and application log data so you can monitor and improve your systems. In Software Telemetry you will learn how to: • Manage toxic telemetry and confidential records • Master multi-tenant techniques and transformation processes • Update to improve the statistical validity of your metrics and dashboards • Make software telemetry emissions easier to parse • Build easily-auditable logging systems • Prevent and handle accidental data leaks • Maintain processes for legal compliance • Justify increased spend on telemetry software Software Telemetry teaches you best practices for operating and updating telemetry systems. These vital systems trace, log, and monitor infrastructure by observing and analyzing the events generated by the system. This practical guide is filled with techniques you can apply to any size of organization, with troubleshooting techniques for every eventuality, and methods to ensure your compliance with standards like GDPR. About the technology Take advantage of the data generated by your IT infrastructure! Telemetry systems provide feedback on what’s happening inside your data center and applications, so you can efficiently monitor, maintain, and audit them. This practical book guides you through instrumenting your systems, setting up centralized logging, doing distributed tracing, and other invaluable telemetry techniques. About the book Software Telemetry shows you how to efficiently collect, store, and analyze system and application log data so you can monitor and improve your systems. Manage the pillars of observability—logs, metrics, and traces—in an end-to-end telemetry system that integrates with your existing infrastructure. You’ll discover how software telemetry benefits both small startups and legacy enterprises. And at a time when data audits are increasingly common, you’ll appreciate the thorough coverage of legal compliance processes, so there’s no reason to panic when a discovery request arrives. What's inside • Multi-tenant techniques and transformation processes • Toxic telemetry and confidential records • Updates to improve the statistical validity of your metrics and dashboards • Revisions that make software telemetry emissions easier to parse About the reader For software developers and infrastructure engineers supporting and building telemetry systems. About the author Jamie Riedesel is a staff engineer at Dropbox with over twenty years of experience in IT. Software Telemetry brief contents contents preface acknowledgments about this book Who should read this book How this book is organized: A road map About the code liveBook discussion forum Other online resources about the author about the cover illustration Chapter 1: Introduction 1.1 Defining the styles of telemetry 1.1.1 Defining centralized logging 1.1.2 Defining metrics 1.1.3 Defining distributed tracing 1.1.4 Defining SIEM 1.2 How telemetry is consumed by different teams 1.2.1 Telemetry use by Operations, DevOps, and SRE teams 1.2.2 Telemetry use by Security and Compliance teams 1.2.3 Telemetry use by Software Engineering and SRE teams 1.2.4 Telemetry use by Customer Support teams 1.2.5 Telemetry use by business intelligence 1.3 Challenges facing telemetry systems 1.3.1 Chronic underinvestment harms decision-making 1.3.2 Diverse needs resist standardization 1.3.3 Information spills and cleaning them up to avoid legal problems 1.3.4 Court orders break your assumptions 1.4 What you will learn Part 1: Telemetry system architecture Chapter 2: The Emitting stage: Creating and submitting telemetry 2.1 Emitting from production code 2.1.1 Emitting telemetry into a log file 2.1.2 Emitting telemetry into the system log 2.1.3 Emitting telemetry into standard output 2.1.4 Formatting telemetry for emissions 2.2 Emitting from hardware 2.2.1 Explaining SNMP 2.2.2 Ingesting telemetry from a Cisco ASA firewall 2.3 Emitting from as-a-Service systems 2.3.1 Emitting events from SaaS systems 2.3.2 Emitting events from IaaS systems Chapter 3: The Shipping stage: Moving and storing telemetry 3.1 Emitter/shipper functions, telemetry from production code 3.1.1 Shipping directly into storage 3.1.2 Shipping through queues and streams 3.1.3 Shipping to SaaS systems 3.2 Shipping between SaaS systems 3.3 Tipping points in Shipping-stage architecture Chapter 4: The Shipping stage: Unifying diverse telemetry formats 4.1 Shipping locally-emitted telemetry 4.1.1 Shipping telemetry from a log file 4.1.2 Shipping telemetry from the system logger 4.1.3 Shipping telemetry from standard output 4.2 Unifying diverse emitting formats 4.2.1 Encoding telemetry into strings 4.2.2 Picking a shipping format 4.2.3 Converting Syslog to JSON or other object-encoding formats 4.2.4 Designing with cardinality in mind Chapter 5: The Presentation stage: Displaying telemetry 5.1 Displaying telemetry in metrics systems 5.1.1 Making pretty pictures with telemetry 5.1.2 Feeding the graphs with aggregation functions 5.1.3 Using aggregations with pdf_pages 5.2 Displaying telemetry in centralized logging systems 5.2.1 Selecting needed features in a display system for centralized logging 5.2.2 Demonstrating centralized logging display 5.3 Displaying telemetry in security systems 5.4 Displaying telemetry distributed tracing systems 5.5 Displaying telemetry in large organizations Chapter 6: Marking up and enriching telemetry 6.1 Markup in the Emitting stage 6.2 Markup and enrichment in the Shipping stage 6.2.1 Applying context-related telemetry in the Shipping stage 6.2.2 Extracting and enriching telemetry in-flight 6.2.3 Converting field types during the Shipping stage 6.3 Enrichment in the Presentation stage 6.4 How telemetry style affects markup and enrichment 6.4.1 Markup and enrichment with centralized logging 6.4.2 Markup and enrichment with SIEM systems 6.4.3 Markup and enrichment with metrics 6.4.4 Markup and enrichment with distributed tracing systems Chapter 7: Handling multitenancy 7.1 How multitenant architectures come about 7.1.1 Evolving multitenancy in an early-stage startup 7.1.2 Evolving multitenancy in a culture of free sharing 7.1.3 Evolving multitenancy in a culture of strong separation 7.2 Designing multitenant telemetry systems 7.2.1 Multitenancy in the Shipping stage 7.2.2 Multitenancy in the Presentation stage Part 2: Use cases revisited: Applying architecture concepts Chapter 8: Growing cloud-based startup 8.1 Telemetry at the small-company stage 8.1.1 Describing the small company’s telemetry system 8.1.2 Analyzing the small company’s telemetry system 8.2 Telemetry at the medium-size company stage 8.2.1 Describing the medium-size company’s telemetry system 8.2.2 Analyzing the medium-size company’s telemetry system 8.3 Telemetry at the large-company stage 8.3.1 Describing the large company’s telemetry system 8.3.2 Analyzing the large company’s telemetry system 8.4 Telemetry at the enterprise stage 8.5 Looking back at all this growth Chapter 9: Nonsoftware business 9.1 Telemetry use in small organizations 9.2 Telemetry use in medium-size organizations 9.3 Telemetry use in large organizations 9.4 Telemetry use in enterprise organizations Chapter 10: Long-established business IT 10.1 Telemetry use in medium-size organizations 10.1.1 Telemetry use in office IT 10.1.2 Telemetry use in production systems 10.2 Telemetry use in large organizations 10.3 Telemetry use in global organizations 10.3.1 Telemetry use in the Booking and Passenger Manifest department 10.3.2 Telemetry use in the Loyalty Programs department Part 3: Techniques for handling telemetry Chapter 11: Optimizing for regular expressions at scale 11.1 Anchoring expressions for speed 11.2 Building expressions to fail fast 11.3 Digging into the Cisco ASA firewall telemetry 11.4 Refining emissions to speed regular-expression performance 11.5 Additional regular-expression resources Chapter 12: Standardized logging and event formats 12.1 Implementing structured logging in your code 12.2 Implementing standards in your code 12.3 Implementing standards in the Shipping stage Chapter 13: Using more nonfile emitting techniques 13.1 Designing for socket- and datagram-based emitters 13.2 Emitting and shipping for container- and serverless-based code 13.2.1 Emitting and shipping from containerd-based code 13.2.2 Emitting and shipping from serverless-based code 13.3 Encrypting UDP-based telemetry Chapter 14: Managing cardinality in telemetry 14.1 Identifying cardinality problems 14.1.1 Cardinality in time-series databases 14.1.2 Cardinality in logging databases 14.2 Lowering the cost of cardinality 14.2.1 Use logging standards to contain cardinality 14.2.2 Using storage-side methods to tame cardinality 14.2.3 Make cardinality someone else’s problem Chapter 15: Ensuring telemetry integrity 15.1 Getting telemetry out of reach of an attacker 15.1.1 Move telemetry too fast to catch 15.1.2 Use ACLs to enforce write-only telemetry 15.1.3 Durable telemetry when using SaaS providers 15.2 Making telemetry harder to mess with 15.2.1 Using access control requirements to defend against attacks 15.2.2 Ensuring configuration integrity in your telemetry systems 15.2.3 Making changes obvious Chapter 16: Redacting and reprocessing telemetry 16.1 Identifying toxic data and where it comes from 16.2 Redacting toxic information spills 16.3 Reprocessing telemetry to support upgrades 16.4 Isolating toxic data to reduce cleanup costs Chapter 17: Building policies for telemetry retention and aggregation 17.1 Creating a retention policy 17.1.1 Building a policy for centralized logging 17.1.2 Building a policy for metrics 17.1.3 Building a policy for distributed tracing 17.1.4 Building a policy for SIEM systems 17.2 Creating an aggregation policy 17.3 Using sampling to reduce costs and increase retention Chapter 18: Surviving legal processes 18.1 Defining the eDiscovery process 18.2 Dealing with records-retention requests 18.2.1 Examining an ELK-based centralized logging system 18.2.2 Examining a Sumo Logic-based centralized logging system 18.3 Dealing with document-production requests 18.3.1 Telemetry in the collection phase 18.3.2 Telemetry in the review phase 18.3.3 Telemetry in the production phase 18.4 Working with lawyers appendix A: Telemetry storage systems A.1 Analyzing Elasticsearch A.1.1 What Elasticsearch is good at A.1.2 What is challenging for Elasticsearch A.2 Analyzing Apache Cassandra A.2.1 What Cassandra is good at A.2.2 What is challenging for Cassandra A.3 Analyzing Grafana Labs’ Loki A.3.1 What Loki is good at A.3.2 What is challenging for Loki A.4 Analyzing MongoDB A.4.1 What MongoDB is good at A.4.2 What is challenging for MongoDB A.5 Analyzing Prometheus A.5.1 What Prometheus is good at A.5.2 What is challenging for Prometheus A.6 Analyzing InfluxDB A.6.1 What InfluxDB is good at A.6.2 What is challenging for InfluxDB A.7 Analyzing Jaeger A.7.1 What Jaeger is good at A.7.2 What is challenging for Jaeger appendix B: Recommendation checklist reference B.1 Telemetry standards, structure, and setting policies B.2 Presentation-stage recommendations B.3 Cardinality management B.4 Telemetry safety and effects B.5 Legal topics appendix C: Exercise answers index Numerics A B C D E F G H I J K L M N O P Q R S T U V W X Y Z