Anatomy of an Incident: Google’s Approach to Incident Management for Production Services

Description

When it comes to system design, failure is inevitable. Scientists and engineers implement solutions based on the available information, without a complete knowledge of the future. You can’t always anticipate the next zero-day event, viral media trend, weather disaster, or shift in technology. But you can be prepared to respond when incidents like these affect your systems. With this report, SRE and DevOps practitioners, IT managers, and engineering leaders will explore methods to help your organization prepare for, respond to, and recover from incidents. With advice from Ayelet Sachto, Adrienne Walcer, and Jessie Yang, you’ll learn how to be prepared to handle failure if and when it happens. Learn the stages of the incident management lifecycle: preparedness, response, recovery, and mitigation Deal proactively with incidents: issues that escalate beyond metrics and alerts Be prepared: practice disaster role playing and incident response exercises Learn the characteristics of the incident-response organizational structure Examine steps to recovery and mitigation after an incident has occurred Conduct postmortems to analyze what went wrong Explore a real-world example from Google: The Mayan Apocalypse Learn how to measure and reduce incidents impact Use postmortems as a tool for prevention and psychological safety Cover Google Copyright Table of Contents Chapter 1. Introduction What Is an Incident? Not Everything Is an Incident Monitoring Alerting The Importance of Actionable Alerts The Incident Management Lifecycle Chapter 2. Practicing Incident Response Readiness (Preparedness) Disaster Role-Playing and Incident Response Exercises Regular Testing Nuanced Testing and Automation Preparing Responders Writing Incident Response Tests Chapter 3. Scaling Incident Management (Response) Component Responders System-of-System (SoS) Responders Incident Response Organizational Structure Common Protocol Trust Respect Transparency Managing Risk The Function of Incident Management and Risk Chapter 4. Mitigation and Recovery Urgent Mitigations Reducing the Impact of Incidents Calculating the Impact of Incidents Reducing the Time to Detect Reducing the Time to Repair Increasing the Time Between Failures Chapter 5. Postmortems and Beyond Psychological Safety Psychological Safety When Implementing Incident Management Practices Psychological Safety When Handling Incidents Additional Tips for Fostering a Psychologically Safe Environment Writing Postmortems Systems Analysis for Organizational Improvement Root Cause Versus Trigger Isolated System Versus Holistic Stack Point-in-Time Versus Trajectory Chapter 6. The Mayan Apocalypse: A Real-World Example Chapter 7. Conclusion and Moving Forward Additional Reading Bibliography Acknowledgments About the Authors