Database Reliability Engineering Designing and Operating Resilient Database Systems

Description

Foreword......Page 5 Preface......Page 6 Why We Wrote This Book......Page 7 Who This Book Is For......Page 8 How This Book Is Organized......Page 9 O’Reilly Safari......Page 12 How to Contact Us......Page 13 1. Introducing Database Reliability Engineering......Page 14 Protect the Data......Page 15 Elimination of Toil......Page 17 Databases Are Not Special Snowflakes......Page 18 Eliminate the Barriers Between Software and Operations......Page 19 Operations Core Overview......Page 20 Hierarchy of Needs......Page 21 Survival and Safety......Page 22 Love and Belonging......Page 23 Esteem......Page 24 Self-actualization......Page 25 Wrapping Up......Page 26 Why Do I Need Service-Level Objectives?......Page 27 Availability......Page 30 Cost or Efficiency......Page 31 Latency Indicators......Page 32 Availability Indicators......Page 36 Throughput Indicators......Page 39 Monitoring and Reporting on SLOs......Page 41 Monitoring Availability......Page 42 Monitoring Throughput......Page 45 Wrapping Up......Page 46 3. Risk Management......Page 48 Unknown Factors and Complexity......Page 49 Human Factors......Page 50 Group Factors......Page 52 What Not to Do......Page 53 A Working Process: Bootstrapping......Page 54 Service Risk Evaluation......Page 55 Architectural Inventory......Page 57 Prioritization......Page 58 Control and Decision Making......Page 62 Ongoing Iterations......Page 66 Wrapping Up......Page 68 4. Operational Visibility......Page 69 Distributed Ephemeral Environments Trending to the Norm......Page 72 Store at High Resolutions for Key Metrics......Page 74 Keep Your Architecture Simple......Page 76 Data In......Page 78 Telemetry/Metrics......Page 80 Logs......Page 82 Bootstrapping Your Monitoring......Page 83 Is the Data Safe?......Page 86 Is the Service Up?......Page 87 Are the Consumers in Pain?......Page 88 Instrumenting the Application......Page 89 Distributed Tracing......Page 90 Instrumenting the Server or Instance......Page 92 Events and Logs......Page 94 Datastore Connection Layer......Page 95 Utilization......Page 96 Errors......Page 97 Throughput and Latency Metrics......Page 99 Commits, Redo, and Journaling......Page 100 Replication State......Page 101 Memory Structures......Page 102 Locking and Concurrency......Page 103 Database Queries......Page 105 Wrapping Up......Page 106 Physical Servers......Page 107 Operating a System and Kernel......Page 108 Cons of Physical Servers......Page 120 Virtualization......Page 121 Storage......Page 122 Use Cases......Page 123 Database as a Service......Page 124 The DBRE and the DBaaS......Page 125 Wrapping Up......Page 126 6. Infrastructure Management......Page 128 Version Control......Page 129 Configuration Definition......Page 130 Building from Configuration......Page 133 Maintaining Configuration......Page 134 Enforcement of Configuration Definitions......Page 135 Infrastructure Definition and Orchestration......Page 136 Monolithic Infrastructure Definitions......Page 137 Separating Vertically......Page 138 Separated Tiers (Horizontal Definitions)......Page 139 Service Catalog......Page 140 Bringing It All Together......Page 142 Development Environments......Page 143 Wrapping Up......Page 144 7. Backup and Recovery......Page 145 Physical versus Logical......Page 146 Full, Incremental, and Differential......Page 147 Recovery Scenarios......Page 148 Planned Recovery Scenarios......Page 149 Unplanned Scenarios......Page 151 Scenario scope......Page 154 Anatomy of a Recovery Strategy......Page 155 Building Block 1: Detection......Page 156 Building Block 2: Tiered Storage......Page 158 Building Block 3: A Varied Toolbox......Page 160 A Recovery Strategy Defined......Page 162 Online, Fast Storage with Full and Incremental Backups......Page 163 Online, Slow Storage with Full and Incremental Backups......Page 164 Offline Storage......Page 165 Object Storage......Page 166 Wrapping Up......Page 167 Education and Collaboration......Page 169 Become a Funnel......Page 170 Domain-Specific Knowledge......Page 171 Integration......Page 175 Prerequisites......Page 176 Testing......Page 179 Test-Friendly Development Practices......Page 180 Post-Commit Testing......Page 181 Full Dataset Testing......Page 182 Operational Tests......Page 184 Deployment......Page 185 Impact Analysis......Page 186 Migration Patterns......Page 187 Manual or Automated......Page 191 Wrapping Up......Page 192 The Purpose of Security......Page 193 Protecting from Accidental Damage......Page 194 Database Security as a Function......Page 195 Education and Collaboration......Page 196 Self-Service......Page 197 Integration and Testing......Page 198 Operational Visibility......Page 199 Vulnerabilities and Exploits......Page 201 STRIDE......Page 202 DREAD......Page 203 Basic Precautions......Page 204 Denial of Service......Page 205 SQL Injection......Page 209 Network and Authentication Protocols......Page 211 Encryption of Data......Page 212 Private Individual Data......Page 213 Confidential/Sensitive Business Data......Page 214 Data in Transit......Page 215 Data in the Database......Page 219 Data in the Filesystem......Page 222 Wrapping Up......Page 225 Data Structure Storage......Page 227 Database Row Storage......Page 228 Sorted-String Tables and Log-Structured Merge Trees......Page 232 Indexing......Page 235 Data Replication......Page 236 Single-Leader......Page 237 Multi-Leader Replication......Page 254 Wrapping Up......Page 262 Conceptual Attributes of a Datastore......Page 263 The Data Model......Page 264 Transactions......Page 268 BASE......Page 274 Internal Attributes of a Datastore......Page 275 The Ubiquitous CAP Theorem Section......Page 276 Consistency Latency Trade-offs......Page 279 Availability......Page 281 Wrapping Up......Page 282 Architectural Components......Page 284 Frontend Datastores......Page 285 Database Proxies......Page 286 Event and Message Systems......Page 289 Caches and Memory Stores......Page 291 Data Architectures......Page 294 Lambda and Kappa......Page 295 Event Sourcing......Page 298 CQRS......Page 299 Wrapping Up......Page 300 A Culture of Database Reliability......Page 301 Breaking-Down Barriers......Page 302 Data Integrity and Recoverability......Page 309 Wrapping Up......Page 310 Index......Page 311