Systems Benchmarking: For Scientists And Engineers

Description

This book serves as both a textbook and handbook on the benchmarking of systems and components used as building blocks of modern information and communication technology applications. It provides theoretical and practical foundations as well as an in-depth exploration of modern benchmarks and benchmark development. The book is divided into two parts: foundations and applications. The first part introduces the foundations of benchmarking as a discipline, covering the three fundamental elements of each benchmarking approach: metrics, workloads, and measurement methodology. The second part focuses on different application areas, presenting contributions in specific fields of benchmark development. These contributions address the unique challenges that arise in the conception and development of benchmarks for specific systems or subsystems, and demonstrate how the foundations and concepts in the first part of the book are being used in existing benchmarks. Further, the book presents a number of concrete applications and case studies based on input from leading benchmark developers from consortia such as the Standard Performance Evaluation Corporation (SPEC) and the Transaction Processing Performance Council (TPC). Providing both practical and theoretical foundations, as well as a detailed discussion of modern benchmarks and their development, the book is intended as a handbook for professionals and researchers working in areas related to benchmarking. It offers an up-to-date point of reference for existing work as well as latest results, research challenges, and future research directions. It also can be used as a textbook for graduate and postgraduate students studying any of the many subjects related to benchmarking. While readers are assumed to be familiar with the principles and practices of computer science, as well as software and systems engineering, no specific expertise in any subfield of these disciplines is required. Foreword by David Patterson......Page 5 Foreword by John R. Mashey......Page 7 Theme......Page 9 Goals, Prerequisites, and Target Audience......Page 11 Book Organization and Outline......Page 12 Acknowledgements......Page 16 Contents......Page 18 About the Authors......Page 25 Part I Foundations......Page 26 1.1 Definition of Benchmark......Page 27 1.2 System Quality Attributes......Page 29 1.3 Types of Benchmarks......Page 31 1.4.1 Fixed-Work Benchmarks......Page 33 14.2 Fixed-Time Benchmarks......Page 35 1.5 Benchmark Quality Criteria......Page 36 1.5.1 Relevance......Page 37 1.5.2 Reproducibility......Page 38 1.5.3 Fairness......Page 39 1.5.5 Usability......Page 41 1.6 Application Scenarios for Benchmarks......Page 42 1.7 Concluding Remarks......Page 43 References......Page 44 Chapter 2 Review of Basic Probability and Statistics......Page 46 2.1 Basic Concepts......Page 47 2.2 Distributions of Random Variables......Page 49 2.3 Independent and Dependent Random Variables......Page 51 2.4 Random Samples and Some Important Statistics......Page 53 2.5 Important Continuous Distributions and Central Limit Theorem......Page 55 2.6 The Bernoulli and Binomial Distributions......Page 59 2.7.1 Regression Analysis......Page 61 2.7.2 Kalman Filter......Page 62 2.7.4 Bayesian Inference......Page 64 2.7.5 Mathematical Optimization......Page 65 2.8 Concluding Remarks......Page 66 References......Page 67 3.1 Definition of Metric......Page 68 3.2 Scales of Measurement......Page 69 3.3.1 Speedup and Relative Change......Page 72 3.3.2 Basic Performance Metrics......Page 73 3.4 Quality Attributes of Good Metrics......Page 74 3.4.1 Motivating Example......Page 75 3.4.2 Quality Attributes......Page 78 3.5.1 Types of Averages......Page 80 3.5.2 Composite Metrics......Page 83 3.5.3 Aggregating Results from Multiple Benchmarks......Page 85 3.5.3.1 Aggregating Ratio Metrics......Page 86 3.5.3.2 Aggregating Normalized Values......Page 89 3.6 Concluding Remarks......Page 92 References......Page 93 Chapter 4 Statistical Measurements......Page 94 4.1 Measurement as a Random Experiment......Page 95 4.2.1 Experimental Errors......Page 97 4.2.2 A Model of Random Errors......Page 99 4.2.3 Estimating Means......Page 102 4.2.3.1 CI Based on the Normal Distribution......Page 103 4.2.3.2 CI Based on the......Page 105 4.2.3.3 Indirect Measurement......Page 109 4.2.3.4 Determining the Number of Measurements Needed......Page 110 4.2.4 Estimating Proportions......Page 111 4.2.4.1 Determining the Number of Measurements Needed......Page 112 4.3 Comparing Alternatives......Page 114 4.3.1 Non-corresponding Measurements......Page 115 4.3.2 Before-and-After Comparisons......Page 118 4.3.3 Comparing Proportions......Page 120 4.4 Concluding Remarks......Page 122 References......Page 123 Chapter 5 Experimental Design......Page 124 5.1 One-Factor Analysis of Variance......Page 125 5.2 Method of Contrasts......Page 130 5.3 Two-Factor Full Factorial Designs......Page 131 5.4 General m-Factor Full Factorial Designs......Page 137 5.5 Fractional Factorial Designs: Plackett–Burman......Page 140 5.6 Case Study......Page 144 5.6.1 Problem Statement......Page 146 5.6.2 Plackett–Burman Design......Page 147 5.6.4 Case Study Summary......Page 149 References......Page 151 6.1 Basic Measurement Strategies......Page 153 6.2 Interval Timers......Page 155 6.2.1 Timer Rollover......Page 156 6.2.2 Timer Accuracy......Page 157 6.2.3 Measuring Short Intervals......Page 159 6.3 Performance Profiling......Page 160 6.4 Event Tracing......Page 162 6.4.1 Call Path Tracing......Page 163 6.4.1.3 Loop Iteration Counts......Page 165 6.4.2 Performance Monitoring and Tracing Tools......Page 166 References......Page 167 Chapter 7 Operational Analysis and Basic Queueing Models......Page 170 7.1 Operational Analysis......Page 171 7.1.2 Service Demand Law......Page 173 7.1.3 Forced Flow Law......Page 174 7.1.4 Little's Law......Page 175 7.1.5 Interactive Response Time Law......Page 176 7.1.7 Performance Bounds......Page 178 7.2 Basic Queueing Theory......Page 181 7.2.1 Single Queues......Page 182 7.2.2 Queueing Networks......Page 184 7.2.4 Response Time Equations......Page 188 7.2.5 Solution Techniques for Queueing Networks......Page 189 7.2.6 Case Study......Page 190 7.2.6.1 Workload Characterization......Page 192 7.2.6.2 Building a Performance Model......Page 195 7.2.6.3 Model Analysis and Validation......Page 197 7.2.7 Conclusions from the Analysis......Page 201 7.3 Concluding Remarks......Page 202 References......Page 203 8.1 Workload Facets and Artifacts......Page 206 8.2 Executable Parts of a Workload......Page 208 8.3.1 Workload Traces......Page 210 8.3.2 Workload Descriptions......Page 212 8.3.2.1 ClosedWorkloads......Page 214 8.3.2.2 Open Workloads......Page 216 8.3.3 System-Metric-Based Workload Descriptions......Page 220 References......Page 221 9.1 Historical Perspective on Computer Systems Benchmarking......Page 223 9.2.1 SPEC's Origin......Page 226 9.2.3 Structure and Organization......Page 228 9.2.4 Open Systems Group (OSG)......Page 229 9.2.5 Graphics and Workstation Performance Group (GWPG)......Page 230 9.2.7 Research Group (RG)......Page 231 9.2.8 Benchmark Development Cycle......Page 232 9.3 Transaction Processing Performance Council (TPC)......Page 234 9.3.2 From Engineering to Marketing, From Benchmarking to Benchmarketing......Page 236 9.3.3 A Progression of Benchmarks......Page 237 9.3.4 Evolution of the TPC Model Over Time......Page 239 9.3.5 The Need and Opportunity for Kit-Based Benchmarks......Page 241 9.3.6 The Virtual World of Computing......Page 243 9.4 Concluding Remarks......Page 244 References......Page 245 Part II Applications......Page 248 10.1 SPEC CPU89......Page 249 10.2 SPEC CPU92......Page 251 10.3 SPEC CPU95......Page 254 10.4 SPEC CPU2000......Page 255 10.5 SPEC CPU2006......Page 256 10.6 SPEC CPU2017......Page 257 10.6.1 SPEC CPU2017 Suites......Page 258 10.6.2 SPEC CPU2017 Workloads......Page 259 10.6.3 SPEC CPU2017 Metrics......Page 261 10.6.4 SPEC CPU2017 Energy Metrics......Page 264 10.6.5 SPEC CPU2017 Full Disclosure Report......Page 265 References......Page 267 Chapter 11 Benchmarking the Energy Efficiency of Servers......Page 269 11.1.1 Device Setup......Page 270 11.1.3 Phases and Intervals......Page 272 11.1.4 Basic Energy-Efficiency Metric......Page 274 11.2.1 Metric Calculation......Page 275 11.3 SPEC Server Efficiency Rating Tool (SERT)......Page 276 11.3.1.1 CPU Workload......Page 277 11.3.1.4 IdleWorkload......Page 278 11.3.2.1 Interval Energy Efficiency......Page 279 11.3.2.4 Final Aggregate Energy Efficiency......Page 280 11.4 Concluding Remarks......Page 281 References......Page 282 Chapter 12 Virtualization Benchmarks......Page 284 12.1.1 SPEC VIRT_SC 2010......Page 285 12.1.1.1 Design......Page 286 12.1.1.2 The Need for a New Benchmark......Page 287 12.1.2 SPEC VIRT_SC 2013......Page 288 12.1.3.1 Design......Page 290 12.1.3.2 SPECvirt Datacenter 2020 Template VMs......Page 292 12.2.1 The VMmark 3 Benchmark......Page 293 12.2.1.1 Workloads......Page 294 12.2.1.2 Scoring Methodology......Page 299 References......Page 301 13.1 Historical Perspective on System File Server Benchmarks......Page 302 13.2 SPEC SFS 2014......Page 303 13.3.1 SPC-1......Page 307 13.3.3 Component and Energy Extension......Page 308 13.4 The IOzone Benchmark......Page 309 13.5 Flexible I/O Tester (fio)......Page 310 13.5.1 Running a Simple Job......Page 311 13.5.2 More Complex Workloads......Page 313 13.5.3 Unusual I/O Patterns......Page 315 13.5.4 ioengines......Page 316 References......Page 317 Chapter 14 TeaStore: A Microservice Reference Application......Page 318 14.1 Requirements on TeaStore......Page 319 14.2 Workload......Page 321 14.3 Architecture......Page 322 14.4.2 Image Provider......Page 325 14.4.4 Recommender......Page 326 14.5 Workload Descriptions for TeaStore......Page 327 14.6 Case Study: TeaStore in Action......Page 329 14.6.1 Energy Efficiency on Homogeneous Systems......Page 330 14.6.2 Energy Efficiency on Heterogeneous Systems......Page 332 References......Page 333 Chapter 15 Elasticity of Cloud Platforms......Page 336 15.1 Defining Elasticity......Page 337 15.2 Elasticity Metrics......Page 339 15.2.1 Provisioning Accuracy......Page 341 15.2.3 Instability......Page 342 15.3 Aggregating Elasticity Metrics......Page 344 15.3.1 Autoscaling Deviation......Page 345 15.3.3 Elastic Speedup Score......Page 346 15.4.1 Overview of Bungee......Page 347 15.5 Case Study......Page 350 References......Page 356 Chapter 16 Performance Isolation......Page 358 16.1 Definition of Performance Isolation......Page 359 16.2 Performance Isolation Metrics......Page 360 16.2.1 Metrics Goals and Requirements......Page 361 16.2.2 QoS-Impact-Based Isolation Metrics......Page 362 16.2.3 Workload-Ratio-Based Isolation Metrics......Page 364 16.2.3.1 Metrics Based on Edge Points......Page 366 16.2.3.2 Metrics Based on Integrals......Page 367 16.2.4.1 Settling Time......Page 368 16.2.4.2 Oscillation......Page 370 16.3.4.2 Unpinned Scenario......Page 375 16.3.5 Effectiveness of the Deployment Options......Page 376 16.3 Case Study......Page 371 16.3.3 Experimental Environment......Page 372 16.3.4 Performance Isolation Metrics in Action......Page 374 16.4.2 Edge-Point-Based Metrics......Page 377 16.4.4 Discussion......Page 378 References......Page 379 Chapter 17 Resource Demand Estimation......Page 382 17.1 Estimation Methods......Page 384 17.1.1 Approximation with Response Times......Page 385 17.1.3 Linear Regression......Page 386 17.1.5 Optimization......Page 387 17.1.6 Machine Learning......Page 388 17.1.8 Gibbs Sampling......Page 389 17.2 Input Parameters......Page 390 17.3 Output Metrics......Page 394 17.4 Robustness......Page 396 17.5 Estimation Accuracy and Execution Time......Page 398 17.6 Library for Resource Demand Estimation (LibReDE)......Page 400 References......Page 401 Chapter 18 Software and System Security......Page 406 18.1.1 Attacks and Common Security Mechanisms......Page 407 18.1.2 Application Scenarios......Page 409 18.2 Current State......Page 410 18.2.1 Workloads......Page 411 18.2.1.1 Workload Drivers......Page 412 18.2.1.3 Exploit Database......Page 413 18.2.1.4 Vulnerability and Attack Injection......Page 414 18.2.1.5 Trace Acquisition......Page 418 18.2.1.6 Trace Generation......Page 419 18.2.2 Metrics......Page 421 18.2.3.1 Attack Detection......Page 427 18.2.3.2 Resource Consumption......Page 429 18.2.3.4 Capacity......Page 430 18.3 Concluding Remarks......Page 431 References......Page 432 Index......Page 439