Benchmarking, Measuring, and Optimizing. 14th BenchCouncil International Symposium, Bench 2022 Virtual Event, November 7–9, 2022 Revised Selected Papers

Description

Preface Organization Invited Talks BenchCouncil Achievement Award Lecture: Benchmarking: An Incomparable Science? BenchCouncil Rising Star Award Lecture: Rethinking Benchmarking in AI: Evaluation-as-a-Service and Dynamic Adversarial Data Collection Towards Fair Machine Learning with Imperfect Information Contents Architecture and System A Quantitative Analysis of OpenMP Task Runtime Systems 1 Introduction 2 Related Work 3 OMPTB: The OpenMP Task Benchmark 4 Experimental Setup 5 Experimental Results 5.1 General Experimental Factors 5.2 Homogeneous Workloads 5.3 Homogeneous Case: Correlation Analysis 5.4 Heterogeneous Workloads 6 Conclusions References EAIBench: An Energy Efficiency Benchmark for AI Training 1 Introduction 2 Related Work 2.1 Benchmark 2.2 Metrics 3 Methodology and Implementation 3.1 Metric and Measurement 3.2 Benchmarks and Configuration Space 3.3 Monitor and Analyzer 3.4 System Under Test 4 Experiment and Result 4.1 Experimental Methodology 4.2 Stability of Epochs 4.3 Model Parameters Analysis 4.4 GPU Comparison 4.5 Different Layers Analysis 5 Conclusion References MSDBench: Understanding the Performance Impact of Isolation Domains on Microservice-Based IoT Deployments 1 Introduction 2 Related Work 3 Benchmark Design 4 Empirical Evaluation 4.1 Microbenchmark Results 4.2 End to End Benchmark Results 4.3 Throughput Results 5 Conclusion References Algorithm and Dataset ShoeMaster: A Benchmark for Sketch2Image Translation of Shoes 1 Introduction 2 Related Work 2.1 Fashion Datasets 2.2 Sketch2Image Translation 3 The ShoeMaster Benchmark 3.1 Shoes Category Knowledge Hierarchy 3.2 Dataset Construction 3.3 Evaluation Metric 4 Experiments 4.1 Approaches 4.2 Methodology 4.3 Results and Analysis 5 Conclusion References Open Source Software Supply Chain Recommendation Based on Heterogeneous Information Network 1 Related Work 1.1 GitHub Open Source Data Collection and Mining 1.2 GitHub Project Recommendation 1.3 Recommendation Methods Based on Heterogeneous Information Networks 2 Proposed Framework 2.1 Data Preprocessing 2.2 Oswhin Construction 2.3 Metapath Selection 2.4 Oswhin-Based Embedding 2.5 Personalized Recommendation 3 Experiment 3.1 Dataset 3.2 Evaluation Metrics 3.3 Experimental Results and Analysis 4 Conclusion References BasicTS: An Open Source Fair Multivariate Time Series Prediction Benchmark 1 Introduction 2 Related Works 3 Benchmark Building 3.1 Design Thoughts 3.2 Implementation of BasicTS 4 Evaluation 4.1 Experimental Setup 4.2 Experimental Results 4.3 Review 5 Conclusion 6 Future Works References Benchmarking Object Detection Models with Mummy Nuts Datasets 1 Introduction 2 Background and Motivation 3 Requirements of Detecting Mummy Nuts with CNNs 3.1 Overview of Object Detection Models 3.2 Requirements of Benchmarking Models 4 Benchmarking Methodology 4.1 Proposed Datasets: Original and Augmented 4.2 Metrics 4.3 Proposed Tools 5 Experiments 5.1 Platform Selection 5.2 Results 5.3 Observations and Discussion 6 Related Work 7 Conclusion and Future Work References Network and Memory An Analysis of Long-Tailed Network Latency Distribution and Background Traffic on Dragonfly+ 1 Introduction 1.1 Motivations 1.2 Contributions 2 Related Work 3 Experimental Setup 3.1 Computing 3.2 Network 3.3 Microbenchmarks and Applications 4 Network Latency Distribution Analysis 4.1 Job Placement Locality and Performance Variability 5 Background Traffic Analysis 5.1 Background Traffic Heuristic 5.2 Correlation Analysis 5.3 The Impact of Background Traffic on Long-Tail 5.4 Application Analysis 6 Discussion 7 Conclusion References MCCBench: A C10M Benchmark Oriented to Interactive Network Services 1 Introduction 2 Motivation 2.1 Benchmark Design 2.2 Benchmark Implementation 3 MCCBench Design 3.1 Workload Definition 3.2 Service Framework 3.3 Performance Evaluation Criteria 4 MCCBench Implementation 4.1 Workloads 4.2 Architecture 4.3 Case Study 5 Evaluation 5.1 Experiment Setup 5.2 Results 5.3 Expansion Experiment 6 Conclusion References STAMP-Rust: Language and Performance Comparison to C on Transactional Benchmarks 1 Introduction 2 Background and Related Work 2.1 The STAMP Benchmark Suite 2.2 Software Transactional Memory in Rust 3 STAMP on Safe Rust 3.1 Type-Level Safety 3.2 Composable Transactions 3.3 The Overhead of Safety 3.4 The Complexity of Using Associative Arrays 4 Analysis of Automatically Generated Benchmarks 5 Evaluation 5.1 Methodology 5.2 Performance Comparison 5.3 Qualitative Analysis 6 Conclusion References Author Index