Principles of High-Performance Processor Design: For High Performance Computing, Deep Neural Networks and Data Science

Description

This book describes how we can design and make efficient processors for high-performance computing, AI, and data science. Although there are many textbooks on the design of processors we do not have a widely accepted definition of the efficiency of a general-purpose computer architecture. Without a definition of the efficiency, it is difficult to make scientific approach to the processor design. In this book, a clear definition of efficiency is given and thus a scientific approach for processor design is made possible.  In chapter 2, the history of the development of high-performance processor is overviewed, to discuss what quantity we can use to measure the efficiency of these processors. The proposed quantity is  the ratio between the minimum possible energy consumption and the actual energy consumption for a given application using a given semiconductor technology. In chapter 3, whether or not this quantity can be used in practice is discussed, for many real-world applications.  In chapter 4, general-purpose processors in the past and present are discussed from this viewpoint. In chapter 5, how we can actually design processors with near-optimal efficiencies is described, and in chapter 6 how we can program such processors.  This book gives a new way to look at the field of the design of high-performance processors. Preface Contents Acronyms 1 Introduction References 2 Traditional Approaches and Their Limitations 2.1 History 2.1.1 CDC 6600 and 7600 2.1.2 Cray-1 2.1.3 The Evolution of Vector Processors 2.1.4 Lessons from the History of Vector-Parallel Architecture 2.1.5 The Evolution of Single-Chip Processors 2.1.6 The Impact of Hierarchical Cache 2.1.7 Alternatives to Cache Memories 2.2 The Need for Quantitative Approach 2.3 What Is Measured and What Is Not References 3 The Lower Limit of Energy Consumption 3.1 Range of Applications We Consider 3.1.1 Structured Mesh 3.1.2 Unstructured Mesh 3.1.3 Particles 3.1.4 Random Graphs 3.1.5 Dense Matrices 3.1.6 Miscellanies 3.1.7 Distribution of Application Types 3.2 Definition of Efficiency 3.3 Structured Mesh 3.3.1 Choice of the Numerical Methods 3.3.2 The Design of An Ideal Processor Architecture for Structured Mesh Calculations 3.4 Unstructured Mesh 3.5 Particles 3.5.1 The Overview of Particle-Based Methods 3.5.2 Short-Range Interactions 3.5.3 Long-Range Interactions 3.6 Random Graphs 3.7 Dense Matrices 3.8 Summary References 4 Analysis of Past and Present Processors 4.1 CDC 6600 4.2 Cray-1 and Its Successors 4.3 x86 Processors 4.3.1 i860 4.3.2 From Pentium to Skylake 4.4 NEC SX-Aurora and Fujitsu A64fx 4.5 SIMD Supercomputers—Illiac IV and TMC CM-2 4.5.1 Illiac IV 4.5.2 CM-2 4.5.3 Problems with Large-Scale SIMD Processors 4.5.3.1 Synchronous System Clock 4.5.3.2 Memory Bandwidth 4.6 GPGPUs 4.7 PEZY Processors and Sunway SW26010 4.7.1 PEZY Processors 4.7.2 Sunway SW26010 4.8 Conclusion References 5 ``Near-Optimal'' Designs 5.1 The Special-Purpose Designs: GRAPE Processors 5.2 The Baseline Design: GRAPE-DR 5.2.1 Design Concept and Architecture 5.2.2 The Efficiency 5.2.3 Software 5.3 Functions Necessary to Widen Application Area 5.3.1 Particles 5.3.2 Dense Matrices 5.3.3 Other Applications 5.3.3.1 Structured Mesh 5.3.3.2 Unstructured Mesh 5.3.3.3 Summary 5.4 An Extreme for Deep Learning: MN-Core/GRAPE-PFN 5.5 A ``General-Purpose'' Design 5.5.1 On-Chip Network for Sorting 5.5.2 Off-Chip DRAM Access 5.5.3 Chip-to-Chip Communications Network for Deep Learning and Unstructured-Mesh Calculations 5.5.4 Support for FP64, FP32 and FP16 or Other Mixed-Precision Operations 5.6 The Reference SIMD Processor 5.6.1 PE 5.6.2 BM 5.6.3 TBM 5.6.4 DRAM Interface 5.6.5 Host Data Interface 5.6.6 Instruction Fetch/Issue References 6 Software 6.1 Traditional Approaches 6.2 How Do We Want to Describe Applications? 6.2.1 Structured Mesh 6.2.2 Unstructured Mesh 6.2.3 Particles 6.2.4 Dense Matrices 6.3 Summary References 7 Present, Past and Future 7.1 Principles of High-Performance Processor Design 7.2 The Current Practice 7.3 Our Past 7.4 GPGPUs and Deep Learning Processors 7.5 The Future References Index