Programming on Parallel Machines: GPU, Multicore, Clusters and More

Description

"Why is this book different from all other parallel programming books?" Suitable for either students or professionals. Practical viewpoint: There is very little theoretical analysis of parallel algorithms, such as O() analysis, maximum theoretical speedup, acyclic graphs and so on. Extensive coverage of "wizardry" aspects, i.e. material known to experienced practitioners but generally not in books, such as coverage of loop iteration scheduling, memory effects of storing large arrays and so on. Appendices cover systems background, crucial in applied work but always just "assumed" to be knowledge possessed by the readers. Considerable attention is paid to techniques for debugging. Uses the main parallel platforms---OpenMP, CUDA and MPI---rather than languages that at this stage are largely experimental, such as the elegant-but-not-yet-mainstream Cilk. Starts with real parallel code right away in Chapter 1, with examples from pthreads, OpenMP and MPI. Constantly evolving: Like all my open source textbooks, this one is constantly evolving. I continue to add new topics, new examples, more timing analyses, and so on, and of course fix bugs and improve the exposition. Prerequisites: The student must be reasonably adept in programming, and have math background through linear algebra. (An appendix to the book reviews the parts of the latter needed for this book.)