Course details

Practical Parallel Programming

PPP Acad. year 2021/2022 Summer semester 5 credits

Current academic year

The course covers architecture and programming of parallel systems with functional and data parallelism. First, the parallel system theory and program parallelization are discussed. The detailed description of most proliferated supercomputing systems, interconnection network typologies and routing algorithms is followed by the architecture of parallel and distributed storage systems. The course goes on in message passing programming in standardized interface MPI. Consequently, techniques for parallel debugging and profiling are discussed. Last part of the course is devoted to the description of parallel programming patterns and case studies from the are of linear algebra, physical systems described by partial differential equations, N-Body systems and Monte-Carlo methods. 

Guarantor

Jaroš Jiří, doc. Ing., Ph.D. (DCSY)

Course coordinator

Kořenek Jan, doc. Ing., Ph.D. (DCSY)

Language of instruction

Czech

Completion

Credit+Examination (written)

Time span

  • 26 hrs lectures
  • 16 hrs pc labs
  • 10 hrs projects

Assessment points

  • 60 pts final exam (written part)
  • 15 pts mid-term test (written part)
  • 25 pts projects

Department

Department of Computer Systems (UPSY)

Lecturer

Jaroš Jiří, doc. Ing., Ph.D. (DCSY)

Instructor

Vrána Roman, Ing.

Subject specific learning outcomes and competences

Overview of principles of current parallel system design and of interconnection networks, communication techniques and algorithms. Survey of parallelization techniques of fundamental scientific problems, knowledge of parallel programming in MPI. Knowledge of basic parallel programming patterns. Practical experience with the work on supercomputers, ability to identify performance issues and propose their solution.
Knowledge of capabilities and limitations of parallel processing, ability to estimate performance of parallel applications. Language means for process/thread communication and synchronization. Competence in hardware-software platforms for high-performance computing and simulations.

Learning objectives

To get familiar with the architecture of distributed supercomputing systems, their interconnection networks and storage. To orientate oneself in parallel systems on the market, be able to assess communication and computing possibilities of a particular architecture and to predict the performance of parallel applications. Learn how to write portable programs using standardized interfaces and languages, specify parallelism and process communication. To learn how to practically use supercoputer for solving complex engineering problems.

Why is the course taught

This course will take you into the area of high performance computing where a single computer is far from being powerful enough to satisfy application demands. The only solution in such cases is to distribute computation across a supercomputing cluster. This course first examines the architecture of top machines, and then focuses on their software equipment. We will learn the MPI library which is an industry standard in high performance computing. Finally, we will introduce a few typical applications ranging such as physical simulation of heat distribution, fluid dynamics, n-body gravitational and Coulomb systems of galaxies and molecules or Monte-Carlo methods. 

Prerequisite knowledge and skills

Von-Neumann computer architecture, computer memory hierarchy, cache memories and their organization, programming in assembly and in C/C++. Knowledge gained in courses PRL and AVS.

Study literature

Fundamental literature

  • Pacecho, P.: Introduction to Parallel Programming. Morgan Kaufman Publishers, 2011, 392 s., ISBN: 9780123742605 URL: download
  • Ananth Grama, Anshul Gupta, George Karypis, Vipin Kumar: Introduction to Parallel Computing, Addison-Wesley, 2003, 978-0201648652.Slides: download
  • Victor Eijkhout: Parallel Programming in MPI and OpenMP Full book: download web version: https://theartofhpc.com/pcse/
  • William Gropp, Ewing Lusk, Anthony Skjellum: Using MPI - 2nd Edition: Portable Parallel Programming with the Message Passing InterfaceUsing MPI - 2nd Edition: Portable Parallel Programming with the Message Passing Interface, MIT Press, 978-0262571326
  • Hennessy, J.L., Patterson, D.A.: Computer Architecture - A Quantitative Approach. 5. vydání, Morgan Kaufman Publishers, Inc., 2012, 1136 s., ISBN 1-55860-596-7.

Syllabus of lectures

  1. Introduction to parallel processing.
  2. Parallel algorithm design and metodology.
  3. MPI: Point-to-point communications.
  4. MPI: Collective communications.
  5. MPI: Communicators and typologies. 
  6. MPI: Data types.
  7. MPI: One-sided communications.
  8. Distributed file systems and MPI-IO.
  9. Knihovny pro paralelní vstup a výstup.
  10. Performance analysis of parallel and distributed applications.
  11. Case studies (Matrix multiplications, BLAS).
  12. Cace studies (Jacobi methods, FDTD).
  13. Interconnection networks: topology and routing algorithms, switching, flow control.

Syllabus of computer exercises

  1. MPI: Point-to-point communications
  2. MPI: Collective communications
  3. MPI: Communicators
  4. MPI: Data types
  5. MPI: One sided communications
  6. MPI: Parallel input and output with MPI-IO
  7. MPI: Parallel input and output with HDF5
  8. Profiling and tracing of parallel applications

 

Syllabus - others, projects and individual work of students

  • A parallel program in MPI on the supercomputer.

Progress assessment

Assessment of a project, 10 hours in total and a midterm examination.

Controlled instruction

  • Missed labs can be substituted in alternative dates (monday or friday)
  • There will be a place for missed labs in the last week of the semester.

Exam prerequisites

To get 20 out of 40 points for projects and midterm examination.

Course inclusion in study plans

Back to top