Publication Details
Large-scale Ultrasound Simulations Using the Hybrid OpenMP/MPI Decomposition
Ultrasound simulations, 2D domain decomposition, OpenMP/MPI, Hybrid programming;
Performance evaluation, Supercomputing, k-Wave toolbox.
The simulation of ultrasound wave propagation through biological tissue has
a wide range of practical applications including planning therapeutic ultrasound
treatments of various brain disorders such as brain tumours, essential tremor,
and Parkinson's disease. The major challenge is to ensure the ultrasound focus is
accurately placed at the desired target within the brain because the skull can
significantly distort it. Performing accurate ultrasound simulations, however,
requires the simulation code to be able to exploit several thousands of processor
cores and work with datasets on the order of tens of TB.We have recently
developed an efficient full-wave ultrasound model based on the pseudospectral
method using pure-MPI with 1D slab domain decomposition that allows simulations
to be performed using up to 1024 compute cores. However, the slab decomposition
limits the number of compute cores to be less or equal to the size of the longest
dimension, which is usually below 1024.
This paper presents an improved implementation that exploits 2D hybrid OpenMP/MPI
decomposition. The 3D grid is first decomposed by MPI processes into slabs. The
slabs are further partitioned into pencils assigned to threads on demand. This
allows 8 to 16 times more compute cores to be employed compared to the pure-MPI
code, while also reducing the amount of communication among processes due to the
efficient use of shared memory within compute nodes.
The hybrid code was tested on the Anselm Supercomputer (IT4Innovations, Czech
Republic) with up to 2048 compute cores and the SuperMUC supercomputer (LRZ,
Germany) with up to 8192 compute cores. The simulation domain sizes ranged from
256^3 to 1024^3 grid points. The experimental results show that the hybrid
decomposition can significantly outperform the pure-MPI one for large simulation
domains and high core counts, where the efficiency remains slightly below 50%.
For a domain size of 1024^3, the hybrid code using 8192 cores enables the
simulations to be accelerated by a factor of 4 compared to the pure-MPI code.
Deployment of the hybrid code has the potential to eventually bring the
simulation times within clinically meaningful timespans, and allow detailed
patient specific treatment plans to be created.
@inproceedings{BUT119818,
author="Jiří {Jaroš} and Vojtěch {Nikl} and Bradley {Treeby}",
title="Large-scale Ultrasound Simulations Using the Hybrid OpenMP/MPI Decomposition",
booktitle="Proceedings of the 3rd International Conference on Exascale Applications and Software",
year="2015",
pages="115--119",
publisher="Association for Computing Machinery",
address="Edinburgh",
isbn="978-0-9926615-1-9",
url="http://www.easc2015.ed.ac.uk/proceedings"
}