Publication Details
k-Dispatch's Performance Modules for Advanced Workflow Submission
Complex scientific workflows describing challenging real-world problems are
composed of a lot of computational tasks which require high performance computing
or cloud facilities to be computed in a sensible time. Most of such tasks are
also written as distributed parallel programs being able to run across multiple
compute nodes. The amount of requested resources per task influences both the
overall execution time (makespan) and the computational cost. Optimal resource
assignment to particular task is thus crucial. Since the exact execution time
cannot be measured for every possible combination of task, input data, and
assigned resources, its prediction may be challenging. This poster introduces the
idea of performance modules implemented within k-Dispatch that employ
space-searching methods together with fitting and machine learning methods. The
optimal amount of resources is found using those methods and evaluated using
a cluster simulator that estimates the workflow makespan quickly. Using the
performance modules, the execution plans for the input workflows are found and
can be submitted to the real cluster. k-Dispatch then monitors those remote
calculations within a provided service.
@misc{BUT179388,
author="Marta {Jaroš} and Jiří {Jaroš}",
title="k-Dispatch's Performance Modules for Advanced Workflow Submission",
year="2022",
pages="1",
address="Soláň",
url="https://www.fit.vut.cz/research/publication/12810/",
note="presentation, poster"
}