k-Dispatch's Performance Modules for Advanced Workflow Submission

• https://events.it4i.cz/event/119/contributions/410/

Complex scientific workflows describing challenging real-world problems are composed of a lot of computational tasks which require high performance computing or cloud facilities to be computed in a sensible time. Most of such tasks are also written as distributed parallel programs being able to run across multiple compute nodes. The amount of requested resources per task influences both the overall execution time (makespan) and the computational cost. Optimal resource assignment to particular task is thus crucial. Since the exact execution time cannot be measured for every possible combination of task, input data, and assigned resources, its prediction may be challenging. This poster introduces the idea of performance modules implemented within k-Dispatch that employ space-searching methods together with fitting and machine learning methods. The optimal amount of resources is found using those methods and evaluated using a cluster simulator that estimates the workflow makespan quickly. Using the performance modules, the execution plans for the input workflows are found and can be submitted to the real cluster. k-Dispatch then monitors those remote calculations within a provided service.