Optimization of Execution Parameters of Moldable Workflows under Incomplete Performance Data

Complex scientific workflows describing challenging realworld problems are composed of many computational tasks
requiring high performance computing or cloud facilities to
be computed in a sensible time. Most of these tasks are usually written as moldable parallel programs being able to run
across various numbers of compute nodes. The amount of
resources assigned to particular tasks may strongly affect the
overall execution and queuing time of the whole workflow
(makespan) as well as the total computational cost.
For this purpose, this paper employs a genetic algorithm
that searches for a good resource distribution over the particular tasks, and a cluster simulator that evaluates makespan and
cost of the developed workflow execution schedule. Since the
exact execution time cannot be measured for every possible
combination of task, input data size, and assigned resources,
several interpolation techniques are used to predict the task
duration for a given amount of compute resources. The best
execution schedules are eventually submitted to a real cluster
with a PBS scheduler to validate the whole technique.
The experimental results confirm the proposed cluster
simulator corresponds to a real PBS job scheduler with a
sufficient fidelity. The investigation of the interpolation techniques showed that incomplete performance data can be
successfully completed by linear and quadratic interpolations
making a maximum mean error below 10%. Finally, the paper
shows it is possible to implement a user defined parameter
which instructs the genetic algorithm to prefer either the
makespan or cost, or find a suitable trade-off.