Backfill

Backfill is the mechanism whereby a lower-priority job can be started before a higher-priority job in such a way that the start of higher-priority job is not delayed.

In the diagram below, assume Job A has just started. Job B, which requires the node currently occupied by Job A plus some additional nodes, will have to wait until Job A has completed. If Job C has a shorter maximum run-time than Job A, Job C will be able to start before Job B, even if Job B has the higher priority and without delaying the start of Job B. In this way, "gaps" in the scheduling plan can be filled and the throughput of the cluster increased.

backfill.png
Backfilling

Note that a job can only then benefit from backfilling if the run-time needed for that job is significantly less than the default maximum and this is specified. As an example for SLURM the following line

#SBATCH --time=2-12:00:00

would set up a limit of 2 days and 12 hours. The shorter the time given, the more the job can benefit from backfilling. However, if the job reaches the time limit, it will be killed by the queueing system. See man sbatch for more details.