(Enterprise) Slurm gres support (for GPUs)

Post Reply
wjg
Posts: 1
Joined: Wed Sep 09, 2020 3:38 am
Full Name: Will Glover
Organization: NYU Shanghai

(Enterprise) Slurm gres support (for GPUs)

Post by wjg »

Hi folks,
It would be really nice to have Webmo support the --gres argument to Slurm's sbatch so that GPUs could be allocated appropriately. As it stands, Terachem fails out of the box for me with the following error:
************************************ WARNING ********************************
No CUDA-compatible devices have been found in your system
If you recently rebooted the machine, run the job using 'root' account
It will initialize the environment and make TeraChem runnable for all users
*****************************************************************************
Exiting...

DIE called at line number 584 in file terachem/int.cpp

I made a small patch (below) to daemon_pbs.cgi in Webmo v20.0.012e that introduces a --gres=gpu:$ppn sbatch argument when "gpu" is added as a constraint. This constraint could be made default for the Terachem interface, so users don't have to enter it manually each time. It's a bit of a hack, because the "gpu" option cannot be used simultaneously with constraints, but it got Terachem and Slurm working for me. Tthe Webmo developers can probably do something better, but I hope this is useful in the meantime.

Code: Select all

381c381,385
<                         my $constraintOption = $constraints ne '' ? "--constraint='$constraints'" : '';
---
>                         my $constraintOption = '';
>                         if ($constraints ne '')
>                         {
>                               $constraintOption = lc $constraints eq 'gpu' ? "--gres=gpu:$ppn" : "--constraint='$constraints'";
>                       }c381c381,385
<                         my $constraintOption = $constraints ne '' ? "--constraint='$constraints'" : '';
---
>                         my $constraintOption = '';
>                         if ($constraints ne '')
>                         {
>                               $constraintOption = lc $constraints eq 'gpu' ? "--gres=gpu:$ppn" : "--constraint='$constraints'";
>                       }

schmidt
Posts: 82
Joined: Sat May 30, 2020 3:00 pm
Full Name: JR Schmidt
Organization: WebMO, LLC

Re: (Enterprise) Slurm gres support (for GPUs)

Post by schmidt »

This is an excellent point. I will add a general consideration (and hopefully elegant solution!) to the issue for a future bugfix release.

Post Reply