buildtest.executors.slurm
¶
This module implements the SlurmExecutor class responsible for submitting jobs to Slurm Scheduler. This class is called in class BuildExecutor when initializing the executors.
Module Contents¶
Classes¶
The SlurmExecutor class is responsible for submitting jobs to Slurm Scheduler. |
|
The SlurmJob class models a Slurm Job ID with helper methods to perform operation against an active slurm job. The SlurmJob class |
Attributes¶
- buildtest.executors.slurm.logger¶
- class buildtest.executors.slurm.SlurmExecutor(name, settings, site_configs, max_pend_time=None)[source]¶
Bases:
buildtest.executors.base.BaseExecutor
The SlurmExecutor class is responsible for submitting jobs to Slurm Scheduler. The SlurmExecutor performs the following steps:
load: load slurm configuration from buildtest configuration file
dispatch: dispatch job to scheduler and acquire job ID
poll: wait for Slurm jobs to finish, if job is pending and exceeds max_pend_time then cancel job
gather: Once job is complete, gather job data
Initiate a base executor, meaning we provide a name (also held by the BuildExecutor base that holds it) and the loaded dictionary of config opts to parse.
- Parameters
name (str) – name of executor
setting (dict) – setting for a given executor defined in configuration file
site_configs (buildtest.config.SiteConfiguration) – Instance of SiteConfiguration class
- type = slurm¶
- dispatch(self, builder)[source]¶
This method is responsible for dispatching job to slurm scheduler and extracting job id. If job id is valid we pass the job to
buildtest.executors.slurm.SlurmJob
class and store object inbuilder.job
.- Parameters
builder (buildtest.buildsystem.base.BuilderBase) – An instance object of BuilderBase type
- poll(self, builder)[source]¶
This method is called during poll stage where we invoke
builder.job.poll()
to get updated job state. If job is pending or suspended we stop timer and check if job needs to be cancelled if time exceedsmax_pend_time
value.- Parameters
builder (buildtest.buildsystem.base.BuilderBase) – An instance object of BuilderBase type
- gather(self, builder)[source]¶
Gather Slurm job data after job completion. In this step we call
builder.job.gather()
, and update builder metadata such as returncode, output and error file.- Parameters
builder (buildtest.buildsystem.base.BuilderBase) – An instance object of BuilderBase type
- class buildtest.executors.slurm.SlurmJob(jobID, cluster=None)[source]¶
Bases:
buildtest.executors.job.Job
The SlurmJob class models a Slurm Job ID with helper methods to perform operation against an active slurm job. The SlurmJob class can poll job to get updated job state, gather job data upon completion of test and cancel job if necessary. We can also retrieve job state and determine if job is running, pending, suspended, or cancelled. Jobs are polled via sacct command which can retrieve pending, running and complete jobs.
- is_pending(self)[source]¶
If job is pending return
True
otherwise returnFalse
. Slurm Job state for pending isPENDING
.
- is_running(self)[source]¶
If job is running return
True
otherwise returnFalse
. Slurm will reportRUNNING
for job state.
- is_suspended(self)[source]¶
If job is suspended return
True
otherwise returnFalse
. Slurm will reportSUSPENDED
for job state.
- is_cancelled(self)[source]¶
If job is cancelled return
True
otherwise returnFalse
. Slurm will reportCANCELLED
for job state.
- is_complete(self)[source]¶
If job is complete return
True
otherwise returnFalse
. Slurm will reportCOMPLETED
for job state.
- is_failed(self)[source]¶
If job failed return
True
otherwise returnFalse
. Slurm will reportFAILED
for job state.
- is_out_of_memory(self)[source]¶
If job is out of memory return
True
otherwise returnFalse
. Slurm will reportOUT_OF_MEMORY
for job state.
- is_timeout(self)[source]¶
If job timed out return
True
otherwise returnFalse
. Slurm will reportTIMEOUT
for job state.
- complete(self)[source]¶
This method is used for gathering job result we assume job is complete if it’s in any of the following state:
COMPLETED
,FAILED
,OUT_OF_MEMORY
,TIMEOUT
- cancel(self)[source]¶
Cancel job by running
scancel <jobid>
. If job is specified to a slurm cluster we cancel job usingscancel <jobid> --clusters=<cluster>
. This method is called if job exceeds max_pend_time.
- poll(self)[source]¶
This method will poll job via
sacct
command to get updated job state by running the following command:sacct -j <jobid> -o State -n -X -P
Slurm will report the job state that can be parsed. Shown below is an example job that is
PENDING
state$ sacct -j 46641229 -o State -n -X -P PENDING
- gather(self)[source]¶
Gather job record which is called after job completion. We use sacct to gather job record and return the job record as a dictionary. The command we run is
sacct -j <jobid> -X -n -P -o <field1>,<field2>,...,<fieldN>
. We retrieve the following format fields from job record:“Account”
“AllocNodes”
“AllocTRES”
“ConsumedEnergyRaw”
“CPUTimeRaw”
“Elapsed”
“End”
“ExitCode”
“JobID”
“JobName”
“NCPUS”
“NNodes”
“QOS”
“ReqMem”
“ReqNodes”
“Start”
“State”
“Submit”
“UID”
“User”
“WorkDir”
The output of sacct is parseable using the pipe symbol (|) and stored into a dict
$ sacct -j 42909266 -X -n -P -o Account,AllocNodes,AllocTRES,ConsumedEnergyRaw,CPUTimeRaw,Elapsed,End,ExitCode,JobID,JobName,NCPUS,NNodes,QOS,ReqMem,ReqNodes,Start,State,Submit,UID,User,WorkDir --clusters=cori nstaff|1|billing=272,cpu=272,energy=262,mem=87G,node=1|262|2176|00:00:08|2021-05-27T18:47:49|0:0|42909266|slurm_metadata|272|1|debug_knl|87Gn|1|2021-05-27T18:47:41|COMPLETED|2021-05-27T18:44:07|92503|siddiq90|/global/u1/s/siddiq90/.buildtest/tests/cori.slurm.knl_debug/metadata/slurm_metadata/0/stage
We retrieve ExitCode and WorkDir via sacct command to get returncode. Slurm will write output and error file in WorkDir location. We run the following command below and parse the output. The ExitCode is in form
<exitcode>:<signal>
which is colon separated list. For more details on Slurm Exit code see https://slurm.schedmd.com/job_exit_code.html$ sacct -j 46294283 --clusters=cori -X -n -P -o ExitCode,Workdir 0:0|/global/u1/s/siddiq90/github/buildtest/var/tests/cori.slurm.knl_debug/hostname/hostname_knl/cd39a853/stage