buildtest.executors.setup
¶
This module is responsible for setup of executors defined in buildtest configuration. The BuildExecutor class initializes the executors and chooses the executor class (LocalExecutor, LSFExecutor, SlurmExecutor, CobaltExecutor) to call depending on executor name.
Module Contents¶
Classes¶
A BuildExecutor is responsible for initialing executors from buildtest configuration |
Attributes¶
- class buildtest.executors.setup.BuildExecutor(site_config, max_pend_time=None)¶
A BuildExecutor is responsible for initialing executors from buildtest configuration file which provides a list of executors. This class keeps track of all executors and provides the following methods:
setup: This method will write executor’s
before_script.sh
that is sourced in each test upon calling executor. run: Responsible for invoking executor’s run method based on builder object which is of type BuilderBase. poll: This is responsible for invokingpoll
method for corresponding executor from the builder object by checking job state- __repr__(self)¶
Return repr(self).
- __str__(self)¶
Return str(self).
- _choose_executor(self, builder)¶
Choose executor is called at the onset of a run and poll stage. Given a builder object we retrieve the executor property
builder.executor
of the builder and check if there is an executor object and of type BaseExecutor.- Parameters
builder (BuilderBase (subclass), required.) – the builder with the loaded Buildspec.
- get(self, name)¶
Given the name of an executor return the executor object which is of subclass of BaseExecutor
- is_cobalt(self, executor_type)¶
- is_local(self, executor_type)¶
- is_lsf(self, executor_type)¶
- is_pbs(self, executor_type)¶
- is_slurm(self, executor_type)¶
- list_executors(self)¶
- poll(self, builders)¶
The poll stage is called after the run stage for builders that require job submission through a batch executor. Given a set of builders object which are instance of BuilderBase, we select the executor object and invoke the poll method for the executor.
If job is pending, running, suspended we poll job
If job is complete we gather job results and mark job complete
Otherwise we mark job incomplete and it will be ignored by buildtest in reporting
Poll all jobs for batch executors (LSF, Slurm, Cobalt, PBS). For slurm we poll until job is in
PENDING
orRUNNING
state. If Slurm job is inFAILED
orCOMPLETED
state we assume job is finished and we gather results. If its in any other state we ignore job and return out of method.For LSF jobs we poll job if it’s in
PEND
orRUN
state, if its inDONE
state we gather results, otherwise we assume job is incomplete and return withignore_job
set toTrue
. This informs buildtest to ignore job when showing report.For Cobalt jobs, we poll if its in
starting
,queued
, orrunning
state. For Cobalt jobs we cannot query job after its complete since JobID is no longer present in queuing system. Therefore, for when job is complete which isdone
orexiting
state, we mark job is complete.For PBS jobs we poll job if its in queued or running stage which corresponds to
Q
andR
in job stage. If job is finished (F
) we gather results. If job is inH
stage we automatically cancel job otherwise we ignore job and mark job complete.- Parameters
builder (list , required) – a list of builder objects for polling. Each element is an instance of BuilderBase (subclass)
- Returns
Return a list of builders
- Return type
list
- run(self, builder)¶
This method implements the executor run implementation. Given a builder object we first detect the correct executor object to use and invoke its
run
method. The executor object is a sub-class of BaseExecutor (i.e LocalExecutor, SlurmExecutor, LSFExecutor,…).- Parameters
builder (BuilderBase (subclass), required.) – the builder with the loaded test configuration.
- setup(self)¶
This method creates directory
var/executors/<executor-name>
for every executor defined in buildtest configuration and write scripts before_script.sh if the fieldbefore_script
is specified in executor section. This method is called after executors are initialized in the class __init__ method.
- buildtest.executors.setup.logger¶