buildtest.executors.setup

This module is responsible for setup of executors defined in buildtest configuration. The BuildExecutor class initializes the executors and chooses the executor class (LocalExecutor, LSFExecutor, SlurmExecutor, CobaltExecutor) to call depending on executor name.

Module Contents

Classes

BuildExecutor

A BuildExecutor is responsible for initialing executors from buildtest configuration

Attributes

logger

class buildtest.executors.setup.BuildExecutor(site_config, max_pend_time=None)

A BuildExecutor is responsible for initialing executors from buildtest configuration file which provides a list of executors. This class keeps track of all executors and provides the following methods:

setup: This method will write executor’s before_script.sh that is sourced in each test upon calling executor. run: Responsible for invoking executor’s run method based on builder object which is of type BuilderBase. poll: This is responsible for invoking poll method for corresponding executor from the builder object by checking job state

__repr__(self)

Return repr(self).

__str__(self)

Return str(self).

_choose_executor(self, builder)

Choose executor is called at the onset of a run and poll stage. Given a builder object we retrieve the executor property builder.executor of the builder and check if there is an executor object and of type BaseExecutor.

Parameters

builder (BuilderBase (subclass), required.) – the builder with the loaded Buildspec.

get(self, name)

Given the name of an executor return the executor object which is of subclass of BaseExecutor

is_cobalt(self, executor_type)

is_local(self, executor_type)

is_lsf(self, executor_type)

is_pbs(self, executor_type)

is_slurm(self, executor_type)

list_executors(self)

poll(self, builders)

The poll stage is called after the run stage for builders that require job submission through a batch executor. Given a set of builders object which are instance of BuilderBase, we select the executor object and invoke the poll method for the executor.

1. If job is pending, running, suspended we poll job

2. If job is complete we gather job results and mark job complete

3. Otherwise we mark job incomplete and it will be ignored by buildtest in reporting

Poll all jobs for batch executors (LSF, Slurm, Cobalt, PBS). For slurm we poll until job is in PENDING or RUNNING state. If Slurm job is in FAILED or COMPLETED state we assume job is finished and we gather results. If its in any other state we ignore job and return out of method.

For LSF jobs we poll job if it’s in PEND or RUN state, if its in DONE state we gather results, otherwise we assume job is incomplete and return with ignore_job set to True. This informs buildtest to ignore job when showing report.

For Cobalt jobs, we poll if its in starting, queued, or running state. For Cobalt jobs we cannot query job after its complete since JobID is no longer present in queuing system. Therefore, for when job is complete which is done or exiting state, we mark job is complete.

For PBS jobs we poll job if its in queued or running stage which corresponds to Q and R in job stage. If job is finished (F) we gather results. If job is in H stage we automatically cancel job otherwise we ignore job and mark job complete.

Parameters

builder (list , required) – a list of builder objects for polling. Each element is an instance of BuilderBase (subclass)

Returns

Return a list of builders

Return type

list

run(self, builder)

This method implements the executor run implementation. Given a builder object we first detect the correct executor object to use and invoke its run method. The executor object is a sub-class of BaseExecutor (i.e LocalExecutor, SlurmExecutor, LSFExecutor,…).

Parameters

builder (BuilderBase (subclass), required.) – the builder with the loaded test configuration.

setup(self)

This method creates directory var/executors/<executor-name> for every executor defined in buildtest configuration and write scripts before_script.sh if the field before_script is specified in executor section. This method is called after executors are initialized in the class __init__ method.

buildtest.executors.setup.logger