This module is responsible for setup of executors defined in buildtest configuration. The BuildExecutor class initializes the executors and chooses the executor class (LocalExecutor, LSFExecutor, SlurmExecutor, CobaltExecutor) to call depending on executor name.
A BuildExecutor is responsible for initialing executors from buildtest configuration
- class buildtest.executors.setup.BuildExecutor(site_config, account=None, maxpendtime=None, pollinterval=None, timeout=None, max_jobs=None)
A BuildExecutor is responsible for initialing executors from buildtest configuration file which provides a list of executors. This class keeps track of all executors and provides the following methods:
setup: This method will write executor’s
before_script.shthat is sourced in each test upon calling executor.
run: Responsible for invoking executor’s run method based on builder object which is of type BuilderBase.
poll: This is responsible for invoking
pollmethod for corresponding executor from the builder object by checking job state
Initialize executors, meaning that we provide the buildtest configuration that are validated, and can instantiate each executor to be available.
site_config (buildtest.config.SiteConfiguration) – instance of SiteConfiguration class that has the buildtest configuration
account (str, optional) – pass account name to charge batch jobs.
maxpendtime (int, optional) – maximum pend time in second until job is cancelled.
pollinterval (int, optional) – Number of seconds to wait until polling batch jobs
max_jobs (int, optional) – Maximum number of jobs to run at a time.
Return a list of executor names
Given the name of an executor return the executor object which is of subclass of BaseExecutor
Return a list of valid builders that were run
Choose executor is called at the onset of a run and poll stage. Given a builder object we retrieve the executor property
builder.executorof the builder and check if there is an executor object and of type BaseExecutor.
builder (buildtest.buildsystem.base.BuilderBase) – An instance object of BuilderBase type
This method creates directory
var/executors/<executor-name>for every executor defined in buildtest configuration and write scripts before_script.sh if the field
before_scriptis specified in executor section. This method is called after executors are initialized in the class __init__ method.
This method will return list of builders that need to run. We need to check any builders that have a job dependency and make sure the dependent jobs are finished prior to running builder. The return method will be a list of builders to run.
This method is responsible for running the build script for each builder async and gather the results. We setup a pool of worker settings by invoking
multiprocessing.pool.Pooland use multiprocessing.pool.Pool.apply_sync() method for running test async which returns an object of type multiprocessing.pool.AsyncResult which holds the result. Next we wait for results to arrive using multiprocessing.pool.AsyncResult.get() method in a infinite loop until all test results are retrieved. The return type is the same builder object which is added to list of valid builders that is returned at end of method.
Poll all until all jobs are complete. At each poll interval, we poll each builder job state. If job is complete or failed we remove job from pending queue. In each interval we sleep and poll jobs until there is no pending jobs.
Print pending jobs in table format during each poll step
This method is invoked by cleaning up any builders that are when exception is raised