buildtest.executors.setup

This module is responsible for setup of executors defined in buildtest configuration. The BuildExecutor class initializes the executors and chooses the executor class (LocalExecutor, LSFExecutor, SlurmExecutor, CobaltExecutor) to call depending on executor name.

Module Contents

Classes

BuildExecutor

A BuildExecutor is responsible for initialing executors from buildtest configuration

Attributes

logger

buildtest.executors.setup.logger

class buildtest.executors.setup.BuildExecutor(site_config, account=None, maxpendtime=None, pollinterval=None, timeout=None, max_jobs=None)

A BuildExecutor is responsible for initialing executors from buildtest configuration file which provides a list of executors. This class keeps track of all executors and provides the following methods:

• setup: This method will write executor’s before_script.sh that is sourced in each test upon calling executor.

• run: Responsible for invoking executor’s run method based on builder object which is of type BuilderBase.

• poll: This is responsible for invoking poll method for corresponding executor from the builder object by checking job state

Initialize executors, meaning that we provide the buildtest configuration that are validated, and can instantiate each executor to be available.

Parameters:

• site_config (buildtest.config.SiteConfiguration) – instance of SiteConfiguration class that has the buildtest configuration

• account (str, optional) – pass account name to charge batch jobs.

• maxpendtime (int, optional) – maximum pend time in second until job is cancelled.

• pollinterval (int, optional) – Number of seconds to wait until polling batch jobs

• max_jobs (int, optional) – Maximum number of jobs to run at a time.

__str__()

Return str(self).

__repr__()

Return repr(self).

names()

Return a list of executor names

get(name)

Given the name of an executor return the executor object which is of subclass of BaseExecutor

get_validbuilders()

Return a list of valid builders that were run

_choose_executor(builder)

Choose executor is called at the onset of a run and poll stage. Given a builder object we retrieve the executor property builder.executor of the builder and check if there is an executor object and of type BaseExecutor.

Parameters:

builder (buildtest.buildsystem.base.BuilderBase) – An instance object of BuilderBase type

setup()

This method creates directory var/executors/<executor-name> for every executor defined in buildtest configuration and write scripts before_script.sh if the field before_script is specified in executor section. This method is called after executors are initialized in the class __init__ method.

select_builders_to_run(builders)

This method will return list of builders that need to run. We need to check any builders that have a job dependency and make sure the dependent jobs are finished prior to running builder. The return method will be a list of builders to run.

run(builders)

This method is responsible for running the build script for each builder async and gather the results. We setup a pool of worker settings by invoking multiprocessing.pool.Pool and use multiprocessing.pool.Pool.apply_sync() method for running test async which returns an object of type multiprocessing.pool.AsyncResult which holds the result. Next we wait for results to arrive using multiprocessing.pool.AsyncResult.get() method in a infinite loop until all test results are retrieved. The return type is the same builder object which is added to list of valid builders that is returned at end of method.

poll(pending_jobs)

Poll all until all jobs are complete. At each poll interval, we poll each builder job state. If job is complete or failed we remove job from pending queue. In each interval we sleep and poll jobs until there is no pending jobs.

_print_job_details(active_jobs)

Print pending jobs in table format during each poll step

Parameters:

• active_jobs (list) – List of builders whose jobs are pending, suspended or running

• completed_jobs (list) – List of builders whose jobs are completed

_cleanup_when_exception()

This method is invoked by cleaning up any builders that are when exception is raised