buildtest.cli.build

This module contains all the methods related to “buildtest build” which is used for building test scripts from a Buildspec

Module Contents

Classes

Tee
BuildTest	This class is an interface to building tests via buildtest build command.

Functions

resolve_testdirectory(→ str)	This method resolves which test directory to select. For example, one
discover_buildspecs(→ Dict[str, List[str]])	This method discovers all buildspecs based on --buildspecs, --tags, --executor
print_discovered_buildspecs(buildspec_dict)	This method will print the discovered buildspecs in table format
discover_buildspecs_by_tags(buildspec_cache, tagnames)	This method discovers buildspecs by tags, using buildtest build --tags option.
discover_buildspecs_by_name(buildspec_cache, names)	This method will discover buildspecs given a list of test names.
discover_buildspecs_by_executor(buildspec_cache, executors)	This method discovers buildspecs by executor name, using buildtest build --executor
discover_by_buildspecs(→ list)	Given a buildspec file specified by the user with buildtest build --buildspec,
print_filters()	This method will print list of filters fields used by buildtest build --filter. This method is invoked by
update_report(valid_builders, report_file[, verbose])	This method will update BUILD_REPORT after every test run performed

Attributes

logger

buildtest.cli.build.logger

class buildtest.cli.build.Tee(filename)

Bases: object

__enter__()

__exit__(exc_type, exc_value, tb)

write(data)

flush()

buildtest.cli.build.resolve_testdirectory(configuration: buildtest.config.SiteConfiguration, testdir: str = None) → str

This method resolves which test directory to select. For example, one can specify test directory via command line buildtest build --testdir <path> or path in configuration file. The default is $HOME/.buildtest/var/tests

Parameters:

• configuration (buildtest.config.SiteConfiguration) – An instance of SiteConfiguration class which contains content of buildtest configuration file

• testdir (str, optional) – Path to test directory specified via command line buildtest build --testdir

Returns:

Path to test directory

Return type:

str

buildtest.cli.build.discover_buildspecs(buildspecs: List[str] | None = None, exclude_buildspecs: List[str] | None = None, name: List[str] | None = None, executors: List[str] | None = None, tags: List[str] | None = None, verbose: bool | None = False) → Dict[str, List[str]]

This method discovers all buildspecs based on –buildspecs, –tags, –executor and excluding buildspecs (–exclude).

Parameters:

• buildspecs (list, optional) – List of input buildspecs passed by argument buildtest build --buildspec

• exclude_buildspecs (list, optional) – List of excluded buildspecs by argument buildtest build --exclude

• name (list, optional) – List of test names to discover buildspecs that are specified by buildtest build --name

• tags (list, optional) – List of input tags for discovering buildspecs by argument buildtest build --tags

• executors (list, optional) – List of input executors for discovering buildspecs by argument buildtest build --executor

• verbose (bool, optional) – Enable verbose output for buildtest that is specified by buildtest --verbose

Returns:

A dictionary containing a list of included, excluded, detected buildspecs and buildspecs detected based on tags and executors

Return type:

dict

buildtest.cli.build.print_discovered_buildspecs(buildspec_dict)

This method will print the discovered buildspecs in table format

Parameters:

buildspec_dict (dict) – A dictionary containing a list of included and excluded buildspecs and breakdown of buildspecs by tags and executors

buildtest.cli.build.discover_buildspecs_by_tags(buildspec_cache, tagnames)

This method discovers buildspecs by tags, using buildtest build --tags option. This method will read BUILDSPEC_CACHE_FILE and search for tags key in buildspec recipe and match with input tag. The input tags are a list of tagnames to search in buildspec with the tags property in buildspec. The return is a list of buildspec files to process.

Parameters:

• buildspec_cache (dict) – Loaded buildspec cache as a dictionary

• tagnames (list) – List of input tags from command line argument buildtest build --tags <tags>

Returns:

first argument is a list of buildspecs discovered for all tag names. The second argument is dictionary breakdown of buildspecs by each tag name

Return type:

list, dict

buildtest.cli.build.discover_buildspecs_by_name(buildspec_cache, names)

This method will discover buildspecs given a list of test names.

Args:

buildspec_cache (dict): Loaded buildspec cache as a dictionary names (list): List of test names to search for buildspecs from cache

Returns:

first argument is a list of buildspecs discovered for all test names. The second argument is dictionary breakdown of buildspecs by each name

Return type:

list, dict

buildtest.cli.build.discover_buildspecs_by_executor(buildspec_cache, executors)

This method discovers buildspecs by executor name, using buildtest build --executor command. This method will read BUILDSPEC_CACHE_FILE and search for executor property in buildspec and match with input executor name. The return is a list of matching buildspec with executor name to process.

Parameters:

• buildspec_cache (dict) – Loaded buildspec cache as a dictionary

• executors (list) – List of input executor name from command line argument buildtest build --executor <name>

Returns:

first argument is a list of buildspecs discovered for all executors. The second argument is dictionary breakdown of buildspecs by each executor name

Return type:

list, dict

buildtest.cli.build.discover_by_buildspecs(buildspec: str) → list

Given a buildspec file specified by the user with buildtest build --buildspec, discover one or more files and return a list for buildtest to process. This method is called once per argument of --buildspec or --exclude option. If its a directory path we recursively find all buildspecs with with .yml extension. If filepath doesn’t exist or file extension is not .yml we return None and capture error in log.

# file path
buildtest build --buildspec tutorials/hello.sh.yml

# directory path
buildtest build --buildspec tutorials

# invalid file path returns None
buildtest build -b /xyz.yml

# invalid file extension
buildtest build -b README.md

Parameters:

buildspec (str) – Full path to buildspec based on argument buildtest build --buildspec

Returns:

List of resolved buildspecs.

Return type:

list

buildtest.cli.build.print_filters()

This method will print list of filters fields used by buildtest build --filter. This method is invoked by running buildtest build --helpfilter.

class buildtest.cli.build.BuildTest(account=None, buildspecs=None, configuration=None, display=None, dry_run=None, exclude_buildspecs=None, exclude_tags=None, executors=None, executor_type=None, filter_buildspecs=None, helpfilter=None, limit=None, max_jobs=None, maxpendtime=None, modulepurge=None, modules=None, name=None, numnodes=None, numprocs=None, poll_interval=None, profile=None, rebuild=None, remove_stagedir=None, report_file=None, rerun=None, retry=None, save_profile=None, strict=None, tags=None, testdir=None, timeout=None, unload_modules=None, validate=None, verbose=None, write_config_file=None)

This class is an interface to building tests via buildtest build command.

The initializer method is responsible for checking input arguments for type check, if any argument fails type check we raise an error. If all arguments pass we assign the values and proceed with building the test.

Parameters:

• account (str, optional) – Project account to charge jobs. This takes input argument buildtest build --account

• buildspecs (list, optional) – list of buildspecs from command line buildtest build --buildspec

• configuration (buildtest.config.SiteConfiguration, optional) – Loaded configuration content which is an instance of SiteConfiguration

• display (list, optional) – Display content of output or test. This is specified via buildtest build --display

• dry_run (bool, optional) – Show a list of tests that will potentially be run without actually running them via buildtest build --dry-run

• exclude_buildspecs (list, optional) – list of excluded buildspecs from command line buildtest build --exclude

• exclude_tags (list, optional) – list if tags to exclude specified via command line buildtest build --exclude-tags

• executor_type (bool, optional) – Filter test by executor type. This option will filter test after discovery by local or batch executors. This can be specified via buildtest build --exec-type

• executors (list, optional) – list of executors passed from command line buildtest build --executors

• filter_buildspecs (dict, optional) – filters buildspecs and tests based on buildtest build --filter argument which is a key/value dictionary that can filter tests based on tags, type, and maintainers

• helpfilter (bool, optional) – Display available filter fields for buildtest build --filter command. This argument is set to True if one specifies buildtest build --helpfilter

• limit (int, optional) – Limit number of tests that can be run. This option is specified by buildtest build --limit

• max_jobs (int, optional) – Maximum number of jobs to run concurrently. This option is specified by buildtest build --max-jobs

• maxpendtime (int, optional) – Specify maximum pending time in seconds for batch job until job is cancelled

• modulepurge (bool, optional) – Determine whether to run ‘module purge’ before running test. This is specified via buildtest build --modulepurge.

• modules (str, optional) – List of modules to load for every test specified via buildtest build --modules.

• name (list, optional) – list of test names to run specified via command line buildtest build --name

• numnodes (list, optional) – List of comma separated nodes values to run batch jobs specified via buildtest build --nodes

• numprocs (list, optional) – List of comma separated process values to run batch jobs specified via buildtest build --procs

• poll_interval (int, optional) – Specify poll interval in seconds for polling batch jobs.

• profile (str, optional) – Profile to load from buildtest configuration specified by buildtest build --profile

• rebuild (int, optional) – Rebuild tests X times based on buildtest build --rebuild option.

• remove_stagedir (bool, optional) – remove stage directory after job completion

• report_file (str, optional) – Location to report file where test data will be written upon completion. This can be specified via buildtest build --report command

• rerun (bool, optional) – Rerun last successful buildtest build command. This is specified via buildtest build --rerun. All other options will be ignored and buildtest will read buildtest options from file BUILDTEST_RERUN_FILE.

• retry (int, optional) – Number of retry for failed jobs

• save_profile (str, optional) – Save profile to buildtest configuration specified by buildtest build --save-profile

• strict (bool, optional) – Enable strict mode for buildtest. This option is specified by buildtest build --strict

• tags (list, optional) – list if tags to discover tests specified via command line buildtest build --tags

• testdir (str) – Path to test directory where tests are written. This argument can be passed from command line buildtest build --testdir

• timeout (int, optional) – Test timeout in seconds specified by buildtest build --timeout

• unload_modules (str, optional) – List of modules to unload for every test specified via buildtest build --unload-modules.

• validate (bool, optional) – Validate given buildspecs and buildtest will stop after parse stage which can be configured via buildtest build --validate option

• verbose (bool, optional) – Enable verbose output for buildtest that is specified by buildtest --verbose

• write_config_file (str, optional) – Write configuration file to specified location. This is specified by buildtest build --write-config-file

buildexecutor

self.system = buildtest_system

if not isinstance(self.system, BuildTestSystem):

self.system = BuildTestSystem()

load_rerun_file()

This will load content of file BUILDTEST_RERUN_FILE that contains a dictionary of key/value pair that keeps track of last buildtest build command. This is used with buildtest build --rerun. Upon loading file we reinitalize all class variables that store argument for buildtest build options

save_rerun_file()

Record buildtest command options and save them into rerun file which is read when invoking buildtest build --rerun.

save_profile_to_configuration()

This method will save profile to configuration file. This method is called when buildtest build --save-profile is invoked. We will open the configuration file and update the profile section, if profile already exist we will override it, otherwise we will insert into the configuration file.

load_profile()

This method will load profile from configuration file and update class variables used for buildtest build command. This method is called when buildtest build --profile is invoked.

_validate_filters()

Check filter fields provided by buildtest build --filter are valid types and supported. Currently supported filter fields are tags, type, maintainers

Raises:

BuildTestError – if input filter field is not valid we raise exception. For type filter we check for value and make sure the schema type is supported

build()

This method is responsible for discovering buildspecs based on input argument. Then we parse the buildspecs and retrieve builder objects for each test. Each builder object will invoke buildtest.buildsystem.base.BuilderBase.build() which will build the test script, and then we run the test and update report.

parse_buildspecs()

Parse all buildspecs by passing buildspec file to buildtest.buildsystem.parser.BuildspecParser class. If buildspec fails validation we skip the buildspec and print all skipped buildspecs. If buildspec passes validation we get all builders by invoking buildtest.buildsystem.builders.Builder class that is responsible for creating builder objects for each test.

Raises:

SystemExit – If no builders are created after parsing buildspecs

build_phase()

This method will build all tests by invoking class method build for each builder that generates testscript in the test directory. If no builders are present upon building test we raise exception and terminate immediately

Raises:

BuildTestError – If no builders are present in build phase

run_phase()

This method will run all builders with the appropriate executor. The buildtest.executors.setup.BuildExecutor class is responsible for orchestrating builder execution to the appropriate executor class. The BuildExecutor contains a list of executors picked up from buildtest configuration. For tests running locally, we get the test metadata and count PASS/FAIL test state which is printed at end in Test Summary. For tests that need batch submission via scheduler, the first stage of run will dispatch job, and state will be unknown. After dispatching all jobs, we will poll jobs until they are complete. The poll section is skipped if all tests are run locally. In poll section we regenerate table with all valid builders and updated test state and returncode and recalculate total pass/fail tests. Any test that failed to run or be dispatched will be skipped during run stage and they will not be recorded in the test report

Returns:

A list of valid builders after running tests

build_success()

Returns status of build whether it was successful or not. The return value is a boolean where True indicates build was successful and False indicates build failed.

print_test_summary(builders)

Print a summary of total pass and fail test with percentage breakdown.

Parameters:

builders (list) – List of builders that ran to completion

_update_build_history(builders)

Write a build history file that is stored in $BUILDTEST_ROOT/var/.history directory summarizing output of build. The history file is a json file named build.json which contains a copy of the build log for troubleshooting. buildtest will create a sub-directory that is incremented such as 0, 1, 2 in $BUILDTEST_ROOT/var/.history which is used to differentiate builds.

Shown below is content of the top-level directory for the history directory. There is one subdirectory for each build ID starting with 0

bash-3.2$ ls -l $BUILDTEST_ROOT/var/.history
total 0
drwxr-xr-x  4 siddiq90  92503  128 Sep  8 13:50 0
drwxr-xr-x  4 siddiq90  92503  128 Sep  8 13:50 1

For every build ID we have a build.json and log file for each build.

bash-3.2$ ls $BUILDTEST_ROOT/var/.history/{0,1}
/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/0:
build.json             buildtest_y3gh46j_.log

/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/1:
build.json             buildtest_a1rjdy59.log

get_build_history_dir()

Return root of build history directory

print_builders_by_type(builders, builder_type)

This method will print the builders in table format. The builders are printed by their

builder type

Parameters:

• builders (list) – A list of builder objects that were run that need to be printed

• builder_type (str) – The builder type corresponding to the list of builders. This type corresponds to the type field in the test

print_batch_builders(builders)

This method will print the builders that were run in batch mode. The builders will be displayed in table format along with builders by numnodes and numprocs.

print_builders(spack_builder, script_builder, batch_builder)

Print detected builders during build phase

buildtest.cli.build.update_report(valid_builders, report_file, verbose=None)

This method will update BUILD_REPORT after every test run performed by buildtest build. If BUILD_REPORT is not created, we will create file and update json file by extracting contents from builder metadata

Parameters:

• valid_builders (list) – List of valid builders that ran to completion

• report_file (str) – Specify location to report file.