Buildtest Command Reference

buildtest is a HPC testing framework for building and running tests.

usage: buildtest [options] [COMMANDS]

Named Arguments

-V, --version

show program’s version number and exit

-c, --configfile

Specify Path to Configuration File

-d, --debug

Stream log messages to stdout

Default: False

-l, --loglevel

Possible choices: DEBUG, INFO, WARNING, ERROR, CRITICAL

Filter log messages based on logging level

Default: 'DEBUG'

--editor

Possible choices: vi, vim, emacs, nano

Select your preferred editor when opening files.

--view-log

Show content of last log

Default: False

--logpath

Print full path to last log file

Default: False

--print-log

Print content of last log without pagination

Default: False

--color

Print output of table with the selected color.

--no-color

Disable colored output

Default: False

--helpcolor

Print available color options in a table format.

Default: False

-r, --report

Specify path to test report file

--listopts

List all options for buildtest

Default: False

--verbose

Enable verbose output

Default: False

COMMANDS

: Possible choices: build, bd, buildspec, bc, config, cg, report, rt, inspect, it, path, p, history, hy, cdash, cd, clean, debugreport, debug, stats, info, show, commands, cmds, docs, tutorial-examples, unittests, test, stylecheck, style

Sub-commands

build (bd)

Build and Run test

buildtest build [-h] [-b BUILDSPEC] [-x EXCLUDE] [-n NAME] [-e EXECUTOR]
                [-xt EXCLUDE_TAGS] [-t TAGS] [--rerun] [-f FILTER]
                [--helpfilter] [-et {local,batch}] [--module-purge]
                [-m MODULES] [-u UNLOAD_MODULES] [--account ACCOUNT]
                [--maxpendtime MAXPENDTIME] [--pollinterval POLLINTERVAL]
                [--procs PROCS [PROCS ...]] [--nodes NODES [NODES ...]]
                [--display {output,test}] [--dry-run] [--limit LIMIT]
                [--max-jobs MAX_JOBS] [--profile PROFILE] [--remove-stagedir]
                [--rebuild REBUILD] [--retry RETRY]
                [--save-profile SAVE_PROFILE] [--strict] [--testdir TESTDIR]
                [--timeout TIMEOUT] [--validate]
                [--write-config-file WRITE_CONFIG_FILE]

discover

Select buildspec file to run based on file, tag, executor

-b, --buildspec

Specify a buildspec (file or directory) to build. A buildspec must end in ‘.yml’ extension.

-x, --exclude

Exclude one or more buildspecs (file or directory) from processing. A buildspec must end in ‘.yml’ extension.

-n, --name

Specify a name of test to run

-e, --executor

Discover buildspecs by executor name found in buildspec cache

-xt, --exclude-tags

Exclude tests by one or more tagnames found in buildspec cache

-t, --tags

Discover buildspecs by tags found in buildspec cache

--rerun

Rerun last successful buildtest build command.

Default: False

filter

Filter tests after selection

-f, --filter

Filter buildspec based on tags, type, or maintainers. Usage: –filter key1=val1,val2;key2=val3;key3=val4,val5

--helpfilter

Show available filter fields used with –filter option

Default: False

-et, --executor-type

Possible choices: local, batch

Filter tests by executor type (local, batch)

module

Module Selection option

--module-purge

Run ‘module purge’ before running any test

Default: False

-m, --modules

Specify a list of modules to load during test execution, to specify multiple modules each one must be comma separated for instance if you want to load ‘gcc’ and ‘python’ module you can do ‘-m gcc,python’

-u, --unload-modules

Specify a list of modules to unload during test execution

batch

Batch Submission options

--account: Specify project account used to charge batch jobs (applicable for batch jobs only)
--maxpendtime: Specify Maximum Pending Time (sec) for job before cancelling job. This only applies for batch job submission.
--pollinterval: Specify Poll Interval (sec) for polling batch jobs
--procs: Specify number of processes to run tests (only applicable with batch jobs). Multiple values can be specified comma separated.
--nodes: Specify number of nodes to run tests (only applicable with batch jobs). Multiple values can be specified comma separated.

extra

All extra options

--display

Possible choices: output, test

Display content of output/error or test

--dry-run

Show a list of tests that will potentially be run without actually running them.

Default: False

--limit

Limit number of tests that can be run.

--max-jobs

Maximum number of jobs that can be run concurrently.

--profile

Specify a profile to load from configuration file

--remove-stagedir

Remove stage directory after job completion.

Default: False

--rebuild

Rebuild test X number of times. Must be a positive number between [1-50]

--retry

Retry failed jobs

Default: 1

--save-profile

Save buildtest command options into a profile and update configuration file

--strict

Enable strict mode for test by setting ‘set -eo pipefail’ in test script

Default: False

--testdir

Specify a custom test directory where to write tests. This overrides configuration file and default location.

--timeout

Specify test timeout in number of seconds

--validate

Validate given buildspecs and control behavior of buildtest build to stop execution after parsing the YAML files.

Default: False

--write-config-file

Specify path to configuration file to write changes when saving profile

buildspec (bc)

Buildspec Interface

buildtest buildspec [-h]  ...

subcommands

Buildspec Interface subcommands

: Possible choices: edit-file, ef, edit-test, et, find, f, maintainers, m, show, s, show-fail, sf, summary, sm, validate, val

Sub-commands

edit-file (ef)

Edit buildspec file based on filename

buildtest buildspec edit-file [-h] [file ...]

Positional Arguments

file: Edit buildspec file in editor

edit-test (et)

Edit buildspec file based on test name

buildtest buildspec edit-test [-h] [name ...]

Positional Arguments

name: Show content of buildspec based on test name

find (f)

Query information from buildspecs cache

buildtest buildspec find [-h] [-c COUNT] [-n] [--pager] [--row-count]
                         [--terse] [-b] [-e] [--group-by-tags]
                         [--group-by-executor] [-p] [-t] [--filter FILTER]
                         [--format FORMAT] [--filterfields] [--formatfields]
                         [--helpfilter] [--helpformat] [-q] [-r] [-s SEARCH]
                         ...

Positional Arguments

: Possible choices: invalid

Named Arguments

-c, --count

Retrieve limited number of rows that get printed

-n, --no-header

Do not print header columns in terse output (–terse)

Default: False

--pager

Enable PAGING when viewing result

Default: False

--row-count

Display number of rows from query shown in table

Default: False

--terse

Print output in machine readable format

Default: False

query

query options to retrieve from buildspec cache

-b, --buildspec

Get all buildspec files from cache

Default: False

-e, --executors

Get all unique executors from buildspecs

Default: False

--group-by-tags

Group tests by tag name

Default: False

--group-by-executor

Group tests by executor name

Default: False

-p, --paths

Print all root buildspec paths

Default: False

-t, --tags

List all available tags

Default: False

filter

filter and format options

--filter

Filter buildspec cache with filter fields in format –filter key1=val1,key2=val2

--format

Format buildspec cache with format fields in format –format field1,field2,…

--filterfields

Print raw filter fields for –filter option for filtering buildspec cache output

Default: False

--formatfields

Print raw format fields for –format option for formatting buildspec cache output

Default: False

--helpfilter

Show filter fields for –filter option for filtering buildspec cache output

Default: False

--helpformat

Show format fields for –format option for formatting buildspec cache output

Default: False

extra

All extra options

-q, --quiet

Don’t print output of buildspec cache when rebuilding cache

Default: False

-r, --rebuild

Rebuild buildspec cache and find all buildspecs again

Default: False

-s, --search

Search for buildspecs to load into cache. This can be a file or a directory. A file must end in ‘.yml’ extension, for directory buildtest will recursively search for all ‘.yml’ extensions. This option can be specified multiple times to search multiple files

Sub-commands

invalid

Show invalid buildspecs

buildtest buildspec find invalid [-h] [--row-count] [--terse] [--pager] [-e]

Named Arguments

--row-count

Display number of rows from query shown in table

Default: False

--terse

Print output in machine readable format

Default: False

--pager

Enable PAGING when viewing result

Default: False

-e, --error

Show error messages

Default: False

maintainers (m)

Query maintainers from buildspecs cache

buildtest buildspec maintainers [-h] [-c COUNT] [-n] [--pager] [--row-count]
                                [--terse] [-b]
                                {find} ...

Named Arguments

-c, --count

Retrieve limited number of rows that get printed

-n, --no-header

Do not print header columns in terse output (–terse)

Default: False

--pager

Enable PAGING when viewing result

Default: False

--row-count

Display number of rows from query shown in table

Default: False

--terse

Print output in machine readable format

Default: False

-b, --breakdown

Breakdown of buildspecs by maintainers

Default: False

Sub-commands

find

Find buildspecs based on maintainer name

buildtest buildspec maintainers find [-h] name

Positional Arguments

name: Find buildspec based on maintainer name

show (s)

Show content of buildspec file

buildtest buildspec show [-h] [--theme Color Themes] [name ...]

Positional Arguments

name: Show content of buildspec based on test name

Named Arguments

--theme

Possible choices: abap, algol, algol_nu, arduino, autumn, bw, borland, coffee, colorful, default, dracula, emacs, friendly_grayscale, friendly, fruity, github-dark, gruvbox-dark, gruvbox-light, igor, inkpot, lightbulb, lilypond, lovelace, manni, material, monokai, murphy, native, nord-darker, nord, one-dark, paraiso-dark, paraiso-light, pastie, perldoc, rainbow_dash, rrt, sas, solarized-dark, solarized-light, staroffice, stata-dark, stata-light, tango, trac, vim, vs, xcode, zenburn

Specify a color theme, Pygments style to use when displaying output. See https://pygments.org/docs/styles/#getting-a-list-of-available-styles for available themese

show-fail (sf)

Show content of buildspec file for all failed tests

buildtest buildspec show-fail [-h] [--theme Color Themes] [name ...]

Positional Arguments

name: Show content of buildspec based on failed test name

Named Arguments

--theme

Possible choices: abap, algol, algol_nu, arduino, autumn, bw, borland, coffee, colorful, default, dracula, emacs, friendly_grayscale, friendly, fruity, github-dark, gruvbox-dark, gruvbox-light, igor, inkpot, lightbulb, lilypond, lovelace, manni, material, monokai, murphy, native, nord-darker, nord, one-dark, paraiso-dark, paraiso-light, pastie, perldoc, rainbow_dash, rrt, sas, solarized-dark, solarized-light, staroffice, stata-dark, stata-light, tango, trac, vim, vs, xcode, zenburn

Specify a color theme, Pygments style to use when displaying output. See https://pygments.org/docs/styles/#getting-a-list-of-available-styles for available themese

summary (sm)

Print summary of buildspec cache

buildtest buildspec summary [-h] [--pager]

Named Arguments

--pager

Enable PAGING when viewing result

Default: False

validate (val)

Validate buildspecs with JSON Schema

buildtest buildspec validate [-h] [-b BUILDSPEC] [-x EXCLUDE] [-e EXECUTOR]
                             [-t TAG] [-n NAME]

Named Arguments

-b, --buildspec: Specify path to buildspec (file, or directory) to validate
-x, --exclude: Specify path to buildspec to exclude (file or directory) during validation
-e, --executor: Specify buildspecs by executor name to validate
-t, --tag: Specify buildspecs by tag name to validate
-n, --name: Specify buildspecs by name to validate

config (cg)

Query buildtest configuration

buildtest config [-h]  ...

subcommands

Query information from buildtest configuration file

: Possible choices: edit, e, path, p, systems, validate, val, view, v, profiles, prof, executors, ex, compilers, co

Sub-commands

edit (e)

Open configuration file in editor

buildtest config edit [-h]  ...

subcommands

Query information about edit

path (p)

Show path to configuration file

buildtest config path [-h]  ...

subcommands

Query information about path

systems

List all available systems

buildtest config systems [-h]  ...

subcommands

Query information about systems

validate (val)

Validate buildtest settings file with schema.

buildtest config validate [-h]  ...

subcommands

Query information about validate

view (v)

View configuration file

buildtest config view [-h] [--pager] [--theme Color Themes]  ...

Named Arguments

--pager

Enable PAGING when viewing result

Default: False

--theme

Possible choices: abap, algol, algol_nu, arduino, autumn, bw, borland, coffee, colorful, default, dracula, emacs, friendly_grayscale, friendly, fruity, github-dark, gruvbox-dark, gruvbox-light, igor, inkpot, lightbulb, lilypond, lovelace, manni, material, monokai, murphy, native, nord-darker, nord, one-dark, paraiso-dark, paraiso-light, pastie, perldoc, rainbow_dash, rrt, sas, solarized-dark, solarized-light, staroffice, stata-dark, stata-light, tango, trac, vim, vs, xcode, zenburn

Specify a color theme, Pygments style to use when displaying output. See https://pygments.org/docs/styles/#getting-a-list-of-available-styles for available themese

subcommands

Query information about view

profiles (prof)

Query profile from buildtest configuration

buildtest config profiles [-h]  ...

subcommands

Query information about profiles

: Possible choices: list, ls, remove, rm

Sub-commands

list (ls)

List all profiles

buildtest config profiles list [-h] [--theme Color Themes] [-y]

Named Arguments

--theme

Possible choices: abap, algol, algol_nu, arduino, autumn, bw, borland, coffee, colorful, default, dracula, emacs, friendly_grayscale, friendly, fruity, github-dark, gruvbox-dark, gruvbox-light, igor, inkpot, lightbulb, lilypond, lovelace, manni, material, monokai, murphy, native, nord-darker, nord, one-dark, paraiso-dark, paraiso-light, pastie, perldoc, rainbow_dash, rrt, sas, solarized-dark, solarized-light, staroffice, stata-dark, stata-light, tango, trac, vim, vs, xcode, zenburn

Specify a color theme, Pygments style to use when displaying output. See https://pygments.org/docs/styles/#getting-a-list-of-available-styles for available themese

-y, --yaml

List Profile details in YAML Format

Default: False

remove (rm)

Remove a profile from configuration

buildtest config profiles remove [-h] [profile_name ...]

Positional Arguments

profile_name: Specify profile name to remove

executors (ex)

Query executors from buildtest configuration

buildtest config executors [-h]  ...

subcommands

Query information about executors

: Possible choices: list, ls, remove, rm

Sub-commands

list (ls)

List all executors

buildtest config executors list [-h] [-j | -y | -d | -i | -a]

Named Arguments

-j, --json

View executor in JSON format

Default: False

-y, --yaml

View executors in YAML format

Default: False

-d, --disabled

Show disabled executors

Default: False

-i, --invalid

Show invalid executors

Default: False

-a, --all

Show all executors

Default: False

remove (rm)

Remove executor from configuration

buildtest config executors remove [-h] [executor_names ...]

Positional Arguments

executor_names: Specify an executor name to remove

compilers (co)

Search compilers

buildtest config compilers [-h]  ...

subcommands

Query information about compilers

: Possible choices: list, test, find, remove, rm

Sub-commands

list

List compilers

buildtest config compilers list [-h] [-j | -y]

Named Arguments

-j, --json

List compiler details in JSON format

Default: False

-y, --yaml

List compiler details in YAML format

Default: False

test

Test each compiler instance by performing module load test

buildtest config compilers test [-h] [compiler_names ...]

Positional Arguments

compiler_names: Specify compiler name to test

find

Find compilers

buildtest config compilers find [-h] [--file FILE] [-d] [-u]
                                [-m MODULEPATH [MODULEPATH ...]]

Named Arguments

--file

Write configuration file to a new file

-d, --detailed

Display detailed output when finding compilers

Default: False

-u, --update

Update configuration file with new compilers

Default: False

-m, --modulepath

Specify a list of directories to search for modules via MODULEPATH to detect compilers

remove (rm)

Remove compilers

buildtest config compilers remove [-h] [compiler_names ...]

Positional Arguments

compiler_names: Specify compiler name to remove

report (rt)

Query test report

buildtest report [-h] [--pager] [--row-count] [--terse] [-n] [-c COUNT]
                 [--filter FILTER] [--helpfilter] [--filterfields]
                 [--helpformat] [--formatfields] [--latest] [--oldest]
                 [-s START] [-e END] [-f | -p] [--format FORMAT | -d]
                 ...

Named Arguments

--pager

Enable PAGING when viewing result

Default: False

--row-count

Display number of rows from query shown in table

Default: False

--terse

Print output in machine readable format

Default: False

-n, --no-header

Do not print header columns in terse output (–terse)

Default: False

-c, --count

Retrieve limited number of rows that get printed

-f, --fail

Retrieve all FAIL tests

Default: False

-p, --pass

Retrieve all PASS tests

Default: False

--format

Format field for printing purposes. Fields must be separated by commas (e.g., –format field1,field2,…).

-d, --detailed

Print a detailed summary of the test results

Default: False

subcommands

Fetch test results from the report file and print them in table format

: Possible choices: clear, c, list, ls, path, p, summary, sm

filter

--filter

Filter report by filter fields. The filter fields must be in key=value format and multiple fields can be comma separated (e.g., –filter key1=val1,key2=val2). For a list of filter fields, run –helpfilter.

--helpfilter

List available filter fields to be used with –filter option

Default: False

--filterfields

Print raw filter fields for –filter option to filter the report

Default: False

format

--helpformat

List available format fields to be used with –format option

Default: False

--formatfields

Print raw format fields for –format option to format the report

Default: False

extra

--latest

Retrieve latest record of a particular test

Default: False

--oldest

Retrieve oldest record of a particular test

Default: False

-s, --start

Filter test by starttime

-e, --end

Filter test by endtime

Sub-commands

clear (c)

Remove all report files

buildtest report clear [-h]

list (ls)

List all report files

buildtest report list [-h]

path (p)

Print full path to the report file being used

buildtest report path [-h]

summary (sm)

Summarize test report

buildtest report summary [-h] [--pager] [--detailed]

Named Arguments

--pager

Enable PAGING when viewing result

Default: False

--detailed, -d

Enable a more detailed report

Default: False

inspect (it)

Inspect a test

buildtest inspect [-h]  ...

subcommands

Inspect Test result based on Test ID or Test Name

: Possible choices: buildspec, b, name, n, query, q, list, ls

Sub-commands

buildspec (b)

Inspect a test based on buildspec

buildtest inspect buildspec [-h] [--pager] [-a] [buildspec ...]

Positional Arguments

buildspec: List of buildspecs to query

Named Arguments

--pager

Enable PAGING when viewing result

Default: False

-a, --all

Fetch all records for a given test

Default: False

name (n)

Specify name of test

buildtest inspect name [-h] [--pager] [name ...]

Positional Arguments

name: Name of test

Named Arguments

--pager

Enable PAGING when viewing result

Default: False

query (q)

Query fields from record

buildtest inspect query [-h] [--pager] [--theme Color Themes] [-b] [-be] [-e]
                        [-o] [-t]
                        [name ...]

Positional Arguments

name: Name of builder to query in report file

Named Arguments

--pager

Enable PAGING when viewing result

Default: False

--theme

Possible choices: abap, algol, algol_nu, arduino, autumn, bw, borland, coffee, colorful, default, dracula, emacs, friendly_grayscale, friendly, fruity, github-dark, gruvbox-dark, gruvbox-light, igor, inkpot, lightbulb, lilypond, lovelace, manni, material, monokai, murphy, native, nord-darker, nord, one-dark, paraiso-dark, paraiso-light, pastie, perldoc, rainbow_dash, rrt, sas, solarized-dark, solarized-light, staroffice, stata-dark, stata-light, tango, trac, vim, vs, xcode, zenburn

Specify a color theme, Pygments style to use when displaying output. See https://pygments.org/docs/styles/#getting-a-list-of-available-styles for available themese

-b, --buildscript

Print build script

Default: False

-be, --buildenv

Print content of build env

Default: False

-e, --error

Print error file

Default: False

-o, --output

Print output file

Default: False

-t, --testpath

Print content of testpath

Default: False

list (ls)

List all test names, ids, and corresponding buildspecs

buildtest inspect list [-h] [--pager] [--row-count] [--terse] [-n] [-b]

Named Arguments

--pager

Enable PAGING when viewing result

Default: False

--row-count

Display number of rows from query shown in table

Default: False

--terse

Print output in machine readable format

Default: False

-n, --no-header

Do not print header columns in terse output (–terse)

Default: False

-b, --builder

List test in builder format

Default: False

path (p)

Show path attributes for a given test

buildtest path [-h] [-be | -t | -o | -e | -b | -s] name

Positional Arguments

name: Name of test

Named Arguments

-be, --buildenv

Show path to build environment

Default: False

-t, --testpath

Show path to test script

Default: False

-o, --outfile

Show path to output file

Default: False

-e, --errfile

Show path to error file

Default: False

-b, --buildscript

Show path to build script

Default: False

-s, --stagedir

Show path to stage directory

Default: False

history (hy)

Query build history

buildtest history [-h]  ...

subcommands

Query build history file

: Possible choices: list, query

Sub-commands

list

List a summary of all builds

buildtest history list [-h] [--pager] [--row-count] [--terse] [-n]

Named Arguments

--pager

Enable PAGING when viewing result

Default: False

--row-count

Display number of rows from query shown in table

Default: False

--terse

Print output in machine readable format

Default: False

-n, --no-header

Do not print header columns in terse output (–terse)

Default: False

query

Query information for a particular build

buildtest history query [-h] [--pager] [-l] [-o] id

Positional Arguments

id: Select a build ID

Named Arguments

--pager

Enable PAGING when viewing result

Default: False

-l, --log

Display logfile for corresponding build id

Default: False

-o, --output

View raw output from buildtest build command

Default: False

cdash

Upload test to CDASH server

buildtest cdash [-h]  ...

subcommands

buildtest CDASH integration

: Possible choices: view, upload

Sub-commands

view

Open CDASH project in web browser

buildtest cdash view [-h]

upload

Upload test results to CDASH server

buildtest cdash upload [-h] [--site SITE] [-o] buildname

Positional Arguments

buildname: Specify Build Name reported in CDASH

Named Arguments

--site

Specify site name reported in CDASH

-o, --open

Open CDASH report in browser

Default: False

cd

Change directory to root of test given a test name

buildtest cd [-h] test

Positional Arguments

test: Change directory to root of test for last run of test.

clean

Remove all generate files from buildtest including test directory, log files, report file, buildspec cache, history files

buildtest clean [-h] [-y]

Named Arguments

-y, --yes

Confirm yes for all prompts

Default: False

debugreport (debug)

Display system information and additional information for debugging purposes.

buildtest debugreport [-h]

stats

Show test statistics for given test

buildtest stats [-h] name

Positional Arguments

name: Name of test

info

Show details regarding current buildtest setup

buildtest info [-h]

show

buildtest command guide

buildtest show [-h]
               {bd,build,bc,buildspec,cdash,cg,config,hy,history,it,inspect,path,rt,report,style,stylecheck,test,unittests}

Positional Arguments

command

Possible choices: bd, build, bc, buildspec, cdash, cg, config, hy, history, it, inspect, path, rt, report, style, stylecheck, test, unittests

Show help message for command.

commands (cmds)

List all buildtest commands

buildtest commands [-h] [-a]

Named Arguments

-a, --with-aliases

Return all buildtest commands including command aliases

Default: False

docs

Undocumented

buildtest docs [-h]

tutorial-examples

Undocumented

buildtest tutorial-examples [-h] [-d] [-w] [--failfast] {aws,spack}

Positional Arguments

examples

Possible choices: aws, spack

Select which tutorial examples to build

Named Arguments

-d, --dryrun

Just print commands that will be generated without running them

Default: False

-w, --write

Write the content of each command to file

Default: False

--failfast

Stop on first failure

Default: False

unittests (test)

Undocumented

buildtest unittests [-h] [-c] [-p PYTESTOPTS] [-s SOURCEFILES]

Named Arguments

-c, --coverage

Enable coverage when running regression test

Default: False

-p, --pytestopts

Specify option to pytest

-s, --sourcefiles

Specify path to file or directory when running regression test

stylecheck (style)

Undocumented

buildtest stylecheck [-h] [--no-black] [--no-isort] [--no-pyflakes] [-a]

Named Arguments

--no-black

Don’t run black style check

Default: False

--no-isort

Don’t run isort style check

Default: False

--no-pyflakes

Don’t run pyflakes check

Default: False

-a, --apply

Apply style checks to codebase.

Default: False

References

GitHub: https://github.com/buildtesters/buildtest Documentation: https://buildtest.readthedocs.io/en/latest/index.html Slack: http://hpcbuildtest.slack.com/

Please report issues at https://github.com/buildtesters/buildtest/issues

Copyright (c) 2021-2024, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy), Shahzeb Siddiqui, and Vanessa Sochat. All rights reserved.