Buildtest Command Reference¶
buildtest is a HPC testing framework for building and running tests.
usage: buildtest [options] [COMMANDS]
Named Arguments¶
- -V, --version
show program’s version number and exit
- -c, --config
Specify Path to Configuration File
- -d, --debug
Print debug messages to screen
Default: False
- --no-color
Disable colored output
Default: False
COMMANDS¶
Possible choices: build, bd, buildspec, bc, config, cg, report, rt, inspect, it, history, hy, edit, et, schema, cdash, cd, clean, path, docs, schemadocs, debugreport, help, h, unittests, stylecheck, style
Sub-commands:¶
build (bd)¶
Build and Run test
buildtest build [-h] [-b BUILDSPEC] [-x EXCLUDE] [-e EXECUTOR] [-t TAGS]
[-f FILTER] [--helpfilter] [--account ACCOUNT]
[--disable-executor-check] [-k] [--maxpendtime MAXPENDTIME]
[--pollinterval POLLINTERVAL] [--rebuild REBUILD] [-r REPORT]
[--retry RETRY] [-s {parse,build}] [--testdir TESTDIR]
[--procs PROCS [PROCS ...]] [--nodes NODES [NODES ...]]
discover¶
select buildspecs
- -b, --buildspec
Specify a buildspec (file or directory) to build. A buildspec must end in ‘.yml’ extension.
- -x, --exclude
Exclude one or more buildspecs (file or directory) from processing. A buildspec must end in ‘.yml’ extension.
- -e, --executor
Discover buildspecs by executor name found in buildspec cache
- -t, --tags
Discover buildspecs by tags found in buildspec cache
filter¶
Filter tests
- -f, --filter
Filter buildspec based on tags, type, or maintainers. Usage: –filter key1=val1,key2=val2
- --helpfilter
Show available filter fields used with –filter option
Default: False
extra¶
All extra options
- --account
Specify project account used to charge batch jobs (applicable for batch jobs only)
- --disable-executor-check
Disable executor check during configuration check. By default these checks are enforced for Local, Slurm, PBS, LSF, and Cobalt Executor.
Default: True
- -k, --keep-stage-dir
Keep stage directory after job completion.
Default: False
- --maxpendtime
Specify Maximum Pending Time (sec) for job before cancelling job. This only applies for batch job submission.
- --pollinterval
Specify Poll Interval (sec) for polling batch jobs
- --rebuild
Rebuild test X number of times. Must be a positive number between [1-50]
- -r, --report
Specify a report file where tests will be written.
- --retry
Retry failed jobs
Default: 1
- -s, --stage
Possible choices: parse, build
control behavior of buildtest build
- --testdir
Specify a custom test directory where to write tests. This overrides configuration file and default location.
- --procs
Specify number of processes to run tests (only applicable with batch jobs). Multiple values can be specified comma separated.
- --nodes
Specify number of nodes to run tests (only applicable with batch jobs). Multiple values can be specified comma separated.
buildspec (bc)¶
Buildspec Interface
buildtest buildspec [-h] ...
Sub-commands:¶
find¶
Query information from buildspecs cache
buildtest buildspec find [-h] [-b] [-e] [--group-by-tags]
[--group-by-executor] [-m] [-mb] [-p] [-t]
[--filter FILTER] [--format FORMAT] [--helpfilter]
[--helpformat] [-n] [--terse] [-r] [--root ROOT]
...
Positional Arguments¶
Possible choices: invalid
Named Arguments¶
- -r, --rebuild
Rebuild buildspec cache and find all buildspecs again
Default: False
- --root
Specify root buildspecs (directory) path to load buildspecs into buildspec cache.
filter and format¶
filter and format options
- --filter
Filter buildspec cache with filter fields in format –filter key1=val1,key2=val2
- --format
Format buildspec cache with format fields in format –format field1,field2,…
- --helpfilter
Show Filter fields for –filter option for filtering buildspec cache output
Default: False
- --helpformat
Show Format fields for –format option for formatting buildspec cache output
Default: False
terse¶
terse options
- -n, --no-header
Print output without header in terse output
Default: False
- --terse
Print output in machine readable format
Default: False
query¶
query options to retrieve from buildspec cache
- -b, --buildspec
Get all buildspec files from cache
Default: False
- -e, --executors
get all unique executors from buildspecs
Default: False
- --group-by-tags
Group tests by tag name
Default: False
- --group-by-executor
Group tests by executor name
Default: False
- -m, --maintainers
Get all maintainers for all buildspecs
Default: False
- -mb, --maintainers-by-buildspecs
Show maintainers breakdown by buildspecs
Default: False
- -p, --paths
print all root buildspec paths
Default: False
- -t, --tags
List all available tags
Default: False
show¶
Show content of buildspec file
buildtest buildspec show [-h] [name [name ...]]
Positional Arguments¶
- name
Show content of buildspec based on test name
validate¶
Validate buildspecs with JSON Schema
buildtest buildspec validate [-h] [-b BUILDSPEC] [-x EXCLUDE] [-e EXECUTOR]
[-t TAG]
Named Arguments¶
- -b, --buildspec
Specify path to buildspec (file, or directory) to validate
- -x, --exclude
Specify path to buildspec to exclude (file or directory) during validation
- -e, --executor
Specify buildspecs by executor name to validate
- -t, --tag
Specify buildspecs by tag name to validate
config (cg)¶
Query buildtest configuration
buildtest config [-h] ...
subcommands¶
Query information from buildtest configuration file
Possible choices: compilers, edit, executors, systems, validate, view
Sub-commands:¶
compilers¶
Search compilers
buildtest config compilers [-h] [-j] [-y] ...
Named Arguments¶
- -j, --json
List compiler details in JSON format
Default: False
- -y, --yaml
List compiler details in YAML format
Default: False
executors¶
Query executors from buildtest configuration
buildtest config executors [-h] [-j | -y | -d | -i]
Named Arguments¶
- -j, --json
View executor in JSON format
Default: False
- -y, --yaml
View executors in YAML format
Default: False
- -d, --disabled
Show disabled executors
Default: False
- -i, --invalid
Show invalid executors
Default: False
report (rt)¶
Query test report
buildtest report [-h] [--filter FILTER] [--format FORMAT] [--helpfilter]
[--helpformat] [--latest] [--oldest] [-n] [-r REPORT] [-t]
...
Named Arguments¶
- --filter
Filter report by filter fields. The filter fields must be a key=value pair and multiple fields can be comma separated in the following format: –filter key1=val1,key2=val2 . For list of filter fields run: –helpfilter.
- --format
format field for printing purposes. For more details see –helpformat for list of available fields. Fields must be separated by comma (usage: –format <field1>,<field2>,…)
- --helpfilter
List available filter fields to be used with –filter option
Default: False
- --helpformat
List of available format fields
Default: False
- --latest
Retrieve latest record of particular test
Default: False
- --oldest
Retrieve oldest record of particular test
Default: False
- -n, --no-header
Don’t print headers column used with terse option (–terse).
Default: False
- -r, --report
Specify a report file to read
Default: “/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.13.0/var/report.json”
- -t, --terse
Print output in machine readable format
Default: False
subcommands¶
Fetch test results from report file and print them in table format
Possible choices: clear, list, summary
inspect (it)¶
Inspect a test based on NAME or ID
buildtest inspect [-h] [-r REPORT] ...
Named Arguments¶
- -r, --report
Specify a report file to load when inspecting test
subcommands¶
Inspect Test result based on Test ID or Test Name
Possible choices: buildspec, name, query, list
Sub-commands:¶
buildspec¶
Inspect a test based on buildspec
buildtest inspect buildspec [-h] [-a] [buildspec [buildspec ...]]
Positional Arguments¶
- buildspec
List of buildspecs to query
Named Arguments¶
- -a, --all
Fetch all records for a given test
Default: False
name¶
Specify name of test
buildtest inspect name [-h] [name [name ...]]
Positional Arguments¶
- name
Name of test
query¶
Query fields from record
buildtest inspect query [-h] [-b] [-e] [-o] [-t] [name [name ...]]
Positional Arguments¶
- name
Name of builder to query in report file
Named Arguments¶
- -b, --buildscript
Print build script
Default: False
- -e, --error
Print error file
Default: False
- -o, --output
Print output file
Default: False
- -t, --testpath
Print content of testpath
Default: False
list¶
List all test names, ids, and corresponding buildspecs
buildtest inspect list [-h] [-n] [-t] [-b]
Named Arguments¶
- -n, --no-header
Print output without header in terse format (–terse)
Default: False
- -t, --terse
Print output in terse format
Default: False
- -b, --builder
List test in builder format
Default: False
history (hy)¶
Query build history
buildtest history [-h] ...
edit (et)¶
Edit a buildspec and validate with schema file
buildtest edit [-h] buildspec
Positional Arguments¶
- buildspec
Open buildspec in editor and validate upon closing file
schema¶
List schema contents and examples
buildtest schema [-h] [-e] [-j] [-n Schema Name]
Named Arguments¶
- -e, --example
Show schema examples
Default: False
- -j, --json
Display json schema file
Default: False
- -n, --name
Possible choices: global.schema.json, definitions.schema.json, settings.schema.json, compiler-v1.0.schema.json, spack-v1.0.schema.json, script-v1.0.schema.json
show schema by name (e.g., script)
cdash¶
Upload test to CDASH server
buildtest cdash [-h] ...
cd¶
change directory to root of test given a test name
buildtest cd [-h] test
Positional Arguments¶
- test
Change directory to root of test for last run of test.
clean¶
Remove all generate files from buildtest including test directory, log files, report file, buildspec cache, history files.
buildtest clean [-h] [-y]
Named Arguments¶
- -y, --yes
Confirm yes for all prompts
Default: False
path¶
Show path attributes for a given test
buildtest path [-h] [-t | -o | -e | -b | -s] name
Positional Arguments¶
- name
Name of test
Named Arguments¶
- -t, --testpath
Show path to test script
Default: False
- -o, --outfile
Show path to output file
Default: False
- -e, --errfile
Show path to error file
Default: False
- -b, --buildscript
Show path to build script
Default: False
- -s, --stagedir
Show path to stage directory
Default: False
debugreport¶
Display system information and additional information for debugging purposes.
buildtest debugreport [-h]
help (h)¶
buildtest command guide
buildtest help [-h]
{build,buildspec,cdash,config,edit,history,inspect,path,report,schema}
Positional Arguments¶
- command
Possible choices: build, buildspec, cdash, config, edit, history, inspect, path, report, schema
Show help message for command
unittests¶
Run buildtest unit tests
buildtest unittests [-h] [-c] [-p PYTESTOPTS] [-s SOURCEFILES]
Named Arguments¶
- -c, --coverage
Enable coverage when running regression test
Default: False
- -p, --pytestopts
Specify option to pytest
- -s, --sourcefiles
Specify path to file or directory when running regression test
stylecheck (style)¶
Run buildtest style checks
buildtest stylecheck [-h] [--no-black] [--no-isort] [--no-pyflakes] [-a]
Named Arguments¶
- --no-black
Don’t run black style check
Default: False
- --no-isort
Don’t run isort style check
Default: False
- --no-pyflakes
Dont’ run pyflakes check
Default: False
- -a, --apply
Apply style checks to codebase.
Default: False
References
GitHub: https://github.com/buildtesters/buildtest Documentation: https://buildtest.readthedocs.io/en/latest/index.html Schema Documentation: https://buildtesters.github.io/buildtest/ Slack: http://hpcbuildtest.slack.com/
Please report issues at https://github.com/buildtesters/buildtest/issues
Copyright (c) 2021-2022, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy), Shahzeb Siddiqui, and Vanessa Sochat. All rights reserved.