Buildtest Command Reference

buildtest is a HPC testing framework for building and running tests.

usage: buildtest [options] [COMMANDS]

Named Arguments

-V, --version

show program’s version number and exit

-c, --config

Specify Path to Configuration File

-d, --debug

Print debug messages to screen

Default: False

--no-color

Disable colored output

Default: False

COMMANDS

Possible choices: build, bd, buildspec, bc, config, cg, report, rt, inspect, it, history, hy, edit, et, schema, cdash, cd, clean, path, docs, schemadocs, debugreport, help, h, unittests, stylecheck, style

Sub-commands:

build (bd)

Build and Run test

buildtest build [-h] [-b BUILDSPEC] [-x EXCLUDE] [-e EXECUTOR] [-t TAGS]
                [-f FILTER] [--helpfilter] [--account ACCOUNT]
                [--disable-executor-check] [-k] [--maxpendtime MAXPENDTIME]
                [--pollinterval POLLINTERVAL] [--rebuild REBUILD] [-r REPORT]
                [--retry RETRY] [-s {parse,build}] [--testdir TESTDIR]
                [--procs PROCS [PROCS ...]] [--nodes NODES [NODES ...]]

discover

select buildspecs

-b, --buildspec

Specify a buildspec (file or directory) to build. A buildspec must end in ‘.yml’ extension.

-x, --exclude

Exclude one or more buildspecs (file or directory) from processing. A buildspec must end in ‘.yml’ extension.

-e, --executor

Discover buildspecs by executor name found in buildspec cache

-t, --tags

Discover buildspecs by tags found in buildspec cache

filter

Filter tests

-f, --filter

Filter buildspec based on tags, type, or maintainers. Usage: –filter key1=val1,key2=val2

--helpfilter

Show available filter fields used with –filter option

Default: False

extra

All extra options

--account

Specify project account used to charge batch jobs (applicable for batch jobs only)

--disable-executor-check

Disable executor check during configuration check. By default these checks are enforced for Local, Slurm, PBS, LSF, and Cobalt Executor.

Default: True

-k, --keep-stage-dir

Keep stage directory after job completion.

Default: False

--maxpendtime

Specify Maximum Pending Time (sec) for job before cancelling job. This only applies for batch job submission.

--pollinterval

Specify Poll Interval (sec) for polling batch jobs

--rebuild

Rebuild test X number of times. Must be a positive number between [1-50]

-r, --report

Specify a report file where tests will be written.

--retry

Retry failed jobs

Default: 1

-s, --stage

Possible choices: parse, build

control behavior of buildtest build

--testdir

Specify a custom test directory where to write tests. This overrides configuration file and default location.

--procs

Specify number of processes to run tests (only applicable with batch jobs). Multiple values can be specified comma separated.

--nodes

Specify number of nodes to run tests (only applicable with batch jobs). Multiple values can be specified comma separated.

buildspec (bc)

Buildspec Interface

buildtest buildspec [-h]  ...

subcommands

Find buildspec from cache file

Possible choices: find, summary, show, validate

Sub-commands:

find

Query information from buildspecs cache

buildtest buildspec find [-h] [-b] [-e] [--group-by-tags]
                         [--group-by-executor] [-m] [-mb] [-p] [-t]
                         [--filter FILTER] [--format FORMAT] [--helpfilter]
                         [--helpformat] [-n] [--terse] [-r] [--root ROOT]
                         ...
Positional Arguments

Possible choices: invalid

Named Arguments
-r, --rebuild

Rebuild buildspec cache and find all buildspecs again

Default: False

--root

Specify root buildspecs (directory) path to load buildspecs into buildspec cache.

filter and format

filter and format options

--filter

Filter buildspec cache with filter fields in format –filter key1=val1,key2=val2

--format

Format buildspec cache with format fields in format –format field1,field2,…

--helpfilter

Show Filter fields for –filter option for filtering buildspec cache output

Default: False

--helpformat

Show Format fields for –format option for formatting buildspec cache output

Default: False

terse

terse options

-n, --no-header

Print output without header in terse output

Default: False

--terse

Print output in machine readable format

Default: False

query

query options to retrieve from buildspec cache

-b, --buildspec

Get all buildspec files from cache

Default: False

-e, --executors

get all unique executors from buildspecs

Default: False

--group-by-tags

Group tests by tag name

Default: False

--group-by-executor

Group tests by executor name

Default: False

-m, --maintainers

Get all maintainers for all buildspecs

Default: False

-mb, --maintainers-by-buildspecs

Show maintainers breakdown by buildspecs

Default: False

-p, --paths

print all root buildspec paths

Default: False

-t, --tags

List all available tags

Default: False

Sub-commands:
invalid

Show invalid buildspecs

buildtest buildspec find invalid [-h] [-e]
Named Arguments
-e, --error

Show error messages

Default: False

summary

Print summary of buildspec cache

buildtest buildspec summary [-h]
show

Show content of buildspec file

buildtest buildspec show [-h] [name [name ...]]
Positional Arguments
name

Show content of buildspec based on test name

validate

Validate buildspecs with JSON Schema

buildtest buildspec validate [-h] [-b BUILDSPEC] [-x EXCLUDE] [-e EXECUTOR]
                             [-t TAG]
Named Arguments
-b, --buildspec

Specify path to buildspec (file, or directory) to validate

-x, --exclude

Specify path to buildspec to exclude (file or directory) during validation

-e, --executor

Specify buildspecs by executor name to validate

-t, --tag

Specify buildspecs by tag name to validate

config (cg)

Query buildtest configuration

buildtest config [-h]  ...

subcommands

Query information from buildtest configuration file

Possible choices: compilers, edit, executors, systems, validate, view

Sub-commands:

compilers

Search compilers

buildtest config compilers [-h] [-j] [-y]  ...
Named Arguments
-j, --json

List compiler details in JSON format

Default: False

-y, --yaml

List compiler details in YAML format

Default: False

subcommands

Find new compilers and add them to detected compiler section

Possible choices: find

Sub-commands:
find

Find compilers

buildtest config compilers find [-h] [-d] [-u]
Named Arguments
-d, --debug

Display Debugging output when finding compilers

Default: False

-u, --update

Update configuration file with new compilers

Default: False

edit

Open configuration file in editor

buildtest config edit [-h]
executors

Query executors from buildtest configuration

buildtest config executors [-h] [-j | -y | -d | -i]
Named Arguments
-j, --json

View executor in JSON format

Default: False

-y, --yaml

View executors in YAML format

Default: False

-d, --disabled

Show disabled executors

Default: False

-i, --invalid

Show invalid executors

Default: False

systems

List all available systems

buildtest config systems [-h]
validate

Validate buildtest settings file with schema.

buildtest config validate [-h]
view

View configuration file

buildtest config view [-h]

report (rt)

Query test report

buildtest report [-h] [--filter FILTER] [--format FORMAT] [--helpfilter]
                 [--helpformat] [--latest] [--oldest] [-n] [-r REPORT] [-t]
                 ...

Named Arguments

--filter

Filter report by filter fields. The filter fields must be a key=value pair and multiple fields can be comma separated in the following format: –filter key1=val1,key2=val2 . For list of filter fields run: –helpfilter.

--format

format field for printing purposes. For more details see –helpformat for list of available fields. Fields must be separated by comma (usage: –format <field1>,<field2>,…)

--helpfilter

List available filter fields to be used with –filter option

Default: False

--helpformat

List of available format fields

Default: False

--latest

Retrieve latest record of particular test

Default: False

--oldest

Retrieve oldest record of particular test

Default: False

-n, --no-header

Don’t print headers column used with terse option (–terse).

Default: False

-r, --report

Specify a report file to read

Default: “/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.13.0/var/report.json”

-t, --terse

Print output in machine readable format

Default: False

subcommands

Fetch test results from report file and print them in table format

Possible choices: clear, list, summary

Sub-commands:

clear

delete report file

buildtest report clear [-h]
list

List all report files

buildtest report list [-h]
summary

Summarize test report

buildtest report summary [-h]

inspect (it)

Inspect a test based on NAME or ID

buildtest inspect [-h] [-r REPORT]  ...

Named Arguments

-r, --report

Specify a report file to load when inspecting test

subcommands

Inspect Test result based on Test ID or Test Name

Possible choices: buildspec, name, query, list

Sub-commands:

buildspec

Inspect a test based on buildspec

buildtest inspect buildspec [-h] [-a] [buildspec [buildspec ...]]
Positional Arguments
buildspec

List of buildspecs to query

Named Arguments
-a, --all

Fetch all records for a given test

Default: False

name

Specify name of test

buildtest inspect name [-h] [name [name ...]]
Positional Arguments
name

Name of test

query

Query fields from record

buildtest inspect query [-h] [-b] [-e] [-o] [-t] [name [name ...]]
Positional Arguments
name

Name of builder to query in report file

Named Arguments
-b, --buildscript

Print build script

Default: False

-e, --error

Print error file

Default: False

-o, --output

Print output file

Default: False

-t, --testpath

Print content of testpath

Default: False

list

List all test names, ids, and corresponding buildspecs

buildtest inspect list [-h] [-n] [-t] [-b]
Named Arguments
-n, --no-header

Print output without header in terse format (–terse)

Default: False

-t, --terse

Print output in terse format

Default: False

-b, --builder

List test in builder format

Default: False

history (hy)

Query build history

buildtest history [-h]  ...

subcommands

Query build history file

Possible choices: list, query

Sub-commands:

list

List a summary of all builds

buildtest history list [-h] [-n] [-t]
Named Arguments
-n, --no-header

Do not print header columns in terse output (–terse)

Default: False

-t, --terse

Print output in machine readable format

Default: False

query

Query information for a particular build

buildtest history query [-h] [-l] [-o] id
Positional Arguments
id

Select a build ID

Named Arguments
-l, --log

Display logfile for corresponding build id

Default: False

-o, --output

view raw output from buildtest build command

Default: False

edit (et)

Edit a buildspec and validate with schema file

buildtest edit [-h] buildspec

Positional Arguments

buildspec

Open buildspec in editor and validate upon closing file

schema

List schema contents and examples

buildtest schema [-h] [-e] [-j] [-n Schema Name]

Named Arguments

-e, --example

Show schema examples

Default: False

-j, --json

Display json schema file

Default: False

-n, --name

Possible choices: global.schema.json, definitions.schema.json, settings.schema.json, compiler-v1.0.schema.json, spack-v1.0.schema.json, script-v1.0.schema.json

show schema by name (e.g., script)

cdash

Upload test to CDASH server

buildtest cdash [-h]  ...

subcommands

buildtest CDASH integeration

Possible choices: view, upload

Sub-commands:

view

Open CDASH project in webbrowser

buildtest cdash view [-h] [--url URL]
Named Arguments
--url

Specify a url to CDASH project

upload

Upload Test to CDASH server

buildtest cdash upload [-h] [-r REPORT] [--site SITE] buildname
Positional Arguments
buildname

Specify Build Name reported in CDASH

Named Arguments
-r, --report

Path to report file to upload test results

--site

Specify site name reported in CDASH

cd

change directory to root of test given a test name

buildtest cd [-h] test

Positional Arguments

test

Change directory to root of test for last run of test.

clean

Remove all generate files from buildtest including test directory, log files, report file, buildspec cache, history files.

buildtest clean [-h] [-y]

Named Arguments

-y, --yes

Confirm yes for all prompts

Default: False

path

Show path attributes for a given test

buildtest path [-h] [-t | -o | -e | -b | -s] name

Positional Arguments

name

Name of test

Named Arguments

-t, --testpath

Show path to test script

Default: False

-o, --outfile

Show path to output file

Default: False

-e, --errfile

Show path to error file

Default: False

-b, --buildscript

Show path to build script

Default: False

-s, --stagedir

Show path to stage directory

Default: False

docs

Open buildtest docs in browser

buildtest docs [-h]

schemadocs

Open buildtest schema docs in browser

buildtest schemadocs [-h]

debugreport

Display system information and additional information for debugging purposes.

buildtest debugreport [-h]

help (h)

buildtest command guide

buildtest help [-h]
               {build,buildspec,cdash,config,edit,history,inspect,path,report,schema}

Positional Arguments

command

Possible choices: build, buildspec, cdash, config, edit, history, inspect, path, report, schema

Show help message for command

unittests

Run buildtest unit tests

buildtest unittests [-h] [-c] [-p PYTESTOPTS] [-s SOURCEFILES]

Named Arguments

-c, --coverage

Enable coverage when running regression test

Default: False

-p, --pytestopts

Specify option to pytest

-s, --sourcefiles

Specify path to file or directory when running regression test

stylecheck (style)

Run buildtest style checks

buildtest stylecheck [-h] [--no-black] [--no-isort] [--no-pyflakes] [-a]

Named Arguments

--no-black

Don’t run black style check

Default: False

--no-isort

Don’t run isort style check

Default: False

--no-pyflakes

Dont’ run pyflakes check

Default: False

-a, --apply

Apply style checks to codebase.

Default: False

References

GitHub: https://github.com/buildtesters/buildtest Documentation: https://buildtest.readthedocs.io/en/latest/index.html Schema Documentation: https://buildtesters.github.io/buildtest/ Slack: http://hpcbuildtest.slack.com/

Please report issues at https://github.com/buildtesters/buildtest/issues

Copyright (c) 2021-2022, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy), Shahzeb Siddiqui, and Vanessa Sochat. All rights reserved.