Buildspecs Interface¶
Now that we learned how to build tests, in this section we will discuss how one can query a buildspec cache. In buildtest, one can load all buildspecs which is equivalent to validating all buildspecs with the appropriate schema. Buildtest will ignore all invalid buildspecs and store them in a separate file.
Note
buildtest bc is an alias for buildtest buildspec command.
Finding Buildspecs - buildtest buildspec find¶
The buildtest buildspec find command is used for finding buildspecs from buildspec
cache. This command is also used for generating the buildspec cache. Shown below is a list of options for
provided for this command.
$ buildtest buildspec find --help
usage: buildtest [options] [COMMANDS] buildspec find [-h] [-b] [-e]
[--group-by-tags]
[--group-by-executor]
[-m] [-mb] [-p] [-t]
[--filter FILTER]
[--format FORMAT]
[--helpfilter]
[--helpformat] [-n]
[--terse] [--pager] [-r]
[--root ROOT]
...
positional arguments:
invalid Show invalid buildspecs
optional arguments:
-h, --help show this help message and exit
--pager Enable PAGING when viewing result
-r, --rebuild Rebuild buildspec cache and find all buildspecs again
--root ROOT Specify root buildspecs (directory) path to load
buildspecs into buildspec cache.
filter and format:
filter and format options
--filter FILTER Filter buildspec cache with filter fields in format
--filter key1=val1,key2=val2
--format FORMAT Format buildspec cache with format fields in format
--format field1,field2,...
--helpfilter Show Filter fields for --filter option for filtering
buildspec cache output
--helpformat Show Format fields for --format option for formatting
buildspec cache output
terse:
terse options
-n, --no-header Print output without header in terse output
--terse Print output in machine readable format
query:
query options to retrieve from buildspec cache
-b, --buildspec Get all buildspec files from cache
-e, --executors get all unique executors from buildspecs
--group-by-tags Group tests by tag name
--group-by-executor Group tests by executor name
-m, --maintainers Get all maintainers for all buildspecs
-mb, --maintainers-by-buildspecs
Show maintainers breakdown by buildspecs
-p, --paths print all root buildspec paths
-t, --tags List all available tags
The buildtest buildspec find command will discover all buildspecs by recursively searching all .yml extensions.
buildtest will validate each buildspec file with the json schema and buildtest will display all valid buildspecs in the output,
all invalid buildspecs will be stored in a file for post-processing.
$ buildtest buildspec find
Buildspec Cache: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/chec
kouts/v0.14.0/var/buildspecs/cache.json
name ┃ type ┃ executor ┃ tags ┃ description
━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━
hello_world │ script │ generic.local.b │ tutorials │ hello world
│ │ ash │ │ example
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
skip │ script │ generic.local.b │ tutorials │ This test is
│ │ ash │ │ skipped
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
unskipped │ script │ generic.local.b │ tutorials │ This test is
│ │ ash │ │ not skipped
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
gcc_version │ script │ generic.local.b │ │ Print gcc
│ │ ash │ │ version
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
foo_bar │ script │ generic.local.s │ tutorials │ prints variable
│ │ h │ │ $FOO
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
circle_area │ script │ generic.local.b │ tutorials │ Calculate
│ │ ash │ python │ circle of area
│ │ │ │ given a radius
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
add_numbers │ script │ generic.local.b │ tutorials │ Add X+Y
│ │ ash │ │
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
bash_login_sheb │ script │ generic.local.b │ tutorials │ customize
ang │ │ ash │ │ shebang line
│ │ │ │ with bash login
│ │ │ │ shell
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
bash_nonlogin_s │ script │ generic.local.b │ tutorials │ customize
hebang │ │ ash │ │ shebang line
│ │ │ │ with default
│ │ │ │ bash (nonlogin)
│ │ │ │ shell
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
python_hello │ script │ generic.local.b │ python │ Hello World
│ │ ash │ │ python
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
executor_regex_ │ script │ generic.local.( │ tutorials │ regular
script_schema │ │ bash|sh) │ │ expression test
│ │ │ │ with executor
│ │ │ │ using script
│ │ │ │ schema
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
bash_env_variab │ script │ generic.local.b │ tutorials │ Declare
les │ │ ash │ │ environment
│ │ │ │ variables in
│ │ │ │ default shell
│ │ │ │ (bash)
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
csh_env_declara │ script │ generic.local.c │ tutorials │ csh shell
tion │ │ sh │ │ example to
│ │ │ │ declare
│ │ │ │ environment
│ │ │ │ variables
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
tcsh_env_declar │ script │ generic.local.c │ tutorials │ tcsh shell
ation │ │ sh │ │ example to
│ │ │ │ declare
│ │ │ │ environment
│ │ │ │ variables
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
sleep │ script │ generic.local.b │ tutorials │ sleep 2 seconds
│ │ ash │ │
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
string_tag │ script │ generic.local.b │ network │ tags can be a
│ │ ash │ │ string
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
list_of_strings │ script │ generic.local.b │ network ping │ tags can be a
_tags │ │ ash │ │ list of strings
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
always_pass │ script │ generic.local.s │ │ This test will
│ │ h │ │ always 'PASS'
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
always_fail │ script │ generic.local.s │ │ This test will
│ │ h │ │ always 'FAIL'
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
test_fail_retur │ script │ generic.local.s │ │ This test will
ncode_match │ │ h │ │ 'FAIL' even if
│ │ │ │ we have
│ │ │ │ returncode
│ │ │ │ match
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
test_pass_retur │ script │ generic.local.s │ │ This test will
ncode_mismatch │ │ h │ │ 'PASS' even if
│ │ │ │ we have
│ │ │ │ returncode
│ │ │ │ mismatch
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
download_stream │ script │ generic.local.b │ │
│ │ ash │ │
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
run_stream │ script │ generic.local.b │ │ job dependency
│ │ ash │ │ on
│ │ │ │ download_stream
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
selinux_disable │ script │ generic.local.b │ tutorials │ Check if
│ │ ash │ │ SELinux is
│ │ │ │ Disabled
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
timelimit_min_m │ script │ generic.local.s │ tutorials │ Run a sleep job
ax │ │ h │ │ for 2 seconds
│ │ │ │ and test pass
│ │ │ │ if its within
│ │ │ │ 1.0-3.0sec
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
timelimit_min │ script │ generic.local.s │ tutorials │ Run a sleep job
│ │ h │ │ for 2 seconds
│ │ │ │ and test pass
│ │ │ │ if its exceeds
│ │ │ │ min time of 1.0
│ │ │ │ sec
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
timelimit_max │ script │ generic.local.s │ tutorials │ Run a sleep job
│ │ h │ │ for 2 seconds
│ │ │ │ and test pass
│ │ │ │ if it's within
│ │ │ │ max time: 5.0
│ │ │ │ sec
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
timelimit_min_f │ script │ generic.local.s │ tutorials │ This test fails
ail │ │ h │ │ because it runs
│ │ │ │ less than
│ │ │ │ mintime of 10
│ │ │ │ second
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
timelimit_max_f │ script │ generic.local.s │ tutorials │ This test fails
ail │ │ h │ │ because it
│ │ │ │ exceeds maxtime
│ │ │ │ of 1.0 second
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
metric_regex_ex │ script │ generic.local.s │ tutorials │ capture result
ample │ │ h │ │ metric from
│ │ │ │ output
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
_bin_sh_shell │ script │ generic.local.s │ tutorials │ /bin/sh shell
│ │ h │ │ example
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
_bin_bash_shell │ script │ generic.local.b │ tutorials │ /bin/bash shell
│ │ ash │ │ example
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
bash_shell │ script │ generic.local.b │ tutorials │ bash shell
│ │ ash │ │ example
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
sh_shell │ script │ generic.local.s │ tutorials │ sh shell
│ │ h │ │ example
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
shell_options │ script │ generic.local.s │ tutorials │ shell options
│ │ h │ │
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
csh_shell │ script │ generic.local.c │ tutorials │ csh shell
│ │ sh │ │ example
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
exit1_fail │ script │ generic.local.b │ tutorials fail │ exit 1 by
│ │ ash │ │ default is FAIL
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
exit1_pass │ script │ generic.local.b │ tutorials pass │ report exit 1
│ │ ash │ │ as PASS
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
returncode_list │ script │ generic.local.b │ tutorials fail │ exit 2 failed
_mismatch │ │ ash │ │ since it failed
│ │ │ │ to match
│ │ │ │ returncode 1
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
returncode_int_ │ script │ generic.local.b │ tutorials pass │ exit 128
match │ │ ash │ │ matches
│ │ │ │ returncode 128
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
variables_bash │ script │ generic.local.b │ tutorials │ Declare shell
│ │ ash │ │ variables in
│ │ │ │ bash
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
status_regex_pa │ script │ generic.local.b │ system │ Pass test based
ss │ │ ash │ │ on regular
│ │ │ │ expression
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
status_regex_fa │ script │ generic.local.b │ system │ Pass test based
il │ │ ash │ │ on regular
│ │ │ │ expression
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
executors_vars_ │ script │ generic.local.( │ tutorials │ Declaring env
env_declaration │ │ bash|sh) │ │ and vars by
│ │ │ │ executors
│ │ │ │ section
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
status_returnco │ script │ generic.local.( │ tutorials │ define status
de_by_executors │ │ bash|sh) │ │ per executor
│ │ │ │ type.
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
executors_sbatc │ script │ generic.local.( │ tutorials │ Declaring env
h_declaration │ │ bash|sh) │ │ and vars by
│ │ │ │ executors
│ │ │ │ section
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
test1 │ script │ generic.local.b │ │ This test will
│ │ ash │ │ pass with exit
│ │ │ │ 1
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
test2 │ script │ generic.local.b │ │ This test will
│ │ ash │ │ run if test1
│ │ │ │ has returncode
│ │ │ │ 1
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
test3 │ script │ generic.local.b │ │ This test will
│ │ ash │ │ run if test1
│ │ │ │ has returncode
│ │ │ │ 1 and test2 has
│ │ │ │ returncode 2
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
runtime_test │ script │ generic.local.b │ │ This test will
│ │ ash │ │ sleep 5 second
│ │ │ │ but will fail
│ │ │ │ due to runtime
│ │ │ │ 2sec
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
runtime_test_pa │ script │ generic.local.b │ │ This test will
ss │ │ ash │ │ run when runtim
│ │ │ │ e_test_pass is
│ │ │ │ PASS
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
runtime_test_fa │ script │ generic.local.b │ │ This test will
il │ │ ash │ │ run when runtim
│ │ │ │ e_test_pass is
│ │ │ │ FAIL
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
jobA │ script │ generic.local.b │ │ no job
│ │ ash │ │ dependency
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
jobB │ script │ generic.local.b │ │ job dependency
│ │ ash │ │ on jobA
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
jobC │ script │ generic.local.b │ │ job dependency
│ │ ash │ │ on jobA and
│ │ │ │ jobB
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
pass_test │ script │ generic.local.b │ │ This test will
│ │ ash │ │ always pass
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
fail_test │ script │ generic.local.b │ │ This test will
│ │ ash │ │ run if test
│ │ │ │ 'pass_test' is
│ │ │ │ in state 'PASS'
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
pass_and_fail_t │ script │ generic.local.b │ │ This test will
est │ │ ash │ │ run if
│ │ │ │ pass_test is
│ │ │ │ 'PASS' and
│ │ │ │ fail_test is
│ │ │ │ 'FAIL'
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
final_test │ script │ generic.local.b │ │ Test will run
│ │ ash │ │ after
│ │ │ │ 'pass_test',
│ │ │ │ 'fail_test',
│ │ │ │ and 'pass_and_f
│ │ │ │ ail_test'
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
root_disk_usage │ script │ generic.local.b │ filesystem │ Check root disk
│ │ ash │ storage │ usage and
│ │ │ │ report if it
│ │ │ │ exceeds
│ │ │ │ threshold
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
systemd_default │ script │ generic.local.b │ system │ check if
_target │ │ ash │ │ default target
│ │ │ │ is multi-user.t
│ │ │ │ arget
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
kernel_swapusag │ script │ generic.local.b │ configuration │ Retrieve Kernel
e │ │ ash │ │ Swap Usage
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
ulimit_filelock │ script │ generic.local.b │ system │ Check if file
_unlimited │ │ ash │ │ lock is set to
│ │ │ │ unlimited in
│ │ │ │ ulimits
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
ulimit_cputime_ │ script │ generic.local.b │ system │ Check if
unlimited │ │ ash │ │ cputime is set
│ │ │ │ to unlimited in
│ │ │ │ ulimits
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
ulimit_stacksiz │ script │ generic.local.b │ system │ Check if stack
e_unlimited │ │ ash │ │ size is set to
│ │ │ │ unlimited in
│ │ │ │ ulimits
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
ulimit_vmsize_u │ script │ generic.local.b │ system │ Check virtual
nlimited │ │ ash │ │ memory size and
│ │ │ │ check if its
│ │ │ │ set to
│ │ │ │ unlimited
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
ulimit_filedesc │ script │ generic.local.b │ system │ Check if open
riptor_4096 │ │ ash │ │ file
│ │ │ │ descriptors
│ │ │ │ limit is set to
│ │ │ │ 4096
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
ulimit_max_user │ script │ generic.local.b │ system │ Check max
_process_2048 │ │ ash │ │ number of user
│ │ │ │ process limit
│ │ │ │ is set to 2048
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
current_user_qu │ script │ generic.local.b │ slurm │ show all
eue │ │ ash │ │ current pending
│ │ │ │ jobs for
│ │ │ │ current user
│ │ │ │ (squeue -u
│ │ │ │ $USER)
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
show_all_jobs │ script │ generic.local.b │ slurm │ show all
│ │ ash │ │ pending +
│ │ │ │ running jobs
│ │ │ │ (squeue -a)
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
nodes_state_dow │ script │ generic.local.b │ slurm │ Show nodes in
n │ │ ash │ │ DOWN state
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
nodes_state_reb │ script │ generic.local.b │ slurm │ Show nodes in
oot │ │ ash │ │ REBOOT state
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
nodes_state_all │ script │ generic.local.b │ slurm │ Show nodes in
ocated │ │ ash │ │ ALLOCATED state
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
nodes_state_com │ script │ generic.local.b │ slurm │ Show nodes in
pleting │ │ ash │ │ COMPLETING
│ │ │ │ state
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
nodes_state_idl │ script │ generic.local.b │ slurm │ Show nodes in
e │ │ ash │ │ IDLE state
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
node_down_fail_ │ script │ generic.local.b │ slurm │ Show nodes
list_reason │ │ ash │ │ DOWN, DRAINED,
│ │ │ │ FAIL or FAILING
│ │ │ │ and list reason
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
dead_nodes │ script │ generic.local.b │ slurm │ Show
│ │ ash │ │ non-responding
│ │ │ │ nodes
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
get_partitions │ script │ generic.local.b │ slurm │ Get all slurm
│ │ ash │ │ partitions
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
sinfo_version │ script │ generic.local.b │ slurm │ get slurm
│ │ ash │ │ version
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
show_accounts │ script │ generic.local.b │ slurm │ run sacctmgr
│ │ ash │ │ list accounts
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
show_users │ script │ generic.local.b │ slurm │ run sacctmgr
│ │ ash │ │ list users
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
show_qos │ script │ generic.local.b │ slurm │ run sacctmgr
│ │ ash │ │ list qos
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
show_tres │ script │ generic.local.b │ slurm │ run sacctmgr
│ │ ash │ │ list tres
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
slurm_config │ script │ generic.local.b │ slurm │ run scontrol
│ │ ash │ │ show config
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
show_partition │ script │ generic.local.b │ slurm │ run scontrol
│ │ ash │ │ show partition
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
show_lsf_config │ script │ generic.local.b │ lsf │ Show LSF
uration │ │ ash │ │ configuration
│ │ │ │ using lsinfo
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
show_lsf_models │ script │ generic.local.b │ lsf │ Show
│ │ ash │ │ information
│ │ │ │ about host
│ │ │ │ models in LSF
│ │ │ │ cluster
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
show_lsf_resour │ script │ generic.local.b │ lsf │ Show
ces │ │ ash │ │ information
│ │ │ │ about LSF
│ │ │ │ resources
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
lsf_version │ script │ generic.local.b │ lsf │ Display lsf
│ │ ash │ │ version using
│ │ │ │ lsinfo
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
show_lsf_user_g │ script │ generic.local.b │ lsf │ Show
roups │ │ ash │ │ information
│ │ │ │ about all LSF
│ │ │ │ user groups
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
display_lsf_hos │ script │ generic.local.b │ lsf │ Show all hosts
ts │ │ ash │ │ in LSF cluster
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
display_hosts_f │ script │ generic.local.b │ lsf │ Show all hosts
ormat │ │ ash │ │ with column
│ │ │ │ hostname and
│ │ │ │ status
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
bhosts_version │ script │ generic.local.b │ lsf │ display version
│ │ ash │ │ from bhosts
│ │ │ │ command
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
show_host_group │ script │ generic.local.b │ lsf │ Show
s │ │ ash │ │ information
│ │ │ │ about host
│ │ │ │ groups using
│ │ │ │ bmgroup
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
show_lsf_queues │ script │ generic.local.b │ lsf │ Show LSF queues
│ │ ash │ │
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
show_lsf_queues │ script │ generic.local.b │ lsf │ Show LSF queues
_formatted │ │ ash │ │ with formatted
│ │ │ │ columns
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
show_lsf_queues │ script │ generic.local.b │ lsf │ Show LSF queues
_current_user │ │ ash │ │ available for
│ │ │ │ current user
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
qsub_version │ script │ generic.local.s │ cobalt │ print version
│ │ h │ │ for qsub
│ │ │ │ command
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
qselect_version │ script │ generic.local.s │ cobalt │ print version
│ │ h │ │ for qselect
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
cqsub_version │ script │ generic.local.s │ cobalt │ print version
│ │ h │ │ for cqsub
│ │ │ │ command
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
qdel_version │ script │ generic.local.s │ cobalt │ print version
│ │ h │ │ for qdel
│ │ │ │ command
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
qmove_version │ script │ generic.local.s │ cobalt │ print version
│ │ h │ │ for qmove
│ │ │ │ command
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
show_jobs │ script │ generic.local.s │ cobalt │ Show all jobs
│ │ h │ │ in queue
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
show_queues │ script │ generic.local.s │ cobalt │ Show all queues
│ │ h │ │
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
inspect_image │ script │ generic.local.b │ containers │ Inspect image
│ │ ash │ singularity │ via
│ │ │ │ 'singularity
│ │ │ │ inspect'
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
pullImage_docke │ script │ generic.local.b │ containers │ Pull image dock
rhub │ │ ash │ singularity │ er://godlovedc/
│ │ │ │ lolcow from
│ │ │ │ DockerHub
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
pullImage_sylab │ script │ generic.local.b │ containers │ Pull image libr
scloud │ │ ash │ singularity │ ary://alpine:la
│ │ │ │ test from
│ │ │ │ Sylabs Cloud
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
pullImage_shub │ script │ generic.local.b │ containers │ Pull image shub
│ │ ash │ singularity │ ://vsoch/singul
│ │ │ │ arity-images
│ │ │ │ from
│ │ │ │ SingularityHub
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
build_sif_from_ │ script │ generic.local.b │ containers │ build sif image
dockerimage │ │ ash │ singularity │ from docker
│ │ │ │ image docker://
│ │ │ │ godlovedc/lolco
│ │ │ │ w
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
build_sandbox_i │ script │ generic.local.b │ containers │ build sandbox
mage │ │ ash │ singularity │ image from
│ │ │ │ docker image do
│ │ │ │ cker://godloved
│ │ │ │ c/lolcow
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
build_remoteima │ script │ generic.local.b │ containers │ build remote
ges │ │ ash │ singularity │ hosted image
│ │ │ │ from AWS
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
runImage │ script │ generic.local.b │ containers │ run container d
│ │ ash │ singularity │ ocker://godlove
│ │ │ │ dc/lolcow
buildtest will load all discovered buildspecs in a cache file (JSON) which is created upon
running buildtest buildspec find. Any subsequent runs will read from cache and update
if any new buildspecs are added. If you make changes to buildspec you should rebuild the
buildspec cache by running:
$ buildtest buildspec find --rebuild
If you want to find all buildspec files in cache you can run buildtest buildspec find --buildspec.
Shown below is an example output.
$ buildtest buildspec find --buildspec
List of Buildspecs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Buildspecs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
...
The buildtest buildspec find --paths will display a list of root directories buildtest will search for
buildspecs when runninh buildtest buildspec find. One can define these directories in the configuration file
or pass them via command line.
$ buildtest buildspec find --paths
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.0/tut
orials
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.0/gen
eral_tests
buildtest will search buildspecs in buildspecs root defined in your configuration,
which is a list of directory paths to search for buildspecs.
If you want to load buildspecs from a directory path, you can specify a directory
via --root option in the format: buildtest buildspec find --root <path> --rebuild.
buildtest will load all valid buildspecs into cache and ignore
the rest. It’s important to add --rebuild if you want to regenerate buildspec cache.
Filtering buildspec¶
Once you have a buildspec cache, we can query the buildspec cache for certain attributes.
When you run buildtest buildspec find it will report all buildspecs from cache which can
be difficult to process. Therefore, we have a filter option (--filter) to restrict our search.
Let’s take a look at the available filter fields that are acceptable with filter option.
$ buildtest buildspec find --helpfilter
Filter Field Description
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Field ┃ Type ┃ Description ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ buildspecs │ Filter tests by buildspec │ FILE │
│ executor │ Filter by executor name │ STRING │
│ tags │ Filter by tag name │ STRING │
│ type │ Filter by schema type │ STRING │
└────────────┴───────────────────────────┴─────────────┘
The --filter option expects an arguments in key=value format as follows:
buildtest buildspec find --filter key1=value1,key2=value2,key3=value3
We can filter buildspec cache by tags=fail which will query all tests with
associated tag field in test.
$ buildtest buildspec find --filter tags=fail
Buildspec Cache: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/chec
kouts/v0.14.0/var/buildspecs/cache.json
name ┃ type ┃ executor ┃ tags ┃ description
━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━
exit1_fail │ script │ generic.local.ba │ tutorials fail │ exit 1 by
│ │ sh │ │ default is FAIL
─────────────────┼────────┼──────────────────┼────────────────┼─────────────────
returncode_list │ script │ generic.local.ba │ tutorials fail │ exit 2 failed
_mismatch │ │ sh │ │ since it failed
│ │ │ │ to match
│ │ │ │ returncode 1
In addition, we can query buildspecs by schema type using type property. In this
example we query all tests by type property
$ buildtest buildspec find --filter type=script
Buildspec Cache: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/chec
kouts/v0.14.0/var/buildspecs/cache.json
name ┃ type ┃ executor ┃ tags ┃ description
━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━
hello_world │ script │ generic.local.b │ tutorials │ hello world
│ │ ash │ │ example
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
skip │ script │ generic.local.b │ tutorials │ This test is
│ │ ash │ │ skipped
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
unskipped │ script │ generic.local.b │ tutorials │ This test is
│ │ ash │ │ not skipped
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
gcc_version │ script │ generic.local.b │ │ Print gcc
│ │ ash │ │ version
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
foo_bar │ script │ generic.local.s │ tutorials │ prints variable
│ │ h │ │ $FOO
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
circle_area │ script │ generic.local.b │ tutorials │ Calculate
│ │ ash │ python │ circle of area
...
Finally, we can combine multiple filter fields separated by comma, in the next example
we can query all buildspecs with tags=tutorials, executor=generic.local.sh, and type=script
$ buildtest buildspec find --filter tags=tutorials,executor=generic.local.sh,type=script
Buildspec Cache: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/chec
kouts/v0.14.0/var/buildspecs/cache.json
name ┃ type ┃ executor ┃ tags ┃ description
━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━
foo_bar │ script │ generic.local.sh │ tutorials │ prints variable
│ │ │ │ $FOO
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
timelimit_min_max │ script │ generic.local.sh │ tutorials │ Run a sleep job
│ │ │ │ for 2 seconds and
│ │ │ │ test pass if its
│ │ │ │ within 1.0-3.0sec
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
timelimit_min │ script │ generic.local.sh │ tutorials │ Run a sleep job
│ │ │ │ for 2 seconds and
│ │ │ │ test pass if its
│ │ │ │ exceeds min time
│ │ │ │ of 1.0 sec
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
timelimit_max │ script │ generic.local.sh │ tutorials │ Run a sleep job
│ │ │ │ for 2 seconds and
│ │ │ │ test pass if it's
│ │ │ │ within max time:
│ │ │ │ 5.0 sec
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
timelimit_min_fail │ script │ generic.local.sh │ tutorials │ This test fails
│ │ │ │ because it runs
│ │ │ │ less than mintime
│ │ │ │ of 10 second
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
timelimit_max_fail │ script │ generic.local.sh │ tutorials │ This test fails
│ │ │ │ because it
│ │ │ │ exceeds maxtime
│ │ │ │ of 1.0 second
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
metric_regex_examp │ script │ generic.local.sh │ tutorials │ capture result
le │ │ │ │ metric from
│ │ │ │ output
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
_bin_sh_shell │ script │ generic.local.sh │ tutorials │ /bin/sh shell
│ │ │ │ example
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
sh_shell │ script │ generic.local.sh │ tutorials │ sh shell example
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
shell_options │ script │ generic.local.sh │ tutorials │ shell options
We can filter output of buildspec cache by buildspec using --filter buildspec=<path> which
expects a path to buildspec file. The buildspec must be in the cache and file path must exist in order to
fetch the result. The path can be absolute or relative path.
In this next example, we will filter cache by file tutorials/pass_returncode.yml and use --format name,buildspec
to format columns. The --format buildspec will show full path to buildspec and name refers to name of test.
For more details on –format see Format buildspec cache.
$ buildtest buildspec find --filter buildspec=tutorials/pass_returncode.yml --format name,buildspec
Buildspec Cache: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/chec
kouts/v0.14.0/var/buildspecs/cache.json
name ┃ buildspec
━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
exit1_fail │ /home/docs/checkouts/readthedocs.org/user_builds/bu
│ ildtest/checkouts/v0.14.0/tutorials/pass_returncode
│ .yml
──────────────────────────┼─────────────────────────────────────────────────────
exit1_pass │ /home/docs/checkouts/readthedocs.org/user_builds/bu
│ ildtest/checkouts/v0.14.0/tutorials/pass_returncode
│ .yml
──────────────────────────┼─────────────────────────────────────────────────────
returncode_list_mismatch │ /home/docs/checkouts/readthedocs.org/user_builds/bu
│ ildtest/checkouts/v0.14.0/tutorials/pass_returncode
│ .yml
──────────────────────────┼─────────────────────────────────────────────────────
returncode_int_match │ /home/docs/checkouts/readthedocs.org/user_builds/bu
│ ildtest/checkouts/v0.14.0/tutorials/pass_returncode
│ .yml
Format buildspec cache¶
We have seen how one can filter buildspecs, but we can also configure which columns to display in the output of buildtest buildspec find. By default, we show a pre-selected format fields in the output, however there are more format fields available that can be configured at the command line.
The format fields are specified in comma separated format such as buildtest buildspec find --format <field1>,<field2>,....
You can see a list of all format fields by --helpformat option as shown below
$ buildtest buildspec find --helpformat
Format Field Description
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Field ┃ Description ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildspec │ Display name of buildspec file │
│ description │ Show description of test │
│ executor │ Display 'executor' property in test │
│ name │ Display name of test │
│ tags │ Display 'tag' property in test │
│ type │ Display 'type' property in test │
└─────────────┴─────────────────────────────────────┘
In the next example, we utilize --format field with --filter option to show
how format fields affect table columns. buildtest will display the table in order of
format fields specified in command line.
$ buildtest buildspec find --format name,description,buildspec --filter tags=tutorials,executor=generic.local.sh
Buildspec Cache: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/chec
kouts/v0.14.0/var/buildspecs/cache.json
name ┃ description ┃ buildspec
━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━
foo_bar │ prints variable $FOO │ /home/docs/checkouts/readt
│ │ hedocs.org/user_builds/bui
│ │ ldtest/checkouts/v0.14.0/t
│ │ utorials/maintainers_examp
│ │ le.yml
──────────────────────┼────────────────────────────┼────────────────────────────
timelimit_min_max │ Run a sleep job for 2 │ /home/docs/checkouts/readt
│ seconds and test pass if │ hedocs.org/user_builds/bui
│ its within 1.0-3.0sec │ ldtest/checkouts/v0.14.0/t
│ │ utorials/runtime_status_te
│ │ st.yml
──────────────────────┼────────────────────────────┼────────────────────────────
timelimit_min │ Run a sleep job for 2 │ /home/docs/checkouts/readt
│ seconds and test pass if │ hedocs.org/user_builds/bui
│ its exceeds min time of │ ldtest/checkouts/v0.14.0/t
│ 1.0 sec │ utorials/runtime_status_te
│ │ st.yml
──────────────────────┼────────────────────────────┼────────────────────────────
timelimit_max │ Run a sleep job for 2 │ /home/docs/checkouts/readt
│ seconds and test pass if │ hedocs.org/user_builds/bui
│ it's within max time: 5.0 │ ldtest/checkouts/v0.14.0/t
│ sec │ utorials/runtime_status_te
│ │ st.yml
──────────────────────┼────────────────────────────┼────────────────────────────
timelimit_min_fail │ This test fails because it │ /home/docs/checkouts/readt
│ runs less than mintime of │ hedocs.org/user_builds/bui
│ 10 second │ ldtest/checkouts/v0.14.0/t
│ │ utorials/runtime_status_te
│ │ st.yml
──────────────────────┼────────────────────────────┼────────────────────────────
timelimit_max_fail │ This test fails because it │ /home/docs/checkouts/readt
│ exceeds maxtime of 1.0 │ hedocs.org/user_builds/bui
│ second │ ldtest/checkouts/v0.14.0/t
│ │ utorials/runtime_status_te
│ │ st.yml
──────────────────────┼────────────────────────────┼────────────────────────────
metric_regex_example │ capture result metric from │ /home/docs/checkouts/readt
│ output │ hedocs.org/user_builds/bui
│ │ ldtest/checkouts/v0.14.0/t
│ │ utorials/metrics_regex.yml
──────────────────────┼────────────────────────────┼────────────────────────────
_bin_sh_shell │ /bin/sh shell example │ /home/docs/checkouts/readt
│ │ hedocs.org/user_builds/bui
│ │ ldtest/checkouts/v0.14.0/t
│ │ utorials/shell_examples.ym
│ │ l
──────────────────────┼────────────────────────────┼────────────────────────────
sh_shell │ sh shell example │ /home/docs/checkouts/readt
│ │ hedocs.org/user_builds/bui
│ │ ldtest/checkouts/v0.14.0/t
│ │ utorials/shell_examples.ym
│ │ l
──────────────────────┼────────────────────────────┼────────────────────────────
shell_options │ shell options │ /home/docs/checkouts/readt
│ │ hedocs.org/user_builds/bui
│ │ ldtest/checkouts/v0.14.0/t
│ │ utorials/shell_examples.ym
│ │ l
Querying buildspec tags¶
If you want to retrieve all unique tags from all buildspecs you can run
buildtest buildspec find --tags. This can be useful if you want to know available
tags in your buildspec cache.
$ buildtest buildspec find --tags
List of Tags
┏━━━━━━━━━━━━━━━┓
┃ Tags ┃
┡━━━━━━━━━━━━━━━┩
│ python │
│ fail │
│ pass │
│ slurm │
│ containers │
│ filesystem │
│ system │
│ configuration │
│ lsf │
│ network │
│ storage │
│ cobalt │
│ singularity │
│ ping │
│ tutorials │
└───────────────┘
In addition, buildtest can group tests by tags via buildtest buildspec find --group-by-tags
which can be useful if you want to know which tests get executed when running buildtest build --tags.
The output is grouped by tag names, followed by name of test and description.
$ buildtest buildspec find --group-by-tags
Tests by Tags
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Tags ┃ Name ┃ Description ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ tutorials │ hello_world │ hello world example │
│ tutorials │ skip │ This test is skipped │
│ tutorials │ unskipped │ This test is not skipped │
│ tutorials │ foo_bar │ prints variable $FOO │
│ tutorials │ circle_area │ Calculate circle of area │
│ │ │ given a radius │
│ tutorials │ add_numbers │ Add X+Y │
│ tutorials │ bash_login_shebang │ customize shebang line with │
│ │ │ bash login shell │
│ tutorials │ bash_nonlogin_shebang │ customize shebang line with │
│ │ │ default bash (nonlogin) shell │
│ tutorials │ executor_regex_script_schema │ regular expression test with │
│ │ │ executor using script schema │
│ tutorials │ bash_env_variables │ Declare environment variables │
│ │ │ in default shell (bash) │
│ tutorials │ csh_env_declaration │ csh shell example to declare │
│ │ │ environment variables │
│ tutorials │ tcsh_env_declaration │ tcsh shell example to declare │
│ │ │ environment variables │
│ tutorials │ sleep │ sleep 2 seconds │
│ tutorials │ selinux_disable │ Check if SELinux is Disabled │
│ tutorials │ timelimit_min_max │ Run a sleep job for 2 seconds │
│ │ │ and test pass if its within │
│ │ │ 1.0-3.0sec │
│ tutorials │ timelimit_min │ Run a sleep job for 2 seconds │
│ │ │ and test pass if its exceeds │
│ │ │ min time of 1.0 sec │
│ tutorials │ timelimit_max │ Run a sleep job for 2 seconds │
│ │ │ and test pass if it's within │
│ │ │ max time: 5.0 sec │
│ tutorials │ timelimit_min_fail │ This test fails because it │
│ │ │ runs less than mintime of 10 │
│ │ │ second │
│ tutorials │ timelimit_max_fail │ This test fails because it │
│ │ │ exceeds maxtime of 1.0 second │
│ tutorials │ metric_regex_example │ capture result metric from │
│ │ │ output │
...
Querying buildspec executor¶
If you want to know all executors in your buildspec cache use the
buildtest buildspec find --executors command. This can be useful when
you want to build by executors (buildtest build --executor).
$ buildtest buildspec find --executors
List of Executors
┏━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Executors ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ generic.local.bash │
│ generic.local.csh │
│ generic.local.(bash|sh) │
│ generic.local.sh │
└─────────────────────────┘
Similar to --group-by-tags, buildtest has an option to group tests by executor
using --group-by-executor option. This will show tests grouped by executor,
name of test and test description. Shown below is an example output.
$ buildtest buildspec find --group-by-executor
Tests by Executors
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Executors ┃ Name ┃ Description ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ generic.local.bash │ hello_world │ hello world example │
│ generic.local.bash │ skip │ This test is skipped │
│ generic.local.bash │ unskipped │ This test is not │
│ │ │ skipped │
│ generic.local.bash │ gcc_version │ Print gcc version │
│ generic.local.bash │ circle_area │ Calculate circle of │
│ │ │ area given a radius │
│ generic.local.bash │ add_numbers │ Add X+Y │
│ generic.local.bash │ bash_login_shebang │ customize shebang line │
│ │ │ with bash login shell │
│ generic.local.bash │ bash_nonlogin_shebang │ customize shebang line │
│ │ │ with default bash │
│ │ │ (nonlogin) shell │
│ generic.local.bash │ python_hello │ Hello World python │
│ generic.local.bash │ bash_env_variables │ Declare environment │
│ │ │ variables in default │
│ │ │ shell (bash) │
│ generic.local.bash │ sleep │ sleep 2 seconds │
│ generic.local.bash │ string_tag │ tags can be a string │
│ generic.local.bash │ list_of_strings_tags │ tags can be a list of │
│ │ │ strings │
│ generic.local.bash │ download_stream │ │
│ generic.local.bash │ run_stream │ job dependency on │
│ │ │ download_stream │
│ generic.local.bash │ selinux_disable │ Check if SELinux is │
│ │ │ Disabled │
│ generic.local.bash │ _bin_bash_shell │ /bin/bash shell example │
...
Query Maintainers¶
When you are writing your buildspecs, you can specify the maintainers field to assign
authors to buildspecs. buildtest can query the maintainers from the cache
once buildspecs are loaded. You can retrieve all maintainers using --maintainers option or -m
short option. In this example, we show all maintainers for buildspecs in buildspec
cache
$ buildtest buildspec find --maintainers
List of Maintainers
┏━━━━━━━━━━━━━━━━━━┓
┃ Maintainers ┃
┡━━━━━━━━━━━━━━━━━━┩
│ @shahzebsiddiqui │
│ @johndoe │
│ @bobsmith │
└──────────────────┘
If you want to see a breakdown of maintainers by buildspec file you can use --maintainers-by-buildspecs
or -mb short option. This can be useful to get correlation between maintainers and the buildspec file.
$ buildtest buildspec find -mb
Breakdown of buildspecs by maintainers
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Maintainers ┃ Buildspec ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ @shahzebsiddiqui │ /home/docs/checkouts/readthedocs.org/user_builds/buildte… │
│ @johndoe │ /home/docs/checkouts/readthedocs.org/user_builds/buildte… │
│ @bobsmith │ /home/docs/checkouts/readthedocs.org/user_builds/buildte… │
└──────────────────┴───────────────────────────────────────────────────────────┘
Terse Output¶
You can use the --terse option to print output of buildtest buildspec find in terse format that can
be useful if you want to parse content of file. In example below, we will print output of tags in terse format, the
first entry tags is the header followed by list of unique tags. The --no-header option
can be used to disable printing of header title.
$ buildtest buildspec find -t --terse
tag
python
fail
pass
slurm
containers
filesystem
system
configuration
lsf
network
storage
cobalt
singularity
ping
tutorials
Invalid Buildspecs - buildtest buildspec find invalid¶
buildtest will store invalid buildspecs in the cache file which can be retrieved using buildtest buildspec find invalid. buildtest
will attempt to parse each buildspec and store error message for every buildspec. If you run without any options it will
report a list of invalid buildspecs as shown below
$ buildtest buildspec find invalid
Invalid Buildspecs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Buildspec ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
└──────────────────────────────────────────────────────────────────────────────┘
If you want to see error messages for each buildspec you can pass the -e or --error option which will display output of
each buildspec followed by error message.
$ buildtest buildspec find invalid -e
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… ─
"'[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.0/tutorials/invalid_buildspec_section.yml]: type badscript is not known to buildtest.'"
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… ─
"['network', 'network'] is not valid under any of the given schemas\n\nFailed validating 'oneOf' in schema['properties']['tags']:\n {'oneOf': [{'type': 'string'},\n {'$ref': '#/definitions/list_of_strings'}]}\n\nOn instance['tags']:\n ['network', 'network']"
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… ─
"'create_burst_buffer_multiple_executors' is too long\n\nFailed validating 'maxLength' in schema['properties']['buildspecs']['propertyNames']:\n {'maxLength': 32, 'pattern': '^[A-Za-z_.][A-Za-z0-9_.]*$'}\n\nOn instance['buildspecs']:\n 'create_burst_buffer_multiple_executors'"
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… ─
'"[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.0/tutorials/invalid_executor.yml]: Unable to find executor: badexecutor in [\'generic.local.bash\', \'generic.local.sh\', \'generic.local.csh\', \'generic.local.zsh\']"'
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… ─
'"[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.0/general_tests/sched/pbs/hostname.yml]: Unable to find executor: generic.pbs.workq in [\'generic.local.bash\', \'generic.local.sh\', \'generic.local.csh\', \'generic.local.zsh\']"'
Cache Summary - buildtest buildspec summary¶
The buildtest buildspec summary command can be used to provide a summary of the buildspec cache. This command
can be used assuming your cache is built via buildtest buildspec find. Shown below is a summary of the cache file.
$ buildtest buildspec summary
╭──────────────────────────────────────────────────────────────────────────────╮
│ │
│ Reading Buildspec Cache File: /home/docs/checkouts/readthedocs.org/use │
│ r_builds/buildtest/checkouts/v0.14.0/var/buildspecs/cache.json │
│ Total Valid Buildspecs: 47 │
│ Total Invalid Buildspecs: 5 │
│ Total Unique Tags: 15 │
│ Total Maintainers: 3 │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
Tag Breakdown Executor Breakdown Maintainers Breakdown
┏━━━━━━━━━━━┳━━━━━━━━━━━━┓┏━━━━━━━━━━━━┳━━━━━━━━━━━━┓┏━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ ┃ total ┃┃ ┃ total ┃┃ ┃ total ┃
┃ tag ┃ tests ┃┃ executor ┃ tests ┃┃ maintaine… ┃ buildspecs ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━┩┡━━━━━━━━━━━━╇━━━━━━━━━━━━┩┡━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ tutorials │ 34 ││ generic.l… │ 84 ││ @shahzebs… │ 2 │
│ python │ 2 ││ generic.l… │ 21 ││ @johndoe │ 1 │
│ network │ 2 ││ generic.l… │ 4 ││ @bobsmith │ 1 │
│ ping │ 1 ││ generic.l… │ 3 │└────────────┴────────────┘
│ fail │ 2 │└────────────┴────────────┘
│ pass │ 2 │
│ system │ 9 │
│ filesyst… │ 1 │
│ storage │ 1 │
│ configur… │ 1 │
│ slurm │ 17 │
│ lsf │ 12 │
│ cobalt │ 7 │
│ containe… │ 8 │
│ singular… │ 8 │
└───────────┴────────────┘
Invalid Buildspecs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Buildspecs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
├──────────────────────────────────────────────────────────────────────────────┤
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
├──────────────────────────────────────────────────────────────────────────────┤
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
├──────────────────────────────────────────────────────────────────────────────┤
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
├──────────────────────────────────────────────────────────────────────────────┤
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… │
└──────────────────────────────────────────────────────────────────────────────┘
Test Breakdown by buildspec
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Tests ┃ Total ┃ Buildspec ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ hello_world │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ skip │ 2 │ /home/docs/checkouts/readthedocs.o… │
│ unskipped │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ gcc_version │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ foo_bar │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ circle_area │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ add_numbers │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ bash_login_shebang │ 2 │ /home/docs/checkouts/readthedocs.o… │
│ bash_nonlogin_shebang │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ python_hello │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ executor_regex_script_schema │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ bash_env_variables │ 3 │ /home/docs/checkouts/readthedocs.o… │
│ csh_env_declaration │ │ │
│ tcsh_env_declaration │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ sleep │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ string_tag │ 2 │ /home/docs/checkouts/readthedocs.o… │
│ list_of_strings_tags │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ always_pass │ 4 │ /home/docs/checkouts/readthedocs.o… │
│ always_fail │ │ │
│ test_fail_returncode_match │ │ │
│ test_pass_returncode_mismatch │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ download_stream │ 2 │ /home/docs/checkouts/readthedocs.o… │
│ run_stream │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ selinux_disable │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ timelimit_min_max │ 5 │ /home/docs/checkouts/readthedocs.o… │
│ timelimit_min │ │ │
│ timelimit_max │ │ │
│ timelimit_min_fail │ │ │
│ timelimit_max_fail │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ metric_regex_example │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ _bin_sh_shell │ 5 │ /home/docs/checkouts/readthedocs.o… │
│ _bin_bash_shell │ │ │
│ bash_shell │ │ │
│ sh_shell │ │ │
│ shell_options │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ csh_shell │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ exit1_fail │ 4 │ /home/docs/checkouts/readthedocs.o… │
│ exit1_pass │ │ │
│ returncode_list_mismatch │ │ │
│ returncode_int_match │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ variables_bash │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ status_regex_pass │ 2 │ /home/docs/checkouts/readthedocs.o… │
│ status_regex_fail │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ executors_vars_env_declaration │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ status_returncode_by_executors │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ executors_sbatch_declaration │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ test1 │ 3 │ /home/docs/checkouts/readthedocs.o… │
│ test2 │ │ │
│ test3 │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ runtime_test │ 3 │ /home/docs/checkouts/readthedocs.o… │
│ runtime_test_pass │ │ │
│ runtime_test_fail │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ jobA │ 3 │ /home/docs/checkouts/readthedocs.o… │
│ jobB │ │ │
│ jobC │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ pass_test │ 4 │ /home/docs/checkouts/readthedocs.o… │
│ fail_test │ │ │
│ pass_and_fail_test │ │ │
│ final_test │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ root_disk_usage │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ systemd_default_target │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ kernel_swapusage │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ ulimit_filelock_unlimited │ 6 │ /home/docs/checkouts/readthedocs.o… │
│ ulimit_cputime_unlimited │ │ │
│ ulimit_stacksize_unlimited │ │ │
│ ulimit_vmsize_unlimited │ │ │
│ ulimit_filedescriptor_4096 │ │ │
│ ulimit_max_user_process_2048 │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ current_user_queue │ 2 │ /home/docs/checkouts/readthedocs.o… │
│ show_all_jobs │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ nodes_state_down │ 9 │ /home/docs/checkouts/readthedocs.o… │
│ nodes_state_reboot │ │ │
│ nodes_state_allocated │ │ │
│ nodes_state_completing │ │ │
│ nodes_state_idle │ │ │
│ node_down_fail_list_reason │ │ │
│ dead_nodes │ │ │
│ get_partitions │ │ │
│ sinfo_version │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ show_accounts │ 4 │ /home/docs/checkouts/readthedocs.o… │
│ show_users │ │ │
│ show_qos │ │ │
│ show_tres │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ slurm_config │ 2 │ /home/docs/checkouts/readthedocs.o… │
│ show_partition │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ show_lsf_configuration │ 4 │ /home/docs/checkouts/readthedocs.o… │
│ show_lsf_models │ │ │
│ show_lsf_resources │ │ │
│ lsf_version │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ show_lsf_user_groups │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ display_lsf_hosts │ 3 │ /home/docs/checkouts/readthedocs.o… │
│ display_hosts_format │ │ │
│ bhosts_version │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ show_host_groups │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ show_lsf_queues │ 3 │ /home/docs/checkouts/readthedocs.o… │
│ show_lsf_queues_formatted │ │ │
│ show_lsf_queues_current_user │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ qsub_version │ 7 │ /home/docs/checkouts/readthedocs.o… │
│ qselect_version │ │ │
│ cqsub_version │ │ │
│ qdel_version │ │ │
│ qmove_version │ │ │
│ show_jobs │ │ │
│ show_queues │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ inspect_image │ 1 │ /home/docs/checkouts/readthedocs.o… │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ pullImage_dockerhub │ 3 │ /home/docs/checkouts/readthedocs.o… │
│ pullImage_sylabscloud │ │ │
│ pullImage_shub │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ build_sif_from_dockerimage │ 3 │ /home/docs/checkouts/readthedocs.o… │
│ build_sandbox_image │ │ │
│ build_remoteimages │ │ │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ runImage │ 1 │ /home/docs/checkouts/readthedocs.o… │
└────────────────────────────────┴───────┴─────────────────────────────────────┘
Validate Buildspecs - buildtest buildspec validate¶
buildtest can validate buildspecs through the buildtest buildspec validate command which provides
analogous options for buildtest build for selecting buildspecs such as -b, -e, -t and -e.
This command can be used to validate buildspecs with the JSON Schema which can be useful if you are writing a buildspec
and want to validate the buildspec without running the test.
Shown below are the available command options.
$ buildtest buildspec validate --help
usage: buildtest [options] [COMMANDS] buildspec validate [-h] [-b BUILDSPEC]
[-x EXCLUDE]
[-e EXECUTOR]
[-t TAG]
optional arguments:
-h, --help show this help message and exit
-b BUILDSPEC, --buildspec BUILDSPEC
Specify path to buildspec (file, or directory) to
validate
-x EXCLUDE, --exclude EXCLUDE
Specify path to buildspec to exclude (file or
directory) during validation
-e EXECUTOR, --executor EXECUTOR
Specify buildspecs by executor name to validate
-t TAG, --tag TAG Specify buildspecs by tag name to validate
The -b option can be used to specify path to buildspec file or directory to validate buildspecs. If its a directory, buildtest will traverse all directories recursively and find any .yml file extensions and attempt to validate each buildspec. Shown below is an example output of what it may look like
$ buildtest buildspec validate -b tutorials/vars.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.0/tutorials/vars.yml
All buildspecs passed validation!!!
If buildtest detects an error during validation, the error message will be displayed to screen as we see in this example
$ buildtest buildspec validate -b tutorials/invalid_tags.yml
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… ─
<ValidationError: "['network', 'network'] is not valid under any of the given schemas">
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.0/tutorials/invalid_tags.yml
1 buildspecs failed to validate
Similarly we can search buildspecs based on tags if you want to validate a group of buildspecs using the -t option. We can
append -t option multiple times to search by multiple tag names. In this next example, we
will validate all buildspecs for python and pass tags.
$ buildtest buildspec validate -t python -t pass
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.0/tutorials/python-hello.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.0/tutorials/python-shell.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.0/tutorials/pass_returncode.yml
All buildspecs passed validation!!!
Show buildspec buildtest buildspec show¶
buildtest can display content of buildspec file given a test name via buildtest buildspec show command which expects a
positional argument that is the name of test. This can be quick way to see content of buildspec without remembering the full path
to the buildspec.
In this next example, we will instruct buildtest to show content of buildspec for test name python_hello.
$ buildtest buildspec show python_hello
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… ─
╭─────────────────────────────────────╮
│ buildspecs: │
│ python_hello: │
│ type: script │
│ description: Hello World python │
│ executor: generic.local.bash │
│ tags: python │
│ run: python hello.py │
│ │
│ │
╰─────────────────────────────────────╯
You can pass multiple arguments to buildtest buildspec show to show content of each test
$ buildtest buildspec show python_hello circle_area
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… ─
╭─────────────────────────────────────╮
│ buildspecs: │
│ python_hello: │
│ type: script │
│ description: Hello World python │
│ executor: generic.local.bash │
│ tags: python │
│ run: python hello.py │
│ │
│ │
╰─────────────────────────────────────╯
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.… ─
╭────────────────────────────────────────────────────────────╮
│ buildspecs: │
│ circle_area: │
│ executor: generic.local.bash │
│ type: script │
│ shell: python │
│ description: "Calculate circle of area given a radius" │
│ tags: │
│ run: | │
│ import math │
│ radius = 2 │
│ area = math.pi * radius * radius │
│ print("Circle Radius ", radius) │
│ print("Area of circle ", area) │
│ │
╰────────────────────────────────────────────────────────────╯
There is bash completion for this command which will show list of test names available in the cache assuming you have run
buildtest buildspec find. If you specify an invalid test name you will get an error followed by list of tests that are available
in the cache
$ buildtest buildspec show XYZ123!
Unable to find test XYZ123! in cache
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.0 │
│ /bin/buildtest:36 in <module> │
│ │
│ 33 │
│ 34 import buildtest.main │
│ 35 │
│ ❱ 36 buildtest.main.main() │
│ 37 │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ __annotations__ = {} │ │
│ │ __builtins__ = <module 'builtins' (built-in)> │ │
│ │ __cached__ = None │ │
│ │ __doc__ = ':"\n# preferred pythons for running │ │
│ │ buildtest\nPREFERRED_PYTHONS="python3 python"\n'+365 │ │
│ │ __file__ = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │ __loader__ = <_frozen_importlib_external.SourceFileLoader object at │ │
│ │ 0x7fdd0377b1d0> │ │
│ │ __name__ = '__main__' │ │
│ │ __package__ = None │ │
│ │ __spec__ = None │ │
│ │ buildtest = <module 'buildtest' from │ │
│ │ '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │ buildtest_file = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │ os = <module 'os' from │ │
│ │ '/home/docs/.asdf/installs/python/3.7.12/lib/python3.… │ │
│ │ prefix = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │ sys = <module 'sys' (built-in)> │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.0 │
│ /buildtest/main.py:160 in main │
│ │
│ 157 │ │ elif args.buildspecs_subcommand == "summary": │
│ 158 │ │ │ summarize_buildspec_cache(configuration) │
│ 159 │ │ elif args.buildspecs_subcommand == "show": │
│ ❱ 160 │ │ │ show_buildspecs(test_names=args.name, configuration=config │
│ 161 │ │ elif args.buildspecs_subcommand == "edit": │
│ 162 │ │ │ edit_buildspec_test(test_names=args.name, configuration=co │
│ 163 │ │ elif args.buildspecs_subcommand == "edit-file": │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ args = Namespace(buildspecs_subcommand='show', │ │
│ │ configfile=None, debug=False, lastlog=False, │ │
│ │ name=['XYZ123!'], no_color=False, report=None, │ │
│ │ subcommands='buildspec') │ │
│ │ config_file = None │ │
│ │ configuration = <buildtest.config.SiteConfiguration object at │ │
│ │ 0x7fdd02b24a10> │ │
│ │ logger = <Logger buildtest (DEBUG)> │ │
│ │ no_color = False │ │
│ │ parser = ArgumentParser(prog='buildtest', usage='%(prog)s │ │
│ │ [options] [COMMANDS]', description='buildtest is a │ │
│ │ HPC testing framework for building and running │ │
│ │ tests.', formatter_class=<class │ │
│ │ 'argparse.RawDescriptionHelpFormatter'>, │ │
│ │ conflict_handler='error', add_help=True) │ │
│ │ report_file = None │ │
│ │ system = <buildtest.system.BuildTestSystem object at │ │
│ │ 0x7fdd01741c10> │ │
│ │ validate_executors = True │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.14.0 │
│ /buildtest/cli/buildspec.py:998 in show_buildspecs │
│ │
│ 995 │ │
│ 996 │ if error: │
│ 997 │ │ raise BuildTestError( │
│ ❱ 998 │ │ │ f"Please select one of the following test: {cache.get_nam │
│ 999 │ │ ) │
│ 1000 │
│ 1001 │
│ │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ cache = <buildtest.cli.buildspec.BuildspecCache object at │ │
│ │ 0x7fdd01b4cc90> │ │
│ │ configuration = <buildtest.config.SiteConfiguration object at │ │
│ │ 0x7fdd02b24a10> │ │
│ │ error = True │ │
│ │ name = 'XYZ123!' │ │
│ │ test_names = ['XYZ123!'] │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
BuildTestError: "Please select one of the following test: ['hello_world',
'skip', 'unskipped', 'gcc_version', 'foo_bar', 'circle_area', 'add_numbers',
'bash_login_shebang', 'bash_nonlogin_shebang', 'python_hello',
'executor_regex_script_schema', 'bash_env_variables', 'csh_env_declaration',
'tcsh_env_declaration', 'sleep', 'string_tag', 'list_of_strings_tags',
'always_pass', 'always_fail', 'test_fail_returncode_match',
'test_pass_returncode_mismatch', 'download_stream', 'run_stream',
'selinux_disable', 'timelimit_min_max', 'timelimit_min', 'timelimit_max',
'timelimit_min_fail', 'timelimit_max_fail', 'metric_regex_example',
'_bin_sh_shell', '_bin_bash_shell', 'bash_shell', 'sh_shell', 'shell_options',
'csh_shell', 'exit1_fail', 'exit1_pass', 'returncode_list_mismatch',
'returncode_int_match', 'variables_bash', 'status_regex_pass',
'status_regex_fail', 'executors_vars_env_declaration',
'status_returncode_by_executors', 'executors_sbatch_declaration', 'test1',
'test2', 'test3', 'runtime_test', 'runtime_test_pass', 'runtime_test_fail',
'jobA', 'jobB', 'jobC', 'pass_test', 'fail_test', 'pass_and_fail_test',
'final_test', 'root_disk_usage', 'systemd_default_target', 'kernel_swapusage',
'ulimit_filelock_unlimited', 'ulimit_cputime_unlimited',
'ulimit_stacksize_unlimited', 'ulimit_vmsize_unlimited',
'ulimit_filedescriptor_4096', 'ulimit_max_user_process_2048',
'current_user_queue', 'show_all_jobs', 'nodes_state_down', 'nodes_state_reboot',
'nodes_state_allocated', 'nodes_state_completing', 'nodes_state_idle',
'node_down_fail_list_reason', 'dead_nodes', 'get_partitions', 'sinfo_version',
'show_accounts', 'show_users', 'show_qos', 'show_tres', 'slurm_config',
'show_partition', 'show_lsf_configuration', 'show_lsf_models',
'show_lsf_resources', 'lsf_version', 'show_lsf_user_groups',
'display_lsf_hosts', 'display_hosts_format', 'bhosts_version',
'show_host_groups', 'show_lsf_queues', 'show_lsf_queues_formatted',
'show_lsf_queues_current_user', 'qsub_version', 'qselect_version',
'cqsub_version', 'qdel_version', 'qmove_version', 'show_jobs', 'show_queues',
'inspect_image', 'pullImage_dockerhub', 'pullImage_sylabscloud',
'pullImage_shub', 'build_sif_from_dockerimage', 'build_sandbox_image',
'build_remoteimages', 'runImage']"
Let’s assume you want to see content of all buildspec that failed test, you can use buildtest report to extract list of test names
that failed as argument to buildtest buildspec show, for instance we see below we have two tests that failed, using --filter state=FAIL will
retrieve all failures, you will want to pipe output to uniq to get unique listing since output of buildtest report will show all test runs and
duplicates can occur if same test is run multiple times.
$ buildtest report --filter state=FAIL --format name --terse --no-header | uniq
exit1_fail
returncode_list_mismatch
Next you can use this as argument to buildtest buildspec show and it will report all buildspec files that correspond to test that failed.
$ buildtest buildspec show $(buildtest report --filter state=FAIL --format name --terse --no-header | uniq)
────────────────────────────────────────────────────────────────────────────── /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/pass_returncode.yml ───────────────────────────────────────────────────────────────────────────────
╭──────────────────────────────────────────────────────────────────────╮
│ buildspecs: │
│ │
│ exit1_fail: │
│ executor: generic.local.bash │
│ type: script │
│ description: exit 1 by default is FAIL │
│ tags: │
│ run: exit 1 │
│ │
│ exit1_pass: │
│ executor: generic.local.bash │
│ type: script │
│ description: report exit 1 as PASS │
│ run: exit 1 │
│ tags: │
│ status: │
│ returncode: [1] │
│ │
│ returncode_list_mismatch: │
│ executor: generic.local.bash │
│ type: script │
│ description: exit 2 failed since it failed to match returncode 1 │
│ run: exit 2 │
│ tags: │
│ status: │
│ returncode: [1, 3] │
│ │
│ returncode_int_match: │
│ executor: generic.local.bash │
│ type: script │
│ description: exit 128 matches returncode 128 │
│ run: exit 128 │
│ tags: │
│ status: │
│ returncode: 128 │
│ │
╰──────────────────────────────────────────────────────────────────────╯
────────────────────────────────────────────────────────────────────────────── /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/pass_returncode.yml ───────────────────────────────────────────────────────────────────────────────
╭──────────────────────────────────────────────────────────────────────╮
│ buildspecs: │
│ │
│ exit1_fail: │
│ executor: generic.local.bash │
│ type: script │
│ description: exit 1 by default is FAIL │
│ tags: │
│ run: exit 1 │
│ │
│ exit1_pass: │
│ executor: generic.local.bash │
│ type: script │
│ description: report exit 1 as PASS │
│ run: exit 1 │
│ tags: │
│ status: │
│ returncode: [1] │
│ │
│ returncode_list_mismatch: │
│ executor: generic.local.bash │
│ type: script │
│ description: exit 2 failed since it failed to match returncode 1 │
│ run: exit 2 │
│ tags: │
│ status: │
│ returncode: [1, 3] │
│ │
│ returncode_int_match: │
│ executor: generic.local.bash │
│ type: script │
│ description: exit 128 matches returncode 128 │
│ run: exit 128 │
│ tags: │
│ status: │
│ returncode: 128 │
│ │
╰──────────────────────────────────────────────────────────────────────╯
Editing buildspecs in your preferred editor¶
buildtest provides an interface to automatically open your buildspecs in editor and validate them after closing file.
You are welcome to open your buildspec in your editor (vim, emacs, nano) but you won’t be able to validate the buildspec
unless you explicitly run the test or use buildtest buildspec validate to see if your buildspec is valid. buildtest comes
with two commands to edit your buildspecs buildtest buildspec edit and buildtest buildspec edit-file which we will
discuss below.
Editing by Test buildtest buildspec edit¶
The buildtest buildspec edit allows one to specify a list of test as positional
arguments to edit in your preferred editor. buildtest will provide tab completion for this
command to show all test available in cache which works similar to buildtest buildspec show command.
For instance, we can see the following test are available as part of command completion
$ buildtest buildspec edit
_bin_bash_shell download_stream nodes_state_down show_host_groups string_tag
_bin_sh_shell executor_regex_script_schema nodes_state_idle show_jobs systemd_default_target
add_numbers executors_sbatch_declaration nodes_state_reboot show_lsf_configuration tcsh_env_declaration
always_fail executors_vars_env_declaration pullImage_dockerhub show_lsf_models test_fail_returncode_match
always_pass exit1_fail pullImage_shub show_lsf_queues test_pass_returncode_mismatch
bash_env_variables exit1_pass pullImage_sylabscloud show_lsf_queues_current_user timelimit_max
bash_login_shebang foo_bar python_hello show_lsf_queues_formatted timelimit_max_fail
bash_nonlogin_shebang gcc_version qdel_version show_lsf_resources timelimit_min
bash_shell get_partitions qmove_version show_lsf_user_groups timelimit_min_fail
bhosts_version hello_world qselect_version show_partition timelimit_min_max
build_remoteimages inspect_image qsub_version show_qos ulimit_cputime_unlimited
build_sandbox_image jobA returncode_int_match show_queues ulimit_filedescriptor_4096
build_sif_from_dockerimage jobB returncode_list_mismatch show_tres ulimit_filelock_unlimited
circle_area jobC root_disk_usage show_users ulimit_max_user_process_2048
cqsub_version kernel_swapusage runImage sinfo_version ulimit_stacksize_unlimited
csh_env_declaration list_of_strings_tags run_stream skip ulimit_vmsize_unlimited
csh_shell lsf_version selinux_disable sleep unskipped
current_user_queue metric_regex_example sh_shell slurm_config variables_bash
dead_nodes node_down_fail_list_reason shell_options status_regex_fail
display_hosts_format nodes_state_allocated show_accounts status_regex_pass
display_lsf_hosts nodes_state_completing show_all_jobs status_returncode_by_executors
Let’s take for instance we want to edit the following test, buildtest will search the buildspec cache and find the buildspec file, open in editor and once changes are written to disk, the next file will be processed until all files are written to disk.
$ buildtest buildspec edit sleep _bin_bash_shell add_numbers
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml is valid
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/shell_examples.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/shell_examples.yml is valid
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/add_numbers.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/add_numbers.yml is valid
If you specify an invalid test, then buildtest will ignore the test and report a message and skip to next test as shown below
$ buildtest buildspec edit invalid_test sleep
Unable to find test invalid_test in cache
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml is valid
Edit buildspecs buildtest buildspec edit-file¶
The buildtest buildspec edit-file command can be used to edit buildspec based on filename as pose to testname.
This command works similar to buildtest buildspec edit where each file is open in editor and validated upon completion.
You can use this command to create new buildspec whereas buildtest buildspec edit only works on existing buildspecs loaded
in cache. You can pass multiple filenames as arguments if you want to edit several files.
$ buildtest buildspec edit-file $BUILDTEST_ROOT/tutorials/sleep.yml
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml is valid