Buildspecs Interface¶
Now that we learned how to build tests, in this section we will discuss how one can query a buildspec cache. In buildtest, one can load all buildspecs which is equivalent to validating all buildspecs with the appropriate schema. Buildtest will ignore all invalid buildspecs and store them in a separate file.
The buildtest buildspec find
command is used for finding buildspecs from buildspec
cache. This command is also used for generating the buildspec cache. Shown below is a list of options for
buildtest buildspec find
.
$ buildtest buildspec find --help
usage: buildtest [options] [COMMANDS] buildspec find [-h] [-b] [-e] [--group-by-tags] [--group-by-executor] [-m] [-mb]
[-p] [-t] [--filter FILTER] [--format FORMAT] [--helpfilter]
[--helpformat] [-n] [--terse] [-r] [--root ROOT]
...
positional arguments:
invalid Show invalid buildspecs
optional arguments:
-h, --help show this help message and exit
-r, --rebuild Rebuild buildspec cache and find all buildspecs again
--root ROOT Specify root buildspecs (directory) path to load buildspecs into buildspec cache.
filter and format:
filter and format options
--filter FILTER Filter buildspec cache with filter fields in format --filter key1=val1,key2=val2
--format FORMAT Format buildspec cache with format fields in format --format field1,field2,...
--helpfilter Show Filter fields for --filter option for filtering buildspec cache output
--helpformat Show Format fields for --format option for formatting buildspec cache output
terse:
terse options
-n, --no-header Print output without header in terse output
--terse Print output in machine readable format
query:
query options to retrieve from buildspec cache
-b, --buildspec Get all buildspec files from cache
-e, --executors get all unique executors from buildspecs
--group-by-tags Group tests by tag name
--group-by-executor Group tests by executor name
-m, --maintainers Get all maintainers for all buildspecs
-mb, --maintainers-by-buildspecs
Show maintainers breakdown by buildspecs
-p, --paths print all root buildspec paths
-t, --tags List all available tags
Finding Buildspecs - buildtest buildspec find
¶
To find all buildspecs you can run buildtest buildspec find
which will discover
all buildspecs by recursively searching all .yml extensions. buildtest will validate each
buildspec file with the json schema and buildtest will display all valid buildspecs in the output,
all invalid buildspecs will be stored in a file for post-processing.
$ buildtest buildspec find
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| name | type | executor | tags | description |
+==================================+==========+=========================+========================+================================================================================+
| skip | script | generic.local.bash | tutorials | This test is skipped |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| unskipped | script | generic.local.bash | tutorials | This test is not skipped |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| status_regex_pass | script | generic.local.bash | system | Pass test based on regular expression |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| status_regex_fail | script | generic.local.bash | system | Pass test based on regular expression |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| metric_regex_example | script | generic.local.sh | tutorials | capture result metric from output |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| executor_regex_script_schema | script | generic.local.(bash|sh) | tutorials | regular expression test with executor using script schema |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| add_numbers | script | generic.local.bash | tutorials | Add X+Y |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| python_hello | script | generic.local.bash | python | Hello World python |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| _bin_sh_shell | script | generic.local.sh | tutorials | /bin/sh shell example |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| _bin_bash_shell | script | generic.local.bash | tutorials | /bin/bash shell example |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| bash_shell | script | generic.local.bash | tutorials | bash shell example |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| sh_shell | script | generic.local.sh | tutorials | sh shell example |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| shell_options | script | generic.local.sh | tutorials | shell options |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| bash_login_shebang | script | generic.local.bash | tutorials | customize shebang line with bash login shell |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| bash_nonlogin_shebang | script | generic.local.bash | tutorials | customize shebang line with default bash (nonlogin) shell |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| run_only_macos_distro | script | generic.local.bash | mac | Run test only if distro is darwin. |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| run_only_linux_distro | script | generic.local.bash | mac | Run test only if distro is CentOS. |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| metric_variable_assignment | script | generic.local.sh | tutorials | capture result metric based on variables and environment variable |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| selinux_disable | script | generic.local.bash | tutorials | Check if SELinux is Disabled |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| exit1_fail | script | generic.local.sh | tutorials fail | exit 1 by default is FAIL |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| exit1_pass | script | generic.local.sh | tutorials pass | report exit 1 as PASS |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| returncode_list_mismatch | script | generic.local.sh | tutorials fail | exit 2 failed since it failed to match returncode 1 |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| returncode_int_match | script | generic.local.sh | tutorials pass | exit 128 matches returncode 128 |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| sleep | script | generic.local.bash | tutorials | sleep 2 seconds |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| csh_shell | script | generic.local.csh | tutorials | csh shell example |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| string_tag | script | generic.local.bash | network | tags can be a string |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| list_of_strings_tags | script | generic.local.bash | network ping | tags can be a list of strings |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| bash_env_variables | script | generic.local.bash | tutorials | Declare environment variables in default shell (bash) |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| csh_env_declaration | script | generic.local.csh | tutorials | csh shell example to declare environment variables |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| tcsh_env_declaration | script | generic.local.csh | tutorials | tcsh shell example to declare environment variables |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| hello_world | script | generic.local.bash | tutorials | hello world example |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| foo_bar | script | generic.local.sh | tutorials | prints variable $FOO |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| variables_bash | script | generic.local.bash | tutorials | Declare shell variables in bash |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| run_only_platform_darwin | script | generic.local.python | tutorials | This test will only run if target platform is Darwin |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| run_only_platform_linux | script | generic.local.python | tutorials | This test will only run if target platform is Linux |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| circle_area | script | generic.local.python | tutorials python | Calculate circle of area given a radius |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| timelimit_min_max | script | generic.local.sh | tutorials | Run a sleep job for 2 seconds and test pass if its within 1.0-3.0sec |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| timelimit_min | script | generic.local.sh | tutorials | Run a sleep job for 2 seconds and test pass if its exceeds min time of 1.0 sec |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| timelimit_max | script | generic.local.sh | tutorials | Run a sleep job for 2 seconds and test pass if it's within max time: 5.0 sec |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| timelimit_min_fail | script | generic.local.sh | tutorials | This test fails because it runs less than mintime of 10 second |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| timelimit_max_fail | script | generic.local.sh | tutorials | This test fails because it exceeds maxtime of 1.0 second |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| run_only_as_root | script | generic.local.bash | tutorials | This test will only run if current user is root |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| spack_test | spack | generic.local.sh | spack | Install bzip2 and run spack test and report results |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| spack_env_directory | spack | generic.local.sh | spack | Concretize m4 in a spack environment named m4 |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| run_pre_post_commands | spack | generic.local.sh | spack | Install zlib |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| remove_environment_automatically | spack | generic.local.sh | spack | remove spack environment automatically before creating a new environment |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| remove_environment_explicit | spack | generic.local.sh | spack | remove spack environment explicitly using the 'rm' property |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| spack_test_results_specs_format | spack | generic.local.sh | spack | Run spack test results with spec format |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| concretize_m4_in_spack_env | spack | generic.local.sh | spack | Concretize m4 in a spack environment named m4 |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| spack_env_create_from_manifest | spack | generic.local.sh | spack | Create spack environment from spack.yaml |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| spack_sbatch_example | spack | generic.local.sh | spack | sbatch directives can be defined in spack schema |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| add_mirror | spack | generic.local.sh | spack | Declare spack mirror |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| add_mirror_in_spack_env | spack | generic.local.sh | spack | Declare spack mirror in spack environment |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| install_zlib | spack | generic.local.sh | spack | Install zlib |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| executors_sbatch_declaration | script | generic.local.(bash|sh) | tutorials | Declaring env and vars by executors section |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| status_returncode_by_executors | script | generic.local.(bash|sh) | tutorials | define status and metrics per executor type. |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| executors_vars_env_declaration | script | generic.local.(bash|sh) | tutorials | Declaring env and vars by executors section |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| openmp_hello_c_example | compiler | generic.local.bash | tutorials compile | OpenMP Hello World C example |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| default_status_regex | compiler | generic.local.bash | tutorials compile | Regular expression check in stdout for gcc group |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| override_status_regex | compiler | generic.local.bash | tutorials compile | Override regular expression for compiler gcc/10.2.0-37fmsw7 |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| override_environmentvars | compiler | generic.local.bash | tutorials compile | override default environment variables |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| custom_run_by_compilers | compiler | generic.local.bash | tutorials compile | Customize binary launch based on compiler |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| vecadd_gnu_exclude | compiler | generic.local.bash | tutorials compile | Vector Addition example with GNU compilers but exclude gcc@10.2.0 |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| hello_c | compiler | generic.local.bash | tutorials compile | Hello World C Compilation |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| vecadd_gnu | compiler | generic.local.bash | tutorials compile | Vector Addition example with GNU compiler |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| hello_f | compiler | generic.local.bash | tutorials compile | Hello World Fortran Compilation |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| pre_post_build_run | compiler | generic.local.bash | tutorials compile | example using pre_build, post_build, pre_run, post_run example |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| metrics_variable_compiler | compiler | generic.local.bash | tutorials compile | define metrics with compiler schema |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| show_lsf_user_groups | script | generic.local.bash | lsf | Show information about all LSF user groups |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| show_host_groups | script | generic.local.bash | lsf | Show information about host groups using bmgroup |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| show_lsf_queues | script | generic.local.bash | lsf | Show LSF queues |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| show_lsf_queues_formatted | script | generic.local.bash | lsf | Show LSF queues with formatted columns |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| show_lsf_queues_current_user | script | generic.local.bash | lsf | Show LSF queues available for current user |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| show_lsf_configuration | script | generic.local.bash | lsf | Show LSF configuration using lsinfo |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| show_lsf_models | script | generic.local.bash | lsf | Show information about host models in LSF cluster |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| show_lsf_resources | script | generic.local.bash | lsf | Show information about LSF resources |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| lsf_version | script | generic.local.bash | lsf | Display lsf version using lsinfo |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| display_lsf_hosts | script | generic.local.bash | lsf | Show all hosts in LSF cluster |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| display_hosts_format | script | generic.local.bash | lsf | Show all hosts with column hostname and status |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| bhosts_version | script | generic.local.bash | lsf | display version from bhosts command |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| current_user_queue | script | generic.local.bash | slurm | show all current pending jobs for current user (squeue -u $USER) |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| show_all_jobs | script | generic.local.bash | slurm | show all pending + running jobs (squeue -a) |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| show_accounts | script | generic.local.bash | slurm | run sacctmgr list accounts |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| show_users | script | generic.local.bash | slurm | run sacctmgr list users |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| show_qos | script | generic.local.bash | slurm | run sacctmgr list qos |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| show_tres | script | generic.local.bash | slurm | run sacctmgr list tres |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| slurm_config | script | generic.local.bash | slurm | run scontrol show config |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| show_partition | script | generic.local.bash | slurm | run scontrol show partition |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| nodes_state_down | script | generic.local.bash | slurm | Show nodes in DOWN state |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| nodes_state_reboot | script | generic.local.bash | slurm | Show nodes in REBOOT state |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| nodes_state_allocated | script | generic.local.bash | slurm | Show nodes in ALLOCATED state |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| nodes_state_completing | script | generic.local.bash | slurm | Show nodes in COMPLETING state |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| nodes_state_idle | script | generic.local.bash | slurm | Show nodes in IDLE state |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| node_down_fail_list_reason | script | generic.local.bash | slurm | Show nodes DOWN, DRAINED, FAIL or FAILING and list reason |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| dead_nodes | script | generic.local.bash | slurm | Show non-responding nodes |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| get_partitions | script | generic.local.bash | slurm | Get all slurm partitions |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| sinfo_version | script | generic.local.bash | slurm | get slurm version |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| qsub_version | script | generic.local.sh | cobalt | print version for qsub command |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| qselect_version | script | generic.local.sh | cobalt | print version for qselect |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| cqsub_version | script | generic.local.sh | cobalt | print version for cqsub command |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| qdel_version | script | generic.local.sh | cobalt | print version for qdel command |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| qmove_version | script | generic.local.sh | cobalt | print version for qmove command |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| show_jobs | script | generic.local.sh | cobalt | Show all jobs in queue |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| show_queues | script | generic.local.sh | cobalt | Show all queues |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| root_disk_usage | script | generic.local.bash | filesystem storage | Check root disk usage and report if it exceeds threshold |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| systemd_default_target | script | generic.local.bash | system | check if default target is multi-user.target |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| ssh_localhost_remotecommand | script | generic.local.bash | ssh | Test if ssh on localhost works and if we can run remote command. |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| kernel_swapusage | script | generic.local.bash | configuration | Retrieve Kernel Swap Usage |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| ulimit_filelock_unlimited | script | generic.local.bash | system | Check if file lock is set to unlimited in ulimits |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| ulimit_cputime_unlimited | script | generic.local.bash | system | Check if cputime is set to unlimited in ulimits |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| ulimit_stacksize_unlimited | script | generic.local.bash | system | Check if stack size is set to unlimited in ulimits |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| ulimit_vmsize_unlimited | script | generic.local.bash | system | Check virtual memory size and check if its set to unlimited |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| ulimit_filedescriptor_4096 | script | generic.local.bash | system | Check if open file descriptors limit is set to 4096 |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| ulimit_max_user_process_2048 | script | generic.local.bash | system | Check max number of user process limit is set to 2048 |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| runImage | script | generic.local.bash | containers singularity | run container docker://godlovedc/lolcow |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| build_sif_from_dockerimage | script | generic.local.bash | containers singularity | build sif image from docker image docker://godlovedc/lolcow |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| build_sandbox_image | script | generic.local.bash | containers singularity | build sandbox image from docker image docker://godlovedc/lolcow |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| build_remoteimages | script | generic.local.bash | containers singularity | build remote hosted image from AWS |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| pullImage_dockerhub | script | generic.local.bash | containers singularity | Pull image docker://godlovedc/lolcow from DockerHub |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| pullImage_sylabscloud | script | generic.local.bash | containers singularity | Pull image library://alpine:latest from Sylabs Cloud |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| pullImage_shub | script | generic.local.bash | containers singularity | Pull image shub://vsoch/singularity-images from SingularityHub |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| inspect_image | script | generic.local.bash | containers singularity | Inspect image via 'singularity inspect' |
+----------------------------------+----------+-------------------------+------------------------+--------------------------------------------------------------------------------+
buildtest will load all discovered buildspecs in a cache file (JSON) which is created upon
running buildtest buildspec find
. Any subsequent runs will read from cache and update
if any new buildspecs are added. If you make changes to buildspec you should rebuild the
buildspec cache by running:
$ buildtest buildspec find --rebuild
If you want to find all buildspec files in cache you can run buildtest buildspec find --buildspec
.
Shown below is an example output.
$ buildtest buildspec find --buildspec
+-------------------------------------------------------------------------------------------------------------------------------------+
| buildspecs |
+=====================================================================================================================================+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/skip_tests.yml |
+-------------------------------------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/status_regex.yml |
+-------------------------------------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/metrics_regex.yml |
+-------------------------------------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/executor_regex_script.yml |
+-------------------------------------------------------------------------------------------------------------------------------------+
...
The buildtest buildspec find --paths
will display a list of root directories buildtest will search for
buildspecs when runninh buildtest buildspec find
. One can define these directories in the configuration file
or pass them via command line.
$ buildtest buildspec find --paths
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests
buildtest will search buildspecs in buildspecs root defined in your configuration,
which is a list of directory paths to search for buildspecs.
If you want to load buildspecs from a directory path, you can specify a directory
via --root
option in the format: buildtest buildspec find --root <path> --rebuild
.
buildtest will load all valid buildspecs into cache and ignore
the rest. It’s important to add --rebuild
if you want to regenerate buildspec cache.
Filtering buildspec¶
Once you have a buildspec cache, we can query the buildspec cache for certain attributes.
When you run buildtest buildspec find it will report all buildspecs from cache which can
be difficult to process. Therefore, we have a filter option (--filter
) to restrict our search.
Let’s take a look at the available filter fields that are acceptable with filter option.
$ buildtest buildspec find --helpfilter
Field Description Type
--------- ------------------------- ------
buildspec Filter tests by buildspec FILE
executor Filter by executor name STRING
tags Filter by tag name STRING
type Filter by schema type STRING
The --filter
option expects an arguments in key=value format as follows:
buildtest buildspec find --filter key1=value1,key2=value2,key3=value3
We can filter buildspec cache by tags=fail
which will query all tests with
associated tag field in test.
$ buildtest buildspec find --filter tags=fail
+--------------------------+--------+------------------+----------------+-----------------------------------------------------+
| name | type | executor | tags | description |
+==========================+========+==================+================+=====================================================+
| exit1_fail | script | generic.local.sh | tutorials fail | exit 1 by default is FAIL |
+--------------------------+--------+------------------+----------------+-----------------------------------------------------+
| returncode_list_mismatch | script | generic.local.sh | tutorials fail | exit 2 failed since it failed to match returncode 1 |
+--------------------------+--------+------------------+----------------+-----------------------------------------------------+
In addition, we can query buildspecs by schema type using type
property. In this
example we query all tests by type property
$ buildtest buildspec find --filter type=script
+--------------------------------+--------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| name | type | executor | tags | description |
+================================+========+=========================+========================+================================================================================+
| skip | script | generic.local.bash | tutorials | This test is skipped |
+--------------------------------+--------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| unskipped | script | generic.local.bash | tutorials | This test is not skipped |
+--------------------------------+--------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| status_regex_pass | script | generic.local.bash | system | Pass test based on regular expression |
+--------------------------------+--------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| status_regex_fail | script | generic.local.bash | system | Pass test based on regular expression |
+--------------------------------+--------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| metric_regex_example | script | generic.local.sh | tutorials | capture result metric from output |
+--------------------------------+--------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| executor_regex_script_schema | script | generic.local.(bash|sh) | tutorials | regular expression test with executor using script schema |
+--------------------------------+--------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| add_numbers | script | generic.local.bash | tutorials | Add X+Y |
+--------------------------------+--------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| python_hello | script | generic.local.bash | python | Hello World python |
+--------------------------------+--------+-------------------------+------------------------+--------------------------------------------------------------------------------+
| _bin_sh_shell | script | generic.local.sh | tutorials | /bin/sh shell example |
+--------------------------------+--------+-------------------------+------------------------+--------------------------------------------------------------------------------+
...
Finally, we can combine multiple filter fields separated by comma, in the next example
we can query all buildspecs with tags=tutorials
, executor=generic.local.sh
, and type=script
$ buildtest buildspec find --filter tags=tutorials,executor=generic.local.sh,type=script
+----------------------------+--------+------------------+----------------+--------------------------------------------------------------------------------+
| name | type | executor | tags | description |
+============================+========+==================+================+================================================================================+
| metric_regex_example | script | generic.local.sh | tutorials | capture result metric from output |
+----------------------------+--------+------------------+----------------+--------------------------------------------------------------------------------+
| _bin_sh_shell | script | generic.local.sh | tutorials | /bin/sh shell example |
+----------------------------+--------+------------------+----------------+--------------------------------------------------------------------------------+
| sh_shell | script | generic.local.sh | tutorials | sh shell example |
+----------------------------+--------+------------------+----------------+--------------------------------------------------------------------------------+
| shell_options | script | generic.local.sh | tutorials | shell options |
+----------------------------+--------+------------------+----------------+--------------------------------------------------------------------------------+
| metric_variable_assignment | script | generic.local.sh | tutorials | capture result metric based on variables and environment variable |
+----------------------------+--------+------------------+----------------+--------------------------------------------------------------------------------+
| exit1_fail | script | generic.local.sh | tutorials fail | exit 1 by default is FAIL |
+----------------------------+--------+------------------+----------------+--------------------------------------------------------------------------------+
| exit1_pass | script | generic.local.sh | tutorials pass | report exit 1 as PASS |
+----------------------------+--------+------------------+----------------+--------------------------------------------------------------------------------+
| returncode_list_mismatch | script | generic.local.sh | tutorials fail | exit 2 failed since it failed to match returncode 1 |
+----------------------------+--------+------------------+----------------+--------------------------------------------------------------------------------+
| returncode_int_match | script | generic.local.sh | tutorials pass | exit 128 matches returncode 128 |
+----------------------------+--------+------------------+----------------+--------------------------------------------------------------------------------+
| foo_bar | script | generic.local.sh | tutorials | prints variable $FOO |
+----------------------------+--------+------------------+----------------+--------------------------------------------------------------------------------+
| timelimit_min_max | script | generic.local.sh | tutorials | Run a sleep job for 2 seconds and test pass if its within 1.0-3.0sec |
+----------------------------+--------+------------------+----------------+--------------------------------------------------------------------------------+
| timelimit_min | script | generic.local.sh | tutorials | Run a sleep job for 2 seconds and test pass if its exceeds min time of 1.0 sec |
+----------------------------+--------+------------------+----------------+--------------------------------------------------------------------------------+
| timelimit_max | script | generic.local.sh | tutorials | Run a sleep job for 2 seconds and test pass if it's within max time: 5.0 sec |
+----------------------------+--------+------------------+----------------+--------------------------------------------------------------------------------+
| timelimit_min_fail | script | generic.local.sh | tutorials | This test fails because it runs less than mintime of 10 second |
+----------------------------+--------+------------------+----------------+--------------------------------------------------------------------------------+
| timelimit_max_fail | script | generic.local.sh | tutorials | This test fails because it exceeds maxtime of 1.0 second |
+----------------------------+--------+------------------+----------------+--------------------------------------------------------------------------------+
We can filter output of buildspec cache by buildspec using --filter buildspec=<path>
which
expects a path to buildspec file. The buildspec must be in the cache and file path must exist in order to
fetch the result. The path can be absolute or relative path.
In this next example, we will filter cache by file tutorials/pass_returncode.yml and use --format name,buildspec
to format columns. The --format buildspec
will show full path to buildspec and name
refers to name of test.
For more details on –format see Format buildspec cache.
$ buildtest buildspec find --filter buildspec=tutorials/pass_returncode.yml --format name,buildspec
+--------------------------+------------------------------------------------------------------------------------------------------------+
| name | buildspec |
+==========================+============================================================================================================+
| exit1_fail | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/pass_returncode.yml |
+--------------------------+------------------------------------------------------------------------------------------------------------+
| exit1_pass | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/pass_returncode.yml |
+--------------------------+------------------------------------------------------------------------------------------------------------+
| returncode_list_mismatch | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/pass_returncode.yml |
+--------------------------+------------------------------------------------------------------------------------------------------------+
| returncode_int_match | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/pass_returncode.yml |
+--------------------------+------------------------------------------------------------------------------------------------------------+
Format buildspec cache¶
We have seen how one can filter buildspecs, but we can also configure which columns to display in the output of buildtest buildspec find. By default, we show a pre-selected format fields in the output, however there are more format fields available that can be configured at the command line.
The format fields are specified in comma separated format such as buildtest buildspec find --format <field1>,<field2>,...
.
You can see a list of all format fields by --helpformat
option as shown below
$ buildtest buildspec find --helpformat
Field Description
----------- -----------------------------------
buildspec Display name of buildspec file
description Show description of test
executor Display 'executor' property in test
name Display name of test
tags Display 'tag' property in test
type Display 'type' property in test
In the next example, we utilize --format
field with --filter
option to show
how format fields affect table columns. buildtest will display the table in order of
format fields specified in command line.
$ buildtest buildspec find --format name,description,buildspec --filter tags=tutorials,executor=generic.local.sh
+----------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| name | description | buildspec |
+============================+================================================================================+================================================================================================================+
| metric_regex_example | capture result metric from output | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/metrics_regex.yml |
+----------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| _bin_sh_shell | /bin/sh shell example | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/shell_examples.yml |
+----------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| sh_shell | sh shell example | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/shell_examples.yml |
+----------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| shell_options | shell options | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/shell_examples.yml |
+----------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| metric_variable_assignment | capture result metric based on variables and environment variable | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/metrics_variable.yml |
+----------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| exit1_fail | exit 1 by default is FAIL | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/pass_returncode.yml |
+----------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| exit1_pass | report exit 1 as PASS | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/pass_returncode.yml |
+----------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| returncode_list_mismatch | exit 2 failed since it failed to match returncode 1 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/pass_returncode.yml |
+----------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| returncode_int_match | exit 128 matches returncode 128 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/pass_returncode.yml |
+----------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| foo_bar | prints variable $FOO | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/maintainers_example.yml |
+----------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| timelimit_min_max | Run a sleep job for 2 seconds and test pass if its within 1.0-3.0sec | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/runtime_status_test.yml |
+----------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| timelimit_min | Run a sleep job for 2 seconds and test pass if its exceeds min time of 1.0 sec | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/runtime_status_test.yml |
+----------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| timelimit_max | Run a sleep job for 2 seconds and test pass if it's within max time: 5.0 sec | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/runtime_status_test.yml |
+----------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| timelimit_min_fail | This test fails because it runs less than mintime of 10 second | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/runtime_status_test.yml |
+----------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| timelimit_max_fail | This test fails because it exceeds maxtime of 1.0 second | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/runtime_status_test.yml |
+----------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
buildtest makes use of python library named tabulate
to generate these tables which are found in commands line like buildtest buildspec find
and buildtest report
.
Querying buildspec tags¶
If you want to retrieve all unique tags from all buildspecs you can run
buildtest buildspec find --tags
. This can be useful if you want to know available
tags in your buildspec cache.
$ buildtest buildspec find --tags
+---------------+
| Tags |
+===============+
| system |
+---------------+
| pass |
+---------------+
| compile |
+---------------+
| slurm |
+---------------+
| tutorials |
+---------------+
| ping |
+---------------+
| singularity |
+---------------+
| fail |
+---------------+
| python |
+---------------+
| storage |
+---------------+
| lsf |
+---------------+
| network |
+---------------+
| filesystem |
+---------------+
| cobalt |
+---------------+
| ssh |
+---------------+
| mac |
+---------------+
| configuration |
+---------------+
| spack |
+---------------+
| containers |
+---------------+
In addition, buildtest can group tests by tags via buildtest buildspec find --group-by-tags
which can be useful if you want to know which tests get executed when running buildtest build --tags
.
The output is grouped by tag names, followed by name of test and description.
$ buildtest buildspec find --group-by-tags
+---------------+----------------------------------+
| tags | name |
+===============+==================================+
| tutorials | skip |
+---------------+----------------------------------+
| tutorials | unskipped |
+---------------+----------------------------------+
| tutorials | metric_regex_example |
+---------------+----------------------------------+
| tutorials | executor_regex_script_schema |
+---------------+----------------------------------+
| tutorials | add_numbers |
+---------------+----------------------------------+
| tutorials | _bin_sh_shell |
+---------------+----------------------------------+
| tutorials | _bin_bash_shell |
+---------------+----------------------------------+
| tutorials | bash_shell |
+---------------+----------------------------------+
| tutorials | sh_shell |
+---------------+----------------------------------+
| tutorials | shell_options |
+---------------+----------------------------------+
| tutorials | bash_login_shebang |
+---------------+----------------------------------+
| tutorials | bash_nonlogin_shebang |
+---------------+----------------------------------+
| tutorials | metric_variable_assignment |
+---------------+----------------------------------+
| tutorials | selinux_disable |
+---------------+----------------------------------+
| tutorials | exit1_fail |
+---------------+----------------------------------+
| tutorials | exit1_pass |
+---------------+----------------------------------+
| tutorials | returncode_list_mismatch |
+---------------+----------------------------------+
| tutorials | returncode_int_match |
+---------------+----------------------------------+
| tutorials | sleep |
+---------------+----------------------------------+
...
Querying buildspec executor¶
If you want to know all executors in your buildspec cache use the
buildtest buildspec find --executors
command. This can be useful when
you want to build by executors (buildtest build --executor
).
$ buildtest buildspec find --executors
+-------------------------+
| executors |
+=========================+
| generic.local.bash |
+-------------------------+
| generic.local.(bash|sh) |
+-------------------------+
| generic.local.csh |
+-------------------------+
| generic.local.python |
+-------------------------+
| generic.local.sh |
+-------------------------+
Similar to --group-by-tags
, buildtest has an option to group tests by executor
using --group-by-executor
option. This will show tests grouped by executor,
name of test and test description. Shown below is an example output.
$ buildtest buildspec find --group-by-executor
+-------------------------+----------------------------------+
| executor | name |
+=========================+==================================+
| generic.local.bash | skip |
+-------------------------+----------------------------------+
| generic.local.bash | unskipped |
+-------------------------+----------------------------------+
| generic.local.bash | status_regex_pass |
+-------------------------+----------------------------------+
| generic.local.bash | status_regex_fail |
+-------------------------+----------------------------------+
| generic.local.bash | add_numbers |
+-------------------------+----------------------------------+
| generic.local.bash | python_hello |
+-------------------------+----------------------------------+
| generic.local.bash | _bin_bash_shell |
+-------------------------+----------------------------------+
| generic.local.bash | bash_shell |
+-------------------------+----------------------------------+
| generic.local.bash | bash_login_shebang |
+-------------------------+----------------------------------+
| generic.local.bash | bash_nonlogin_shebang |
+-------------------------+----------------------------------+
| generic.local.bash | run_only_macos_distro |
+-------------------------+----------------------------------+
| generic.local.bash | run_only_linux_distro |
+-------------------------+----------------------------------+
| generic.local.bash | selinux_disable |
+-------------------------+----------------------------------+
| generic.local.bash | sleep |
+-------------------------+----------------------------------+
...
Query Maintainers¶
When you are writing your buildspecs, you can specify the maintainers
field to assign
authors to buildspecs. buildtest can query the maintainers from the cache
once buildspecs are loaded. You can retrieve all maintainers using --maintainers
option or -m
short option. In this example, we show all maintainers for buildspecs in buildspec
cache
$ buildtest buildspec find --maintainers
+------------------+
| maintainers |
+==================+
| @shahzebsiddiqui |
+------------------+
| @johndoe |
+------------------+
| @bobsmith |
+------------------+
If you want to see a breakdown of maintainers by buildspec file you can use --maintainers-by-buildspecs
or -mb
short option. This can be useful to get correlation between maintainers and the buildspec file.
$ buildtest buildspec find -mb
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| maintainers | buildspec |
+==================+====================================================================================================================================================================================================================================+
| @shahzebsiddiqui | ['/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/hello_world.yml', '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/configuration/ulimits.yml'] |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| @johndoe | ['/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/maintainers_example.yml'] |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| @bobsmith | ['/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/maintainers_example.yml'] |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Terse Output¶
You can use the --terse
option to print output of buildtest buildspec find
in terse format that can
be useful if you want to parse content of file. In example below, we will print output of tags in terse format, the
first entry tags
is the header followed by list of unique tags. The --no-header
option
can be used to disable printing of header title.
$ buildtest buildspec find -t --terse
tag
system
pass
compile
slurm
tutorials
ping
singularity
fail
python
storage
lsf
network
filesystem
cobalt
ssh
mac
configuration
spack
containers
Invalid Buildspecs - buildtest buildspec find invalid
¶
buildtest will store invalid buildspecs in the cache file which can be retrieved using buildtest buildspec find invalid
. buildtest
will attempt to parse each buildspec and store error message for every buildspec. If you run without any options it will
report a list of invalid buildspecs as shown below
$ buildtest buildspec find invalid
+---------------------------------------------------------------------------------------------------------------------------------+
| buildspecs |
+=================================================================================================================================+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/invalid_executor.yml |
+---------------------------------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/invalid_tags.yml |
+---------------------------------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/burstbuffer_datawarp_executors.yml |
+---------------------------------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/invalid_buildspec_section.yml |
+---------------------------------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/spack/spack_multiple_executor_sbatch.yml |
+---------------------------------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/spack/env_install.yml |
+---------------------------------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/sched/pbs/hostname.yml |
+---------------------------------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/sched/pbs/batch.yml |
+---------------------------------------------------------------------------------------------------------------------------------+
If you want to see error messages for each buildspec you can pass the -e
or --error
option which will display output of
each buildspec followed by error message.
$ buildtest buildspec find invalid -e
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/invalid_executor.yml
________________________________________________________________________________
"[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/invalid_executor.yml]: Unable to find executor: badexecutor in ['generic.local.bash', 'generic.local.sh', 'generic.local.csh', 'generic.local.zsh', 'generic.local.python']"
________________________________________________________________________________
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/invalid_tags.yml
________________________________________________________________________________
['network', 'network'] is not valid under any of the given schemas
Failed validating 'oneOf' in schema['properties']['tags']:
{'oneOf': [{'type': 'string'},
{'$ref': '#/definitions/list_of_strings'}]}
On instance['tags']:
['network', 'network']
________________________________________________________________________________
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/burstbuffer_datawarp_executors.yml
________________________________________________________________________________
'create_burst_buffer_multiple_executors' is too long
Failed validating 'maxLength' in schema['properties']['buildspecs']['propertyNames']:
{'maxLength': 32, 'pattern': '^[A-Za-z_.][A-Za-z0-9_.]*$'}
On instance['buildspecs']:
'create_burst_buffer_multiple_executors'
________________________________________________________________________________
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/invalid_buildspec_section.yml
________________________________________________________________________________
'[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/invalid_buildspec_section.yml]: type badscript is not known to buildtest.'
________________________________________________________________________________
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/spack/spack_multiple_executor_sbatch.yml
________________________________________________________________________________
Additional properties are not allowed ('post_cmd' was unexpected)
Failed validating 'additionalProperties' in schema:
{'$id': 'spack-v1.0.schema.json',
'$schema': 'http://json-schema.org/draft-07/schema#',
'additionalProperties': False,
'definitions': {'env': {'additionalProperties': False,
'description': 'Used for managing spack '
'environment using ``spack '
'env`` command. ',
'properties': {'activate': {'additionalProperties': False,
'description': 'Activate '
'a '
'spack '
'environment '
'via '
'``spack '
'env '
'activate``',
'properties': {'dir': {'description': 'Activate '
'spack '
'environment '
'from '
'directory.',
'type': 'string'},
'name': {'description': 'Name '
'of '
'spack '
'environment '
'to '
'activate. '
'In '
'order '
'to '
'activate '
'spack '
'environment '
'``my-project`` '
'you '
'need '
'to '
'run '
'``spack '
'env '
'activate '
'my-project`` '
'which '
'is '
'specified '
'by '
'``name: '
'my-project``.',
'type': 'string'},
'options': {'description': 'Pass '
'options '
'to '
'``spack '
'env '
'activate`` '
'command',
'type': 'string'}},
'type': 'object'},
'concretize': {'description': 'If '
'``concretize: '
'true`` '
'is '
'set, '
'we '
'will '
'concretize '
'spack '
'environment '
'by '
'running '
'``spack '
'concretize '
'-f`` '
'otherwise '
'this '
'line '
'will '
'be '
'ignored.',
'type': 'boolean'},
'create': {'additionalProperties': False,
'description': 'Create '
'a '
'spack '
'environment '
'via '
'``spack '
'env '
'create``',
'properties': {'dir': {'description': 'Create '
'a '
'spack '
'environment '
'in '
'a '
'specific '
'directory. '
'This '
'will '
'run '
'``spack '
'env '
'create '
'-d '
'<dir>``. '
'Directory '
'path '
'does '
'not '
'have '
'to '
'exist '
'prior '
'to '
'execution '
'however '
'user '
'must '
'have '
'appropriate '
'ACL '
'in-order '
'to '
'create '
'directory.',
'type': 'string'},
'manifest': {'description': 'Specify '
'path '
'to '
'spack '
'manifest '
'file '
'(``spack.yaml`` '
'or '
'``spack.lock``) '
'when '
'creating '
'environment',
'type': 'string'},
'name': {'description': 'Name '
'of '
'spack '
'environment '
'to '
'create',
'type': 'string'},
'options': {'description': 'Pass '
'options '
'to '
'``spack '
'env '
'create`` '
'command',
'type': 'string'},
'remove_environment': {'default': False,
'description': 'Remove '
'existing '
'spack '
'environment '
'before '
'creating '
'new '
'environment. '
'If '
'set '
'to '
'``True`` '
'we '
'will '
'run '
'``spack '
'env '
'rm '
'-y '
'<name>``.',
'type': 'boolean'}},
'type': 'object'},
'mirror': {'$ref': 'definitions.schema.json#/definitions/env',
'description': 'Add '
'mirror '
'in '
'spack '
'environment '
'by '
'running '
'``spack '
'mirror '
'add``'},
'rm': {'additionalProperties': False,
'description': 'Remove '
'an '
'existing '
'spack '
'environment '
'via '
'``spack '
'env '
'rm``.',
'properties': {'name': {'description': 'Remove '
'spack '
'environment '
'by '
'name. '
'This '
'will '
'run '
'``spack '
'env '
'rm '
'-y '
'<name>``.',
'type': 'string'}},
'required': ['name'],
'type': 'object'},
'specs': {'$ref': 'definitions.schema.json#/definitions/list_of_strings',
'description': 'Add '
'specs '
'to '
'environment '
'by '
'running '
'``spack '
'add '
'<specs>``. '
'The '
'``specs`` '
'is a '
'list '
'of '
'string '
'which '
'expect '
'the '
'argument '
'to '
'be '
'name '
'of '
'spack '
'package.'}},
'type': 'object'},
'install': {'additionalProperties': False,
'description': 'Install spack packages '
'using ``spack install`` '
'command',
'properties': {'options': {'description': 'Pass '
'options '
'to '
'``spack '
'install`` '
'command',
'type': 'string'},
'specs': {'$ref': 'definitions.schema.json#/definitions/list_of_strings',
'description': 'List '
'of '
'specs '
'to '
'install '
'using '
'``spack '
'install`` '
'command'}},
'type': 'object'},
'test': {'additionalProperties': False,
'properties': {'remove_tests': {'description': 'Remove '
'all '
'test '
'suites '
'in '
'spack '
'before '
'running '
'test '
'via '
'``spack '
'test '
'run``. '
'If '
'set '
'to '
'``True`` '
'we '
'will '
'run '
'``spack '
'test '
'remove '
'-y`` '
'which '
'will '
'remove '
'all '
'test '
'suites.',
'type': 'boolean'},
'results': {'additionalProperties': False,
'anyOf': [{'required': ['specs']},
{'required': ['suite']},
{'required': ['specs',
'suite']}],
'description': 'View '
'test '
'results '
'via '
'``spack '
'test '
'results`` '
'after '
'running '
'tests '
'via '
'``spack '
'test '
'run``. '
'Results '
'can '
'be '
'viewed '
'using '
'suitename '
'or '
'installed '
'specs '
'or '
'both.',
'properties': {'option': {'description': 'Pass '
'options '
'to '
'``spack '
'test '
'results``',
'type': 'string'},
'specs': {'$ref': 'definitions.schema.json#/definitions/list_of_strings',
'description': 'Report '
'result '
'by '
'spec '
'name '
'by '
'running '
'``spack '
'test '
'run '
'-- '
'<specs>``.'},
'suite': {'$ref': 'definitions.schema.json#/definitions/list_of_strings',
'description': 'Report '
'results '
'by '
'suite '
'name '
'by '
'running '
'``spack '
'test '
'results '
'<suite>``.'}},
'type': 'object'},
'run': {'additionalProperties': False,
'description': 'Run '
'tests '
'using '
'spack '
'via '
'``spack '
'test '
'run`` '
'command. '
'This '
'command '
'requires '
'specs '
'are '
'installed '
'in '
'your '
'spack '
'instance '
'prior '
'to '
'running '
'tests.',
'properties': {'option': {'description': 'Pass '
'options '
'to '
'``spack '
'test '
'run``',
'type': 'string'},
'specs': {'$ref': 'definitions.schema.json#/definitions/list_of_strings',
'description': 'List '
'of '
'specs '
'to '
'run '
'tests '
'by '
'running '
'``spack '
'test '
'run '
'<specs>``.'}},
'required': ['specs'],
'type': 'object'}},
'required': ['run'],
'type': 'object'}},
'description': 'The spack schema is referenced using ``type: spack`` '
'which is used for generating tests using spack '
'package manager',
'properties': {'BB': {'$ref': 'definitions.schema.json#/definitions/BB'},
'DW': {'$ref': 'definitions.schema.json#/definitions/DW'},
'batch': {'$ref': 'definitions.schema.json#/definitions/batch'},
'bsub': {'$ref': 'definitions.schema.json#/definitions/bsub'},
'cobalt': {'$ref': 'definitions.schema.json#/definitions/cobalt'},
'description': {'$ref': 'definitions.schema.json#/definitions/description'},
'env': {'$ref': 'definitions.schema.json#/definitions/env'},
'executor': {'$ref': 'definitions.schema.json#/definitions/executor'},
'executors': {'$ref': 'definitions.schema.json#/definitions/executors'},
'metrics': {'$ref': 'definitions.schema.json#/definitions/metrics'},
'pbs': {'$ref': 'definitions.schema.json#/definitions/pbs'},
'post_cmds': {'description': 'Shell commands run after '
'spack',
'type': 'string'},
'pre_cmds': {'description': 'Shell commands run before '
'spack',
'type': 'string'},
'sbatch': {'$ref': 'definitions.schema.json#/definitions/sbatch'},
'skip': {'$ref': 'definitions.schema.json#/definitions/skip'},
'spack': {'additionalProperties': False,
'description': 'Entry point to spack '
'configuration',
'properties': {'compiler_find': {'description': 'Run '
'``spack '
'compiler '
'find`` '
'if '
'set '
'to '
'``True``. '
'This '
'is '
'run '
'right '
'after '
'sourcing '
'spack '
'startup '
'script.',
'type': 'boolean'},
'env': {'$ref': '#definitions/env',
'description': 'Manage '
'spack '
'environments '
'via '
'``spack '
'env`` '
'command'},
'install': {'$ref': '#definitions/install',
'description': 'Install '
'spack '
'packages '
'by '
'running '
'``spack '
'install``. '},
'mirror': {'$ref': 'definitions.schema.json#/definitions/env',
'description': 'Add '
'mirror '
'by '
'running '
'``spack '
'mirror '
'add``'},
'root': {'type': 'string'},
'test': {'$ref': '#definitions/test',
'description': 'Entry '
'point '
'to '
'``spack '
'test``'},
'verify_spack': {'default': True,
'description': 'This '
'boolean '
'will '
'determine '
'if '
'we '
'need '
'to '
'check '
'for '
'file '
'existence '
'where '
'spack '
'is '
'cloned '
'via '
'``root`` '
'property '
'and '
'file '
'**$SPACK_ROOT/share/spack/setup-env.sh** '
'exists. '
'These '
'checks '
'can '
'be '
'disabled '
'by '
'setting '
'this '
'to '
'``False`` '
'which '
'can '
'be '
'useful '
'if '
'you '
'dont '
'want '
'buildtest '
'to '
'raise '
'exception '
'during '
'test '
'generation '
'process '
'and '
'test '
'is '
'skipped.',
'type': 'boolean'}},
'required': ['root'],
'type': 'object'},
'status': {'$ref': 'definitions.schema.json#/definitions/status'},
'tags': {'$ref': 'definitions.schema.json#/definitions/tags'},
'type': {'description': 'Select schema type to use '
'when validating buildspec. '
"This must be set to 'spack'",
'pattern': '^spack$',
'type': 'string'},
'vars': {'$ref': 'definitions.schema.json#/definitions/env'}},
'required': ['type', 'executor', 'spack'],
'title': 'spack schema version 1.0',
'type': 'object'}
On instance:
{'batch': {'cpucount': '8', 'timelimit': '30'},
'description': 'sbatch directives can be defined in spack schema',
'executor': 'generic.local.(sh|bash)',
'executors': {'generic.local.bash': {'sbatch': ['-N 8']},
'generic.local.sh': {'sbatch': ['-N 1']}},
'post_cmd': 'rm -rf $SPACK_ROOT',
'pre_cmds': 'cd /tmp\ngit clone https://github.com/spack/spack\n',
'spack': {'env': {'activate': {'name': 'm4'},
'concretize': True,
'specs': ['m4']},
'root': '/tmp/spack'},
'tags': ['spack'],
'type': 'spack'}
________________________________________________________________________________
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/spack/env_install.yml
________________________________________________________________________________
Additional properties are not allowed ('option' was unexpected)
Failed validating 'additionalProperties' in schema['properties']['spack']['properties']['install']:
{'additionalProperties': False,
'description': 'Install spack packages using ``spack install`` '
'command',
'properties': {'options': {'description': 'Pass options to ``spack '
'install`` command',
'type': 'string'},
'specs': {'$ref': 'definitions.schema.json#/definitions/list_of_strings',
'description': 'List of specs to install '
'using ``spack install`` '
'command'}},
'type': 'object'}
On instance['spack']['install']:
{'option': '--keep-prefix'}
________________________________________________________________________________
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/sched/pbs/hostname.yml
________________________________________________________________________________
"[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/sched/pbs/hostname.yml]: Unable to find executor: generic.pbs.workq in ['generic.local.bash', 'generic.local.sh', 'generic.local.csh', 'generic.local.zsh', 'generic.local.python']"
________________________________________________________________________________
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/sched/pbs/batch.yml
________________________________________________________________________________
"[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/sched/pbs/batch.yml]: Unable to find executor: generic.pbs.workq in ['generic.local.bash', 'generic.local.sh', 'generic.local.csh', 'generic.local.zsh', 'generic.local.python']"
________________________________________________________________________________
Cache Summary - buildtest buildspec summary
¶
The buildtest buildspec summary
command can be used to provide a summary of the buildspec cache. This command
can be used assuming your cache is built via buildtest buildspec find
. Shown below is a summary of the cache file.
$ buildtest buildspec summary
Reading Buildspec Cache File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/var/buildspecs/cache.json
Search Paths: ['/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials', '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests']
Total Valid Buildspecs: 65
Total Invalid Buildspecs: 8
Total Unique Tags: 19
Total Unique Executors: 5
Total Maintainers: 3
Unique Tags: ['system', 'pass', 'compile', 'slurm', 'tutorials', 'ping', 'singularity', 'fail', 'python', 'storage', 'lsf', 'network', 'filesystem', 'cobalt', 'ssh', 'mac', 'configuration', 'spack', 'containers']
Unique Executors: ['generic.local.bash', 'generic.local.(bash|sh)', 'generic.local.csh', 'generic.local.python', 'generic.local.sh']
Unique Maintainers: ['@shahzebsiddiqui', '@johndoe', '@bobsmith']
Tag Breakdowns
______________________________
+---------------+---------+
| name | total |
+===============+=========+
| tutorials | 49 |
+---------------+---------+
| system | 9 |
+---------------+---------+
| python | 2 |
+---------------+---------+
| mac | 2 |
+---------------+---------+
| fail | 2 |
+---------------+---------+
| pass | 2 |
+---------------+---------+
| network | 2 |
+---------------+---------+
| ping | 1 |
+---------------+---------+
| spack | 12 |
+---------------+---------+
| compile | 11 |
+---------------+---------+
| lsf | 12 |
+---------------+---------+
| slurm | 17 |
+---------------+---------+
| cobalt | 7 |
+---------------+---------+
| filesystem | 1 |
+---------------+---------+
| storage | 1 |
+---------------+---------+
| ssh | 1 |
+---------------+---------+
| configuration | 1 |
+---------------+---------+
| containers | 8 |
+---------------+---------+
| singularity | 8 |
+---------------+---------+
Executor Breakdowns
______________________________
+-------------------------+---------+
| name | total |
+=========================+=========+
| generic.local.bash | 78 |
+-------------------------+---------+
| generic.local.sh | 34 |
+-------------------------+---------+
| generic.local.(bash|sh) | 4 |
+-------------------------+---------+
| generic.local.csh | 3 |
+-------------------------+---------+
| generic.local.python | 3 |
+-------------------------+---------+
Test Breakdown by buildspecs
______________________________
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| buildspec | total |
+=====================================================================================================================================+=========+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/skip_tests.yml | 2 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/status_regex.yml | 2 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/metrics_regex.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/executor_regex_script.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/add_numbers.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/python-hello.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/shell_examples.yml | 5 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/shebang.yml | 2 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/run_only_distro.yml | 2 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/metrics_variable.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/selinux.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/pass_returncode.yml | 4 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/sleep.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/csh_shell_examples.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/tags_example.yml | 2 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/environment.yml | 3 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/hello_world.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/maintainers_example.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/vars.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/run_only_platform.yml | 2 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/python-shell.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/runtime_status_test.yml | 5 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/root_user.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/spack/spack_test.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/spack/env_create_directory.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/spack/pre_post_cmds.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/spack/remove_environment_example.yml | 2 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/spack/spack_test_specs.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/spack/concretize_m4.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/spack/env_create_manifest.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/spack/spack_sbatch.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/spack/mirror_example.yml | 2 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/spack/install_zlib.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/script/executor_scheduler.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/script/status_by_executors.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/script/multiple_executors.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/compilers/openmp_hello.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/compilers/compiler_status_regex.yml | 2 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/compilers/envvar_override.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/compilers/custom_run.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/compilers/compiler_exclude.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/compilers/gnu_hello_c.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/compilers/vecadd.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/compilers/gnu_hello_fortran.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/compilers/pre_post_build_run.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/compilers/metrics_openmp.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/sched/lsf/bugroup.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/sched/lsf/bmgroups.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/sched/lsf/bqueues.yml | 3 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/sched/lsf/lsinfo.yml | 4 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/sched/lsf/bhosts.yml | 3 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/sched/slurm/squeue.yml | 2 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/sched/slurm/sacctmgr.yml | 4 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/sched/slurm/scontrol.yml | 2 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/sched/slurm/sinfo.yml | 9 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/sched/cobalt/commands.yml | 7 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/configuration/disk_usage.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/configuration/systemd-default-target.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/configuration/ssh_localhost.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/configuration/kernel_state.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/configuration/ulimits.yml | 6 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/containers/singularity/run.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/containers/singularity/build.yml | 3 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/containers/singularity/pull.yml | 3 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/general_tests/containers/singularity/inspect.yml | 1 |
+-------------------------------------------------------------------------------------------------------------------------------------+---------+
Validate Buildspecs - buildtest buildspec validate
¶
buildtest can validate buildspecs through the buildtest buildspec validate
command which provides
analogous options for buildtest build
for selecting buildspecs such as -b
, -e
, -t
and -e
.
This command can be used to validate buildspecs with the JSON Schema which can be useful if you are writing a buildspec
and want to validate the buildspec without running the test.
Shown below are the available command options.
$ buildtest buildspec validate --help
usage: buildtest [options] [COMMANDS] buildspec validate [-h] [-b BUILDSPEC] [-x EXCLUDE] [-e EXECUTOR] [-t TAG]
optional arguments:
-h, --help show this help message and exit
-b BUILDSPEC, --buildspec BUILDSPEC
Specify path to buildspec (file, or directory) to validate
-x EXCLUDE, --exclude EXCLUDE
Specify path to buildspec to exclude (file or directory) during validation
-e EXECUTOR, --executor EXECUTOR
Specify buildspecs by executor name to validate
-t TAG, --tag TAG Specify buildspecs by tag name to validate
The -b option can be used to specify path to buildspec file or directory to validate buildspecs. If its a directory, buildtest will traverse all directories recursively and find any .yml file extensions and attempt to validate each buildspec. Shown below is an example output of what it may look like
$ buildtest buildspec validate -b tutorials/vars.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/vars.yml
All buildspecs passed validation!!!
If buildtest detects an error during validation, the error message will be displayed to screen as we see in this example
$ buildtest buildspec validate -b tutorials/invalid_tags.yml
file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/invalid_tags.yml
________________________________________________________________________________
['network', 'network'] is not valid under any of the given schemas
Failed validating 'oneOf' in schema['properties']['tags']:
{'oneOf': [{'type': 'string'},
{'$ref': '#/definitions/list_of_strings'}]}
On instance['tags']:
['network', 'network']
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/invalid_tags.yml
There were 1 buildspecs that failed validation
Similarly we can search buildspecs based on tags if you want to validate a group of buildspecs using the -t
option. We can
append -t
option multiple times to search by multiple tag names. In this next example, we
will validate all buildspecs for python and pass tags.
$ buildtest buildspec validate -t python -t pass
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/python-hello.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/pass_returncode.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/python-shell.yml
All buildspecs passed validation!!!
Finally we can also search by executors using the -e
option which can be appended to search by
multiple executors.
$ buildtest buildspec validate -e generic.local.csh
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/environment.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/csh_shell_examples.yml
All buildspecs passed validation!!!
Edit buildspecs buildtest edit
¶
The buildtest edit
command can be used to edit buildspec with your preferred editor
defined by environment $EDITOR, if this environment is not set buildtest will resort to vim
.
Once you make change, the file will be written back to disk and validated with the jsonschema.
If it passes validation you will see a message such as follows:
$ buildtest edit tutorials/vars.yml
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest.tmp/tutorials/vars.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest.tmp/tutorials/vars.yml is valid
If there is an error during validation, buildtest will print the exception to stdout and it is your
responsibility to fix the buildspec based on error message. In example below, the user provided an invalid
value for type
field.
$ buildtest edit tutorials/vars.yml
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest.tmp/tutorials/vars.yml
Traceback (most recent call last):
File "/Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest", line 17, in <module>
buildtest.main.main()
File "/Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/main.py", line 103, in main
edit_buildspec(args.buildspec, configuration)
File "/Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/cli/edit.py", line 23, in edit_buildspec
BuildspecParser(buildspec, be)
File "/Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/buildsystem/parser.py", line 74, in __init__
self._validate()
File "/Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/buildsystem/parser.py", line 185, in _validate
self._check_schema_type(test)
File "/Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/buildsystem/parser.py", line 101, in _check_schema_type
raise BuildspecError(self.buildspec, msg)
buildtest.exceptions.BuildspecError: '[/Users/siddiq90/Documents/GitHubDesktop/buildtest.tmp/tutorials/vars.yml]: type script123 is not known to buildtest.'
Show buildspec buildtest buildspec show
¶
buildtest can display content of buildspec file given a test name via buildtest buildspec show
command which expects a
positional argument that is the name of test. This can be quick way to see content of buildspec without remembering the full path
to the buildspec.
In this next example, we will instruct buildtest to show content of buildspec for test name python_hello.
$ buildtest buildspec show python_hello
version: "1.0"
buildspecs:
python_hello:
type: script
description: Hello World python
executor: generic.local.bash
tags: python
run: python hello.py
buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/python-hello.yml
There is bash completion for this command which will show list of test names available in the cache assuming you have run
buildtest buildspec find
. If you specify an invalid test name you will get an error followed by list of tests that are available
in the cache
$ buildtest buildspec show XYZ123!
Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest", line 17, in <module>
buildtest.main.main()
File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/buildtest/main.py", line 118, in main
show_buildspecs(name=args.name, configuration=configuration)
File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/buildtest/cli/buildspec.py", line 964, in show_buildspecs
f"{name} not in cache. Please select one of the following test: {cache.get_names()}"
buildtest.exceptions.BuildTestError: "XYZ123! not in cache. Please select one of the following test: ['skip', 'unskipped', 'status_regex_pass', 'status_regex_fail', 'metric_regex_example', 'executor_regex_script_schema', 'add_numbers', 'python_hello', '_bin_sh_shell', '_bin_bash_shell', 'bash_shell', 'sh_shell', 'shell_options', 'bash_login_shebang', 'bash_nonlogin_shebang', 'run_only_macos_distro', 'run_only_linux_distro', 'metric_variable_assignment', 'selinux_disable', 'exit1_fail', 'exit1_pass', 'returncode_list_mismatch', 'returncode_int_match', 'sleep', 'csh_shell', 'string_tag', 'list_of_strings_tags', 'bash_env_variables', 'csh_env_declaration', 'tcsh_env_declaration', 'hello_world', 'foo_bar', 'variables_bash', 'run_only_platform_darwin', 'run_only_platform_linux', 'circle_area', 'timelimit_min_max', 'timelimit_min', 'timelimit_max', 'timelimit_min_fail', 'timelimit_max_fail', 'run_only_as_root', 'spack_test', 'spack_env_directory', 'run_pre_post_commands', 'remove_environment_automatically', 'remove_environment_explicit', 'spack_test_results_specs_format', 'concretize_m4_in_spack_env', 'spack_env_create_from_manifest', 'spack_sbatch_example', 'add_mirror', 'add_mirror_in_spack_env', 'install_zlib', 'executors_sbatch_declaration', 'status_returncode_by_executors', 'executors_vars_env_declaration', 'openmp_hello_c_example', 'default_status_regex', 'override_status_regex', 'override_environmentvars', 'custom_run_by_compilers', 'vecadd_gnu_exclude', 'hello_c', 'vecadd_gnu', 'hello_f', 'pre_post_build_run', 'metrics_variable_compiler', 'show_lsf_user_groups', 'show_host_groups', 'show_lsf_queues', 'show_lsf_queues_formatted', 'show_lsf_queues_current_user', 'show_lsf_configuration', 'show_lsf_models', 'show_lsf_resources', 'lsf_version', 'display_lsf_hosts', 'display_hosts_format', 'bhosts_version', 'current_user_queue', 'show_all_jobs', 'show_accounts', 'show_users', 'show_qos', 'show_tres', 'slurm_config', 'show_partition', 'nodes_state_down', 'nodes_state_reboot', 'nodes_state_allocated', 'nodes_state_completing', 'nodes_state_idle', 'node_down_fail_list_reason', 'dead_nodes', 'get_partitions', 'sinfo_version', 'qsub_version', 'qselect_version', 'cqsub_version', 'qdel_version', 'qmove_version', 'show_jobs', 'show_queues', 'root_disk_usage', 'systemd_default_target', 'ssh_localhost_remotecommand', 'kernel_swapusage', 'ulimit_filelock_unlimited', 'ulimit_cputime_unlimited', 'ulimit_stacksize_unlimited', 'ulimit_vmsize_unlimited', 'ulimit_filedescriptor_4096', 'ulimit_max_user_process_2048', 'runImage', 'build_sif_from_dockerimage', 'build_sandbox_image', 'build_remoteimages', 'pullImage_dockerhub', 'pullImage_sylabscloud', 'pullImage_shub', 'inspect_image']"