Buildspecs Interface

Now that we learned how to build tests, in this section we will discuss how one can query a buildspec cache. In buildtest, one can load all buildspecs which is equivalent to validating all buildspecs with the appropriate schema. Buildtest will ignore all invalid buildspecs and store them in a separate file.

The buildtest buildspec find command is used for finding buildspecs from buildspec cache. This command is also used for generating the buildspec cache. Shown below is a list of options for buildtest buildspec find.

$ buildtest buildspec find --help
usage: buildtest [options] [COMMANDS] buildspec find [-h] [--root ROOT] [-r] [-t] [-b] [-e] [-p] [--group-by-tags]
                                                     [--group-by-executor] [-m] [-mb] [--filter FILTER]
                                                     [--format FORMAT] [--helpfilter] [--helpformat] [--terse]

optional arguments:
  -h, --help            show this help message and exit
  --root ROOT           Specify root buildspecs (directory) path to load buildspecs into buildspec cache.
  -r, --rebuild         Rebuild buildspec cache and find all buildspecs again
  -t, --tags            List all available tags
  -b, --buildspec       Get all buildspec files from cache
  -e, --executors       get all unique executors from buildspecs
  -p, --paths           print all root buildspec paths
  --group-by-tags       Group tests by tag name
  --group-by-executor   Group tests by executor name
  -m, --maintainers     Get all maintainers for all buildspecs
  -mb, --maintainers-by-buildspecs
                        Show maintainers breakdown by buildspecs
  --filter FILTER       Filter buildspec cache with filter fields in format --filter key1=val1,key2=val2
  --format FORMAT       Format buildspec cache with format fields in format --format field1,field2,...
  --helpfilter          Show Filter fields for --filter option for filtering buildspec cache output
  --helpformat          Show Format fields for --format option for formatting buildspec cache output
  --terse               Print output in machine readable format

Finding Buildspecs

To find all buildspecs you can run buildtest buildspec find which will discover all buildspecs by recursively searching all .yml extensions. buildtest will validate each buildspec file with the json schema and buildtest will display all valid buildspecs in the output, all invalid buildspecs will be stored in a file for post-processing.

$ buildtest buildspec find
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| name                             | type     | executor                | tags                          | description                                                                    |
+==================================+==========+=========================+===============================+================================================================================+
| skip                             | script   | generic.local.bash      | ['tutorials']                 | This test is skipped                                                           |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| unskipped                        | script   | generic.local.bash      | ['tutorials']                 | This test is not skipped                                                       |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| executor_regex_script_schema     | script   | generic.local.(bash|sh) | ['tutorials']                 | regular expression test with executor using script schema                      |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| add_numbers                      | script   | generic.local.bash      | ['tutorials']                 | Add X+Y                                                                        |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| python_hello                     | script   | generic.local.bash      | python                        | Hello World python                                                             |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| _bin_sh_shell                    | script   | generic.local.sh        | ['tutorials']                 | /bin/sh shell example                                                          |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| _bin_bash_shell                  | script   | generic.local.bash      | ['tutorials']                 | /bin/bash shell example                                                        |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| bash_shell                       | script   | generic.local.bash      | ['tutorials']                 | bash shell example                                                             |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| sh_shell                         | script   | generic.local.sh        | ['tutorials']                 | sh shell example                                                               |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| shell_options                    | script   | generic.local.sh        | ['tutorials']                 | shell options                                                                  |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| bash_login_shebang               | script   | generic.local.bash      | tutorials                     | customize shebang line with bash login shell                                   |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| bash_nonlogin_shebang            | script   | generic.local.bash      | tutorials                     | customize shebang line with default bash (nonlogin) shell                      |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| run_only_macos_distro            | script   | generic.local.bash      | ['mac']                       | Run test only if distro is darwin.                                             |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| run_only_linux_distro            | script   | generic.local.bash      | ['mac']                       | Run test only if distro is CentOS.                                             |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| selinux_disable                  | script   | generic.local.bash      | ['tutorials']                 | Check if SELinux is Disabled                                                   |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| exit1_fail                       | script   | generic.local.sh        | ['tutorials', 'fail']         | exit 1 by default is FAIL                                                      |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| exit1_pass                       | script   | generic.local.sh        | ['tutorials', 'pass']         | report exit 1 as PASS                                                          |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| returncode_list_mismatch         | script   | generic.local.sh        | ['tutorials', 'fail']         | exit 2 failed since it failed to match returncode 1                            |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| returncode_int_match             | script   | generic.local.sh        | ['tutorials', 'pass']         | exit 128 matches returncode 128                                                |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| sleep                            | script   | generic.local.bash      | ['tutorials']                 | sleep 2 seconds                                                                |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| csh_shell                        | script   | generic.local.csh       | ['tutorials']                 | csh shell example                                                              |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| string_tag                       | script   | generic.local.bash      | network                       | tags can be a string                                                           |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| list_of_strings_tags             | script   | generic.local.bash      | ['network', 'ping']           | tags can be a list of strings                                                  |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| bash_env_variables               | script   | generic.local.bash      | ['tutorials']                 | Declare environment variables in default shell (bash)                          |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| csh_env_declaration              | script   | generic.local.csh       | ['tutorials']                 | csh shell example to declare environment variables                             |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| tcsh_env_declaration             | script   | generic.local.csh       | ['tutorials']                 | tcsh shell example to declare environment variables                            |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| hello_world                      | script   | generic.local.bash      | tutorials                     | hello world example                                                            |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| variables_bash                   | script   | generic.local.bash      | ['tutorials']                 | Declare shell variables in bash                                                |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| run_only_platform_darwin         | script   | generic.local.python    | ['tutorials']                 | This test will only run if target platform is Darwin                           |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| run_only_platform_linux          | script   | generic.local.python    | ['tutorials']                 | This test will only run if target platform is Linux                            |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| circle_area                      | script   | generic.local.python    | ['tutorials', 'python']       | Calculate circle of area given a radius                                        |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| timelimit_min_max                | script   | generic.local.sh        | ['tutorials']                 | Run a sleep job for 2 seconds and test pass if its within 1.0-3.0sec           |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| timelimit_min                    | script   | generic.local.sh        | ['tutorials']                 | Run a sleep job for 2 seconds and test pass if its exceeds min time of 1.0 sec |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| timelimit_max                    | script   | generic.local.sh        | ['tutorials']                 | Run a sleep job for 2 seconds and test pass if it's within max time: 5.0 sec   |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| timelimit_min_fail               | script   | generic.local.sh        | ['tutorials']                 | This test fails because it runs less than mintime of 10 second                 |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| timelimit_max_fail               | script   | generic.local.sh        | ['tutorials']                 | This test fails because it exceeds maxtime of 1.0 second                       |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| run_only_as_root                 | script   | generic.local.bash      | ['tutorials']                 | This test will only run if current user is root                                |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| spack_test                       | spack    | generic.local.sh        | ['spack']                     | Install bzip2 and run spack test and report results                            |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| spack_env_directory              | spack    | generic.local.sh        | ['spack']                     | Concretize m4 in a spack environment named m4                                  |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| run_pre_post_commands            | spack    | generic.local.sh        | ['spack']                     | Install zlib                                                                   |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| remove_environment_automatically | spack    | generic.local.sh        | ['spack']                     | remove spack environment automatically before creating a new environment       |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| remove_environment_explicit      | spack    | generic.local.sh        | ['spack']                     | remove spack environment explicitly using the 'rm' property                    |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| spack_test_results_specs_format  | spack    | generic.local.sh        | ['spack']                     | Run spack test results with spec format                                        |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| concretize_m4_in_spack_env       | spack    | generic.local.sh        | ['spack']                     | Concretize m4 in a spack environment named m4                                  |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| spack_env_create_from_manifest   | spack    | generic.local.sh        | ['spack']                     | Create spack environment from spack.yaml                                       |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| spack_sbatch_example             | spack    | generic.local.sh        | ['spack']                     | sbatch directives can be defined in spack schema                               |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| add_mirror                       | spack    | generic.local.sh        | ['spack']                     | Declare spack mirror                                                           |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| add_mirror_in_spack_env          | spack    | generic.local.sh        | ['spack']                     | Declare spack mirror in spack environment                                      |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| install_zlib                     | spack    | generic.local.sh        | ['spack']                     | Install zlib                                                                   |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| openmp_hello_c_example           | compiler | generic.local.bash      | ['tutorials', 'compile']      | OpenMP Hello World C example                                                   |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| default_status_regex             | compiler | generic.local.bash      | ['tutorials', 'compile']      | Regular expression check in stdout for gcc group                               |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| override_status_regex            | compiler | generic.local.bash      | ['tutorials', 'compile']      | Override regular expression for compiler gcc/10.2.0-37fmsw7                    |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| override_environmentvars         | compiler | generic.local.bash      | ['tutorials', 'compile']      | override default environment variables                                         |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| custom_run_by_compilers          | compiler | generic.local.bash      | ['tutorials', 'compile']      | Customize binary launch based on compiler                                      |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| vecadd_gnu_exclude               | compiler | generic.local.bash      | ['tutorials', 'compile']      | Vector Addition example with GNU compilers but exclude gcc@10.2.0              |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| hello_c                          | compiler | generic.local.bash      | ['tutorials', 'compile']      | Hello World C Compilation                                                      |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| vecadd_gnu                       | compiler | generic.local.bash      | ['tutorials', 'compile']      | Vector Addition example with GNU compiler                                      |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| hello_f                          | compiler | generic.local.bash      | ['tutorials', 'compile']      | Hello World Fortran Compilation                                                |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| pre_post_build_run               | compiler | generic.local.bash      | ['tutorials', 'compile']      | example using pre_build, post_build, pre_run, post_run example                 |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| show_lsf_user_groups             | script   | generic.local.bash      | lsf                           | Show information about all LSF user groups                                     |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| show_host_groups                 | script   | generic.local.bash      | lsf                           | Show information about host groups using bmgroup                               |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| show_lsf_queues                  | script   | generic.local.bash      | lsf                           | Show LSF queues                                                                |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| show_lsf_queues_formatted        | script   | generic.local.bash      | lsf                           | Show LSF queues with formatted columns                                         |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| show_lsf_queues_current_user     | script   | generic.local.bash      | lsf                           | Show LSF queues available for current user                                     |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| show_lsf_configuration           | script   | generic.local.bash      | lsf                           | Show LSF configuration using lsinfo                                            |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| show_lsf_models                  | script   | generic.local.bash      | lsf                           | Show information about host models in LSF cluster                              |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| show_lsf_resources               | script   | generic.local.bash      | lsf                           | Show information about LSF resources                                           |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| lsf_version                      | script   | generic.local.bash      | lsf                           | Display lsf version using lsinfo                                               |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| display_lsf_hosts                | script   | generic.local.bash      | lsf                           | Show all hosts in LSF cluster                                                  |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| display_hosts_format             | script   | generic.local.bash      | lsf                           | Show all hosts with column hostname and status                                 |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| bhosts_version                   | script   | generic.local.bash      | lsf                           | display version from bhosts command                                            |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| current_user_queue               | script   | generic.local.bash      | ['slurm']                     | show all current pending jobs for current user (squeue -u $USER)               |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| show_all_jobs                    | script   | generic.local.bash      | ['slurm']                     | show all pending + running jobs (squeue -a)                                    |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| show_accounts                    | script   | generic.local.bash      | ['slurm']                     | run sacctmgr list accounts                                                     |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| show_users                       | script   | generic.local.bash      | ['slurm']                     | run sacctmgr list users                                                        |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| show_qos                         | script   | generic.local.bash      | ['slurm']                     | run sacctmgr list qos                                                          |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| show_tres                        | script   | generic.local.bash      | ['slurm']                     | run sacctmgr list tres                                                         |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| slurm_config                     | script   | generic.local.bash      | ['slurm']                     | run scontrol show config                                                       |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| show_partition                   | script   | generic.local.bash      | ['slurm']                     | run scontrol show partition                                                    |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| nodes_state_down                 | script   | generic.local.bash      | ['slurm']                     | Show nodes in DOWN state                                                       |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| nodes_state_reboot               | script   | generic.local.bash      | ['slurm']                     | Show nodes in REBOOT state                                                     |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| nodes_state_allocated            | script   | generic.local.bash      | ['slurm']                     | Show nodes in ALLOCATED state                                                  |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| nodes_state_completing           | script   | generic.local.bash      | ['slurm']                     | Show nodes in COMPLETING state                                                 |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| nodes_state_idle                 | script   | generic.local.bash      | ['slurm']                     | Show nodes in IDLE state                                                       |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| node_down_fail_list_reason       | script   | generic.local.bash      | ['slurm']                     | Show nodes DOWN, DRAINED, FAIL or FAILING and list reason                      |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| dead_nodes                       | script   | generic.local.bash      | ['slurm']                     | Show non-responding nodes                                                      |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| get_partitions                   | script   | generic.local.bash      | ['slurm']                     | Get all slurm partitions                                                       |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| sinfo_version                    | script   | generic.local.bash      | ['slurm']                     | get slurm version                                                              |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| qsub_version                     | script   | generic.local.sh        | ['cobalt']                    | print version for qsub command                                                 |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| qselect_version                  | script   | generic.local.sh        | ['cobalt']                    | print version for qselect                                                      |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| cqsub_version                    | script   | generic.local.sh        | ['cobalt']                    | print version for cqsub command                                                |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| qdel_version                     | script   | generic.local.sh        | ['cobalt']                    | print version for qdel command                                                 |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| qmove_version                    | script   | generic.local.sh        | ['cobalt']                    | print version for qmove command                                                |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| show_jobs                        | script   | generic.local.sh        | ['cobalt']                    | Show all jobs in queue                                                         |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| show_queues                      | script   | generic.local.sh        | ['cobalt']                    | Show all queues                                                                |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| root_disk_usage                  | script   | generic.local.bash      | ['filesystem', 'storage']     | Check root disk usage and report if it exceeds threshold                       |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| systemd_default_target           | script   | generic.local.bash      | ['system']                    | check if default target is multi-user.target                                   |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| ssh_localhost_remotecommand      | script   | generic.local.bash      | ['ssh']                       | Test if ssh on localhost works and if we can run remote command.               |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| kernel_swapusage                 | script   | generic.local.bash      | ['configuration']             | Retrieve Kernel Swap Usage                                                     |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| ulimit_filelock_unlimited        | script   | generic.local.bash      | ['system']                    | Check if file lock is set to unlimited in ulimits                              |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| ulimit_cputime_unlimited         | script   | generic.local.bash      | ['system']                    | Check if cputime is set to unlimited in ulimits                                |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| ulimit_stacksize_unlimited       | script   | generic.local.bash      | ['system']                    | Check if stack size is set to unlimited in ulimits                             |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| ulimit_vmsize_unlimited          | script   | generic.local.bash      | ['system']                    | Check virtual memory size and check if its set to unlimited                    |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| ulimit_filedescriptor_4096       | script   | generic.local.bash      | ['system']                    | Check if open file descriptors limit is set to 4096                            |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| ulimit_max_user_process_2048     | script   | generic.local.bash      | ['system']                    | Check max number of user process limit is set to 2048                          |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| runImage                         | script   | generic.local.bash      | ['containers', 'singularity'] | run container docker://godlovedc/lolcow                                        |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| build_sif_from_dockerimage       | script   | generic.local.bash      | ['containers', 'singularity'] | build sif image from docker image docker://godlovedc/lolcow                    |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| build_sandbox_image              | script   | generic.local.bash      | ['containers', 'singularity'] | build sandbox image from docker image docker://godlovedc/lolcow                |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| build_remoteimages               | script   | generic.local.bash      | ['containers', 'singularity'] | build remote hosted image from AWS                                             |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| pullImage_dockerhub              | script   | generic.local.bash      | ['containers', 'singularity'] | Pull image docker://godlovedc/lolcow from DockerHub                            |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| pullImage_sylabscloud            | script   | generic.local.bash      | ['containers', 'singularity'] | Pull image library://alpine:latest from Sylabs Cloud                           |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| pullImage_shub                   | script   | generic.local.bash      | ['containers', 'singularity'] | Pull image shub://vsoch/singularity-images from SingularityHub                 |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| inspect_image                    | script   | generic.local.bash      | ['containers', 'singularity'] | Inspect image via 'singularity inspect'                                        |
+----------------------------------+----------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+

buildtest will load all discovered buildspecs in a cache file (JSON) which is created upon running buildtest buildspec find. Any subsequent runs will read from cache and update if any new buildspecs are added. If you make changes to buildspec you should rebuild the buildspec cache by running:

$ buildtest buildspec find --rebuild

If you want to find all buildspec files in cache you can run buildtest buildspec find --buildspec. Shown below is an example output.

$ buildtest buildspec find --buildspec
+-------------------------------------------------------------------------------------------------------------------------------------+
| buildspecs                                                                                                                          |
+=====================================================================================================================================+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/skip_tests.yml                               |
+-------------------------------------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/executor_regex_script.yml                    |
+-------------------------------------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/add_numbers.yml                              |
+-------------------------------------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/python-hello.yml                             |
+-------------------------------------------------------------------------------------------------------------------------------------+
...

The buildtest buildspec find --paths will display a list of root directories buildtest will search for buildspecs when runninh buildtest buildspec find. One can define these directories in the configuration file or pass them via command line.

$ buildtest buildspec find --paths
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/general_tests

buildtest will search buildspecs in buildspecs root defined in your configuration, which is a list of directory paths to search for buildspecs. If you want to load buildspecs from a directory path, you can specify a directory via --root option in the format: buildtest buildspec find --root <path> --rebuild. buildtest will load all valid buildspecs into cache and ignore the rest. It’s important to add --rebuild if you want to regenerate buildspec cache.

Filtering buildspec

Once you have a buildspec cache, we can query the buildspec cache for certain attributes. When you run buildtest buildspec find it will report all buildspecs from cache which can be difficult to process. Therefore, we have a filter option (--filter) to restrict our search. Let’s take a look at the available filter fields that are acceptable with filter option.

$ buildtest buildspec find --helpfilter
Field     Description              Type
--------  -----------------------  ------
executor  Filter by executor name  STRING
tags      Filter by tag name       STRING
type      Filter by schema type    STRING

The --filter option expects an arguments in key=value format as follows:

buildtest buildspec find --filter key1=value1,key2=value2,key3=value3

We can filter buildspec cache by tags=fail which will query all tests with associated tag field in test.

$ buildtest buildspec find --filter tags=fail
+--------------------------+--------+------------------+-----------------------+-----------------------------------------------------+
| name                     | type   | executor         | tags                  | description                                         |
+==========================+========+==================+=======================+=====================================================+
| exit1_fail               | script | generic.local.sh | ['tutorials', 'fail'] | exit 1 by default is FAIL                           |
+--------------------------+--------+------------------+-----------------------+-----------------------------------------------------+
| returncode_list_mismatch | script | generic.local.sh | ['tutorials', 'fail'] | exit 2 failed since it failed to match returncode 1 |
+--------------------------+--------+------------------+-----------------------+-----------------------------------------------------+

In addition, we can query buildspecs by schema type using type property. In this example we query all tests by type property

$ buildtest buildspec find --filter type=script
+------------------------------+--------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| name                         | type   | executor                | tags                          | description                                                                    |
+==============================+========+=========================+===============================+================================================================================+
| skip                         | script | generic.local.bash      | ['tutorials']                 | This test is skipped                                                           |
+------------------------------+--------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| unskipped                    | script | generic.local.bash      | ['tutorials']                 | This test is not skipped                                                       |
+------------------------------+--------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| executor_regex_script_schema | script | generic.local.(bash|sh) | ['tutorials']                 | regular expression test with executor using script schema                      |
+------------------------------+--------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| add_numbers                  | script | generic.local.bash      | ['tutorials']                 | Add X+Y                                                                        |
+------------------------------+--------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| python_hello                 | script | generic.local.bash      | python                        | Hello World python                                                             |
+------------------------------+--------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| _bin_sh_shell                | script | generic.local.sh        | ['tutorials']                 | /bin/sh shell example                                                          |
+------------------------------+--------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| _bin_bash_shell              | script | generic.local.bash      | ['tutorials']                 | /bin/bash shell example                                                        |
+------------------------------+--------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| bash_shell                   | script | generic.local.bash      | ['tutorials']                 | bash shell example                                                             |
+------------------------------+--------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
| sh_shell                     | script | generic.local.sh        | ['tutorials']                 | sh shell example                                                               |
+------------------------------+--------+-------------------------+-------------------------------+--------------------------------------------------------------------------------+
...

Finally, we can combine multiple filter fields separated by comma, in the next example we can query all buildspecs with tags=tutorials, executor=generic.local.sh, and type=script

$ buildtest buildspec find --filter tags=tutorials,executor=generic.local.sh,type=script
+--------------------------+--------+------------------+-----------------------+--------------------------------------------------------------------------------+
| name                     | type   | executor         | tags                  | description                                                                    |
+==========================+========+==================+=======================+================================================================================+
| _bin_sh_shell            | script | generic.local.sh | ['tutorials']         | /bin/sh shell example                                                          |
+--------------------------+--------+------------------+-----------------------+--------------------------------------------------------------------------------+
| sh_shell                 | script | generic.local.sh | ['tutorials']         | sh shell example                                                               |
+--------------------------+--------+------------------+-----------------------+--------------------------------------------------------------------------------+
| shell_options            | script | generic.local.sh | ['tutorials']         | shell options                                                                  |
+--------------------------+--------+------------------+-----------------------+--------------------------------------------------------------------------------+
| exit1_fail               | script | generic.local.sh | ['tutorials', 'fail'] | exit 1 by default is FAIL                                                      |
+--------------------------+--------+------------------+-----------------------+--------------------------------------------------------------------------------+
| exit1_pass               | script | generic.local.sh | ['tutorials', 'pass'] | report exit 1 as PASS                                                          |
+--------------------------+--------+------------------+-----------------------+--------------------------------------------------------------------------------+
| returncode_list_mismatch | script | generic.local.sh | ['tutorials', 'fail'] | exit 2 failed since it failed to match returncode 1                            |
+--------------------------+--------+------------------+-----------------------+--------------------------------------------------------------------------------+
| returncode_int_match     | script | generic.local.sh | ['tutorials', 'pass'] | exit 128 matches returncode 128                                                |
+--------------------------+--------+------------------+-----------------------+--------------------------------------------------------------------------------+
| timelimit_min_max        | script | generic.local.sh | ['tutorials']         | Run a sleep job for 2 seconds and test pass if its within 1.0-3.0sec           |
+--------------------------+--------+------------------+-----------------------+--------------------------------------------------------------------------------+
| timelimit_min            | script | generic.local.sh | ['tutorials']         | Run a sleep job for 2 seconds and test pass if its exceeds min time of 1.0 sec |
+--------------------------+--------+------------------+-----------------------+--------------------------------------------------------------------------------+
| timelimit_max            | script | generic.local.sh | ['tutorials']         | Run a sleep job for 2 seconds and test pass if it's within max time: 5.0 sec   |
+--------------------------+--------+------------------+-----------------------+--------------------------------------------------------------------------------+
| timelimit_min_fail       | script | generic.local.sh | ['tutorials']         | This test fails because it runs less than mintime of 10 second                 |
+--------------------------+--------+------------------+-----------------------+--------------------------------------------------------------------------------+
| timelimit_max_fail       | script | generic.local.sh | ['tutorials']         | This test fails because it exceeds maxtime of 1.0 second                       |
+--------------------------+--------+------------------+-----------------------+--------------------------------------------------------------------------------+

Format buildspec cache

We have seen how one can filter buildspecs, but we can also configure which columns to display in the output of buildtest buildspec find. By default, we show a pre-selected format fields in the output, however there are more format fields available that can be configured at the command line.

The format fields are specified in comma separated format such as buildtest buildspec find --format <field1>,<field2>,.... You can see a list of all format fields by --helpformat option as shown below

$ buildtest buildspec find --helpformat
Field        Description
-----------  -----------------------
description  Format by description
executor     Format by executor type
file         Format by file
name         Format by test name
tags         Format by tag name
type         Format by schema type

In the next example, we utilize --format field with --filter option to show how format fields affect table columns. buildtest will display the table in order of format fields specified in command line.

$ buildtest buildspec find --format name,description,file --filter tags=tutorials,executor=generic.local.sh
+--------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| name                     | description                                                                    | file                                                                                                           |
+==========================+================================================================================+================================================================================================================+
| _bin_sh_shell            | /bin/sh shell example                                                          | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/shell_examples.yml      |
+--------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| sh_shell                 | sh shell example                                                               | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/shell_examples.yml      |
+--------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| shell_options            | shell options                                                                  | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/shell_examples.yml      |
+--------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| exit1_fail               | exit 1 by default is FAIL                                                      | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/pass_returncode.yml     |
+--------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| exit1_pass               | report exit 1 as PASS                                                          | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/pass_returncode.yml     |
+--------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| returncode_list_mismatch | exit 2 failed since it failed to match returncode 1                            | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/pass_returncode.yml     |
+--------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| returncode_int_match     | exit 128 matches returncode 128                                                | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/pass_returncode.yml     |
+--------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| timelimit_min_max        | Run a sleep job for 2 seconds and test pass if its within 1.0-3.0sec           | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/runtime_status_test.yml |
+--------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| timelimit_min            | Run a sleep job for 2 seconds and test pass if its exceeds min time of 1.0 sec | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/runtime_status_test.yml |
+--------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| timelimit_max            | Run a sleep job for 2 seconds and test pass if it's within max time: 5.0 sec   | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/runtime_status_test.yml |
+--------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| timelimit_min_fail       | This test fails because it runs less than mintime of 10 second                 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/runtime_status_test.yml |
+--------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| timelimit_max_fail       | This test fails because it exceeds maxtime of 1.0 second                       | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/runtime_status_test.yml |
+--------------------------+--------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------+

buildtest makes use of python library named tabulate to generate these tables which are found in commands line like buildtest buildspec find and buildtest report.

Querying buildspec tags

If you want to retrieve all unique tags from all buildspecs you can run buildtest buildspec find --tags. This can be useful if you want to know available tags in your buildspec cache.

$ buildtest buildspec find --tags
+---------------+
| Tags          |
+===============+
| singularity   |
+---------------+
| lsf           |
+---------------+
| python        |
+---------------+
| compile       |
+---------------+
| configuration |
+---------------+
| pass          |
+---------------+
| fail          |
+---------------+
| mac           |
+---------------+
| ssh           |
+---------------+
| containers    |
+---------------+
| slurm         |
+---------------+
| system        |
+---------------+
| spack         |
+---------------+
| tutorials     |
+---------------+
| filesystem    |
+---------------+
| storage       |
+---------------+
| cobalt        |
+---------------+
| network       |
+---------------+
| ping          |
+---------------+

In addition, buildtest can group tests by tags via buildtest buildspec find --group-by-tags which can be useful if you want to know which tests get executed when running buildtest build --tags. The output is grouped by tag names, followed by name of test and description.

$ buildtest buildspec find --group-by-tags
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tags          | name                             | description                                                                    |
+===============+==================================+================================================================================+
| tutorials     | skip                             | This test is skipped                                                           |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | unskipped                        | This test is not skipped                                                       |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | executor_regex_script_schema     | regular expression test with executor using script schema                      |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | add_numbers                      | Add X+Y                                                                        |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | _bin_sh_shell                    | /bin/sh shell example                                                          |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | _bin_bash_shell                  | /bin/bash shell example                                                        |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | bash_shell                       | bash shell example                                                             |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | sh_shell                         | sh shell example                                                               |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | shell_options                    | shell options                                                                  |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | bash_login_shebang               | customize shebang line with bash login shell                                   |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | bash_nonlogin_shebang            | customize shebang line with default bash (nonlogin) shell                      |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | selinux_disable                  | Check if SELinux is Disabled                                                   |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | exit1_fail                       | exit 1 by default is FAIL                                                      |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | exit1_pass                       | report exit 1 as PASS                                                          |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | returncode_list_mismatch         | exit 2 failed since it failed to match returncode 1                            |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | returncode_int_match             | exit 128 matches returncode 128                                                |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | sleep                            | sleep 2 seconds                                                                |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | csh_shell                        | csh shell example                                                              |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
| tutorials     | bash_env_variables               | Declare environment variables in default shell (bash)                          |
+---------------+----------------------------------+--------------------------------------------------------------------------------+
...

Querying buildspec executor

If you want to know all executors in your buildspec cache use the buildtest buildspec find --executors command. This can be useful when you want to build by executors (buildtest build --executor).

$ buildtest buildspec find --executors
+-------------------------+
| executors               |
+=========================+
| generic.local.(bash|sh) |
+-------------------------+
| generic.local.bash      |
+-------------------------+
| generic.local.python    |
+-------------------------+
| generic.local.sh        |
+-------------------------+
| generic.local.csh       |
+-------------------------+

Similar to --group-by-tags, buildtest has an option to group tests by executor using --group-by-executor option. This will show tests grouped by executor, name of test and test description. Shown below is an example output.

$ buildtest buildspec find --group-by-executor
+-------------------------+----------------------------------+--------------------------------------------------------------------------------+
| executor                | name                             | description                                                                    |
+=========================+==================================+================================================================================+
| generic.local.bash      | skip                             | This test is skipped                                                           |
+-------------------------+----------------------------------+--------------------------------------------------------------------------------+
| generic.local.bash      | unskipped                        | This test is not skipped                                                       |
+-------------------------+----------------------------------+--------------------------------------------------------------------------------+
| generic.local.bash      | add_numbers                      | Add X+Y                                                                        |
+-------------------------+----------------------------------+--------------------------------------------------------------------------------+
| generic.local.bash      | python_hello                     | Hello World python                                                             |
+-------------------------+----------------------------------+--------------------------------------------------------------------------------+
| generic.local.bash      | _bin_bash_shell                  | /bin/bash shell example                                                        |
+-------------------------+----------------------------------+--------------------------------------------------------------------------------+
| generic.local.bash      | bash_shell                       | bash shell example                                                             |
+-------------------------+----------------------------------+--------------------------------------------------------------------------------+
| generic.local.bash      | bash_login_shebang               | customize shebang line with bash login shell                                   |
+-------------------------+----------------------------------+--------------------------------------------------------------------------------+
| generic.local.bash      | bash_nonlogin_shebang            | customize shebang line with default bash (nonlogin) shell                      |
+-------------------------+----------------------------------+--------------------------------------------------------------------------------+
| generic.local.bash      | run_only_macos_distro            | Run test only if distro is darwin.                                             |
+-------------------------+----------------------------------+--------------------------------------------------------------------------------+
| generic.local.bash      | run_only_linux_distro            | Run test only if distro is CentOS.                                             |
+-------------------------+----------------------------------+--------------------------------------------------------------------------------+
| generic.local.bash      | selinux_disable                  | Check if SELinux is Disabled                                                   |
+-------------------------+----------------------------------+--------------------------------------------------------------------------------+
| generic.local.bash      | sleep                            | sleep 2 seconds                                                                |
+-------------------------+----------------------------------+--------------------------------------------------------------------------------+
| generic.local.bash      | string_tag                       | tags can be a string                                                           |
+-------------------------+----------------------------------+--------------------------------------------------------------------------------+
| generic.local.bash      | list_of_strings_tags             | tags can be a list of strings                                                  |
+-------------------------+----------------------------------+--------------------------------------------------------------------------------+
...

Query Maintainers in buildspecs

When you are writing your buildspecs, you can specify the maintainers field to assign authors to buildspecs. buildtest can query the maintainers from the cache once buildspecs are loaded. You can retrieve all maintainers using --maintainers option or -m short option. In this example, we show all maintainers for buildspecs in buildspec cache

$ buildtest buildspec find --maintainers
+------------------+
| maintainers      |
+==================+
| @shahzebsiddiqui |
+------------------+

If you want to see a breakdown of maintainers by buildspec file you can use --maintainers-by-buildspecs or -mb short option. This can be useful to get correlation between maintainers and the buildspec file.

$ buildtest buildspec find -mb
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| maintainers      | buildspec                                                                                                                                                                                                                          |
+==================+====================================================================================================================================================================================================================================+
| @shahzebsiddiqui | ['/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/hello_world.yml', '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/general_tests/configuration/ulimits.yml'] |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Validate Buildspecs

buildtest can validate buildspecs through the buildtest buildspec validate command which provides analogous options for buildtest build for selecting buildspecs such as -b, -e, -t and -e. This command can be used to validate buildspecs with the JSON Schema which can be useful if you are writing a buildspec and want to validate the buildspec without running the test.

Shown below are the available command options.

$ buildtest buildspec validate --help
usage: buildtest [options] [COMMANDS] buildspec validate [-h] [-b BUILDSPEC] [-x EXCLUDE] [-t TAG] [-e EXECUTOR]

optional arguments:
  -h, --help            show this help message and exit
  -b BUILDSPEC, --buildspec BUILDSPEC
                        Specify path to buildspec (file, or directory) to validate
  -x EXCLUDE, --exclude EXCLUDE
                        Specify path to buildspec to exclude (file or directory) during validation
  -t TAG, --tag TAG     Specify buildspecs by tag name to validate
  -e EXECUTOR, --executor EXECUTOR
                        Specify buildspecs by executor name to validate

The -b option can be used to specify path to buildspec file or directory to validate buildspecs. If its a directory, buildtest will traverse all directories recursively and find any .yml file extensions and attempt to validate each buildspec. Shown below is an example output of what it may look like

$ buildtest buildspec validate -b tutorials/vars.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/vars.yml
All buildspecs passed validation!!!

If buildtest detects an error during validation, the error message will be displayed to screen as we see in this example

$ buildtest buildspec validate -b tutorials/invalid_tags.yml

file:  /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/invalid_tags.yml
________________________________________________________________________________
['network', 'network'] is not valid under any of the given schemas

Failed validating 'oneOf' in schema['properties']['tags']:
    {'oneOf': [{'type': 'string'},
               {'$ref': '#/definitions/list_of_strings'}]}

On instance['tags']:
    ['network', 'network']


Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/invalid_tags.yml
There were 1 buildspecs that failed validation

Similarly we can search buildspecs based on tags if you want to validate a group of buildspecs using the -t option. We can append -t option multiple times to search by multiple tag names. In this next example, we will validate all buildspecs for python and pass tags.

$ buildtest buildspec validate -t python -t pass
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/pass_returncode.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/python-shell.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/python-hello.yml
All buildspecs passed validation!!!

Finally we can also search by executors using the -e option which can be appended to search by multiple executors.

$ buildtest buildspec validate -e generic.local.csh
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/csh_shell_examples.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/environment.yml
All buildspecs passed validation!!!