Buildspecs Interface

Now that we learned how to build tests, in this section we will discuss how one can query a buildspec cache. In buildtest, one can load all buildspecs which is equivalent to validating all buildspecs with the appropriate schema. Buildtest will ignore all invalid buildspecs and store them in a separate file.

Note

buildtest bc is an alias for buildtest buildspec command.

Finding Buildspecs - buildtest buildspec find

The buildtest buildspec find command is used for finding buildspecs from buildspec cache. This command is also used for generating the buildspec cache. Shown below is a list of options for provided for this command.

$ buildtest buildspec find --help
usage: buildtest [options] [COMMANDS] buildspec find [-h] [-b] [-e]
                                                     [--group-by-tags]
                                                     [--group-by-executor]
                                                     [-m] [-mb] [-p] [-t]
                                                     [--filter FILTER]
                                                     [--format FORMAT]
                                                     [--helpfilter]
                                                     [--helpformat] [-n]
                                                     [--terse] [-r]
                                                     [--root ROOT]
                                                     ...

positional arguments:
  
    invalid             Show invalid buildspecs

optional arguments:
  -h, --help            show this help message and exit
  -r, --rebuild         Rebuild buildspec cache and find all buildspecs again
  --root ROOT           Specify root buildspecs (directory) path to load
                        buildspecs into buildspec cache.

filter and format:
  filter and format options

  --filter FILTER       Filter buildspec cache with filter fields in format
                        --filter key1=val1,key2=val2
  --format FORMAT       Format buildspec cache with format fields in format
                        --format field1,field2,...
  --helpfilter          Show Filter fields for --filter option for filtering
                        buildspec cache output
  --helpformat          Show Format fields for --format option for formatting
                        buildspec cache output

terse:
  terse options

  -n, --no-header       Print output without header in terse output
  --terse               Print output in machine readable format

query:
  query options to retrieve from buildspec cache

  -b, --buildspec       Get all buildspec files from cache
  -e, --executors       get all unique executors from buildspecs
  --group-by-tags       Group tests by tag name
  --group-by-executor   Group tests by executor name
  -m, --maintainers     Get all maintainers for all buildspecs
  -mb, --maintainers-by-buildspecs
                        Show maintainers breakdown by buildspecs
  -p, --paths           print all root buildspec paths
  -t, --tags            List all available tags

The buildtest buildspec find command will discover all buildspecs by recursively searching all .yml extensions. buildtest will validate each buildspec file with the json schema and buildtest will display all valid buildspecs in the output, all invalid buildspecs will be stored in a file for post-processing.

$ buildtest buildspec find
                                buildspec cache                                 
┏━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ name           ┃ type   ┃ executor        ┃ tags           ┃ description     ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ run_only_macos │ script │ generic.local.b │ mac            │ Run test only   │
│ _distro        │        │ ash             │                │ if distro is    │
│                │        │                 │                │ darwin.         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ run_only_linux │ script │ generic.local.b │ mac            │ Run test only   │
│ _distro        │        │ ash             │                │ if distro is    │
│                │        │                 │                │ CentOS.         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ hello_world    │ script │ generic.local.b │ tutorials      │ hello world     │
│                │        │ ash             │                │ example         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ skip           │ script │ generic.local.b │ tutorials      │ This test is    │
│                │        │ ash             │                │ skipped         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ unskipped      │ script │ generic.local.b │ tutorials      │ This test is    │
│                │        │ ash             │                │ not skipped     │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ foo_bar        │ script │ generic.local.s │ tutorials      │ prints variable │
│                │        │ h               │                │ $FOO            │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ circle_area    │ script │ generic.local.b │ tutorials      │ Calculate       │
│                │        │ ash             │ python         │ circle of area  │
│                │        │                 │                │ given a radius  │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ add_numbers    │ script │ generic.local.b │ tutorials      │ Add X+Y         │
│                │        │ ash             │                │                 │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ bash_login_she │ script │ generic.local.b │ tutorials      │ customize       │
│ bang           │        │ ash             │                │ shebang line    │
│                │        │                 │                │ with bash login │
│                │        │                 │                │ shell           │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ bash_nonlogin_ │ script │ generic.local.b │ tutorials      │ customize       │
│ shebang        │        │ ash             │                │ shebang line    │
│                │        │                 │                │ with default    │
│                │        │                 │                │ bash (nonlogin) │
│                │        │                 │                │ shell           │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ python_hello   │ script │ generic.local.b │ python         │ Hello World     │
│                │        │ ash             │                │ python          │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ executor_regex │ script │ generic.local.( │ tutorials      │ regular         │
│ _script_schema │        │ bash|sh)        │                │ expression test │
│                │        │                 │                │ with executor   │
│                │        │                 │                │ using script    │
│                │        │                 │                │ schema          │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ bash_env_varia │ script │ generic.local.b │ tutorials      │ Declare         │
│ bles           │        │ ash             │                │ environment     │
│                │        │                 │                │ variables in    │
│                │        │                 │                │ default shell   │
│                │        │                 │                │ (bash)          │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ csh_env_declar │ script │ generic.local.c │ tutorials      │ csh shell       │
│ ation          │        │ sh              │                │ example to      │
│                │        │                 │                │ declare         │
│                │        │                 │                │ environment     │
│                │        │                 │                │ variables       │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ tcsh_env_decla │ script │ generic.local.c │ tutorials      │ tcsh shell      │
│ ration         │        │ sh              │                │ example to      │
│                │        │                 │                │ declare         │
│                │        │                 │                │ environment     │
│                │        │                 │                │ variables       │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ sleep          │ script │ generic.local.b │ tutorials      │ sleep 2 seconds │
│                │        │ ash             │                │                 │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ run_only_platf │ script │ generic.local.b │ tutorials      │ This test will  │
│ orm_darwin     │        │ ash             │                │ only run if     │
│                │        │                 │                │ target platform │
│                │        │                 │                │ is Darwin       │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ run_only_platf │ script │ generic.local.b │ tutorials      │ This test will  │
│ orm_linux      │        │ ash             │                │ only run if     │
│                │        │                 │                │ target platform │
│                │        │                 │                │ is Linux        │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ string_tag     │ script │ generic.local.b │ network        │ tags can be a   │
│                │        │ ash             │                │ string          │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ list_of_string │ script │ generic.local.b │ network ping   │ tags can be a   │
│ s_tags         │        │ ash             │                │ list of strings │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ always_pass    │ script │ generic.local.s │                │ This test will  │
│                │        │ h               │                │ always 'PASS'   │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ always_fail    │ script │ generic.local.s │                │ This test will  │
│                │        │ h               │                │ always 'FAIL'   │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ test_fail_retu │ script │ generic.local.s │                │ This test will  │
│ rncode_match   │        │ h               │                │ 'FAIL' even if  │
│                │        │                 │                │ we have         │
│                │        │                 │                │ returncode      │
│                │        │                 │                │ match           │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ test_pass_retu │ script │ generic.local.s │                │ This test will  │
│ rncode_mismatc │        │ h               │                │ 'PASS' even if  │
│ h              │        │                 │                │ we have         │
│                │        │                 │                │ returncode      │
│                │        │                 │                │ mismatch        │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ selinux_disabl │ script │ generic.local.b │ tutorials      │ Check if        │
│ e              │        │ ash             │                │ SELinux is      │
│                │        │                 │                │ Disabled        │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ timelimit_min_ │ script │ generic.local.s │ tutorials      │ Run a sleep job │
│ max            │        │ h               │                │ for 2 seconds   │
│                │        │                 │                │ and test pass   │
│                │        │                 │                │ if its within   │
│                │        │                 │                │ 1.0-3.0sec      │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ timelimit_min  │ script │ generic.local.s │ tutorials      │ Run a sleep job │
│                │        │ h               │                │ for 2 seconds   │
│                │        │                 │                │ and test pass   │
│                │        │                 │                │ if its exceeds  │
│                │        │                 │                │ min time of 1.0 │
│                │        │                 │                │ sec             │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ timelimit_max  │ script │ generic.local.s │ tutorials      │ Run a sleep job │
│                │        │ h               │                │ for 2 seconds   │
│                │        │                 │                │ and test pass   │
│                │        │                 │                │ if it's within  │
│                │        │                 │                │ max time: 5.0   │
│                │        │                 │                │ sec             │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ timelimit_min_ │ script │ generic.local.s │ tutorials      │ This test fails │
│ fail           │        │ h               │                │ because it runs │
│                │        │                 │                │ less than       │
│                │        │                 │                │ mintime of 10   │
│                │        │                 │                │ second          │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ timelimit_max_ │ script │ generic.local.s │ tutorials      │ This test fails │
│ fail           │        │ h               │                │ because it      │
│                │        │                 │                │ exceeds maxtime │
│                │        │                 │                │ of 1.0 second   │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ metric_regex_e │ script │ generic.local.s │ tutorials      │ capture result  │
│ xample         │        │ h               │                │ metric from     │
│                │        │                 │                │ output          │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ _bin_sh_shell  │ script │ generic.local.s │ tutorials      │ /bin/sh shell   │
│                │        │ h               │                │ example         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ _bin_bash_shel │ script │ generic.local.b │ tutorials      │ /bin/bash shell │
│ l              │        │ ash             │                │ example         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ bash_shell     │ script │ generic.local.b │ tutorials      │ bash shell      │
│                │        │ ash             │                │ example         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ sh_shell       │ script │ generic.local.s │ tutorials      │ sh shell        │
│                │        │ h               │                │ example         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ shell_options  │ script │ generic.local.s │ tutorials      │ shell options   │
│                │        │ h               │                │                 │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ csh_shell      │ script │ generic.local.c │ tutorials      │ csh shell       │
│                │        │ sh              │                │ example         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ run_only_as_ro │ script │ generic.local.b │ tutorials      │ This test will  │
│ ot             │        │ ash             │                │ only run if     │
│                │        │                 │                │ current user is │
│                │        │                 │                │ root            │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ exit1_fail     │ script │ generic.local.b │ tutorials fail │ exit 1 by       │
│                │        │ ash             │                │ default is FAIL │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ exit1_pass     │ script │ generic.local.b │ tutorials pass │ report exit 1   │
│                │        │ ash             │                │ as PASS         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ returncode_lis │ script │ generic.local.b │ tutorials fail │ exit 2 failed   │
│ t_mismatch     │        │ ash             │                │ since it failed │
│                │        │                 │                │ to match        │
│                │        │                 │                │ returncode 1    │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ returncode_int │ script │ generic.local.b │ tutorials pass │ exit 128        │
│ _match         │        │ ash             │                │ matches         │
│                │        │                 │                │ returncode 128  │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ metric_variabl │ script │ generic.local.s │ tutorials      │ capture result  │
│ e_assignment   │        │ h               │                │ metric based on │
│                │        │                 │                │ variables and   │
│                │        │                 │                │ environment     │
│                │        │                 │                │ variable        │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ variables_bash │ script │ generic.local.b │ tutorials      │ Declare shell   │
│                │        │ ash             │                │ variables in    │
│                │        │                 │                │ bash            │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ status_regex_p │ script │ generic.local.b │ system         │ Pass test based │
│ ass            │        │ ash             │                │ on regular      │
│                │        │                 │                │ expression      │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ status_regex_f │ script │ generic.local.b │ system         │ Pass test based │
│ ail            │        │ ash             │                │ on regular      │
│                │        │                 │                │ expression      │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ executors_vars │ script │ generic.local.( │ tutorials      │ Declaring env   │
│ _env_declarati │        │ bash|sh)        │                │ and vars by     │
│ on             │        │                 │                │ executors       │
│                │        │                 │                │ section         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ status_returnc │ script │ generic.local.( │ tutorials      │ define status   │
│ ode_by_executo │        │ bash|sh)        │                │ and metrics per │
│ rs             │        │                 │                │ executor type.  │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ executors_sbat │ script │ generic.local.( │ tutorials      │ Declaring env   │
│ ch_declaration │        │ bash|sh)        │                │ and vars by     │
│                │        │                 │                │ executors       │
│                │        │                 │                │ section         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ root_disk_usag │ script │ generic.local.b │ filesystem     │ Check root disk │
│ e              │        │ ash             │ storage        │ usage and       │
│                │        │                 │                │ report if it    │
│                │        │                 │                │ exceeds         │
│                │        │                 │                │ threshold       │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ systemd_defaul │ script │ generic.local.b │ system         │ check if        │
│ t_target       │        │ ash             │                │ default target  │
│                │        │                 │                │ is multi-user.t │
│                │        │                 │                │ arget           │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ kernel_swapusa │ script │ generic.local.b │ configuration  │ Retrieve Kernel │
│ ge             │        │ ash             │                │ Swap Usage      │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ ulimit_fileloc │ script │ generic.local.b │ system         │ Check if file   │
│ k_unlimited    │        │ ash             │                │ lock is set to  │
│                │        │                 │                │ unlimited in    │
│                │        │                 │                │ ulimits         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ ulimit_cputime │ script │ generic.local.b │ system         │ Check if        │
│ _unlimited     │        │ ash             │                │ cputime is set  │
│                │        │                 │                │ to unlimited in │
│                │        │                 │                │ ulimits         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ ulimit_stacksi │ script │ generic.local.b │ system         │ Check if stack  │
│ ze_unlimited   │        │ ash             │                │ size is set to  │
│                │        │                 │                │ unlimited in    │
│                │        │                 │                │ ulimits         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ ulimit_vmsize_ │ script │ generic.local.b │ system         │ Check virtual   │
│ unlimited      │        │ ash             │                │ memory size and │
│                │        │                 │                │ check if its    │
│                │        │                 │                │ set to          │
│                │        │                 │                │ unlimited       │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ ulimit_filedes │ script │ generic.local.b │ system         │ Check if open   │
│ criptor_4096   │        │ ash             │                │ file            │
│                │        │                 │                │ descriptors     │
│                │        │                 │                │ limit is set to │
│                │        │                 │                │ 4096            │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ ulimit_max_use │ script │ generic.local.b │ system         │ Check max       │
│ r_process_2048 │        │ ash             │                │ number of user  │
│                │        │                 │                │ process limit   │
│                │        │                 │                │ is set to 2048  │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ current_user_q │ script │ generic.local.b │ slurm          │ show all        │
│ ueue           │        │ ash             │                │ current pending │
│                │        │                 │                │ jobs for        │
│                │        │                 │                │ current user    │
│                │        │                 │                │ (squeue -u      │
│                │        │                 │                │ $USER)          │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ show_all_jobs  │ script │ generic.local.b │ slurm          │ show all        │
│                │        │ ash             │                │ pending +       │
│                │        │                 │                │ running jobs    │
│                │        │                 │                │ (squeue -a)     │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ nodes_state_do │ script │ generic.local.b │ slurm          │ Show nodes in   │
│ wn             │        │ ash             │                │ DOWN state      │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ nodes_state_re │ script │ generic.local.b │ slurm          │ Show nodes in   │
│ boot           │        │ ash             │                │ REBOOT state    │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ nodes_state_al │ script │ generic.local.b │ slurm          │ Show nodes in   │
│ located        │        │ ash             │                │ ALLOCATED state │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ nodes_state_co │ script │ generic.local.b │ slurm          │ Show nodes in   │
│ mpleting       │        │ ash             │                │ COMPLETING      │
│                │        │                 │                │ state           │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ nodes_state_id │ script │ generic.local.b │ slurm          │ Show nodes in   │
│ le             │        │ ash             │                │ IDLE state      │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ node_down_fail │ script │ generic.local.b │ slurm          │ Show nodes      │
│ _list_reason   │        │ ash             │                │ DOWN, DRAINED,  │
│                │        │                 │                │ FAIL or FAILING │
│                │        │                 │                │ and list reason │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ dead_nodes     │ script │ generic.local.b │ slurm          │ Show            │
│                │        │ ash             │                │ non-responding  │
│                │        │                 │                │ nodes           │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ get_partitions │ script │ generic.local.b │ slurm          │ Get all slurm   │
│                │        │ ash             │                │ partitions      │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ sinfo_version  │ script │ generic.local.b │ slurm          │ get slurm       │
│                │        │ ash             │                │ version         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ show_accounts  │ script │ generic.local.b │ slurm          │ run sacctmgr    │
│                │        │ ash             │                │ list accounts   │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ show_users     │ script │ generic.local.b │ slurm          │ run sacctmgr    │
│                │        │ ash             │                │ list users      │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ show_qos       │ script │ generic.local.b │ slurm          │ run sacctmgr    │
│                │        │ ash             │                │ list qos        │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ show_tres      │ script │ generic.local.b │ slurm          │ run sacctmgr    │
│                │        │ ash             │                │ list tres       │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ slurm_config   │ script │ generic.local.b │ slurm          │ run scontrol    │
│                │        │ ash             │                │ show config     │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ show_partition │ script │ generic.local.b │ slurm          │ run scontrol    │
│                │        │ ash             │                │ show partition  │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ show_lsf_confi │ script │ generic.local.b │ lsf            │ Show LSF        │
│ guration       │        │ ash             │                │ configuration   │
│                │        │                 │                │ using lsinfo    │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ show_lsf_model │ script │ generic.local.b │ lsf            │ Show            │
│ s              │        │ ash             │                │ information     │
│                │        │                 │                │ about host      │
│                │        │                 │                │ models in LSF   │
│                │        │                 │                │ cluster         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ show_lsf_resou │ script │ generic.local.b │ lsf            │ Show            │
│ rces           │        │ ash             │                │ information     │
│                │        │                 │                │ about LSF       │
│                │        │                 │                │ resources       │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ lsf_version    │ script │ generic.local.b │ lsf            │ Display lsf     │
│                │        │ ash             │                │ version using   │
│                │        │                 │                │ lsinfo          │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ show_lsf_user_ │ script │ generic.local.b │ lsf            │ Show            │
│ groups         │        │ ash             │                │ information     │
│                │        │                 │                │ about all LSF   │
│                │        │                 │                │ user groups     │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ display_lsf_ho │ script │ generic.local.b │ lsf            │ Show all hosts  │
│ sts            │        │ ash             │                │ in LSF cluster  │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ display_hosts_ │ script │ generic.local.b │ lsf            │ Show all hosts  │
│ format         │        │ ash             │                │ with column     │
│                │        │                 │                │ hostname and    │
│                │        │                 │                │ status          │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ bhosts_version │ script │ generic.local.b │ lsf            │ display version │
│                │        │ ash             │                │ from bhosts     │
│                │        │                 │                │ command         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ show_host_grou │ script │ generic.local.b │ lsf            │ Show            │
│ ps             │        │ ash             │                │ information     │
│                │        │                 │                │ about host      │
│                │        │                 │                │ groups using    │
│                │        │                 │                │ bmgroup         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ show_lsf_queue │ script │ generic.local.b │ lsf            │ Show LSF queues │
│ s              │        │ ash             │                │                 │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ show_lsf_queue │ script │ generic.local.b │ lsf            │ Show LSF queues │
│ s_formatted    │        │ ash             │                │ with formatted  │
│                │        │                 │                │ columns         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ show_lsf_queue │ script │ generic.local.b │ lsf            │ Show LSF queues │
│ s_current_user │        │ ash             │                │ available for   │
│                │        │                 │                │ current user    │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ qsub_version   │ script │ generic.local.s │ cobalt         │ print version   │
│                │        │ h               │                │ for qsub        │
│                │        │                 │                │ command         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ qselect_versio │ script │ generic.local.s │ cobalt         │ print version   │
│ n              │        │ h               │                │ for qselect     │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ cqsub_version  │ script │ generic.local.s │ cobalt         │ print version   │
│                │        │ h               │                │ for cqsub       │
│                │        │                 │                │ command         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ qdel_version   │ script │ generic.local.s │ cobalt         │ print version   │
│                │        │ h               │                │ for qdel        │
│                │        │                 │                │ command         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ qmove_version  │ script │ generic.local.s │ cobalt         │ print version   │
│                │        │ h               │                │ for qmove       │
│                │        │                 │                │ command         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ show_jobs      │ script │ generic.local.s │ cobalt         │ Show all jobs   │
│                │        │ h               │                │ in queue        │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ show_queues    │ script │ generic.local.s │ cobalt         │ Show all queues │
│                │        │ h               │                │                 │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ inspect_image  │ script │ generic.local.b │ containers     │ Inspect image   │
│                │        │ ash             │ singularity    │ via             │
│                │        │                 │                │ 'singularity    │
│                │        │                 │                │ inspect'        │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ pullImage_dock │ script │ generic.local.b │ containers     │ Pull image dock │
│ erhub          │        │ ash             │ singularity    │ er://godlovedc/ │
│                │        │                 │                │ lolcow from     │
│                │        │                 │                │ DockerHub       │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ pullImage_syla │ script │ generic.local.b │ containers     │ Pull image libr │
│ bscloud        │        │ ash             │ singularity    │ ary://alpine:la │
│                │        │                 │                │ test from       │
│                │        │                 │                │ Sylabs Cloud    │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ pullImage_shub │ script │ generic.local.b │ containers     │ Pull image shub │
│                │        │ ash             │ singularity    │ ://vsoch/singul │
│                │        │                 │                │ arity-images    │
│                │        │                 │                │ from            │
│                │        │                 │                │ SingularityHub  │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ build_sif_from │ script │ generic.local.b │ containers     │ build sif image │
│ _dockerimage   │        │ ash             │ singularity    │ from docker     │
│                │        │                 │                │ image docker:// │
│                │        │                 │                │ godlovedc/lolco │
│                │        │                 │                │ w               │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ build_sandbox_ │ script │ generic.local.b │ containers     │ build sandbox   │
│ image          │        │ ash             │ singularity    │ image from      │
│                │        │                 │                │ docker image do │
│                │        │                 │                │ cker://godloved │
│                │        │                 │                │ c/lolcow        │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ build_remoteim │ script │ generic.local.b │ containers     │ build remote    │
│ ages           │        │ ash             │ singularity    │ hosted image    │
│                │        │                 │                │ from AWS        │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ runImage       │ script │ generic.local.b │ containers     │ run container d │
│                │        │ ash             │ singularity    │ ocker://godlove │
│                │        │                 │                │ dc/lolcow       │
└────────────────┴────────┴─────────────────┴────────────────┴─────────────────┘

buildtest will load all discovered buildspecs in a cache file (JSON) which is created upon running buildtest buildspec find. Any subsequent runs will read from cache and update if any new buildspecs are added. If you make changes to buildspec you should rebuild the buildspec cache by running:

$ buildtest buildspec find --rebuild

If you want to find all buildspec files in cache you can run buildtest buildspec find --buildspec. Shown below is an example output.

$ buildtest buildspec find --buildspec
                               List of Buildspecs                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Buildspecs                                                                   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
...

The buildtest buildspec find --paths will display a list of root directories buildtest will search for buildspecs when runninh buildtest buildspec find. One can define these directories in the configuration file or pass them via command line.

$ buildtest buildspec find --paths
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/general_tests

buildtest will search buildspecs in buildspecs root defined in your configuration, which is a list of directory paths to search for buildspecs. If you want to load buildspecs from a directory path, you can specify a directory via --root option in the format: buildtest buildspec find --root <path> --rebuild. buildtest will load all valid buildspecs into cache and ignore the rest. It’s important to add --rebuild if you want to regenerate buildspec cache.

Filtering buildspec

Once you have a buildspec cache, we can query the buildspec cache for certain attributes. When you run buildtest buildspec find it will report all buildspecs from cache which can be difficult to process. Therefore, we have a filter option (--filter) to restrict our search. Let’s take a look at the available filter fields that are acceptable with filter option.

$ buildtest buildspec find --helpfilter
             Filter Field Description              
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ Field      ┃ Description               ┃ Type   ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ buildspecs │ Filter tests by buildspec │ FILE   │
│ executor   │ Filter by executor name   │ STRING │
│ tags       │ Filter by tag name        │ STRING │
│ type       │ Filter by schema type     │ STRING │
└────────────┴───────────────────────────┴────────┘

The --filter option expects an arguments in key=value format as follows:

buildtest buildspec find --filter key1=value1,key2=value2,key3=value3

We can filter buildspec cache by tags=fail which will query all tests with associated tag field in test.

$ buildtest buildspec find --filter tags=fail
                                buildspec cache                                 
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ name            ┃ type   ┃ executor       ┃ tags           ┃ description     ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ exit1_fail      │ script │ generic.local. │ tutorials fail │ exit 1 by       │
│                 │        │ bash           │                │ default is FAIL │
├─────────────────┼────────┼────────────────┼────────────────┼─────────────────┤
│ returncode_list │ script │ generic.local. │ tutorials fail │ exit 2 failed   │
│ _mismatch       │        │ bash           │                │ since it failed │
│                 │        │                │                │ to match        │
│                 │        │                │                │ returncode 1    │
└─────────────────┴────────┴────────────────┴────────────────┴─────────────────┘

In addition, we can query buildspecs by schema type using type property. In this example we query all tests by type property

$ buildtest buildspec find --filter type=script
                                buildspec cache                                 
┏━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ name           ┃ type   ┃ executor        ┃ tags           ┃ description     ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ run_only_macos │ script │ generic.local.b │ mac            │ Run test only   │
│ _distro        │        │ ash             │                │ if distro is    │
│                │        │                 │                │ darwin.         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ run_only_linux │ script │ generic.local.b │ mac            │ Run test only   │
│ _distro        │        │ ash             │                │ if distro is    │
│                │        │                 │                │ CentOS.         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ hello_world    │ script │ generic.local.b │ tutorials      │ hello world     │
│                │        │ ash             │                │ example         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ skip           │ script │ generic.local.b │ tutorials      │ This test is    │
│                │        │ ash             │                │ skipped         │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
│ unskipped      │ script │ generic.local.b │ tutorials      │ This test is    │
│                │        │ ash             │                │ not skipped     │
├────────────────┼────────┼─────────────────┼────────────────┼─────────────────┤
...

Finally, we can combine multiple filter fields separated by comma, in the next example we can query all buildspecs with tags=tutorials, executor=generic.local.sh, and type=script

$ buildtest buildspec find --filter tags=tutorials,executor=generic.local.sh,type=script
                                buildspec cache                                 
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
┃ name             ┃ type   ┃ executor         ┃ tags      ┃ description       ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│ foo_bar          │ script │ generic.local.sh │ tutorials │ prints variable   │
│                  │        │                  │           │ $FOO              │
├──────────────────┼────────┼──────────────────┼───────────┼───────────────────┤
│ timelimit_min_ma │ script │ generic.local.sh │ tutorials │ Run a sleep job   │
│ x                │        │                  │           │ for 2 seconds and │
│                  │        │                  │           │ test pass if its  │
│                  │        │                  │           │ within 1.0-3.0sec │
├──────────────────┼────────┼──────────────────┼───────────┼───────────────────┤
│ timelimit_min    │ script │ generic.local.sh │ tutorials │ Run a sleep job   │
│                  │        │                  │           │ for 2 seconds and │
│                  │        │                  │           │ test pass if its  │
│                  │        │                  │           │ exceeds min time  │
│                  │        │                  │           │ of 1.0 sec        │
├──────────────────┼────────┼──────────────────┼───────────┼───────────────────┤
│ timelimit_max    │ script │ generic.local.sh │ tutorials │ Run a sleep job   │
│                  │        │                  │           │ for 2 seconds and │
│                  │        │                  │           │ test pass if it's │
│                  │        │                  │           │ within max time:  │
│                  │        │                  │           │ 5.0 sec           │
├──────────────────┼────────┼──────────────────┼───────────┼───────────────────┤
│ timelimit_min_fa │ script │ generic.local.sh │ tutorials │ This test fails   │
│ il               │        │                  │           │ because it runs   │
│                  │        │                  │           │ less than mintime │
│                  │        │                  │           │ of 10 second      │
├──────────────────┼────────┼──────────────────┼───────────┼───────────────────┤
│ timelimit_max_fa │ script │ generic.local.sh │ tutorials │ This test fails   │
│ il               │        │                  │           │ because it        │
│                  │        │                  │           │ exceeds maxtime   │
│                  │        │                  │           │ of 1.0 second     │
├──────────────────┼────────┼──────────────────┼───────────┼───────────────────┤
│ metric_regex_exa │ script │ generic.local.sh │ tutorials │ capture result    │
│ mple             │        │                  │           │ metric from       │
│                  │        │                  │           │ output            │
├──────────────────┼────────┼──────────────────┼───────────┼───────────────────┤
│ _bin_sh_shell    │ script │ generic.local.sh │ tutorials │ /bin/sh shell     │
│                  │        │                  │           │ example           │
├──────────────────┼────────┼──────────────────┼───────────┼───────────────────┤
│ sh_shell         │ script │ generic.local.sh │ tutorials │ sh shell example  │
├──────────────────┼────────┼──────────────────┼───────────┼───────────────────┤
│ shell_options    │ script │ generic.local.sh │ tutorials │ shell options     │
├──────────────────┼────────┼──────────────────┼───────────┼───────────────────┤
│ metric_variable_ │ script │ generic.local.sh │ tutorials │ capture result    │
│ assignment       │        │                  │           │ metric based on   │
│                  │        │                  │           │ variables and     │
│                  │        │                  │           │ environment       │
│                  │        │                  │           │ variable          │
└──────────────────┴────────┴──────────────────┴───────────┴───────────────────┘

We can filter output of buildspec cache by buildspec using --filter buildspec=<path> which expects a path to buildspec file. The buildspec must be in the cache and file path must exist in order to fetch the result. The path can be absolute or relative path.

In this next example, we will filter cache by file tutorials/pass_returncode.yml and use --format name,buildspec to format columns. The --format buildspec will show full path to buildspec and name refers to name of test. For more details on –format see Format buildspec cache.

$ buildtest buildspec find --filter buildspec=tutorials/pass_returncode.yml --format name,buildspec
                                buildspec cache                                 
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                     ┃ buildspec                                         ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ exit1_fail               │ /home/docs/checkouts/readthedocs.org/user_builds/ │
│                          │ buildtest/checkouts/devel/tutorials/pass_returnco │
│                          │ de.yml                                            │
├──────────────────────────┼───────────────────────────────────────────────────┤
│ exit1_pass               │ /home/docs/checkouts/readthedocs.org/user_builds/ │
│                          │ buildtest/checkouts/devel/tutorials/pass_returnco │
│                          │ de.yml                                            │
├──────────────────────────┼───────────────────────────────────────────────────┤
│ returncode_list_mismatch │ /home/docs/checkouts/readthedocs.org/user_builds/ │
│                          │ buildtest/checkouts/devel/tutorials/pass_returnco │
│                          │ de.yml                                            │
├──────────────────────────┼───────────────────────────────────────────────────┤
│ returncode_int_match     │ /home/docs/checkouts/readthedocs.org/user_builds/ │
│                          │ buildtest/checkouts/devel/tutorials/pass_returnco │
│                          │ de.yml                                            │
└──────────────────────────┴───────────────────────────────────────────────────┘

Format buildspec cache

We have seen how one can filter buildspecs, but we can also configure which columns to display in the output of buildtest buildspec find. By default, we show a pre-selected format fields in the output, however there are more format fields available that can be configured at the command line.

The format fields are specified in comma separated format such as buildtest buildspec find --format <field1>,<field2>,.... You can see a list of all format fields by --helpformat option as shown below

$ buildtest buildspec find --helpformat
              Format Field Description               
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Field       ┃ Description                         ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildspec   │ Display name of buildspec file      │
│ description │ Show description of test            │
│ executor    │ Display 'executor' property in test │
│ name        │ Display name of test                │
│ tags        │ Display 'tag' property in test      │
│ type        │ Display 'type' property in test     │
└─────────────┴─────────────────────────────────────┘

In the next example, we utilize --format field with --filter option to show how format fields affect table columns. buildtest will display the table in order of format fields specified in command line.

$ buildtest buildspec find --format name,description,buildspec --filter tags=tutorials,executor=generic.local.sh
                                buildspec cache                                 
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                    ┃ description              ┃ buildspec               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ foo_bar                 │ prints variable $FOO     │ /home/docs/checkouts/re │
│                         │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/maintai │
│                         │                          │ ners_example.yml        │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ timelimit_min_max       │ Run a sleep job for 2    │ /home/docs/checkouts/re │
│                         │ seconds and test pass if │ adthedocs.org/user_buil │
│                         │ its within 1.0-3.0sec    │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/runtime │
│                         │                          │ _status_test.yml        │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ timelimit_min           │ Run a sleep job for 2    │ /home/docs/checkouts/re │
│                         │ seconds and test pass if │ adthedocs.org/user_buil │
│                         │ its exceeds min time of  │ ds/buildtest/checkouts/ │
│                         │ 1.0 sec                  │ devel/tutorials/runtime │
│                         │                          │ _status_test.yml        │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ timelimit_max           │ Run a sleep job for 2    │ /home/docs/checkouts/re │
│                         │ seconds and test pass if │ adthedocs.org/user_buil │
│                         │ it's within max time:    │ ds/buildtest/checkouts/ │
│                         │ 5.0 sec                  │ devel/tutorials/runtime │
│                         │                          │ _status_test.yml        │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ timelimit_min_fail      │ This test fails because  │ /home/docs/checkouts/re │
│                         │ it runs less than        │ adthedocs.org/user_buil │
│                         │ mintime of 10 second     │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/runtime │
│                         │                          │ _status_test.yml        │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ timelimit_max_fail      │ This test fails because  │ /home/docs/checkouts/re │
│                         │ it exceeds maxtime of    │ adthedocs.org/user_buil │
│                         │ 1.0 second               │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/runtime │
│                         │                          │ _status_test.yml        │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ metric_regex_example    │ capture result metric    │ /home/docs/checkouts/re │
│                         │ from output              │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/metrics │
│                         │                          │ _regex.yml              │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ _bin_sh_shell           │ /bin/sh shell example    │ /home/docs/checkouts/re │
│                         │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/shell_e │
│                         │                          │ xamples.yml             │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ sh_shell                │ sh shell example         │ /home/docs/checkouts/re │
│                         │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/shell_e │
│                         │                          │ xamples.yml             │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ shell_options           │ shell options            │ /home/docs/checkouts/re │
│                         │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/shell_e │
│                         │                          │ xamples.yml             │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ metric_variable_assignm │ capture result metric    │ /home/docs/checkouts/re │
│ ent                     │ based on variables and   │ adthedocs.org/user_buil │
│                         │ environment variable     │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/metrics │
│                         │                          │ _variable.yml           │
└─────────────────────────┴──────────────────────────┴─────────────────────────┘

Querying buildspec tags

If you want to retrieve all unique tags from all buildspecs you can run buildtest buildspec find --tags. This can be useful if you want to know available tags in your buildspec cache.

$ buildtest buildspec find --tags
  List of Tags   
┏━━━━━━━━━━━━━━━┓
┃ Tags          ┃
┡━━━━━━━━━━━━━━━┩
│ network       │
│ singularity   │
│ configuration │
│ ping          │
│ fail          │
│ pass          │
│ storage       │
│ slurm         │
│ lsf           │
│ tutorials     │
│ cobalt        │
│ containers    │
│ filesystem    │
│ mac           │
│ system        │
│ python        │
└───────────────┘

In addition, buildtest can group tests by tags via buildtest buildspec find --group-by-tags which can be useful if you want to know which tests get executed when running buildtest build --tags. The output is grouped by tag names, followed by name of test and description.

$ buildtest buildspec find --group-by-tags
                                 Tests by Tags                                  
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Tags          ┃ Name                         ┃ Description                   ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ mac           │ run_only_macos_distro        │ Run test only if distro is    │
│               │                              │ darwin.                       │
│ mac           │ run_only_linux_distro        │ Run test only if distro is    │
│               │                              │ CentOS.                       │
│ tutorials     │ hello_world                  │ hello world example           │
│ tutorials     │ skip                         │ This test is skipped          │
│ tutorials     │ unskipped                    │ This test is not skipped      │
│ tutorials     │ foo_bar                      │ prints variable $FOO          │
│ tutorials     │ circle_area                  │ Calculate circle of area      │
│               │                              │ given a radius                │
│ tutorials     │ add_numbers                  │ Add X+Y                       │
│ tutorials     │ bash_login_shebang           │ customize shebang line with   │
│               │                              │ bash login shell              │
│ tutorials     │ bash_nonlogin_shebang        │ customize shebang line with   │
│               │                              │ default bash (nonlogin) shell │
│ tutorials     │ executor_regex_script_schema │ regular expression test with  │
│               │                              │ executor using script schema  │
│ tutorials     │ bash_env_variables           │ Declare environment variables │
│               │                              │ in default shell (bash)       │
│ tutorials     │ csh_env_declaration          │ csh shell example to declare  │
│               │                              │ environment variables         │
│ tutorials     │ tcsh_env_declaration         │ tcsh shell example to declare │
│               │                              │ environment variables         │
│ tutorials     │ sleep                        │ sleep 2 seconds               │
│ tutorials     │ run_only_platform_darwin     │ This test will only run if    │
│               │                              │ target platform is Darwin     │
│ tutorials     │ run_only_platform_linux      │ This test will only run if    │
│               │                              │ target platform is Linux      │
│ tutorials     │ selinux_disable              │ Check if SELinux is Disabled  │
│ tutorials     │ timelimit_min_max            │ Run a sleep job for 2 seconds │
│               │                              │ and test pass if its within   │
│               │                              │ 1.0-3.0sec                    │
│ tutorials     │ timelimit_min                │ Run a sleep job for 2 seconds │
│               │                              │ and test pass if its exceeds  │
│               │                              │ min time of 1.0 sec           │
│ tutorials     │ timelimit_max                │ Run a sleep job for 2 seconds │
│               │                              │ and test pass if it's within  │
...

Querying buildspec executor

If you want to know all executors in your buildspec cache use the buildtest buildspec find --executors command. This can be useful when you want to build by executors (buildtest build --executor).

$ buildtest buildspec find --executors
     List of Executors     
┏━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Executors               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ generic.local.sh        │
│ generic.local.csh       │
│ generic.local.bash      │
│ generic.local.(bash|sh) │
└─────────────────────────┘

Similar to --group-by-tags, buildtest has an option to group tests by executor using --group-by-executor option. This will show tests grouped by executor, name of test and test description. Shown below is an example output.

$ buildtest buildspec find --group-by-executor
                               Tests by Executors                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Executors               ┃ Name                     ┃ Description             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ generic.local.bash      │ run_only_macos_distro    │ Run test only if distro │
│                         │                          │ is darwin.              │
│ generic.local.bash      │ run_only_linux_distro    │ Run test only if distro │
│                         │                          │ is CentOS.              │
│ generic.local.bash      │ hello_world              │ hello world example     │
│ generic.local.bash      │ skip                     │ This test is skipped    │
│ generic.local.bash      │ unskipped                │ This test is not        │
│                         │                          │ skipped                 │
│ generic.local.bash      │ circle_area              │ Calculate circle of     │
│                         │                          │ area given a radius     │
│ generic.local.bash      │ add_numbers              │ Add X+Y                 │
│ generic.local.bash      │ bash_login_shebang       │ customize shebang line  │
│                         │                          │ with bash login shell   │
│ generic.local.bash      │ bash_nonlogin_shebang    │ customize shebang line  │
│                         │                          │ with default bash       │
│                         │                          │ (nonlogin) shell        │
│ generic.local.bash      │ python_hello             │ Hello World python      │
│ generic.local.bash      │ bash_env_variables       │ Declare environment     │
│                         │                          │ variables in default    │
│                         │                          │ shell (bash)            │
│ generic.local.bash      │ sleep                    │ sleep 2 seconds         │
│ generic.local.bash      │ run_only_platform_darwin │ This test will only run │
│                         │                          │ if target platform is   │
│                         │                          │ Darwin                  │
│ generic.local.bash      │ run_only_platform_linux  │ This test will only run │
│                         │                          │ if target platform is   │
│                         │                          │ Linux                   │
...

Query Maintainers

When you are writing your buildspecs, you can specify the maintainers field to assign authors to buildspecs. buildtest can query the maintainers from the cache once buildspecs are loaded. You can retrieve all maintainers using --maintainers option or -m short option. In this example, we show all maintainers for buildspecs in buildspec cache

$ buildtest buildspec find --maintainers
List of Maintainers 
┏━━━━━━━━━━━━━━━━━━┓
┃ Maintainers      ┃
┡━━━━━━━━━━━━━━━━━━┩
│ @shahzebsiddiqui │
│ @johndoe         │
│ @bobsmith        │
└──────────────────┘

If you want to see a breakdown of maintainers by buildspec file you can use --maintainers-by-buildspecs or -mb short option. This can be useful to get correlation between maintainers and the buildspec file.

$ buildtest buildspec find -mb
                     Breakdown of buildspecs by maintainers                     
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Maintainers      ┃ Buildspec                                                 ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ @shahzebsiddiqui │ /home/docs/checkouts/readthedocs.org/user_builds/buildtes │
│                  │ t/checkouts/devel/tutorials/hello_world.yml:/home/docs/ch │
│                  │ eckouts/readthedocs.org/user_builds/buildtest/checkouts/d │
│                  │ evel/general_tests/configuration/ulimits.yml              │
│ @johndoe         │ /home/docs/checkouts/readthedocs.org/user_builds/buildtes │
│                  │ t/checkouts/devel/tutorials/maintainers_example.yml       │
│ @bobsmith        │ /home/docs/checkouts/readthedocs.org/user_builds/buildtes │
│                  │ t/checkouts/devel/tutorials/maintainers_example.yml       │
└──────────────────┴───────────────────────────────────────────────────────────┘

Terse Output

You can use the --terse option to print output of buildtest buildspec find in terse format that can be useful if you want to parse content of file. In example below, we will print output of tags in terse format, the first entry tags is the header followed by list of unique tags. The --no-header option can be used to disable printing of header title.

$ buildtest buildspec find -t --terse
tag
network
singularity
configuration
ping
fail
pass
storage
slurm
lsf
tutorials
cobalt
containers
filesystem
mac
system
python

Invalid Buildspecs - buildtest buildspec find invalid

buildtest will store invalid buildspecs in the cache file which can be retrieved using buildtest buildspec find invalid. buildtest will attempt to parse each buildspec and store error message for every buildspec. If you run without any options it will report a list of invalid buildspecs as shown below

$ buildtest buildspec find invalid
                               Invalid Buildspecs                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Buildspec                                                                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
└──────────────────────────────────────────────────────────────────────────────┘

If you want to see error messages for each buildspec you can pass the -e or --error option which will display output of each buildspec followed by error message.

$ buildtest buildspec find invalid -e
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… ─
"'[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/invalid_buildspec_section.yml]: type badscript is not known to buildtest.'"
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… ─
"['network', 'network'] is not valid under any of the given schemas\n\nFailed validating 'oneOf' in schema['properties']['tags']:\n    {'oneOf': [{'type': 'string'},\n               {'$ref': '#/definitions/list_of_strings'}]}\n\nOn instance['tags']:\n    ['network', 'network']"
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… ─
"'create_burst_buffer_multiple_executors' is too long\n\nFailed validating 'maxLength' in schema['properties']['buildspecs']['propertyNames']:\n    {'maxLength': 32, 'pattern': '^[A-Za-z_.][A-Za-z0-9_.]*$'}\n\nOn instance['buildspecs']:\n    'create_burst_buffer_multiple_executors'"
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… ─
'"[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/invalid_executor.yml]: Unable to find executor: badexecutor in [\'generic.local.bash\', \'generic.local.sh\', \'generic.local.csh\', \'generic.local.zsh\']"'
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… ─
'"[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/general_tests/sched/pbs/hostname.yml]: Unable to find executor: generic.pbs.workq in [\'generic.local.bash\', \'generic.local.sh\', \'generic.local.csh\', \'generic.local.zsh\']"'
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… ─
'"[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/general_tests/sched/pbs/batch.yml]: Unable to find executor: generic.pbs.workq in [\'generic.local.bash\', \'generic.local.sh\', \'generic.local.csh\', \'generic.local.zsh\']"'

Cache Summary - buildtest buildspec summary

The buildtest buildspec summary command can be used to provide a summary of the buildspec cache. This command can be used assuming your cache is built via buildtest buildspec find. Shown below is a summary of the cache file.

$ buildtest buildspec summary
╭──────────────────────────────────────────────────────────────────────────────╮
│                                                                              │
│     Reading Buildspec Cache File:   /home/docs/checkouts/readthedocs.org/use │
│ r_builds/buildtest/checkouts/devel/var/buildspecs/cache.json                 │
│     Total Valid Buildspecs:         45                                       │
│     Total Invalid Buildspecs:       6                                        │
│     Total Unique Tags:              16                                       │
│     Total Maintainers:              3                                        │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
      Tag Breakdown           Executor Breakdown        Maintainers Breakdown   
┏━━━━━━━━━━━┳━━━━━━━━━━━━┓┏━━━━━━━━━━━━┳━━━━━━━━━━━━┓┏━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃           ┃ total      ┃┃            ┃ total      ┃┃            ┃ total      ┃
┃ tag       ┃ tests      ┃┃ executor   ┃ tests      ┃┃ maintaine… ┃ buildspecs ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━┩┡━━━━━━━━━━━━╇━━━━━━━━━━━━┩┡━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ mac       │ 2          ││ generic.l… │ 73         ││ @shahzebs… │ 2          │
│ tutorials │ 38         ││ generic.l… │ 22         ││ @johndoe   │ 1          │
│ python    │ 2          ││ generic.l… │ 4          ││ @bobsmith  │ 1          │
│ network   │ 2          ││ generic.l… │ 3          │└────────────┴────────────┘
│ ping      │ 1          │└────────────┴────────────┘                           
│ fail      │ 2          │                                                      
│ pass      │ 2          │                                                      
│ system    │ 9          │                                                      
│ filesyst… │ 1          │                                                      
│ storage   │ 1          │                                                      
│ configur… │ 1          │                                                      
│ slurm     │ 17         │                                                      
│ lsf       │ 12         │                                                      
│ cobalt    │ 7          │                                                      
│ containe… │ 8          │                                                      
│ singular… │ 8          │                                                      
└───────────┴────────────┘                                                      
                                                                                
                                                                                
                                                                                
                               Invalid Buildspecs                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Buildspecs                                                                   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
├──────────────────────────────────────────────────────────────────────────────┤
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
├──────────────────────────────────────────────────────────────────────────────┤
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
├──────────────────────────────────────────────────────────────────────────────┤
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
├──────────────────────────────────────────────────────────────────────────────┤
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
├──────────────────────────────────────────────────────────────────────────────┤
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… │
└──────────────────────────────────────────────────────────────────────────────┘
                          Test Breakdown by buildspec                           
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Tests                          ┃ Total ┃ Buildspec                           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ run_only_macos_distro          │ 2     │ /home/docs/checkouts/readthedocs.or │
│ run_only_linux_distro          │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/run_only_distro.yml  │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ hello_world                    │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/hello_world.yml      │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ skip                           │ 2     │ /home/docs/checkouts/readthedocs.or │
│ unskipped                      │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/skip_tests.yml       │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ foo_bar                        │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/maintainers_example. │
│                                │       │ yml                                 │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ circle_area                    │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/python-shell.yml     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ add_numbers                    │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/add_numbers.yml      │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ bash_login_shebang             │ 2     │ /home/docs/checkouts/readthedocs.or │
│ bash_nonlogin_shebang          │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/shebang.yml          │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ python_hello                   │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/python-hello.yml     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ executor_regex_script_schema   │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/executor_regex_scrip │
│                                │       │ t.yml                               │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ bash_env_variables             │ 3     │ /home/docs/checkouts/readthedocs.or │
│ csh_env_declaration            │       │ g/user_builds/buildtest/checkouts/d │
│ tcsh_env_declaration           │       │ evel/tutorials/environment.yml      │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ sleep                          │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/sleep.yml            │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ run_only_platform_darwin       │ 2     │ /home/docs/checkouts/readthedocs.or │
│ run_only_platform_linux        │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/run_only_platform.ym │
│                                │       │ l                                   │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ string_tag                     │ 2     │ /home/docs/checkouts/readthedocs.or │
│ list_of_strings_tags           │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/tags_example.yml     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ always_pass                    │ 4     │ /home/docs/checkouts/readthedocs.or │
│ always_fail                    │       │ g/user_builds/buildtest/checkouts/d │
│ test_fail_returncode_match     │       │ evel/tutorials/explicit_state.yml   │
│ test_pass_returncode_mismatch  │       │                                     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ selinux_disable                │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/selinux.yml          │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ timelimit_min_max              │ 5     │ /home/docs/checkouts/readthedocs.or │
│ timelimit_min                  │       │ g/user_builds/buildtest/checkouts/d │
│ timelimit_max                  │       │ evel/tutorials/runtime_status_test. │
│ timelimit_min_fail             │       │ yml                                 │
│ timelimit_max_fail             │       │                                     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ metric_regex_example           │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/metrics_regex.yml    │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ _bin_sh_shell                  │ 5     │ /home/docs/checkouts/readthedocs.or │
│ _bin_bash_shell                │       │ g/user_builds/buildtest/checkouts/d │
│ bash_shell                     │       │ evel/tutorials/shell_examples.yml   │
│ sh_shell                       │       │                                     │
│ shell_options                  │       │                                     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ csh_shell                      │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/csh_shell_examples.y │
│                                │       │ ml                                  │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ run_only_as_root               │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/root_user.yml        │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ exit1_fail                     │ 4     │ /home/docs/checkouts/readthedocs.or │
│ exit1_pass                     │       │ g/user_builds/buildtest/checkouts/d │
│ returncode_list_mismatch       │       │ evel/tutorials/pass_returncode.yml  │
│ returncode_int_match           │       │                                     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ metric_variable_assignment     │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/metrics_variable.yml │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ variables_bash                 │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/vars.yml             │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ status_regex_pass              │ 2     │ /home/docs/checkouts/readthedocs.or │
│ status_regex_fail              │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/status_regex.yml     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ executors_vars_env_declaration │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/script/multiple_exec │
│                                │       │ utors.yml                           │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ status_returncode_by_executors │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/script/status_by_exe │
│                                │       │ cutors.yml                          │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ executors_sbatch_declaration   │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/tutorials/script/executor_sche │
│                                │       │ duler.yml                           │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ root_disk_usage                │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/general_tests/configuration/di │
│                                │       │ sk_usage.yml                        │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ systemd_default_target         │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/general_tests/configuration/sy │
│                                │       │ stemd-default-target.yml            │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ kernel_swapusage               │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/general_tests/configuration/ke │
│                                │       │ rnel_state.yml                      │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ ulimit_filelock_unlimited      │ 6     │ /home/docs/checkouts/readthedocs.or │
│ ulimit_cputime_unlimited       │       │ g/user_builds/buildtest/checkouts/d │
│ ulimit_stacksize_unlimited     │       │ evel/general_tests/configuration/ul │
│ ulimit_vmsize_unlimited        │       │ imits.yml                           │
│ ulimit_filedescriptor_4096     │       │                                     │
│ ulimit_max_user_process_2048   │       │                                     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ current_user_queue             │ 2     │ /home/docs/checkouts/readthedocs.or │
│ show_all_jobs                  │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/general_tests/sched/slurm/sque │
│                                │       │ ue.yml                              │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ nodes_state_down               │ 9     │ /home/docs/checkouts/readthedocs.or │
│ nodes_state_reboot             │       │ g/user_builds/buildtest/checkouts/d │
│ nodes_state_allocated          │       │ evel/general_tests/sched/slurm/sinf │
│ nodes_state_completing         │       │ o.yml                               │
│ nodes_state_idle               │       │                                     │
│ node_down_fail_list_reason     │       │                                     │
│ dead_nodes                     │       │                                     │
│ get_partitions                 │       │                                     │
│ sinfo_version                  │       │                                     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ show_accounts                  │ 4     │ /home/docs/checkouts/readthedocs.or │
│ show_users                     │       │ g/user_builds/buildtest/checkouts/d │
│ show_qos                       │       │ evel/general_tests/sched/slurm/sacc │
│ show_tres                      │       │ tmgr.yml                            │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ slurm_config                   │ 2     │ /home/docs/checkouts/readthedocs.or │
│ show_partition                 │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/general_tests/sched/slurm/scon │
│                                │       │ trol.yml                            │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ show_lsf_configuration         │ 4     │ /home/docs/checkouts/readthedocs.or │
│ show_lsf_models                │       │ g/user_builds/buildtest/checkouts/d │
│ show_lsf_resources             │       │ evel/general_tests/sched/lsf/lsinfo │
│ lsf_version                    │       │ .yml                                │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ show_lsf_user_groups           │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/general_tests/sched/lsf/bugrou │
│                                │       │ p.yml                               │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ display_lsf_hosts              │ 3     │ /home/docs/checkouts/readthedocs.or │
│ display_hosts_format           │       │ g/user_builds/buildtest/checkouts/d │
│ bhosts_version                 │       │ evel/general_tests/sched/lsf/bhosts │
│                                │       │ .yml                                │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ show_host_groups               │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/general_tests/sched/lsf/bmgrou │
│                                │       │ ps.yml                              │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ show_lsf_queues                │ 3     │ /home/docs/checkouts/readthedocs.or │
│ show_lsf_queues_formatted      │       │ g/user_builds/buildtest/checkouts/d │
│ show_lsf_queues_current_user   │       │ evel/general_tests/sched/lsf/bqueue │
│                                │       │ s.yml                               │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ qsub_version                   │ 7     │ /home/docs/checkouts/readthedocs.or │
│ qselect_version                │       │ g/user_builds/buildtest/checkouts/d │
│ cqsub_version                  │       │ evel/general_tests/sched/cobalt/com │
│ qdel_version                   │       │ mands.yml                           │
│ qmove_version                  │       │                                     │
│ show_jobs                      │       │                                     │
│ show_queues                    │       │                                     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ inspect_image                  │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/general_tests/containers/singu │
│                                │       │ larity/inspect.yml                  │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ pullImage_dockerhub            │ 3     │ /home/docs/checkouts/readthedocs.or │
│ pullImage_sylabscloud          │       │ g/user_builds/buildtest/checkouts/d │
│ pullImage_shub                 │       │ evel/general_tests/containers/singu │
│                                │       │ larity/pull.yml                     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ build_sif_from_dockerimage     │ 3     │ /home/docs/checkouts/readthedocs.or │
│ build_sandbox_image            │       │ g/user_builds/buildtest/checkouts/d │
│ build_remoteimages             │       │ evel/general_tests/containers/singu │
│                                │       │ larity/build.yml                    │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ runImage                       │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/d │
│                                │       │ evel/general_tests/containers/singu │
│                                │       │ larity/run.yml                      │
└────────────────────────────────┴───────┴─────────────────────────────────────┘

Validate Buildspecs - buildtest buildspec validate

buildtest can validate buildspecs through the buildtest buildspec validate command which provides analogous options for buildtest build for selecting buildspecs such as -b, -e, -t and -e. This command can be used to validate buildspecs with the JSON Schema which can be useful if you are writing a buildspec and want to validate the buildspec without running the test.

Shown below are the available command options.

$ buildtest buildspec validate --help
usage: buildtest [options] [COMMANDS] buildspec validate [-h] [-b BUILDSPEC]
                                                         [-x EXCLUDE]
                                                         [-e EXECUTOR]
                                                         [-t TAG]

optional arguments:
  -h, --help            show this help message and exit
  -b BUILDSPEC, --buildspec BUILDSPEC
                        Specify path to buildspec (file, or directory) to
                        validate
  -x EXCLUDE, --exclude EXCLUDE
                        Specify path to buildspec to exclude (file or
                        directory) during validation
  -e EXECUTOR, --executor EXECUTOR
                        Specify buildspecs by executor name to validate
  -t TAG, --tag TAG     Specify buildspecs by tag name to validate

The -b option can be used to specify path to buildspec file or directory to validate buildspecs. If its a directory, buildtest will traverse all directories recursively and find any .yml file extensions and attempt to validate each buildspec. Shown below is an example output of what it may look like

$ buildtest buildspec validate -b tutorials/vars.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/vars.yml
All buildspecs passed validation!!!

If buildtest detects an error during validation, the error message will be displayed to screen as we see in this example

$ buildtest buildspec validate -b tutorials/invalid_tags.yml
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… ─
<ValidationError: "['network', 'network'] is not valid under any of the given schemas">


Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/invalid_tags.yml
1 buildspecs failed to validate

Similarly we can search buildspecs based on tags if you want to validate a group of buildspecs using the -t option. We can append -t option multiple times to search by multiple tag names. In this next example, we will validate all buildspecs for python and pass tags.

$ buildtest buildspec validate -t python -t pass
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/python-hello.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/pass_returncode.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/python-shell.yml
All buildspecs passed validation!!!

Edit buildspecs buildtest edit

Note

buildtest et is an alias for buildtest edit command.

The buildtest edit command can be used to edit buildspec with your preferred editor defined by environment $EDITOR, if this environment is not set buildtest will resort to vim. Once you make change, the file will be written back to disk and validated with the jsonschema. If it passes validation you will see a message such as follows:

$ buildtest edit tutorials/vars.yml
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest.tmp/tutorials/vars.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest.tmp/tutorials/vars.yml is valid

If there is an error during validation, buildtest will print the exception to stdout and it is your responsibility to fix the buildspec based on error message. In example below, the user provided an invalid value for type field.

$ buildtest edit tutorials/vars.yml
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest.tmp/tutorials/vars.yml
Traceback (most recent call last):
  File "/Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest", line 17, in <module>
    buildtest.main.main()
  File "/Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/main.py", line 103, in main
    edit_buildspec(args.buildspec, configuration)
  File "/Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/cli/edit.py", line 23, in edit_buildspec
    BuildspecParser(buildspec, be)
  File "/Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/buildsystem/parser.py", line 74, in __init__
    self._validate()
  File "/Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/buildsystem/parser.py", line 185, in _validate
    self._check_schema_type(test)
  File "/Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/buildsystem/parser.py", line 101, in _check_schema_type
    raise BuildspecError(self.buildspec, msg)
buildtest.exceptions.BuildspecError: '[/Users/siddiq90/Documents/GitHubDesktop/buildtest.tmp/tutorials/vars.yml]: type script123 is not known to buildtest.'

Show buildspec buildtest buildspec show

buildtest can display content of buildspec file given a test name via buildtest buildspec show command which expects a positional argument that is the name of test. This can be quick way to see content of buildspec without remembering the full path to the buildspec.

In this next example, we will instruct buildtest to show content of buildspec for test name python_hello.

$ buildtest buildspec show python_hello
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… ─
╭─────────────────────────────────────╮
│ version: "1.0"                      │
│ buildspecs:                         │
│   python_hello:                     │
│     type: script                    │
│     description: Hello World python │
│     executor: generic.local.bash    │
│     tags: python                    │
│     run: python hello.py            │
│                                     │
│                                     │
╰─────────────────────────────────────╯

There is bash completion for this command which will show list of test names available in the cache assuming you have run buildtest buildspec find. If you specify an invalid test name you will get an error followed by list of tests that are available in the cache

$ buildtest buildspec show XYZ123!
Unable to find test XYZ123! in cache
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ in/buildtest:36 in <module>                                                  │
│                                                                              │
│   33                                                                         │
│   34 import buildtest.main                                                   │
│   35                                                                         │
│ ❱ 36 buildtest.main.main()                                                   │
│   37                                                                         │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ __annotations__ = {}                                                     │ │
│ │    __builtins__ = <module 'builtins' (built-in)>                         │ │
│ │      __cached__ = None                                                   │ │
│ │         __doc__ = ':"\n# preferred pythons for running                   │ │
│ │                   buildtest\nPREFERRED_PYTHONS="python3 python"\n'+365   │ │
│ │        __file__ = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │      __loader__ = <_frozen_importlib_external.SourceFileLoader object at │ │
│ │                   0x7f7388295110>                                        │ │
│ │        __name__ = '__main__'                                             │ │
│ │     __package__ = None                                                   │ │
│ │        __spec__ = None                                                   │ │
│ │       buildtest = <module 'buildtest' from                               │ │
│ │                   '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │  buildtest_file = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │              os = <module 'os' from                                      │ │
│ │                   '/home/docs/.asdf/installs/python/3.7.12/lib/python3.… │ │
│ │          prefix = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │             sys = <module 'sys' (built-in)>                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/main.py:148 in main                                                 │
│                                                                              │
│   145 │   │   elif args.buildspecs_subcommand == "summary":                  │
│   146 │   │   │   summarize_buildspec_cache(configuration)                   │
│   147 │   │   elif args.buildspecs_subcommand == "show":                     │
│ ❱ 148 │   │   │   show_buildspecs(test_names=args.name, configuration=config │
│   149 │   │   elif args.buildspecs_subcommand == "validate":                 │
│   150 │   │   │   buildspec_validate(                                        │
│   151 │   │   │   │   buildspecs=args.buildspec,                             │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │               args = Namespace(buildspecs_subcommand='show',             │ │
│ │                      configfile=None, debug=False, name=['XYZ123!'],     │ │
│ │                      no_color=False, subcommands='buildspec')            │ │
│ │        config_file = None                                                │ │
│ │      configuration = <buildtest.config.SiteConfiguration object at       │ │
│ │                      0x7f7386268e50>                                     │ │
│ │             logger = <Logger buildtest (DEBUG)>                          │ │
│ │           no_color = False                                               │ │
│ │             parser = ArgumentParser(prog='buildtest', usage='%(prog)s    │ │
│ │                      [options] [COMMANDS]', description='buildtest is a  │ │
│ │                      HPC testing framework for building and running      │ │
│ │                      tests.', formatter_class=<class                     │ │
│ │                      'argparse.RawDescriptionHelpFormatter'>,            │ │
│ │                      conflict_handler='error', add_help=True)            │ │
│ │             system = <buildtest.system.BuildTestSystem object at         │ │
│ │                      0x7f7385ed9810>                                     │ │
│ │ validate_executors = True                                                │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/cli/buildspec.py:851 in show_buildspecs                             │
│                                                                              │
│    848 │                                                                     │
│    849 │   if error:                                                         │
│    850 │   │   raise BuildTestError(                                         │
│ ❱  851 │   │   │   f"Please select one of the following test: {cache.get_nam │
│    852 │   │   )                                                             │
│    853 │   # print("\nbuildspec: ", buildspec)                               │
│    854                                                                       │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │         cache = <buildtest.cli.buildspec.BuildspecCache object at        │ │
│ │                 0x7f7385f7f810>                                          │ │
│ │ configuration = <buildtest.config.SiteConfiguration object at            │ │
│ │                 0x7f7386268e50>                                          │ │
│ │         error = True                                                     │ │
│ │          name = 'XYZ123!'                                                │ │
│ │    test_names = ['XYZ123!']                                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
BuildTestError: "Please select one of the following test: 
['run_only_macos_distro', 'run_only_linux_distro', 'hello_world', 'skip', 
'unskipped', 'foo_bar', 'circle_area', 'add_numbers', 'bash_login_shebang', 
'bash_nonlogin_shebang', 'python_hello', 'executor_regex_script_schema', 
'bash_env_variables', 'csh_env_declaration', 'tcsh_env_declaration', 'sleep', 
'run_only_platform_darwin', 'run_only_platform_linux', 'string_tag', 
'list_of_strings_tags', 'always_pass', 'always_fail', 
'test_fail_returncode_match', 'test_pass_returncode_mismatch', 
'selinux_disable', 'timelimit_min_max', 'timelimit_min', 'timelimit_max', 
'timelimit_min_fail', 'timelimit_max_fail', 'metric_regex_example', 
'_bin_sh_shell', '_bin_bash_shell', 'bash_shell', 'sh_shell', 'shell_options', 
'csh_shell', 'run_only_as_root', 'exit1_fail', 'exit1_pass', 
'returncode_list_mismatch', 'returncode_int_match', 
'metric_variable_assignment', 'variables_bash', 'status_regex_pass', 
'status_regex_fail', 'executors_vars_env_declaration', 
'status_returncode_by_executors', 'executors_sbatch_declaration', 
'root_disk_usage', 'systemd_default_target', 'kernel_swapusage', 
'ulimit_filelock_unlimited', 'ulimit_cputime_unlimited', 
'ulimit_stacksize_unlimited', 'ulimit_vmsize_unlimited', 
'ulimit_filedescriptor_4096', 'ulimit_max_user_process_2048', 
'current_user_queue', 'show_all_jobs', 'nodes_state_down', 'nodes_state_reboot',
'nodes_state_allocated', 'nodes_state_completing', 'nodes_state_idle', 
'node_down_fail_list_reason', 'dead_nodes', 'get_partitions', 'sinfo_version', 
'show_accounts', 'show_users', 'show_qos', 'show_tres', 'slurm_config', 
'show_partition', 'show_lsf_configuration', 'show_lsf_models', 
'show_lsf_resources', 'lsf_version', 'show_lsf_user_groups', 
'display_lsf_hosts', 'display_hosts_format', 'bhosts_version', 
'show_host_groups', 'show_lsf_queues', 'show_lsf_queues_formatted', 
'show_lsf_queues_current_user', 'qsub_version', 'qselect_version', 
'cqsub_version', 'qdel_version', 'qmove_version', 'show_jobs', 'show_queues', 
'inspect_image', 'pullImage_dockerhub', 'pullImage_sylabscloud', 
'pullImage_shub', 'build_sif_from_dockerimage', 'build_sandbox_image', 
'build_remoteimages', 'runImage']"