Buildspecs Interface

Now that we learned how to build tests, in this section we will discuss how one can query a buildspec cache. In buildtest, one can load all buildspecs which is equivalent to validating all buildspecs with the appropriate schema. Buildtest will ignore all invalid buildspecs and store them in a separate file.

Note

buildtest bc is an alias for buildtest buildspec command.

Finding Buildspecs - buildtest buildspec find

The buildtest buildspec find command is used for finding buildspecs from buildspec cache. This command is also used for generating the buildspec cache. Shown below is a list of options for provided for this command.

buildtest buildspec find --help
$ buildtest buildspec find --help
usage: buildtest [options] [COMMANDS] buildspec find [-h] [-b] [-e]
                                                     [--group-by-tags]
                                                     [--group-by-executor]
                                                     [-p] [-t]
                                                     [--filter FILTER]
                                                     [--format FORMAT]
                                                     [--helpfilter]
                                                     [--helpformat]
                                                     [--filterfields]
                                                     [--formatfields] [-n]
                                                     [--terse] [--count COUNT]
                                                     [--pager] [-r]
                                                     [--root ROOT]
                                                     [--color COLOR] [-q]
                                                     ...

positional arguments:
  
    invalid            Show invalid buildspecs

optional arguments:
  -h, --help           show this help message and exit
  --count COUNT        Limit number of entries queried in output
  --pager              Enable PAGING when viewing result
  -r, --rebuild        Rebuild buildspec cache and find all buildspecs again
  --root ROOT          Specify root buildspecs (directory) path to load
                       buildspecs into buildspec cache.
  --color COLOR        Print output of table with the selected color.
  -q, --quiet          Don't print output of buildspec cache when rebuilding
                       cache

filter and format:
  filter and format options

  --filter FILTER      Filter buildspec cache with filter fields in format
                       --filter key1=val1,key2=val2
  --format FORMAT      Format buildspec cache with format fields in format
                       --format field1,field2,...
  --helpfilter         Show Filter fields for --filter option for filtering
                       buildspec cache output
  --helpformat         Show Format fields for --format option for formatting
                       buildspec cache output
  --filterfields       Print raw Filter fields for --filter option for
                       filtering builspec cache output
  --formatfields       Print raw Format fields for --format option for
                       formatting builspec cache output

terse:
  terse options

  -n, --no-header      Print output without header in terse output
  --terse              Print output in machine readable format

query:
  query options to retrieve from buildspec cache

  -b, --buildspec      Get all buildspec files from cache
  -e, --executors      get all unique executors from buildspecs
  --group-by-tags      Group tests by tag name
  --group-by-executor  Group tests by executor name
  -p, --paths          print all root buildspec paths
  -t, --tags           List all available tags

The buildtest buildspec find command will discover all buildspecs by recursively searching all .yml extensions. buildtest will validate each buildspec file with the json schema and buildtest will display all valid buildspecs in the output, all invalid buildspecs will be stored in a file for post-processing.

buildtest buildspec find
$ buildtest buildspec find
Buildspec Cache: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/var/buildspecs/cache.json
 name            ┃ type   ┃ executor        ┃ tags            ┃ description     
━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━
 _bin_sh_shell   │ script │ generic.local.s │ tutorials       │ /bin/sh shell   
                 │        │ h               │                 │ example         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 _bin_bash_shell │ script │ generic.local.b │ tutorials       │ /bin/bash shell 
                 │        │ ash             │                 │ example         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 bash_shell      │ script │ generic.local.b │ tutorials       │ bash shell      
                 │        │ ash             │                 │ example         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 sh_shell        │ script │ generic.local.s │ tutorials       │ sh shell        
                 │        │ h               │                 │ example         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 shell_options   │ script │ generic.local.s │ tutorials       │ shell options   
                 │        │ h               │                 │                 
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 add_numbers     │ script │ generic.local.b │ tutorials       │ Add X+Y         
                 │        │ ash             │                 │                 
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 summary_example │ script │ generic.local.b │ tutorials       │ The summary     
                 │        │ ash             │                 │ field can be a  
                 │        │                 │                 │ multi-line      
                 │        │                 │                 │ string and      
                 │        │                 │                 │ exceed 80 char  
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 sleep           │ script │ generic.local.b │ tutorials       │ sleep 2 seconds 
                 │        │ ash             │                 │                 
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 bash_env_variab │ script │ generic.local.b │ tutorials       │ Declare         
 les             │        │ ash             │                 │ environment     
                 │        │                 │                 │ variables in    
                 │        │                 │                 │ default shell   
                 │        │                 │                 │ (bash)          
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 csh_env_declara │ script │ generic.local.c │ tutorials       │ csh shell       
 tion            │        │ sh              │                 │ example to      
                 │        │                 │                 │ declare         
                 │        │                 │                 │ environment     
                 │        │                 │                 │ variables       
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 tcsh_env_declar │ script │ generic.local.c │ tutorials       │ tcsh shell      
 ation           │        │ sh              │                 │ example to      
                 │        │                 │                 │ declare         
                 │        │                 │                 │ environment     
                 │        │                 │                 │ variables       
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 csh_shell       │ script │ generic.local.c │ tutorials       │ csh shell       
                 │        │ sh              │                 │ example         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 python_hello    │ script │ generic.local.b │ python          │ Hello World     
                 │        │ ash             │                 │ python          
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 hello_world     │ script │ generic.local.b │ tutorials       │ hello world     
                 │        │ ash             │                 │ example         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 variables_bash  │ script │ generic.local.b │ tutorials       │ Declare shell   
                 │        │ ash             │                 │ variables in    
                 │        │                 │                 │ bash            
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 foo_bar         │ script │ generic.local.s │ tutorials       │ prints variable 
                 │        │ h               │                 │ $FOO            
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 bash_login_sheb │ script │ generic.local.b │ tutorials       │ customize       
 ang             │        │ ash             │                 │ shebang line    
                 │        │                 │                 │ with bash login 
                 │        │                 │                 │ shell           
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 bash_nonlogin_s │ script │ generic.local.b │ tutorials       │ customize       
 hebang          │        │ ash             │                 │ shebang line    
                 │        │                 │                 │ with default    
                 │        │                 │                 │ bash (nonlogin) 
                 │        │                 │                 │ shell           
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 skip            │ script │ generic.local.b │ tutorials       │ This test is    
                 │        │ ash             │                 │ skipped         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 unskipped       │ script │ generic.local.b │ tutorials       │ This test is    
                 │        │ ash             │                 │ not skipped     
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 wrongexecutor   │ script │ badexecutor     │ tutorials fail  │ valid test but  
                 │        │                 │                 │ invalid         
                 │        │                 │                 │ executor name   
                 │        │                 │                 │ so test won't   
                 │        │                 │                 │ run             
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 create_burst_bu │ script │ generic.local.( │ jobs            │ Create a burst  
 ffer_executors  │        │ sh|bash)        │                 │ buffer for      
                 │        │                 │                 │ multiple        
                 │        │                 │                 │ executors       
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 skip_all_tests  │ script │ generic.local.b │ tutorials       │ All test in     
                 │        │ ash             │                 │ this buildspec  
                 │        │                 │                 │ are skipped     
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 this_test_is_al │ script │ generic.local.b │ tutorials       │ This test is    
 so_skipped      │        │ ash             │                 │ also skipped    
                 │        │                 │                 │ even if skip is 
                 │        │                 │                 │ defined in test 
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 circle_area     │ script │ generic.local.b │ tutorials       │ Calculate       
                 │        │ ash             │ python          │ circle of area  
                 │        │                 │                 │ given a radius  
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 metric_regex_ex │ script │ generic.local.s │ tutorials       │ capture result  
 ample           │        │ h               │                 │ metric from     
                 │        │                 │                 │ output          
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 gcc_version     │ script │ generic.local.b │                 │ Print gcc       
                 │        │ ash             │                 │ version         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 string_tag      │ script │ generic.local.b │ network         │ tags can be a   
                 │        │ ash             │                 │ string          
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 list_of_strings │ script │ generic.local.b │ network ping    │ tags can be a   
 _tags           │        │ ash             │                 │ list of strings 
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 status_returnco │ script │ generic.local.( │ tutorials       │ define status   
 de_by_executors │        │ bash|sh)        │                 │ per executor    
                 │        │                 │                 │ type.           
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 executors_sbatc │ script │ generic.local.( │ tutorials       │ Declaring env   
 h_declaration   │        │ bash|sh)        │                 │ and vars by     
                 │        │                 │                 │ executors       
                 │        │                 │                 │ section         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 executors_vars_ │ script │ generic.local.( │ tutorials       │ Declaring env   
 env_declaration │        │ bash|sh)        │                 │ and vars by     
                 │        │                 │                 │ executors       
                 │        │                 │                 │ section         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 multiple_execut │ script │ generic.local.( │ tutorials       │ run test with   
 ors             │        │ bash|sh)        │                 │ executor        
                 │        │                 │                 │ generic.local.b 
                 │        │                 │                 │ ash and         
                 │        │                 │                 │ generic.local.s 
                 │        │                 │                 │ h executor      
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 jobA            │ script │ generic.local.b │                 │ no job          
                 │        │ ash             │                 │ dependency      
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 jobB            │ script │ generic.local.b │                 │ job dependency  
                 │        │ ash             │                 │ on jobA         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 jobC            │ script │ generic.local.b │                 │ job dependency  
                 │        │ ash             │                 │ on jobA and     
                 │        │                 │                 │ jobB            
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 runtime_test    │ script │ generic.local.b │                 │ This test will  
                 │        │ ash             │                 │ sleep 5 second  
                 │        │                 │                 │ but will fail   
                 │        │                 │                 │ due to runtime  
                 │        │                 │                 │ 2sec            
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 runtime_test_pa │ script │ generic.local.b │                 │ This test will  
 ss              │        │ ash             │                 │ run when        
                 │        │                 │                 │ runtime_test_pa 
                 │        │                 │                 │ ss is PASS      
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 runtime_test_fa │ script │ generic.local.b │                 │ This test will  
 il              │        │ ash             │                 │ run when        
                 │        │                 │                 │ runtime_test_pa 
                 │        │                 │                 │ ss is FAIL      
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 pass_test       │ script │ generic.local.b │                 │ This test will  
                 │        │ ash             │                 │ always pass     
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 fail_test       │ script │ generic.local.b │                 │ This test will  
                 │        │ ash             │                 │ run if test     
                 │        │                 │                 │ 'pass_test' is  
                 │        │                 │                 │ in state 'PASS' 
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 pass_and_fail_t │ script │ generic.local.b │                 │ This test will  
 est             │        │ ash             │                 │ run if          
                 │        │                 │                 │ pass_test is    
                 │        │                 │                 │ 'PASS' and      
                 │        │                 │                 │ fail_test is    
                 │        │                 │                 │ 'FAIL'          
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 final_test      │ script │ generic.local.b │                 │ Test will run   
                 │        │ ash             │                 │ after           
                 │        │                 │                 │ 'pass_test',    
                 │        │                 │                 │ 'fail_test',    
                 │        │                 │                 │ and             
                 │        │                 │                 │ 'pass_and_fail_ 
                 │        │                 │                 │ test'           
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 test1           │ script │ generic.local.b │                 │ This test will  
                 │        │ ash             │                 │ pass with exit  
                 │        │                 │                 │ 1               
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 test2           │ script │ generic.local.b │                 │ This test will  
                 │        │ ash             │                 │ run if test1    
                 │        │                 │                 │ has returncode  
                 │        │                 │                 │ 1               
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 test3           │ script │ generic.local.b │                 │ This test will  
                 │        │ ash             │                 │ run if test1    
                 │        │                 │                 │ has returncode  
                 │        │                 │                 │ 1 and test2 has 
                 │        │                 │                 │ returncode 2    
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 exit1_fail      │ script │ generic.local.b │ tutorials fail  │ exit 1 by       
                 │        │ ash             │                 │ default is FAIL 
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 exit1_pass      │ script │ generic.local.b │ tutorials pass  │ report exit 1   
                 │        │ ash             │                 │ as PASS         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 returncode_list │ script │ generic.local.b │ tutorials fail  │ exit 2 failed   
 _mismatch       │        │ ash             │                 │ since it failed 
                 │        │                 │                 │ to match        
                 │        │                 │                 │ returncode 1    
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 returncode_int_ │ script │ generic.local.b │ tutorials pass  │ exit 128        
 match           │        │ ash             │                 │ matches         
                 │        │                 │                 │ returncode 128  
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 timelimit_min_m │ script │ generic.local.s │ tutorials       │ Run a sleep job 
 ax              │        │ h               │                 │ for 2 seconds   
                 │        │                 │                 │ and test pass   
                 │        │                 │                 │ if its within   
                 │        │                 │                 │ 1.0-3.0sec      
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 timelimit_min   │ script │ generic.local.s │ tutorials       │ Run a sleep job 
                 │        │ h               │                 │ for 2 seconds   
                 │        │                 │                 │ and test pass   
                 │        │                 │                 │ if its exceeds  
                 │        │                 │                 │ min time of 1.0 
                 │        │                 │                 │ sec             
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 timelimit_max   │ script │ generic.local.s │ tutorials       │ Run a sleep job 
                 │        │ h               │                 │ for 2 seconds   
                 │        │                 │                 │ and test pass   
                 │        │                 │                 │ if it's within  
                 │        │                 │                 │ max time: 5.0   
                 │        │                 │                 │ sec             
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 timelimit_min_f │ script │ generic.local.s │ tutorials       │ This test fails 
 ail             │        │ h               │                 │ because it runs 
                 │        │                 │                 │ less than       
                 │        │                 │                 │ mintime of 10   
                 │        │                 │                 │ second          
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 timelimit_max_f │ script │ generic.local.s │ tutorials       │ This test fails 
 ail             │        │ h               │                 │ because it      
                 │        │                 │                 │ exceeds maxtime 
                 │        │                 │                 │ of 1.0 second   
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 status_regex_pa │ script │ generic.local.b │ system          │ Pass test based 
 ss              │        │ ash             │                 │ on regular      
                 │        │                 │                 │ expression      
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 status_regex_fa │ script │ generic.local.b │ system          │ Pass test based 
 il              │        │ ash             │                 │ on regular      
                 │        │                 │                 │ expression      
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 always_pass     │ script │ generic.local.s │                 │ This test will  
                 │        │ h               │                 │ always 'PASS'   
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 always_fail     │ script │ generic.local.s │                 │ This test will  
                 │        │ h               │                 │ always 'FAIL'   
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 test_fail_retur │ script │ generic.local.s │                 │ This test will  
 ncode_match     │        │ h               │                 │ 'FAIL' even if  
                 │        │                 │                 │ we have         
                 │        │                 │                 │ returncode      
                 │        │                 │                 │ match           
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 test_pass_retur │ script │ generic.local.s │                 │ This test will  
 ncode_mismatch  │        │ h               │                 │ 'PASS' even if  
                 │        │                 │                 │ we have         
                 │        │                 │                 │ returncode      
                 │        │                 │                 │ mismatch        
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 root_disk_usage │ script │ generic.local.b │ filesystem      │ Check root disk 
                 │        │ ash             │ storage         │ usage and       
                 │        │                 │                 │ report if it    
                 │        │                 │                 │ exceeds         
                 │        │                 │                 │ threshold       
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 ulimit_filelock │ script │ generic.local.b │ system          │ Check if file   
 _unlimited      │        │ ash             │                 │ lock is set to  
                 │        │                 │                 │ unlimited in    
                 │        │                 │                 │ ulimits         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 ulimit_cputime_ │ script │ generic.local.b │ system          │ Check if        
 unlimited       │        │ ash             │                 │ cputime is set  
                 │        │                 │                 │ to unlimited in 
                 │        │                 │                 │ ulimits         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 ulimit_stacksiz │ script │ generic.local.b │ system          │ Check if stack  
 e_unlimited     │        │ ash             │                 │ size is set to  
                 │        │                 │                 │ unlimited in    
                 │        │                 │                 │ ulimits         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 ulimit_vmsize_u │ script │ generic.local.b │ system          │ Check virtual   
 nlimited        │        │ ash             │                 │ memory size and 
                 │        │                 │                 │ check if its    
                 │        │                 │                 │ set to          
                 │        │                 │                 │ unlimited       
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 ulimit_filedesc │ script │ generic.local.b │ system          │ Check if open   
 riptor_4096     │        │ ash             │                 │ file            
                 │        │                 │                 │ descriptors     
                 │        │                 │                 │ limit is set to 
                 │        │                 │                 │ 4096            
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 ulimit_max_user │ script │ generic.local.b │ system          │ Check max       
 _process_2048   │        │ ash             │                 │ number of user  
                 │        │                 │                 │ process limit   
                 │        │                 │                 │ is set to 2048  
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 systemd_default │ script │ generic.local.b │ system          │ check if        
 _target         │        │ ash             │                 │ default target  
                 │        │                 │                 │ is              
                 │        │                 │                 │ multi-user.targ 
                 │        │                 │                 │ et              
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 kernel_swapusag │ script │ generic.local.b │ configuration   │ Retrieve Kernel 
 e               │        │ ash             │                 │ Swap Usage      
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 pullImage_docke │ script │ generic.local.b │ containers      │ Pull image      
 rhub            │        │ ash             │ singularity     │ docker://godlov 
                 │        │                 │                 │ edc/lolcow from 
                 │        │                 │                 │ DockerHub       
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 pullImage_sylab │ script │ generic.local.b │ containers      │ Pull image      
 scloud          │        │ ash             │ singularity     │ library://alpin 
                 │        │                 │                 │ e:latest from   
                 │        │                 │                 │ Sylabs Cloud    
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 pullImage_shub  │ script │ generic.local.b │ containers      │ Pull image      
                 │        │ ash             │ singularity     │ shub://vsoch/si 
                 │        │                 │                 │ ngularity-image 
                 │        │                 │                 │ s from          
                 │        │                 │                 │ SingularityHub  
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 runImage        │ script │ generic.local.b │ containers      │ run container   
                 │        │ ash             │ singularity     │ docker://godlov 
                 │        │                 │                 │ edc/lolcow      
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 inspect_image   │ script │ generic.local.b │ containers      │ Inspect image   
                 │        │ ash             │ singularity     │ via             
                 │        │                 │                 │ 'singularity    
                 │        │                 │                 │ inspect'        
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 build_sif_from_ │ script │ generic.local.b │ containers      │ build sif image 
 dockerimage     │        │ ash             │ singularity     │ from docker     
                 │        │                 │                 │ image           
                 │        │                 │                 │ docker://godlov 
                 │        │                 │                 │ edc/lolcow      
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 build_sandbox_i │ script │ generic.local.b │ containers      │ build sandbox   
 mage            │        │ ash             │ singularity     │ image from      
                 │        │                 │                 │ docker image    
                 │        │                 │                 │ docker://godlov 
                 │        │                 │                 │ edc/lolcow      
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 build_remoteima │ script │ generic.local.b │ containers      │ build remote    
 ges             │        │ ash             │ singularity     │ hosted image    
                 │        │                 │                 │ from AWS        
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 nodes_state_dow │ script │ generic.local.b │ slurm           │ Show nodes in   
 n               │        │ ash             │                 │ DOWN state      
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 nodes_state_reb │ script │ generic.local.b │ slurm           │ Show nodes in   
 oot             │        │ ash             │                 │ REBOOT state    
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 nodes_state_all │ script │ generic.local.b │ slurm           │ Show nodes in   
 ocated          │        │ ash             │                 │ ALLOCATED state 
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 nodes_state_com │ script │ generic.local.b │ slurm           │ Show nodes in   
 pleting         │        │ ash             │                 │ COMPLETING      
                 │        │                 │                 │ state           
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 nodes_state_idl │ script │ generic.local.b │ slurm           │ Show nodes in   
 e               │        │ ash             │                 │ IDLE state      
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 node_down_fail_ │ script │ generic.local.b │ slurm           │ Show nodes      
 list_reason     │        │ ash             │                 │ DOWN, DRAINED,  
                 │        │                 │                 │ FAIL or FAILING 
                 │        │                 │                 │ and list reason 
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 dead_nodes      │ script │ generic.local.b │ slurm           │ Show            
                 │        │ ash             │                 │ non-responding  
                 │        │                 │                 │ nodes           
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 get_partitions  │ script │ generic.local.b │ slurm           │ Get all slurm   
                 │        │ ash             │                 │ partitions      
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 sinfo_version   │ script │ generic.local.b │ slurm           │ get slurm       
                 │        │ ash             │                 │ version         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 show_accounts   │ script │ generic.local.b │ slurm           │ run sacctmgr    
                 │        │ ash             │                 │ list accounts   
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 show_users      │ script │ generic.local.b │ slurm           │ run sacctmgr    
                 │        │ ash             │                 │ list users      
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 show_qos        │ script │ generic.local.b │ slurm           │ run sacctmgr    
                 │        │ ash             │                 │ list qos        
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 show_tres       │ script │ generic.local.b │ slurm           │ run sacctmgr    
                 │        │ ash             │                 │ list tres       
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 slurm_config    │ script │ generic.local.b │ slurm           │ run scontrol    
                 │        │ ash             │                 │ show config     
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 show_partition  │ script │ generic.local.b │ slurm           │ run scontrol    
                 │        │ ash             │                 │ show partition  
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 current_user_qu │ script │ generic.local.b │ slurm           │ show all        
 eue             │        │ ash             │                 │ current pending 
                 │        │                 │                 │ jobs for        
                 │        │                 │                 │ current user    
                 │        │                 │                 │ (squeue -u      
                 │        │                 │                 │ $USER)          
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 show_all_jobs   │ script │ generic.local.b │ slurm           │ show all        
                 │        │ ash             │                 │ pending +       
                 │        │                 │                 │ running jobs    
                 │        │                 │                 │ (squeue -a)     
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 qsub_version    │ script │ generic.local.s │ cobalt          │ print version   
                 │        │ h               │                 │ for qsub        
                 │        │                 │                 │ command         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 qselect_version │ script │ generic.local.s │ cobalt          │ print version   
                 │        │ h               │                 │ for qselect     
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 cqsub_version   │ script │ generic.local.s │ cobalt          │ print version   
                 │        │ h               │                 │ for cqsub       
                 │        │                 │                 │ command         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 qdel_version    │ script │ generic.local.s │ cobalt          │ print version   
                 │        │ h               │                 │ for qdel        
                 │        │                 │                 │ command         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 qmove_version   │ script │ generic.local.s │ cobalt          │ print version   
                 │        │ h               │                 │ for qmove       
                 │        │                 │                 │ command         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 show_jobs       │ script │ generic.local.s │ cobalt          │ Show all jobs   
                 │        │ h               │                 │ in queue        
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 show_queues     │ script │ generic.local.s │ cobalt          │ Show all queues 
                 │        │ h               │                 │                 
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 pbs_sleep       │ script │ generic.pbs.wor │                 │                 
                 │        │ kq              │                 │                 
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 display_lsf_hos │ script │ generic.local.b │ lsf             │ Show all hosts  
 ts              │        │ ash             │                 │ in LSF cluster  
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 display_hosts_f │ script │ generic.local.b │ lsf             │ Show all hosts  
 ormat           │        │ ash             │                 │ with column     
                 │        │                 │                 │ hostname and    
                 │        │                 │                 │ status          
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 bhosts_version  │ script │ generic.local.b │ lsf             │ display version 
                 │        │ ash             │                 │ from bhosts     
                 │        │                 │                 │ command         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 show_lsf_queues │ script │ generic.local.b │ lsf             │ Show LSF queues 
                 │        │ ash             │                 │                 
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 show_lsf_queues │ script │ generic.local.b │ lsf             │ Show LSF queues 
 _formatted      │        │ ash             │                 │ with formatted  
                 │        │                 │                 │ columns         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 show_lsf_queues │ script │ generic.local.b │ lsf             │ Show LSF queues 
 _current_user   │        │ ash             │                 │ available for   
                 │        │                 │                 │ current user    
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 show_lsf_config │ script │ generic.local.b │ lsf             │ Show LSF        
 uration         │        │ ash             │                 │ configuration   
                 │        │                 │                 │ using lsinfo    
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 show_lsf_models │ script │ generic.local.b │ lsf             │ Show            
                 │        │ ash             │                 │ information     
                 │        │                 │                 │ about host      
                 │        │                 │                 │ models in LSF   
                 │        │                 │                 │ cluster         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 show_lsf_resour │ script │ generic.local.b │ lsf             │ Show            
 ces             │        │ ash             │                 │ information     
                 │        │                 │                 │ about LSF       
                 │        │                 │                 │ resources       
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 lsf_version     │ script │ generic.local.b │ lsf             │ Display lsf     
                 │        │ ash             │                 │ version using   
                 │        │                 │                 │ lsinfo          
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 show_host_group │ script │ generic.local.b │ lsf             │ Show            
 s               │        │ ash             │                 │ information     
                 │        │                 │                 │ about host      
                 │        │                 │                 │ groups using    
                 │        │                 │                 │ bmgroup         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 show_lsf_user_g │ script │ generic.local.b │ lsf             │ Show            
 roups           │        │ ash             │                 │ information     
                 │        │                 │                 │ about all LSF   
                 │        │                 │                 │ user groups

buildtest will load all discovered buildspecs in a cache file (JSON) which is created upon running buildtest buildspec find. Any subsequent runs will read from cache and update if any new buildspecs are added. If you make changes to buildspec you should rebuild the buildspec cache by running:

$ buildtest buildspec find --rebuild

The --quiet option can be used to suppress output when using buildtest buildspec find this can be useful if you want to rebuild the cache without seeing output of cache. Take for instance the following command

buildtest buildspec find --quiet --rebuild
$ buildtest buildspec find --quiet --rebuild
Clearing cache file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/var/buildspecs/cache.json
Buildspec Paths: ['/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/tutorials', '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/general_tests']
Updating buildspec cache file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/var/buildspecs/cache.json

If you want to limit the number of entries to display in output, you can use --count option which expects a positive number. For instance let’s limit output to 5 entries, we can run the following

buildtest buildspec find --count=5
$ buildtest buildspec find --count=5
Buildspec Cache: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/var/buildspecs/cache.json
 name            ┃ type   ┃ executor           ┃ tags      ┃ description        
━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━
 _bin_sh_shell   │ script │ generic.local.sh   │ tutorials │ /bin/sh shell      
                 │        │                    │           │ example            
─────────────────┼────────┼────────────────────┼───────────┼────────────────────
 _bin_bash_shell │ script │ generic.local.bash │ tutorials │ /bin/bash shell    
                 │        │                    │           │ example            
─────────────────┼────────┼────────────────────┼───────────┼────────────────────
 bash_shell      │ script │ generic.local.bash │ tutorials │ bash shell example 
─────────────────┼────────┼────────────────────┼───────────┼────────────────────
 sh_shell        │ script │ generic.local.sh   │ tutorials │ sh shell example   
─────────────────┼────────┼────────────────────┼───────────┼────────────────────
 shell_options   │ script │ generic.local.sh   │ tutorials │ shell options

You can also color the output via --color=<COLOR> which will color output of each entry with the selected color.

Finding buildspec files

If you want to find all buildspec files in cache you can run buildtest buildspec find --buildspec. Shown below is an example output.

buildtest buildspec find --buildspec
$ buildtest buildspec find --buildspec
                               List of Buildspecs                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Buildspecs                                                                   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/tu │
│ torials/shell_examples.yml                                                   │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/tu │
│ torials/add_numbers.yml                                                      │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/tu │
│ torials/summary_example.yml                                                  │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/tu │
...

Find root paths where buildspecs are searched

The buildtest buildspec find --paths will display a list of root directories buildtest will search for buildspecs when running buildtest buildspec find. One can define these directories in the configuration file or pass them via command line.

buildtest buildspec find --paths
$ buildtest buildspec find --paths
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/tutorials
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/general_tests

buildtest will search buildspecs in buildspecs root defined in your configuration, which is a list of directory paths to search for buildspecs. If you want to load buildspecs from a directory path, you can specify a directory via --root option in the format: buildtest buildspec find --root <path> --rebuild. buildtest will load all valid buildspecs into cache and ignore the rest. It’s important to add --rebuild if you want to regenerate buildspec cache.

Filtering buildspec

Once you have a buildspec cache, we can query the buildspec cache for certain attributes. When you run buildtest buildspec find it will report all buildspecs from cache which can be difficult to process. Therefore, we have a filter option (--filter) to restrict our search. Let’s take a look at the available filter fields that are acceptable with filter option.

buildtest buildspec find --helpfilter
$ buildtest buildspec find --helpfilter
                Filter Field Description                
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Field      ┃ Type                      ┃ Description ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ buildspecs │ Filter tests by buildspec │ FILE        │
├────────────┼───────────────────────────┼─────────────┤
│ executor   │ Filter by executor name   │ STRING      │
├────────────┼───────────────────────────┼─────────────┤
│ tags       │ Filter by tag name        │ STRING      │
├────────────┼───────────────────────────┼─────────────┤
│ type       │ Filter by schema type     │ STRING      │
└────────────┴───────────────────────────┴─────────────┘

The --filter option expects an arguments in key=value format as follows:

buildtest buildspec find --filter key1=value1,key2=value2,key3=value3

We can filter buildspec cache by tags=fail which will query all tests with associated tag field in test.

buildtest buildspec find --filter tags=fail
$ buildtest buildspec find --filter tags=fail
Buildspec Cache: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/var/buildspecs/cache.json
 name            ┃ type   ┃ executor         ┃ tags           ┃ description     
━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━
 wrongexecutor   │ script │ badexecutor      │ tutorials fail │ valid test but  
                 │        │                  │                │ invalid         
                 │        │                  │                │ executor name   
                 │        │                  │                │ so test won't   
                 │        │                  │                │ run             
─────────────────┼────────┼──────────────────┼────────────────┼─────────────────
 exit1_fail      │ script │ generic.local.ba │ tutorials fail │ exit 1 by       
                 │        │ sh               │                │ default is FAIL 
─────────────────┼────────┼──────────────────┼────────────────┼─────────────────
 returncode_list │ script │ generic.local.ba │ tutorials fail │ exit 2 failed   
 _mismatch       │        │ sh               │                │ since it failed 
                 │        │                  │                │ to match        
                 │        │                  │                │ returncode 1

In addition, we can query buildspecs by schema type using type property. In this example we query all tests by type property

buildtest buildspec find --filter type=script
$ buildtest buildspec find --filter type=script
Buildspec Cache: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/var/buildspecs/cache.json
 name            ┃ type   ┃ executor        ┃ tags            ┃ description     
━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━
 _bin_sh_shell   │ script │ generic.local.s │ tutorials       │ /bin/sh shell   
                 │        │ h               │                 │ example         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 _bin_bash_shell │ script │ generic.local.b │ tutorials       │ /bin/bash shell 
                 │        │ ash             │                 │ example         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 bash_shell      │ script │ generic.local.b │ tutorials       │ bash shell      
                 │        │ ash             │                 │ example         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 sh_shell        │ script │ generic.local.s │ tutorials       │ sh shell        
                 │        │ h               │                 │ example         
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 shell_options   │ script │ generic.local.s │ tutorials       │ shell options   
                 │        │ h               │                 │                 
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
 add_numbers     │ script │ generic.local.b │ tutorials       │ Add X+Y         
                 │        │ ash             │                 │                 
─────────────────┼────────┼─────────────────┼─────────────────┼─────────────────
...

Finally, we can combine multiple filter fields separated by comma, in the next example we can query all buildspecs with tags=tutorials, executor=generic.local.sh, and type=script

buildtest buildspec find --filter tags=tutorials,executor=generic.local.sh,type=script
$ buildtest buildspec find --filter tags=tutorials,executor=generic.local.sh,type=script
Buildspec Cache: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/var/buildspecs/cache.json
 name               ┃ type   ┃ executor         ┃ tags      ┃ description       
━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━
 _bin_sh_shell      │ script │ generic.local.sh │ tutorials │ /bin/sh shell     
                    │        │                  │           │ example           
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
 sh_shell           │ script │ generic.local.sh │ tutorials │ sh shell example  
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
 shell_options      │ script │ generic.local.sh │ tutorials │ shell options     
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
 foo_bar            │ script │ generic.local.sh │ tutorials │ prints variable   
                    │        │                  │           │ $FOO              
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
 metric_regex_examp │ script │ generic.local.sh │ tutorials │ capture result    
 le                 │        │                  │           │ metric from       
                    │        │                  │           │ output            
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
 timelimit_min_max  │ script │ generic.local.sh │ tutorials │ Run a sleep job   
                    │        │                  │           │ for 2 seconds and 
                    │        │                  │           │ test pass if its  
                    │        │                  │           │ within 1.0-3.0sec 
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
 timelimit_min      │ script │ generic.local.sh │ tutorials │ Run a sleep job   
                    │        │                  │           │ for 2 seconds and 
                    │        │                  │           │ test pass if its  
                    │        │                  │           │ exceeds min time  
                    │        │                  │           │ of 1.0 sec        
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
 timelimit_max      │ script │ generic.local.sh │ tutorials │ Run a sleep job   
                    │        │                  │           │ for 2 seconds and 
                    │        │                  │           │ test pass if it's 
                    │        │                  │           │ within max time:  
                    │        │                  │           │ 5.0 sec           
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
 timelimit_min_fail │ script │ generic.local.sh │ tutorials │ This test fails   
                    │        │                  │           │ because it runs   
                    │        │                  │           │ less than mintime 
                    │        │                  │           │ of 10 second      
────────────────────┼────────┼──────────────────┼───────────┼───────────────────
 timelimit_max_fail │ script │ generic.local.sh │ tutorials │ This test fails   
                    │        │                  │           │ because it        
                    │        │                  │           │ exceeds maxtime   
                    │        │                  │           │ of 1.0 second

We can filter output of buildspec cache by buildspec using --filter buildspec=<path> which expects a path to buildspec file. The buildspec must be in the cache and file path must exist in order to fetch the result. The path can be absolute or relative path.

In this next example, we will filter cache by file tutorials/test_status/pass_returncode.yml and use --format name,buildspec to format columns. The --format buildspec will show full path to buildspec and name refers to name of test. For more details on –format see Format buildspec cache.

buildtest buildspec find --filter buildspec=tutorials/test_status/pass_returncode.yml --format name,buildspec
$ buildtest buildspec find --filter buildspec=tutorials/test_status/pass_returncode.yml --format name,buildspec
Buildspec Cache: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/var/buildspecs/cache.json
 name                     ┃ buildspec                                           
━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 exit1_fail               │ /home/docs/checkouts/readthedocs.org/user_builds/bu 
                          │ ildtest/checkouts/v1.0/tutorials/test_status/pass_r 
                          │ eturncode.yml                                       
──────────────────────────┼─────────────────────────────────────────────────────
 exit1_pass               │ /home/docs/checkouts/readthedocs.org/user_builds/bu 
                          │ ildtest/checkouts/v1.0/tutorials/test_status/pass_r 
                          │ eturncode.yml                                       
──────────────────────────┼─────────────────────────────────────────────────────
 returncode_list_mismatch │ /home/docs/checkouts/readthedocs.org/user_builds/bu 
                          │ ildtest/checkouts/v1.0/tutorials/test_status/pass_r 
                          │ eturncode.yml                                       
──────────────────────────┼─────────────────────────────────────────────────────
 returncode_int_match     │ /home/docs/checkouts/readthedocs.org/user_builds/bu 
                          │ ildtest/checkouts/v1.0/tutorials/test_status/pass_r 
                          │ eturncode.yml

Format buildspec cache

We have seen how one can filter buildspecs, but we can also configure which columns to display in the output of buildtest buildspec find. By default, we show a pre-selected format fields in the output, however there are more format fields available that can be configured at the command line.

The format fields are specified in comma separated format such as buildtest buildspec find --format <field1>,<field2>,.... You can see a list of all format fields by --helpformat option as shown below

buildtest buildspec find --helpformat
$ buildtest buildspec find --helpformat
              Format Field Description               
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Field       ┃ Description                         ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildspec   │ Display name of buildspec file      │
├─────────────┼─────────────────────────────────────┤
│ description │ Show description of test            │
├─────────────┼─────────────────────────────────────┤
│ executor    │ Display 'executor' property in test │
├─────────────┼─────────────────────────────────────┤
│ name        │ Display name of test                │
├─────────────┼─────────────────────────────────────┤
│ tags        │ Display 'tag' property in test      │
├─────────────┼─────────────────────────────────────┤
│ type        │ Display 'type' property in test     │
└─────────────┴─────────────────────────────────────┘

In the next example, we utilize --format field with --filter option to show how format fields affect table columns. buildtest will display the table in order of format fields specified in command line.

buildtest buildspec find --format name,description,buildspec --filter tags=tutorials,executor=generic.local.sh
$ buildtest buildspec find --format name,description,buildspec --filter tags=tutorials,executor=generic.local.sh
Buildspec Cache: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/var/buildspecs/cache.json
 name                 ┃ description                ┃ buildspec                  
━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 _bin_sh_shell        │ /bin/sh shell example      │ /home/docs/checkouts/readt 
                      │                            │ hedocs.org/user_builds/bui 
                      │                            │ ldtest/checkouts/v1.0/tuto 
                      │                            │ rials/shell_examples.yml   
──────────────────────┼────────────────────────────┼────────────────────────────
 sh_shell             │ sh shell example           │ /home/docs/checkouts/readt 
                      │                            │ hedocs.org/user_builds/bui 
                      │                            │ ldtest/checkouts/v1.0/tuto 
                      │                            │ rials/shell_examples.yml   
──────────────────────┼────────────────────────────┼────────────────────────────
 shell_options        │ shell options              │ /home/docs/checkouts/readt 
                      │                            │ hedocs.org/user_builds/bui 
                      │                            │ ldtest/checkouts/v1.0/tuto 
                      │                            │ rials/shell_examples.yml   
──────────────────────┼────────────────────────────┼────────────────────────────
 foo_bar              │ prints variable $FOO       │ /home/docs/checkouts/readt 
                      │                            │ hedocs.org/user_builds/bui 
                      │                            │ ldtest/checkouts/v1.0/tuto 
                      │                            │ rials/maintainers_example. 
                      │                            │ yml                        
──────────────────────┼────────────────────────────┼────────────────────────────
 metric_regex_example │ capture result metric from │ /home/docs/checkouts/readt 
                      │ output                     │ hedocs.org/user_builds/bui 
                      │                            │ ldtest/checkouts/v1.0/tuto 
                      │                            │ rials/metrics_regex.yml    
──────────────────────┼────────────────────────────┼────────────────────────────
 timelimit_min_max    │ Run a sleep job for 2      │ /home/docs/checkouts/readt 
                      │ seconds and test pass if   │ hedocs.org/user_builds/bui 
                      │ its within 1.0-3.0sec      │ ldtest/checkouts/v1.0/tuto 
                      │                            │ rials/test_status/runtime_ 
                      │                            │ status_test.yml            
──────────────────────┼────────────────────────────┼────────────────────────────
 timelimit_min        │ Run a sleep job for 2      │ /home/docs/checkouts/readt 
                      │ seconds and test pass if   │ hedocs.org/user_builds/bui 
                      │ its exceeds min time of    │ ldtest/checkouts/v1.0/tuto 
                      │ 1.0 sec                    │ rials/test_status/runtime_ 
                      │                            │ status_test.yml            
──────────────────────┼────────────────────────────┼────────────────────────────
 timelimit_max        │ Run a sleep job for 2      │ /home/docs/checkouts/readt 
                      │ seconds and test pass if   │ hedocs.org/user_builds/bui 
                      │ it's within max time: 5.0  │ ldtest/checkouts/v1.0/tuto 
                      │ sec                        │ rials/test_status/runtime_ 
                      │                            │ status_test.yml            
──────────────────────┼────────────────────────────┼────────────────────────────
 timelimit_min_fail   │ This test fails because it │ /home/docs/checkouts/readt 
                      │ runs less than mintime of  │ hedocs.org/user_builds/bui 
                      │ 10 second                  │ ldtest/checkouts/v1.0/tuto 
                      │                            │ rials/test_status/runtime_ 
                      │                            │ status_test.yml            
──────────────────────┼────────────────────────────┼────────────────────────────
 timelimit_max_fail   │ This test fails because it │ /home/docs/checkouts/readt 
                      │ exceeds maxtime of 1.0     │ hedocs.org/user_builds/bui 
                      │ second                     │ ldtest/checkouts/v1.0/tuto 
                      │                            │ rials/test_status/runtime_ 
                      │                            │ status_test.yml

Querying buildspec tags

If you want to retrieve all unique tags from all buildspecs you can run buildtest buildspec find --tags. This can be useful if you want to know available tags in your buildspec cache.

buildtest buildspec find --tags
$ buildtest buildspec find --tags
  List of Tags   
┏━━━━━━━━━━━━━━━┓
┃ Tags          ┃
┡━━━━━━━━━━━━━━━┩
│ tutorials     │
│ cobalt        │
│ containers    │
│ lsf           │
│ slurm         │
│ singularity   │
│ fail          │
│ storage       │
│ jobs          │
│ python        │
│ network       │
│ pass          │
│ system        │
│ filesystem    │
│ configuration │
│ ping          │
└───────────────┘

In addition, buildtest can group tests by tags via buildtest buildspec find --group-by-tags which can be useful if you want to know which tests get executed when running buildtest build --tags. The output is grouped by tag names, followed by name of test and description.

buildtest buildspec find --group-by-tags
$ buildtest buildspec find --group-by-tags
                                 Tests by Tags                                  
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Tags          ┃ Name                         ┃ Description                   ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ tutorials     │ _bin_sh_shell                │ /bin/sh shell example         │
├───────────────┼──────────────────────────────┼───────────────────────────────┤
│ tutorials     │ _bin_bash_shell              │ /bin/bash shell example       │
├───────────────┼──────────────────────────────┼───────────────────────────────┤
│ tutorials     │ bash_shell                   │ bash shell example            │
├───────────────┼──────────────────────────────┼───────────────────────────────┤
│ tutorials     │ sh_shell                     │ sh shell example              │
├───────────────┼──────────────────────────────┼───────────────────────────────┤
│ tutorials     │ shell_options                │ shell options                 │
├───────────────┼──────────────────────────────┼───────────────────────────────┤
│ tutorials     │ add_numbers                  │ Add X+Y                       │
├───────────────┼──────────────────────────────┼───────────────────────────────┤
│ tutorials     │ summary_example              │ The summary field can be a    │
│               │                              │ multi-line string and exceed  │
│               │                              │ 80 char                       │
├───────────────┼──────────────────────────────┼───────────────────────────────┤
│ tutorials     │ sleep                        │ sleep 2 seconds               │
├───────────────┼──────────────────────────────┼───────────────────────────────┤
│ tutorials     │ bash_env_variables           │ Declare environment variables │
│               │                              │ in default shell (bash)       │
├───────────────┼──────────────────────────────┼───────────────────────────────┤
│ tutorials     │ csh_env_declaration          │ csh shell example to declare  │
│               │                              │ environment variables         │
├───────────────┼──────────────────────────────┼───────────────────────────────┤
│ tutorials     │ tcsh_env_declaration         │ tcsh shell example to declare │
│               │                              │ environment variables         │
├───────────────┼──────────────────────────────┼───────────────────────────────┤
│ tutorials     │ csh_shell                    │ csh shell example             │
├───────────────┼──────────────────────────────┼───────────────────────────────┤
│ tutorials     │ hello_world                  │ hello world example           │
├───────────────┼──────────────────────────────┼───────────────────────────────┤
│ tutorials     │ variables_bash               │ Declare shell variables in    │
│               │                              │ bash                          │
├───────────────┼──────────────────────────────┼───────────────────────────────┤
│ tutorials     │ foo_bar                      │ prints variable $FOO          │
├───────────────┼──────────────────────────────┼───────────────────────────────┤
│ tutorials     │ bash_login_shebang           │ customize shebang line with   │
...

Querying buildspec executor

If you want to know all executors in your buildspec cache use the buildtest buildspec find --executors command. This can be useful when you want to build by executors (buildtest build --executor).

buildtest buildspec find --executors
$ buildtest buildspec find --executors
     List of Executors     
┏━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Executors               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ generic.local.bash      │
│ badexecutor             │
│ generic.local.sh        │
│ generic.local.csh       │
│ generic.local.(sh|bash) │
│ generic.pbs.workq       │
│ generic.local.(bash|sh) │
└─────────────────────────┘

Similar to --group-by-tags, buildtest has an option to group tests by executor using --group-by-executor option. This will show tests grouped by executor, name of test and test description. Shown below is an example output.

buildtest buildspec find --group-by-executor
$ buildtest buildspec find --group-by-executor
                               Tests by Executors                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Executors               ┃ Name                     ┃ Description             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ generic.local.sh        │ _bin_sh_shell            │ /bin/sh shell example   │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ generic.local.sh        │ sh_shell                 │ sh shell example        │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ generic.local.sh        │ shell_options            │ shell options           │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ generic.local.sh        │ foo_bar                  │ prints variable $FOO    │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ generic.local.sh        │ metric_regex_example     │ capture result metric   │
│                         │                          │ from output             │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ generic.local.sh        │ timelimit_min_max        │ Run a sleep job for 2   │
│                         │                          │ seconds and test pass   │
│                         │                          │ if its within           │
│                         │                          │ 1.0-3.0sec              │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ generic.local.sh        │ timelimit_min            │ Run a sleep job for 2   │
│                         │                          │ seconds and test pass   │
│                         │                          │ if its exceeds min time │
│                         │                          │ of 1.0 sec              │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ generic.local.sh        │ timelimit_max            │ Run a sleep job for 2   │
│                         │                          │ seconds and test pass   │
│                         │                          │ if it's within max      │
│                         │                          │ time: 5.0 sec           │
├─────────────────────────┼──────────────────────────┼─────────────────────────┤
│ generic.local.sh        │ timelimit_min_fail       │ This test fails because │
...

Terse Output

You can use the --terse option to print output of buildtest buildspec find in terse format that can be useful if you want to parse content of file. In example below, we will print output of tags in terse format, the first entry tags is the header followed by list of unique tags. The --no-header option can be used to disable printing of header title.

buildtest buildspec find -t --terse
$ buildtest buildspec find -t --terse
tag
tutorials
cobalt
containers
lsf
slurm
singularity
fail
storage
jobs
python
network
pass
system
filesystem
configuration
ping

You can also use --count with terse option, note that heading is not counted as an element, the –count will only limit number of entries reported from the buildspec cache. Shown below we retrieve 5 test results in terse mode and disable heading via -n option.

buildtest buildspec find --terse -n --count=5
$ buildtest buildspec find --terse -n --count=5
_bin_sh_shell|script|generic.local.sh|tutorials|/bin/sh shell example
_bin_bash_shell|script|generic.local.bash|tutorials|/bin/bash shell example
bash_shell|script|generic.local.bash|tutorials|bash shell example
sh_shell|script|generic.local.sh|tutorials|sh shell example
shell_options|script|generic.local.sh|tutorials|shell options

Invalid Buildspecs - buildtest buildspec find invalid

buildtest will store invalid buildspecs in the cache file which can be retrieved using buildtest buildspec find invalid. buildtest will attempt to parse each buildspec and store error message for every buildspec. If you run without any options it will report a list of invalid buildspecs as shown below

buildtest buildspec find invalid
$ buildtest buildspec find invalid
                               Invalid Buildspecs                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Buildspec                                                                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/t… │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/t… │
└──────────────────────────────────────────────────────────────────────────────┘

If you want to see error messages for each buildspec you can pass the -e or --error option which will display output of each buildspec followed by error message.

buildtest buildspec find -e
$ buildtest buildspec find invalid -e
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/t… ─
"['network', 'network'] is not valid under any of the given schemas\n\nFailed validating 'oneOf' in schema['properties']['tags']:\n    {'oneOf': [{'type': 'string'},\n               {'$ref': '#/definitions/list_of_strings'}]}\n\nOn instance['tags']:\n    ['network', 'network']"
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/t… ─
"'[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/tutorials/invalid_buildspec_section.yml]: type badscript is not known to buildtest.'"

Query Maintainers (buildtest buildspec maintainers)

buildtest keeps track of maintainers (i.e authors) for a given buildspec provided that you specify the maintainers property. This is stored in the buildspec cache which can be used to query some interesting details.

Shown below is the help for buildtest buildspec maintainers --help

buildtest buildspec maintainers --help
$ buildtest buildspec maintainers --help
usage: buildtest [options] [COMMANDS] buildspec maintainers
       [-h] [-l] [-b] [--terse] [-n] {find} ...

positional arguments:
  {find}
    find           Find buildspecs based on maintainer name

optional arguments:
  -h, --help       show this help message and exit
  -l, --list       List all maintainers
  -b, --breakdown  Breakdown of buildspecs by maintainers
  --terse          Print output in machine readable format
  -n, --no-header  Print output without header in terse output

If you want to see a listing of all maintainers you can use the --list as shown below

buildtest buildspec maintainers --list
$ buildtest buildspec maintainers --list
┏━━━━━━━━━━━━━━━━━━┓
┃ Maintainers      ┃
┡━━━━━━━━━━━━━━━━━━┩
│ @shahzebsiddiqui │
│ @johndoe         │
│ @bobsmith        │
└──────────────────┘

If you prefer a machine readable format, then you can use --terse and --no-header.

buildtest buildspec maintainers --list --terse --no-header
$ buildtest buildspec maintainers --list --terse --no-header
@shahzebsiddiqui
@johndoe
@bobsmith

If you want to see a breakdown of all buildspecs by maintainers you can use –breakdown which will display the following information

buildtest buildspec maintainers --breakdown
$ buildtest buildspec maintainers --breakdown
                     Breakdown of buildspecs by maintainers                     
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Maintainers      ┃ Buildspec                                                 ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ @shahzebsiddiqui │ /home/docs/checkouts/readthedocs.org/user_builds/buildtes │
│                  │ t/checkouts/v1.0/tutorials/hello_world.yml:/home/docs/che │
│                  │ ckouts/readthedocs.org/user_builds/buildtest/checkouts/v1 │
│                  │ .0/general_tests/configuration/ulimits.yml                │
├──────────────────┼───────────────────────────────────────────────────────────┤
│ @johndoe         │ /home/docs/checkouts/readthedocs.org/user_builds/buildtes │
│                  │ t/checkouts/v1.0/tutorials/maintainers_example.yml        │
├──────────────────┼───────────────────────────────────────────────────────────┤
│ @bobsmith        │ /home/docs/checkouts/readthedocs.org/user_builds/buildtes │
│                  │ t/checkouts/v1.0/tutorials/maintainers_example.yml        │
└──────────────────┴───────────────────────────────────────────────────────────┘

The buildtest buildspec maintainers find command can be used to report buildspec given a maintainer name which works similar to –breakdown but doesn’t report information for all maintainers. Shown below, we query all buildspecs by maintainer @shahzebsiddiqui

buildtest buildspec maintainers find @shahzebsiddiqui
$ buildtest buildspec maintainers find @shahzebsiddiqui
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/tutorials/hello_world.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/general_tests/configuration/ulimits.yml

Cache Summary - buildtest buildspec summary

The buildtest buildspec summary command can be used to provide a summary of the buildspec cache. This command can be used assuming your cache is built via buildtest buildspec find. Shown below is a summary of the cache file.

buildtest buildspec summary
$ buildtest buildspec summary
╭──────────────────────────────────────────────────────────────────────────────╮
│                                                                              │
│     Reading Buildspec Cache File:   /home/docs/checkouts/readthedocs.org/use │
│     Total Valid Buildspecs:         50                                       │
│     Total Invalid Buildspecs:       2                                        │
│     Total Unique Tags:              16                                       │
│     Total Maintainers:              3                                        │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
      Tag Breakdown           Executor Breakdown        Maintainers Breakdown   
┏━━━━━━━━━━━┳━━━━━━━━━━━━┓┏━━━━━━━━━━━━┳━━━━━━━━━━━━┓┏━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃           ┃ total      ┃┃            ┃ total      ┃┃ maintainer ┃ total      ┃
┃ tag       ┃ tests      ┃┃ executor   ┃ tests      ┃┃ s          ┃ buildspecs ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━┩┡━━━━━━━━━━━━╇━━━━━━━━━━━━┩┡━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ tutorials │ 37         ││ generic.lo │ 21         ││ @shahzebsi │ 2          │
│ python    │ 2          ││ cal.sh     │            ││ ddiqui     │            │
│ fail      │ 3          ││ generic.lo │ 84         ││ @johndoe   │ 1          │
│ jobs      │ 1          ││ cal.bash   │            ││ @bobsmith  │ 1          │
│ network   │ 2          ││ generic.lo │ 3          │└────────────┴────────────┘
│ ping      │ 1          ││ cal.csh    │            │                           
│ pass      │ 2          ││ badexecuto │ 1          │                           
│ system    │ 9          ││ r          │            │                           
│ filesyste │ 1          ││ generic.lo │ 1          │                           
│ m         │            ││ cal.(sh|ba │            │                           
│ storage   │ 1          ││ sh)        │            │                           
                               Invalid Buildspecs                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Buildspecs                                                                   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/tu │
│ torials/invalid_tags.yml                                                     │
├──────────────────────────────────────────────────────────────────────────────┤
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/tu │
│ torials/invalid_buildspec_section.yml                                        │
                          Test Breakdown by buildspec                           
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Tests                          ┃ Total ┃ Buildspec                           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ _bin_sh_shell                  │ 5     │ /home/docs/checkouts/readthedocs.or │
│ _bin_bash_shell                │       │ g/user_builds/buildtest/checkouts/v │
│ bash_shell                     │       │ 1.0/tutorials/shell_examples.yml    │
│ sh_shell                       │       │                                     │
│ shell_options                  │       │                                     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ add_numbers                    │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/add_numbers.yml       │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ summary_example                │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/summary_example.yml   │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ sleep                          │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/sleep.yml             │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ bash_env_variables             │ 3     │ /home/docs/checkouts/readthedocs.or │
│ csh_env_declaration            │       │ g/user_builds/buildtest/checkouts/v │
│ tcsh_env_declaration           │       │ 1.0/tutorials/environment.yml       │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ csh_shell                      │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/csh_shell_examples.ym │
│                                │       │ l                                   │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ python_hello                   │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/python-hello.yml      │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ hello_world                    │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/hello_world.yml       │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ variables_bash                 │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/vars.yml              │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ foo_bar                        │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/maintainers_example.y │
│                                │       │ ml                                  │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ bash_login_shebang             │ 2     │ /home/docs/checkouts/readthedocs.or │
│ bash_nonlogin_shebang          │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/shebang.yml           │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ skip                           │ 2     │ /home/docs/checkouts/readthedocs.or │
│ unskipped                      │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/skip_tests.yml        │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ wrongexecutor                  │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/invalid_executor.yml  │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ create_burst_buffer_executors  │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/burstbuffer_datawarp_ │
│                                │       │ executors.yml                       │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ skip_all_tests                 │ 2     │ /home/docs/checkouts/readthedocs.or │
│ this_test_is_also_skipped      │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/skip_buildspec.yml    │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ circle_area                    │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/python-shell.yml      │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ metric_regex_example           │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/metrics_regex.yml     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ gcc_version                    │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/gcc_version.yml       │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ string_tag                     │ 2     │ /home/docs/checkouts/readthedocs.or │
│ list_of_strings_tags           │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/tags_example.yml      │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ status_returncode_by_executors │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/multi_executors/statu │
│                                │       │ s_by_executors.yml                  │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ executors_sbatch_declaration   │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/multi_executors/execu │
│                                │       │ tor_scheduler.yml                   │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ executors_vars_env_declaration │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/multi_executors/execu │
│                                │       │ tors_var_env_declaration.yml        │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ multiple_executors             │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/multi_executors/execu │
│                                │       │ tor_regex_script.yml                │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ jobA                           │ 3     │ /home/docs/checkouts/readthedocs.or │
│ jobB                           │       │ g/user_builds/buildtest/checkouts/v │
│ jobC                           │       │ 1.0/tutorials/job_dependency/ex1.ym │
│                                │       │ l                                   │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ runtime_test                   │ 3     │ /home/docs/checkouts/readthedocs.or │
│ runtime_test_pass              │       │ g/user_builds/buildtest/checkouts/v │
│ runtime_test_fail              │       │ 1.0/tutorials/job_dependency/ex4.ym │
│                                │       │ l                                   │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ pass_test                      │ 4     │ /home/docs/checkouts/readthedocs.or │
│ fail_test                      │       │ g/user_builds/buildtest/checkouts/v │
│ pass_and_fail_test             │       │ 1.0/tutorials/job_dependency/ex3.ym │
│ final_test                     │       │ l                                   │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ test1                          │ 3     │ /home/docs/checkouts/readthedocs.or │
│ test2                          │       │ g/user_builds/buildtest/checkouts/v │
│ test3                          │       │ 1.0/tutorials/job_dependency/ex2.ym │
│                                │       │ l                                   │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ exit1_fail                     │ 4     │ /home/docs/checkouts/readthedocs.or │
│ exit1_pass                     │       │ g/user_builds/buildtest/checkouts/v │
│ returncode_list_mismatch       │       │ 1.0/tutorials/test_status/pass_retu │
│ returncode_int_match           │       │ rncode.yml                          │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ timelimit_min_max              │ 5     │ /home/docs/checkouts/readthedocs.or │
│ timelimit_min                  │       │ g/user_builds/buildtest/checkouts/v │
│ timelimit_max                  │       │ 1.0/tutorials/test_status/runtime_s │
│ timelimit_min_fail             │       │ tatus_test.yml                      │
│ timelimit_max_fail             │       │                                     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ status_regex_pass              │ 2     │ /home/docs/checkouts/readthedocs.or │
│ status_regex_fail              │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/tutorials/test_status/status_re │
│                                │       │ gex.yml                             │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ always_pass                    │ 4     │ /home/docs/checkouts/readthedocs.or │
│ always_fail                    │       │ g/user_builds/buildtest/checkouts/v │
│ test_fail_returncode_match     │       │ 1.0/tutorials/test_status/explicit_ │
│ test_pass_returncode_mismatch  │       │ state.yml                           │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ root_disk_usage                │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/general_tests/configuration/dis │
│                                │       │ k_usage.yml                         │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ ulimit_filelock_unlimited      │ 6     │ /home/docs/checkouts/readthedocs.or │
│ ulimit_cputime_unlimited       │       │ g/user_builds/buildtest/checkouts/v │
│ ulimit_stacksize_unlimited     │       │ 1.0/general_tests/configuration/uli │
│ ulimit_vmsize_unlimited        │       │ mits.yml                            │
│ ulimit_filedescriptor_4096     │       │                                     │
│ ulimit_max_user_process_2048   │       │                                     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ systemd_default_target         │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/general_tests/configuration/sys │
│                                │       │ temd-default-target.yml             │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ kernel_swapusage               │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/general_tests/configuration/ker │
│                                │       │ nel_state.yml                       │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ pullImage_dockerhub            │ 3     │ /home/docs/checkouts/readthedocs.or │
│ pullImage_sylabscloud          │       │ g/user_builds/buildtest/checkouts/v │
│ pullImage_shub                 │       │ 1.0/general_tests/containers/singul │
│                                │       │ arity/pull.yml                      │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ runImage                       │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/general_tests/containers/singul │
│                                │       │ arity/run.yml                       │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ inspect_image                  │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/general_tests/containers/singul │
│                                │       │ arity/inspect.yml                   │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ build_sif_from_dockerimage     │ 3     │ /home/docs/checkouts/readthedocs.or │
│ build_sandbox_image            │       │ g/user_builds/buildtest/checkouts/v │
│ build_remoteimages             │       │ 1.0/general_tests/containers/singul │
│                                │       │ arity/build.yml                     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ nodes_state_down               │ 9     │ /home/docs/checkouts/readthedocs.or │
│ nodes_state_reboot             │       │ g/user_builds/buildtest/checkouts/v │
│ nodes_state_allocated          │       │ 1.0/general_tests/sched/slurm/sinfo │
│ nodes_state_completing         │       │ .yml                                │
│ nodes_state_idle               │       │                                     │
│ node_down_fail_list_reason     │       │                                     │
│ dead_nodes                     │       │                                     │
│ get_partitions                 │       │                                     │
│ sinfo_version                  │       │                                     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ show_accounts                  │ 4     │ /home/docs/checkouts/readthedocs.or │
│ show_users                     │       │ g/user_builds/buildtest/checkouts/v │
│ show_qos                       │       │ 1.0/general_tests/sched/slurm/sacct │
│ show_tres                      │       │ mgr.yml                             │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ slurm_config                   │ 2     │ /home/docs/checkouts/readthedocs.or │
│ show_partition                 │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/general_tests/sched/slurm/scont │
│                                │       │ rol.yml                             │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ current_user_queue             │ 2     │ /home/docs/checkouts/readthedocs.or │
│ show_all_jobs                  │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/general_tests/sched/slurm/squeu │
│                                │       │ e.yml                               │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ qsub_version                   │ 7     │ /home/docs/checkouts/readthedocs.or │
│ qselect_version                │       │ g/user_builds/buildtest/checkouts/v │
│ cqsub_version                  │       │ 1.0/general_tests/sched/cobalt/comm │
│ qdel_version                   │       │ ands.yml                            │
│ qmove_version                  │       │                                     │
│ show_jobs                      │       │                                     │
│ show_queues                    │       │                                     │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ pbs_sleep                      │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/general_tests/sched/pbs/hostnam │
│                                │       │ e.yml                               │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ display_lsf_hosts              │ 3     │ /home/docs/checkouts/readthedocs.or │
│ display_hosts_format           │       │ g/user_builds/buildtest/checkouts/v │
│ bhosts_version                 │       │ 1.0/general_tests/sched/lsf/bhosts. │
│                                │       │ yml                                 │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ show_lsf_queues                │ 3     │ /home/docs/checkouts/readthedocs.or │
│ show_lsf_queues_formatted      │       │ g/user_builds/buildtest/checkouts/v │
│ show_lsf_queues_current_user   │       │ 1.0/general_tests/sched/lsf/bqueues │
│                                │       │ .yml                                │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ show_lsf_configuration         │ 4     │ /home/docs/checkouts/readthedocs.or │
│ show_lsf_models                │       │ g/user_builds/buildtest/checkouts/v │
│ show_lsf_resources             │       │ 1.0/general_tests/sched/lsf/lsinfo. │
│ lsf_version                    │       │ yml                                 │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ show_host_groups               │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/general_tests/sched/lsf/bmgroup │
│                                │       │ s.yml                               │
├────────────────────────────────┼───────┼─────────────────────────────────────┤
│ show_lsf_user_groups           │ 1     │ /home/docs/checkouts/readthedocs.or │
│                                │       │ g/user_builds/buildtest/checkouts/v │
│                                │       │ 1.0/general_tests/sched/lsf/bugroup │
│                                │       │ .yml                                │
└────────────────────────────────┴───────┴─────────────────────────────────────┘

Validate Buildspecs - buildtest buildspec validate

buildtest can validate buildspecs through the buildtest buildspec validate command which provides analogous options for buildtest build for selecting buildspecs such as -b, -e, -t and -e. This command can be used to validate buildspecs with the JSON Schema which can be useful if you are writing a buildspec and want to validate the buildspec without running the test.

Shown below are the available command options.

buildtest buildspec validate --help
$ buildtest buildspec validate --help
usage: buildtest [options] [COMMANDS] buildspec validate [-h] [-b BUILDSPEC]
                                                         [-x EXCLUDE]
                                                         [-e EXECUTOR]
                                                         [-t TAG]

optional arguments:
  -h, --help            show this help message and exit
  -b BUILDSPEC, --buildspec BUILDSPEC
                        Specify path to buildspec (file, or directory) to
                        validate
  -x EXCLUDE, --exclude EXCLUDE
                        Specify path to buildspec to exclude (file or
                        directory) during validation
  -e EXECUTOR, --executor EXECUTOR
                        Specify buildspecs by executor name to validate
  -t TAG, --tag TAG     Specify buildspecs by tag name to validate

The -b option can be used to specify path to buildspec file or directory to validate buildspecs. If its a directory, buildtest will traverse all directories recursively and find any .yml file extensions and attempt to validate each buildspec. Shown below is an example output of what it may look like

buildtest buildspec validate -b tutorials/vars.yml
$ buildtest buildspec validate -b tutorials/vars.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/tutorials/vars.yml
All buildspecs passed validation!!!

If buildtest detects an error during validation, the error message will be displayed to screen with a non-zero returncode.

buildtest buildspec validate -b tutorials/invalid_tags.yml
$ buildtest buildspec validate -b tutorials/invalid_tags.yml
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/t… ─
<ValidationError: "['network', 'network'] is not valid under any of the given schemas">


Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/tutorials/invalid_tags.yml
1 buildspecs failed to validate

Similarly we can search buildspecs based on tags if you want to validate a group of buildspecs using the -t option. We can append -t option multiple times to search by multiple tag names. In this next example, we will validate all buildspecs for python and pass tags.

buildtest buildspec validate -t python -t pass
$ buildtest buildspec validate -t python -t pass
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/tutorials/python-hello.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/tutorials/python-shell.yml
Processing buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/tutorials/test_status/pass_returncode.yml
All buildspecs passed validation!!!

Show buildspec buildtest buildspec show

buildtest can display content of buildspec file given a test name via buildtest buildspec show command which expects a positional argument that is the name of test. This can be quick way to see content of buildspec without remembering the full path to the buildspec.

In this next example, we will instruct buildtest to show content of buildspec for test name python_hello.

buildtest buildspec show python_hello
$ buildtest buildspec show python_hello
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/t… ─
buildspecs:                                                                     
  python_hello:                                                                 
    type: script                                                                
    description: Hello World python                                             
    executor: generic.local.bash                                                
    tags: python                                                                
    run: python hello.py

You can pass multiple arguments to buildtest buildspec show to show content of each test

buildtest buildspec show python_hello circle_area
$ buildtest buildspec show python_hello circle_area
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/t… ─
buildspecs:                                                                     
  python_hello:                                                                 
    type: script                                                                
    description: Hello World python                                             
    executor: generic.local.bash                                                
    tags: python                                                                
    run: python hello.py                                                        
                                                                                
                                                                                
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/t… ─
buildspecs:                                                                     
  circle_area:                                                                  
    executor: generic.local.bash                                                
    type: script                                                                
    shell: python                                                               
    description: "Calculate circle of area given a radius"                      
    tags: [tutorials, python]                                                   
    run: |                                                                      
      import math                                                               
      radius = 2                                                                
      area = math.pi * radius * radius                                          
      print("Circle Radius ", radius)                                           
      print("Area of circle ", area)

There is bash completion for this command which will show list of test names available in the cache assuming you have run buildtest buildspec find. If you specify an invalid test name you will get an error followed by list of tests that are available in the cache

buildtest buildspec show python_hello
$ buildtest buildspec show XYZ123!
Unable to find test XYZ123! in cache

You can use --theme option to define the color scheme used for printing content of buildspecs. The available comlor schemes can be found at https://pygments.org/docs/styles/#getting-a-list-of-available-styles. buildtest supports tab completion on the available themes which you can see below

$  buildtest bc show --theme
abap                autumn              default             friendly_grayscale  igor                manni               native              pastie              sas                 stata-dark          vim
algol               borland             dracula             fruity              inkpot              material            one-dark            perldoc             solarized-dark      stata-light         vs
algol_nu            bw                  emacs               gruvbox-dark        lilypond            monokai             paraiso-dark        rainbow_dash        solarized-light     tango               xcode
arduino             colorful            friendly            gruvbox-light       lovelace            murphy              paraiso-light       rrt                 stata               trac                zenburn

Show fail buildspec buildtest buildspec show-fail

buildtest can display content of buildspec file of all failed tests via buildtest buildspec show-fail command. This can be quick way to see content of buildspec file given a failed test name such as buildtest buildspec show-fail exit1_fail.

buildtest buildspec show-fail exit1_fail
$ buildtest buildspec show-fail exit1_fail
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.0/t… ─
buildspecs:                                                                     
                                                                                
  exit1_fail:                                                                   
    executor: generic.local.bash                                                
    type: script                                                                
    description: exit 1 by default is FAIL                                      
    tags: [tutorials, fail]                                                     
    run: exit 1                                                                 
                                                                                
  exit1_pass:                                                                   
    executor: generic.local.bash                                                
    type: script                                                                
    description: report exit 1 as PASS                                          
    run: exit 1                                                                 
    tags: [tutorials, pass]                                                     
    status:                                                                     
      returncode: [1]                                                           
                                                                                
  returncode_list_mismatch:                                                     
    executor: generic.local.bash                                                
    type: script                                                                
    description: exit 2 failed since it failed to match returncode 1            
    run: exit 2                                                                 
    tags: [tutorials, fail]                                                     
    status:                                                                     
      returncode: [1, 3]                                                        
                                                                                
  returncode_int_match:                                                         
    executor: generic.local.bash                                                
    type: script                                                                
    description: exit 128 matches returncode 128                                
    run: exit 128                                                               
    tags: [tutorials, pass]                                                     
    status:                                                                     
      returncode: 128

If you run buildtest buildspec show-fail without any argument, then buildtest will show content of all failed tests with corresponding buildspec. buildtest will automatically filter out duplicate buildspec entries where multiple test correspond to same buildspec to avoid printing content of buildspec multiple times.

Editing buildspecs in your preferred editor

buildtest provides an interface to automatically open your buildspecs in editor and validate them after closing file. You are welcome to open your buildspec in your editor (vim, emacs, nano) but you won’t be able to validate the buildspec unless you explicitly run the test or use buildtest buildspec validate to see if your buildspec is valid. buildtest comes with two commands to edit your buildspecs buildtest buildspec edit-test and buildtest buildspec edit-file which we will discuss below.

Editing by Test buildtest buildspec edit-test

The buildtest buildspec edit-test allows one to specify a list of test as positional arguments to edit-test in your preferred editor. buildtest will provide tab completion for this command to show all test available in cache which works similar to buildtest buildspec show command.

For instance, we can see the following test are available as part of command completion

$ buildtest buildspec edit-test
_bin_bash_shell                 download_stream                 nodes_state_down                show_host_groups                string_tag
_bin_sh_shell                   executor_regex_script_schema    nodes_state_idle                show_jobs                       systemd_default_target
add_numbers                     executors_sbatch_declaration    nodes_state_reboot              show_lsf_configuration          tcsh_env_declaration
always_fail                     executors_vars_env_declaration  pullImage_dockerhub             show_lsf_models                 test_fail_returncode_match
always_pass                     exit1_fail                      pullImage_shub                  show_lsf_queues                 test_pass_returncode_mismatch
bash_env_variables              exit1_pass                      pullImage_sylabscloud           show_lsf_queues_current_user    timelimit_max
bash_login_shebang              foo_bar                         python_hello                    show_lsf_queues_formatted       timelimit_max_fail
bash_nonlogin_shebang           gcc_version                     qdel_version                    show_lsf_resources              timelimit_min
bash_shell                      get_partitions                  qmove_version                   show_lsf_user_groups            timelimit_min_fail
bhosts_version                  hello_world                     qselect_version                 show_partition                  timelimit_min_max
build_remoteimages              inspect_image                   qsub_version                    show_qos                        ulimit_cputime_unlimited
build_sandbox_image             jobA                            returncode_int_match            show_queues                     ulimit_filedescriptor_4096
build_sif_from_dockerimage      jobB                            returncode_list_mismatch        show_tres                       ulimit_filelock_unlimited
circle_area                     jobC                            root_disk_usage                 show_users                      ulimit_max_user_process_2048
cqsub_version                   kernel_swapusage                runImage                        sinfo_version                   ulimit_stacksize_unlimited
csh_env_declaration             list_of_strings_tags            run_stream                      skip                            ulimit_vmsize_unlimited
csh_shell                       lsf_version                     selinux_disable                 sleep                           unskipped
current_user_queue              metric_regex_example            sh_shell                        slurm_config                    variables_bash
dead_nodes                      node_down_fail_list_reason      shell_options                   status_regex_fail
display_hosts_format            nodes_state_allocated           show_accounts                   status_regex_pass
display_lsf_hosts               nodes_state_completing          show_all_jobs                   status_returncode_by_executors

Let’s take for instance we want to edit the following test, buildtest will search the buildspec cache and find the buildspec file, open in editor and once changes are written to disk, the next file will be processed until all files are written to disk.

$ buildtest buildspec edit-test sleep _bin_bash_shell add_numbers
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml is valid
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/shell_examples.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/shell_examples.yml is valid
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/add_numbers.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/add_numbers.yml is valid

If you specify an invalid test, then buildtest will ignore the test and report a message and skip to next test as shown below

$ buildtest buildspec edit-test invalid_test sleep
Unable to find test invalid_test in cache
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml is valid

Edit buildspecs buildtest buildspec edit-file

The buildtest buildspec edit-file command can be used to edit buildspec based on filename as pose to testname. This command works similar to buildtest buildspec edit-test where each file is open in editor and validated upon completion. You can use this command to create new buildspec whereas buildtest buildspec edit-test only works on existing buildspecs loaded in cache. You can pass multiple filenames as arguments if you want to edit several files.

$ buildtest buildspec edit-file $BUILDTEST_ROOT/tutorials/sleep.yml
  Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml
  /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml is valid