Query Test Report

buildtest keeps track of all tests and results in a JSON file. This file is read by buildtest report command to extract certain fields from JSON file and display them in table format. Shown below is command usage to query test reports.

buildtest report --help
$ buildtest report --help
usage: buildtest [options] [COMMANDS] report [-h] [--pager] [--row-count]
                                             [--terse] [-n] [-c COUNT]
                                             [--filter FILTER] [--helpfilter]
                                             [--filterfields] [--helpformat]
                                             [--formatfields] [--latest]
                                             [--oldest] [-s START] [-e END]
                                             [-f | -p] [--format FORMAT | -d]
                                             ...

optional arguments:
  -h, --help            show this help message and exit
  --pager               Enable PAGING when viewing result
  --row-count           Display number of rows from query shown in table
  --terse               Print output in machine readable format
  -n, --no-header       Do not print header columns in terse output (--terse)
  -c COUNT, --count COUNT
                        Retrieve limited number of rows that get printed
  -f, --fail            Retrieve all FAIL tests
  -p, --pass            Retrieve all PASS tests
  --format FORMAT       Format field for printing purposes. Fields must be
                        separated by commas (e.g., --format
                        field1,field2,...).
  -d, --detailed        Print a detailed summary of the test results

subcommands:
  Fetch test results from the report file and print them in table format

  
    clear (c)           Remove all report files
    list (ls)           List all report files
    path (p)            Print full path to the report file being used
    summary (sm)        Summarize test report

filter:
  --filter FILTER       Filter report by filter fields. The filter fields must
                        be in key=value format and multiple fields can be
                        comma separated (e.g., --filter key1=val1,key2=val2).
                        For a list of filter fields, run --helpfilter.
  --helpfilter          List available filter fields to be used with --filter
                        option
  --filterfields        Print raw filter fields for --filter option to filter
                        the report

format:
  --helpformat          List available format fields to be used with --format
                        option
  --formatfields        Print raw format fields for --format option to format
                        the report

extra:
  --latest              Retrieve latest record of a particular test
  --oldest              Retrieve oldest record of a particular test
  -s START, --start START
                        Filter test by starttime
  -e END, --end END     Filter test by endtime

You may run buildtest report without any option, and buildtest will display all test results with default format fields. To see a list of all format fields, click here.

buildtest report
$ buildtest report
                 Report File: /tmp/tmpjcb9dx2w/var/report.json                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ name                         ┃ id         ┃ state  ┃ runtime   ┃ returncode  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ variables_bash               │ 17e5a942   │ PASS   │ 0.009769  │ 0           │
├──────────────────────────────┼────────────┼────────┼───────────┼─────────────┤
│ variables_bash               │ f0ba0142   │ PASS   │ 0.009573  │ 0           │
├──────────────────────────────┼────────────┼────────┼───────────┼─────────────┤
│ variables_bash               │ 14f5700c   │ PASS   │ 0.009655  │ 0           │
├──────────────────────────────┼────────────┼────────┼───────────┼─────────────┤
│ variables_bash               │ 141e90a4   │ PASS   │ 0.009529  │ 0           │
├──────────────────────────────┼────────────┼────────┼───────────┼─────────────┤
│ variables_bash               │ 71c7fbeb   │ PASS   │ 0.010044  │ 0           │
├──────────────────────────────┼────────────┼────────┼───────────┼─────────────┤
│ exit1_pass                   │ 31743897   │ PASS   │ 0.014586  │ 1           │
├──────────────────────────────┼────────────┼────────┼───────────┼─────────────┤
│ exit1_pass                   │ 9167ac0e   │ PASS   │ 0.014208  │ 1           │
├──────────────────────────────┼────────────┼────────┼───────────┼─────────────┤
│ exit1_pass                   │ 72516f5e   │ PASS   │ 0.014722  │ 1           │
├──────────────────────────────┼────────────┼────────┼───────────┼─────────────┤
...

You can run buildtest report path command to get the full path to the report file being used. This is helpful when the report file is not known in that case one can print the file by doing cat $(buildtest report path).

buildtest report path
$ buildtest report path
/tmp/tmpjcb9dx2w/var/report.json

The buildtest report command will fetch all records and display them in a table format, if you want to know how many records are displayed in the table you can use buildtest report --row-count which will display a raw count of records in the table. In the example below we will run buildtest report --row-count which will show number of tests in the report.

buildtest report --row-count
$ buildtest report --row-count
25

You may find it useful to fetch all failed records and determine pass/fail criteria as part of your shell script, this can be done by running buildtest report --row-count --fail as shown below.

buildtest report --row-count --fail
$ buildtest report --row-count --fail
25

Note

buildtest rt is an alias for buildtest report command.

Detailed Reports (buildtest report --detailed)

The buildtest report –detailed option can be used to show a detailed test report that may be of interest when examining a test. This option is synonymous to running buildtest report --format name,id,user,state,returncode, runtime,outfile,errfile,buildspec. Shown below is an example output.

buildtest report --detailed
$ buildtest report --detailed
                 Report File: /tmp/tmpjcb9dx2w/var/report.json                  
┏━━━━━━━━┳━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┓
┃        ┃        ┃      ┃       ┃ return ┃ runtim ┃ outfil ┃ errfil ┃ buildsp ┃
┃ name   ┃ id     ┃ user ┃ state ┃ code   ┃ e      ┃ e      ┃ e      ┃ ec      ┃
┡━━━━━━━━╇━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━┩
│ variab │ 17e5a9 │ docs │ PASS  │ 0      │ 0.0097 │ /tmp/t │ /tmp/t │ /home/d │
│ les_ba │ 42     │      │       │        │ 69     │ mpjcb9 │ mpjcb9 │ ocs/che │
│ sh     │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│        │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/va │ ash/va │ buildte │
│        │        │      │       │        │        │ rs/var │ rs/var │ st/chec │
│        │        │      │       │        │        │ iables │ iables │ kouts/s │
│        │        │      │       │        │        │ _bash/ │ _bash/ │ table/t │
│        │        │      │       │        │        │ 17e5a9 │ 17e5a9 │ utorial │
│        │        │      │       │        │        │ 42/var │ 42/var │ s/vars. │
│        │        │      │       │        │        │ iables │ iables │ yml     │
│        │        │      │       │        │        │ _bash. │ _bash. │         │
│        │        │      │       │        │        │ out    │ err    │         │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ variab │ f0ba01 │ docs │ PASS  │ 0      │ 0.0095 │ /tmp/t │ /tmp/t │ /home/d │
│ les_ba │ 42     │      │       │        │ 73     │ mpjcb9 │ mpjcb9 │ ocs/che │
│ sh     │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│        │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/va │ ash/va │ buildte │
│        │        │      │       │        │        │ rs/var │ rs/var │ st/chec │
│        │        │      │       │        │        │ iables │ iables │ kouts/s │
│        │        │      │       │        │        │ _bash/ │ _bash/ │ table/t │
│        │        │      │       │        │        │ f0ba01 │ f0ba01 │ utorial │
│        │        │      │       │        │        │ 42/var │ 42/var │ s/vars. │
│        │        │      │       │        │        │ iables │ iables │ yml     │
│        │        │      │       │        │        │ _bash. │ _bash. │         │
│        │        │      │       │        │        │ out    │ err    │         │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ variab │ 14f570 │ docs │ PASS  │ 0      │ 0.0096 │ /tmp/t │ /tmp/t │ /home/d │
│ les_ba │ 0c     │      │       │        │ 55     │ mpjcb9 │ mpjcb9 │ ocs/che │
│ sh     │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│        │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/va │ ash/va │ buildte │
│        │        │      │       │        │        │ rs/var │ rs/var │ st/chec │
│        │        │      │       │        │        │ iables │ iables │ kouts/s │
│        │        │      │       │        │        │ _bash/ │ _bash/ │ table/t │
│        │        │      │       │        │        │ 14f570 │ 14f570 │ utorial │
│        │        │      │       │        │        │ 0c/var │ 0c/var │ s/vars. │
│        │        │      │       │        │        │ iables │ iables │ yml     │
│        │        │      │       │        │        │ _bash. │ _bash. │         │
│        │        │      │       │        │        │ out    │ err    │         │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ variab │ 141e90 │ docs │ PASS  │ 0      │ 0.0095 │ /tmp/t │ /tmp/t │ /home/d │
│ les_ba │ a4     │      │       │        │ 29     │ mpjcb9 │ mpjcb9 │ ocs/che │
│ sh     │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│        │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/va │ ash/va │ buildte │
│        │        │      │       │        │        │ rs/var │ rs/var │ st/chec │
│        │        │      │       │        │        │ iables │ iables │ kouts/s │
│        │        │      │       │        │        │ _bash/ │ _bash/ │ table/t │
│        │        │      │       │        │        │ 141e90 │ 141e90 │ utorial │
│        │        │      │       │        │        │ a4/var │ a4/var │ s/vars. │
│        │        │      │       │        │        │ iables │ iables │ yml     │
│        │        │      │       │        │        │ _bash. │ _bash. │         │
│        │        │      │       │        │        │ out    │ err    │         │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ variab │ 71c7fb │ docs │ PASS  │ 0      │ 0.0100 │ /tmp/t │ /tmp/t │ /home/d │
│ les_ba │ eb     │      │       │        │ 44     │ mpjcb9 │ mpjcb9 │ ocs/che │
│ sh     │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│        │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/va │ ash/va │ buildte │
│        │        │      │       │        │        │ rs/var │ rs/var │ st/chec │
│        │        │      │       │        │        │ iables │ iables │ kouts/s │
│        │        │      │       │        │        │ _bash/ │ _bash/ │ table/t │
│        │        │      │       │        │        │ 71c7fb │ 71c7fb │ utorial │
│        │        │      │       │        │        │ eb/var │ eb/var │ s/vars. │
│        │        │      │       │        │        │ iables │ iables │ yml     │
│        │        │      │       │        │        │ _bash. │ _bash. │         │
│        │        │      │       │        │        │ out    │ err    │         │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ exit1_ │ 317438 │ docs │ PASS  │ 1      │ 0.0145 │ /tmp/t │ /tmp/t │ /home/d │
│ pass   │ 97     │      │       │        │ 86     │ mpjcb9 │ mpjcb9 │ ocs/che │
│        │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│        │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/exit │ e/exit │ table/t │
│        │        │      │       │        │        │ 1_pass │ 1_pass │ utorial │
│        │        │      │       │        │        │ /31743 │ /31743 │ s/test_ │
│        │        │      │       │        │        │ 897/ex │ 897/ex │ status/ │
│        │        │      │       │        │        │ it1_pa │ it1_pa │ pass_re │
│        │        │      │       │        │        │ ss.out │ ss.err │ turncod │
│        │        │      │       │        │        │        │        │ e.yml   │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ exit1_ │ 9167ac │ docs │ PASS  │ 1      │ 0.0142 │ /tmp/t │ /tmp/t │ /home/d │
│ pass   │ 0e     │      │       │        │ 08     │ mpjcb9 │ mpjcb9 │ ocs/che │
│        │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│        │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/exit │ e/exit │ table/t │
│        │        │      │       │        │        │ 1_pass │ 1_pass │ utorial │
│        │        │      │       │        │        │ /9167a │ /9167a │ s/test_ │
│        │        │      │       │        │        │ c0e/ex │ c0e/ex │ status/ │
│        │        │      │       │        │        │ it1_pa │ it1_pa │ pass_re │
│        │        │      │       │        │        │ ss.out │ ss.err │ turncod │
│        │        │      │       │        │        │        │        │ e.yml   │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ exit1_ │ 72516f │ docs │ PASS  │ 1      │ 0.0147 │ /tmp/t │ /tmp/t │ /home/d │
│ pass   │ 5e     │      │       │        │ 22     │ mpjcb9 │ mpjcb9 │ ocs/che │
│        │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│        │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/exit │ e/exit │ table/t │
│        │        │      │       │        │        │ 1_pass │ 1_pass │ utorial │
│        │        │      │       │        │        │ /72516 │ /72516 │ s/test_ │
│        │        │      │       │        │        │ f5e/ex │ f5e/ex │ status/ │
│        │        │      │       │        │        │ it1_pa │ it1_pa │ pass_re │
│        │        │      │       │        │        │ ss.out │ ss.err │ turncod │
│        │        │      │       │        │        │        │        │ e.yml   │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ exit1_ │ 1a292a │ docs │ PASS  │ 1      │ 0.0142 │ /tmp/t │ /tmp/t │ /home/d │
│ pass   │ 6a     │      │       │        │ 81     │ mpjcb9 │ mpjcb9 │ ocs/che │
│        │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│        │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/exit │ e/exit │ table/t │
│        │        │      │       │        │        │ 1_pass │ 1_pass │ utorial │
│        │        │      │       │        │        │ /1a292 │ /1a292 │ s/test_ │
│        │        │      │       │        │        │ a6a/ex │ a6a/ex │ status/ │
│        │        │      │       │        │        │ it1_pa │ it1_pa │ pass_re │
│        │        │      │       │        │        │ ss.out │ ss.err │ turncod │
│        │        │      │       │        │        │        │        │ e.yml   │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ exit1_ │ fca13c │ docs │ PASS  │ 1      │ 0.0140 │ /tmp/t │ /tmp/t │ /home/d │
│ pass   │ f8     │      │       │        │ 01     │ mpjcb9 │ mpjcb9 │ ocs/che │
│        │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│        │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/exit │ e/exit │ table/t │
│        │        │      │       │        │        │ 1_pass │ 1_pass │ utorial │
│        │        │      │       │        │        │ /fca13 │ /fca13 │ s/test_ │
│        │        │      │       │        │        │ cf8/ex │ cf8/ex │ status/ │
│        │        │      │       │        │        │ it1_pa │ it1_pa │ pass_re │
│        │        │      │       │        │        │ ss.out │ ss.err │ turncod │
│        │        │      │       │        │        │        │        │ e.yml   │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ exit1_ │ 70cb6c │ docs │ PASS  │ 1      │ 0.0146 │ /tmp/t │ /tmp/t │ /home/d │
│ pass   │ 0f     │      │       │        │ 8      │ mpjcb9 │ mpjcb9 │ ocs/che │
│        │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│        │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/exit │ e/exit │ table/t │
│        │        │      │       │        │        │ 1_pass │ 1_pass │ utorial │
│        │        │      │       │        │        │ /70cb6 │ /70cb6 │ s/test_ │
│        │        │      │       │        │        │ c0f/ex │ c0f/ex │ status/ │
│        │        │      │       │        │        │ it1_pa │ it1_pa │ pass_re │
│        │        │      │       │        │        │ ss.out │ ss.err │ turncod │
│        │        │      │       │        │        │        │        │ e.yml   │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ exit1_ │ e35f68 │ docs │ PASS  │ 1      │ 0.0140 │ /tmp/t │ /tmp/t │ /home/d │
│ pass   │ 95     │      │       │        │ 48     │ mpjcb9 │ mpjcb9 │ ocs/che │
│        │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│        │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/exit │ e/exit │ table/t │
│        │        │      │       │        │        │ 1_pass │ 1_pass │ utorial │
│        │        │      │       │        │        │ /e35f6 │ /e35f6 │ s/test_ │
│        │        │      │       │        │        │ 895/ex │ 895/ex │ status/ │
│        │        │      │       │        │        │ it1_pa │ it1_pa │ pass_re │
│        │        │      │       │        │        │ ss.out │ ss.err │ turncod │
│        │        │      │       │        │        │        │        │ e.yml   │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ exit1_ │ d4e668 │ docs │ PASS  │ 1      │ 0.0148 │ /tmp/t │ /tmp/t │ /home/d │
│ pass   │ ef     │      │       │        │ 18     │ mpjcb9 │ mpjcb9 │ ocs/che │
│        │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│        │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/exit │ e/exit │ table/t │
│        │        │      │       │        │        │ 1_pass │ 1_pass │ utorial │
│        │        │      │       │        │        │ /d4e66 │ /d4e66 │ s/test_ │
│        │        │      │       │        │        │ 8ef/ex │ 8ef/ex │ status/ │
│        │        │      │       │        │        │ it1_pa │ it1_pa │ pass_re │
│        │        │      │       │        │        │ ss.out │ ss.err │ turncod │
│        │        │      │       │        │        │        │        │ e.yml   │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ exit1_ │ d3fb6c │ docs │ PASS  │ 1      │ 0.0141 │ /tmp/t │ /tmp/t │ /home/d │
│ pass   │ 56     │      │       │        │ 85     │ mpjcb9 │ mpjcb9 │ ocs/che │
│        │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│        │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/exit │ e/exit │ table/t │
│        │        │      │       │        │        │ 1_pass │ 1_pass │ utorial │
│        │        │      │       │        │        │ /d3fb6 │ /d3fb6 │ s/test_ │
│        │        │      │       │        │        │ c56/ex │ c56/ex │ status/ │
│        │        │      │       │        │        │ it1_pa │ it1_pa │ pass_re │
│        │        │      │       │        │        │ ss.out │ ss.err │ turncod │
│        │        │      │       │        │        │        │        │ e.yml   │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ return │ 84a02a │ docs │ FAIL  │ 2      │ 0.0151 │ /tmp/t │ /tmp/t │ /home/d │
│ code_l │ 2a     │      │       │        │ 89     │ mpjcb9 │ mpjcb9 │ ocs/che │
│ ist_mi │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│ smatch │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/retu │ e/retu │ table/t │
│        │        │      │       │        │        │ rncode │ rncode │ utorial │
│        │        │      │       │        │        │ _list_ │ _list_ │ s/test_ │
│        │        │      │       │        │        │ mismat │ mismat │ status/ │
│        │        │      │       │        │        │ ch/84a │ ch/84a │ pass_re │
│        │        │      │       │        │        │ 02a2a/ │ 02a2a/ │ turncod │
│        │        │      │       │        │        │ return │ return │ e.yml   │
│        │        │      │       │        │        │ code_l │ code_l │         │
│        │        │      │       │        │        │ ist_mi │ ist_mi │         │
│        │        │      │       │        │        │ smatch │ smatch │         │
│        │        │      │       │        │        │ .out   │ .err   │         │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ return │ c9a38d │ docs │ FAIL  │ 2      │ 0.0143 │ /tmp/t │ /tmp/t │ /home/d │
│ code_l │ 26     │      │       │        │ 5      │ mpjcb9 │ mpjcb9 │ ocs/che │
│ ist_mi │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│ smatch │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/retu │ e/retu │ table/t │
│        │        │      │       │        │        │ rncode │ rncode │ utorial │
│        │        │      │       │        │        │ _list_ │ _list_ │ s/test_ │
│        │        │      │       │        │        │ mismat │ mismat │ status/ │
│        │        │      │       │        │        │ ch/c9a │ ch/c9a │ pass_re │
│        │        │      │       │        │        │ 38d26/ │ 38d26/ │ turncod │
│        │        │      │       │        │        │ return │ return │ e.yml   │
│        │        │      │       │        │        │ code_l │ code_l │         │
│        │        │      │       │        │        │ ist_mi │ ist_mi │         │
│        │        │      │       │        │        │ smatch │ smatch │         │
│        │        │      │       │        │        │ .out   │ .err   │         │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ return │ 4b8846 │ docs │ FAIL  │ 2      │ 0.0142 │ /tmp/t │ /tmp/t │ /home/d │
│ code_l │ a2     │      │       │        │ 49     │ mpjcb9 │ mpjcb9 │ ocs/che │
│ ist_mi │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│ smatch │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/retu │ e/retu │ table/t │
│        │        │      │       │        │        │ rncode │ rncode │ utorial │
│        │        │      │       │        │        │ _list_ │ _list_ │ s/test_ │
│        │        │      │       │        │        │ mismat │ mismat │ status/ │
│        │        │      │       │        │        │ ch/4b8 │ ch/4b8 │ pass_re │
│        │        │      │       │        │        │ 846a2/ │ 846a2/ │ turncod │
│        │        │      │       │        │        │ return │ return │ e.yml   │
│        │        │      │       │        │        │ code_l │ code_l │         │
│        │        │      │       │        │        │ ist_mi │ ist_mi │         │
│        │        │      │       │        │        │ smatch │ smatch │         │
│        │        │      │       │        │        │ .out   │ .err   │         │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ return │ f5327c │ docs │ FAIL  │ 2      │ 0.0144 │ /tmp/t │ /tmp/t │ /home/d │
│ code_l │ 7d     │      │       │        │ 71     │ mpjcb9 │ mpjcb9 │ ocs/che │
│ ist_mi │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│ smatch │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/retu │ e/retu │ table/t │
│        │        │      │       │        │        │ rncode │ rncode │ utorial │
│        │        │      │       │        │        │ _list_ │ _list_ │ s/test_ │
│        │        │      │       │        │        │ mismat │ mismat │ status/ │
│        │        │      │       │        │        │ ch/f53 │ ch/f53 │ pass_re │
│        │        │      │       │        │        │ 27c7d/ │ 27c7d/ │ turncod │
│        │        │      │       │        │        │ return │ return │ e.yml   │
│        │        │      │       │        │        │ code_l │ code_l │         │
│        │        │      │       │        │        │ ist_mi │ ist_mi │         │
│        │        │      │       │        │        │ smatch │ smatch │         │
│        │        │      │       │        │        │ .out   │ .err   │         │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ return │ ee6d33 │ docs │ FAIL  │ 2      │ 0.0149 │ /tmp/t │ /tmp/t │ /home/d │
│ code_l │ c8     │      │       │        │ 02     │ mpjcb9 │ mpjcb9 │ ocs/che │
│ ist_mi │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│ smatch │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/retu │ e/retu │ table/t │
│        │        │      │       │        │        │ rncode │ rncode │ utorial │
│        │        │      │       │        │        │ _list_ │ _list_ │ s/test_ │
│        │        │      │       │        │        │ mismat │ mismat │ status/ │
│        │        │      │       │        │        │ ch/ee6 │ ch/ee6 │ pass_re │
│        │        │      │       │        │        │ d33c8/ │ d33c8/ │ turncod │
│        │        │      │       │        │        │ return │ return │ e.yml   │
│        │        │      │       │        │        │ code_l │ code_l │         │
│        │        │      │       │        │        │ ist_mi │ ist_mi │         │
│        │        │      │       │        │        │ smatch │ smatch │         │
│        │        │      │       │        │        │ .out   │ .err   │         │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ return │ a2a4cb │ docs │ FAIL  │ 2      │ 0.0141 │ /tmp/t │ /tmp/t │ /home/d │
│ code_l │ fd     │      │       │        │ 75     │ mpjcb9 │ mpjcb9 │ ocs/che │
│ ist_mi │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│ smatch │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/retu │ e/retu │ table/t │
│        │        │      │       │        │        │ rncode │ rncode │ utorial │
│        │        │      │       │        │        │ _list_ │ _list_ │ s/test_ │
│        │        │      │       │        │        │ mismat │ mismat │ status/ │
│        │        │      │       │        │        │ ch/a2a │ ch/a2a │ pass_re │
│        │        │      │       │        │        │ 4cbfd/ │ 4cbfd/ │ turncod │
│        │        │      │       │        │        │ return │ return │ e.yml   │
│        │        │      │       │        │        │ code_l │ code_l │         │
│        │        │      │       │        │        │ ist_mi │ ist_mi │         │
│        │        │      │       │        │        │ smatch │ smatch │         │
│        │        │      │       │        │        │ .out   │ .err   │         │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ return │ 6187c2 │ docs │ FAIL  │ 2      │ 0.0161 │ /tmp/t │ /tmp/t │ /home/d │
│ code_l │ 99     │      │       │        │ 05     │ mpjcb9 │ mpjcb9 │ ocs/che │
│ ist_mi │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│ smatch │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/retu │ e/retu │ table/t │
│        │        │      │       │        │        │ rncode │ rncode │ utorial │
│        │        │      │       │        │        │ _list_ │ _list_ │ s/test_ │
│        │        │      │       │        │        │ mismat │ mismat │ status/ │
│        │        │      │       │        │        │ ch/618 │ ch/618 │ pass_re │
│        │        │      │       │        │        │ 7c299/ │ 7c299/ │ turncod │
│        │        │      │       │        │        │ return │ return │ e.yml   │
│        │        │      │       │        │        │ code_l │ code_l │         │
│        │        │      │       │        │        │ ist_mi │ ist_mi │         │
│        │        │      │       │        │        │ smatch │ smatch │         │
│        │        │      │       │        │        │ .out   │ .err   │         │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ return │ b70d4e │ docs │ FAIL  │ 2      │ 0.0145 │ /tmp/t │ /tmp/t │ /home/d │
│ code_l │ 27     │      │       │        │ 3      │ mpjcb9 │ mpjcb9 │ ocs/che │
│ ist_mi │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│ smatch │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/retu │ e/retu │ table/t │
│        │        │      │       │        │        │ rncode │ rncode │ utorial │
│        │        │      │       │        │        │ _list_ │ _list_ │ s/test_ │
│        │        │      │       │        │        │ mismat │ mismat │ status/ │
│        │        │      │       │        │        │ ch/b70 │ ch/b70 │ pass_re │
│        │        │      │       │        │        │ d4e27/ │ d4e27/ │ turncod │
│        │        │      │       │        │        │ return │ return │ e.yml   │
│        │        │      │       │        │        │ code_l │ code_l │         │
│        │        │      │       │        │        │ ist_mi │ ist_mi │         │
│        │        │      │       │        │        │ smatch │ smatch │         │
│        │        │      │       │        │        │ .out   │ .err   │         │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ return │ fd4203 │ docs │ PASS  │ 128    │ 0.0145 │ /tmp/t │ /tmp/t │ /home/d │
│ code_i │ 2f     │      │       │        │        │ mpjcb9 │ mpjcb9 │ ocs/che │
│ nt_mat │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│ ch     │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/retu │ e/retu │ table/t │
│        │        │      │       │        │        │ rncode │ rncode │ utorial │
│        │        │      │       │        │        │ _int_m │ _int_m │ s/test_ │
│        │        │      │       │        │        │ atch/f │ atch/f │ status/ │
│        │        │      │       │        │        │ d42032 │ d42032 │ pass_re │
│        │        │      │       │        │        │ f/retu │ f/retu │ turncod │
│        │        │      │       │        │        │ rncode │ rncode │ e.yml   │
│        │        │      │       │        │        │ _int_m │ _int_m │         │
│        │        │      │       │        │        │ atch.o │ atch.e │         │
│        │        │      │       │        │        │ ut     │ rr     │         │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ return │ 97233a │ docs │ PASS  │ 128    │ 0.0141 │ /tmp/t │ /tmp/t │ /home/d │
│ code_i │ 17     │      │       │        │ 62     │ mpjcb9 │ mpjcb9 │ ocs/che │
│ nt_mat │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│ ch     │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/retu │ e/retu │ table/t │
│        │        │      │       │        │        │ rncode │ rncode │ utorial │
│        │        │      │       │        │        │ _int_m │ _int_m │ s/test_ │
│        │        │      │       │        │        │ atch/9 │ atch/9 │ status/ │
│        │        │      │       │        │        │ 7233a1 │ 7233a1 │ pass_re │
│        │        │      │       │        │        │ 7/retu │ 7/retu │ turncod │
│        │        │      │       │        │        │ rncode │ rncode │ e.yml   │
│        │        │      │       │        │        │ _int_m │ _int_m │         │
│        │        │      │       │        │        │ atch.o │ atch.e │         │
│        │        │      │       │        │        │ ut     │ rr     │         │
├────────┼────────┼──────┼───────┼────────┼────────┼────────┼────────┼─────────┤
│ return │ c23115 │ docs │ PASS  │ 128    │ 0.0142 │ /tmp/t │ /tmp/t │ /home/d │
│ code_i │ dd     │      │       │        │ 54     │ mpjcb9 │ mpjcb9 │ ocs/che │
│ nt_mat │        │      │       │        │        │ dx2w/v │ dx2w/v │ ckouts/ │
│ ch     │        │      │       │        │        │ ar/tes │ ar/tes │ readthe │
│        │        │      │       │        │        │ ts/gen │ ts/gen │ docs.or │
│        │        │      │       │        │        │ eric.l │ eric.l │ g/user_ │
│        │        │      │       │        │        │ ocal.b │ ocal.b │ builds/ │
│        │        │      │       │        │        │ ash/pa │ ash/pa │ buildte │
│        │        │      │       │        │        │ ss_ret │ ss_ret │ st/chec │
│        │        │      │       │        │        │ urncod │ urncod │ kouts/s │
│        │        │      │       │        │        │ e/retu │ e/retu │ table/t │
│        │        │      │       │        │        │ rncode │ rncode │ utorial │
│        │        │      │       │        │        │ _int_m │ _int_m │ s/test_ │
│        │        │      │       │        │        │ atch/c │ atch/c │ status/ │
│        │        │      │       │        │        │ 23115d │ 23115d │ pass_re │
│        │        │      │       │        │        │ d/retu │ d/retu │ turncod │
│        │        │      │       │        │        │ rncode │ rncode │ e.yml   │
│        │        │      │       │        │        │ _int_m │ _int_m │         │
│        │        │      │       │        │        │ atch.o │ atch.e │         │
│        │        │      │       │        │        │ ut     │ rr     │         │
└────────┴────────┴──────┴───────┴────────┴────────┴────────┴────────┴─────────┘

Note

The --detailed and --format options are mutually exclusive options because both options will alter the format fields when displaying test results.

You will get the following error if you specify both options as shown below

buildtest report --detailed --format name
$ buildtest report --detailed --format name
usage: buildtest [options] [COMMANDS] report [-h] [--pager] [--row-count]
                                             [--terse] [-n] [-c COUNT]
                                             [--filter FILTER] [--helpfilter]
                                             [--filterfields] [--helpformat]
                                             [--formatfields] [--latest]
                                             [--oldest] [-s START] [-e END]
                                             [-f | -p] [--format FORMAT | -d]
                                             ...
buildtest [options] [COMMANDS] report: error: argument --format: not allowed with argument -d/--detailed

Format Reports (buildtest report --format)

The buildtest report command displays a default format fields that can be changed using the --format option. The report file (JSON) contains many more fields and we expose some of the fields with the –format option. To see a list of available format fields you can run buildtest report --helpformat. This option will list all format fields with their description.

buildtest report --helpformat
$ buildtest report --helpformat
                          Format Fields                          
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Field      ┃ Description                                      ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildspec  │ Buildspec File                                   │
│ buildenv   │ Show build environment file for test             │
│ command    │ Command executed                                 │
│ compiler   │ Retrieve compiler used for test                  │
│ endtime    │ End Time for test                                │
│ errfile    │ Error File                                       │
│ executor   │ Name of executor used for running test           │
│ hostname   │ Hostname of machine where job was submitted from │
│ full_id    │ Fully qualified build identifier                 │
│ id         │ Unique build identifier                          │
│ metrics    │ List all metrics for test                        │
│ name       │ Name of test                                     │
│ outfile    │ Output file                                      │
│ returncode │ Return Code                                      │
│ runtime    │ Total runtime in seconds                         │
│ schemafile │ Schema file used for validating test             │
│ starttime  │ Start time of test                               │
│ state      │ State of test (PASS/FAIL)                        │
│ tags       │ Tag name                                         │
│ testroot   │ Root of test directory                           │
│ testpath   │ Path to test                                     │
│ user       │ User who ran the test                            │
└────────────┴──────────────────────────────────────────────────┘

To print just the raw format fields use the command buildtest report --formatfields.

buildtest report --formatfields
$ buildtest report --formatfields
buildspec
buildenv
command
compiler
endtime
errfile
executor
hostname
full_id
id
metrics
name
outfile
returncode
runtime
schemafile
starttime
state
tags
testroot
testpath
user

Format Field Usage

The --format field are specified in comma separated format (i.e --format <field1>,<field2>). In this example we format table by fields --format id,executor,state,returncode.

buildtest rt --format name,id,executor,state,returncode
$ buildtest rt --format name,id,executor,state,returncode
                 Report File: /tmp/tmpjcb9dx2w/var/report.json                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓
┃ name                    ┃ id       ┃ executor           ┃ state ┃ returncode ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩
│ variables_bash          │ 17e5a942 │ generic.local.bash │ PASS  │ 0          │
├─────────────────────────┼──────────┼────────────────────┼───────┼────────────┤
│ variables_bash          │ f0ba0142 │ generic.local.bash │ PASS  │ 0          │
├─────────────────────────┼──────────┼────────────────────┼───────┼────────────┤
│ variables_bash          │ 14f5700c │ generic.local.bash │ PASS  │ 0          │
├─────────────────────────┼──────────┼────────────────────┼───────┼────────────┤
│ variables_bash          │ 141e90a4 │ generic.local.bash │ PASS  │ 0          │
├─────────────────────────┼──────────┼────────────────────┼───────┼────────────┤
│ variables_bash          │ 71c7fbeb │ generic.local.bash │ PASS  │ 0          │
├─────────────────────────┼──────────┼────────────────────┼───────┼────────────┤
│ exit1_pass              │ 31743897 │ generic.local.bash │ PASS  │ 1          │
├─────────────────────────┼──────────┼────────────────────┼───────┼────────────┤
│ exit1_pass              │ 9167ac0e │ generic.local.bash │ PASS  │ 1          │
├─────────────────────────┼──────────┼────────────────────┼───────┼────────────┤
│ exit1_pass              │ 72516f5e │ generic.local.bash │ PASS  │ 1          │
├─────────────────────────┼──────────┼────────────────────┼───────┼────────────┤
│ exit1_pass              │ 1a292a6a │ generic.local.bash │ PASS  │ 1          │
...

Filter Reports (buildtest report --filter)

The buildtest report command will display all tests results, which can be quite long depending on number of tests so therefore we need a mechanism to filter the test results. The --filter option can be used to filter out tests in the output based on filter fields. First, lets see the available filter fields by run buildtest report --helpfilter which shows a list of filter fields and their description.

buildtest report --helpfilter
$ buildtest report --helpfilter
                       Filter Fields                        
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Field      ┃ Description                ┃ Expected Value ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ buildspec  │ Filter by buildspec file   │ FILE           │
│ name       │ Filter by test name        │ STRING         │
│ executor   │ Filter by executor name    │ STRING         │
│ returncode │ Filter tests by returncode │ INT            │
│ state      │ Filter by test state       │ PASS/FAIL      │
│ tags       │ Filter tests by tag name   │ STRING         │
└────────────┴────────────────────────────┴────────────────┘

To print just the raw filter fields use the command buildtest report --filterfields.

buildtest report --filterfields
$ buildtest report --filterfields
buildspec
name
executor
returncode
state
tags

The --filter option expects arguments in key=value format. You can specify multiple filter delimited by comma. buildtest will treat multiple filters as logical AND operation. The filter option can be used with --format field. Let’s see some examples to illustrate the point.

Filter by returncode (--filter returncode)

If you want to retrieve all tests with a given returncode, we can use the returncode property. For instance, let’s retrieve all tests with returncode of 2 by setting --filter returncode=2.

buildtest rt --filter returncode=2 --format=name,id,returncode
$ buildtest rt --filter returncode=2 --format=name,id,returncode
                 Report File: /tmp/tmpjcb9dx2w/var/report.json                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ name                                     ┃ id             ┃ returncode       ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ returncode_list_mismatch                 │ 84a02a2a       │ 2                │
├──────────────────────────────────────────┼────────────────┼──────────────────┤
│ returncode_list_mismatch                 │ c9a38d26       │ 2                │
├──────────────────────────────────────────┼────────────────┼──────────────────┤
│ returncode_list_mismatch                 │ 4b8846a2       │ 2                │
├──────────────────────────────────────────┼────────────────┼──────────────────┤
│ returncode_list_mismatch                 │ f5327c7d       │ 2                │
├──────────────────────────────────────────┼────────────────┼──────────────────┤
│ returncode_list_mismatch                 │ ee6d33c8       │ 2                │
├──────────────────────────────────────────┼────────────────┼──────────────────┤
│ returncode_list_mismatch                 │ a2a4cbfd       │ 2                │
├──────────────────────────────────────────┼────────────────┼──────────────────┤
│ returncode_list_mismatch                 │ 6187c299       │ 2                │
├──────────────────────────────────────────┼────────────────┼──────────────────┤
│ returncode_list_mismatch                 │ b70d4e27       │ 2                │
└──────────────────────────────────────────┴────────────────┴──────────────────┘

Note

buildtest automatically converts returncode to integer when matching returncode, so --filter returncode="2" will work too

Filter by test name (--filter name)

If you want to filter by test name, use the name attribute in filter option. Let’s assume we want to filter all tests by name exit1_pass, this can be achieved by setting filter field as follows: --filter name=exit1_pass. Shown below is an example using name filter field to filter test results.

buildtest rt --filter name=exit1_pass --format=name,id,returncode,state
$ buildtest rt --filter name=exit1_pass --format=name,id,returncode,state
                 Report File: /tmp/tmpjcb9dx2w/var/report.json                  
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ name                 ┃ id                ┃ returncode           ┃ state      ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ exit1_pass           │ 31743897          │ 1                    │ PASS       │
├──────────────────────┼───────────────────┼──────────────────────┼────────────┤
│ exit1_pass           │ 9167ac0e          │ 1                    │ PASS       │
├──────────────────────┼───────────────────┼──────────────────────┼────────────┤
│ exit1_pass           │ 72516f5e          │ 1                    │ PASS       │
├──────────────────────┼───────────────────┼──────────────────────┼────────────┤
│ exit1_pass           │ 1a292a6a          │ 1                    │ PASS       │
├──────────────────────┼───────────────────┼──────────────────────┼────────────┤
│ exit1_pass           │ fca13cf8          │ 1                    │ PASS       │
├──────────────────────┼───────────────────┼──────────────────────┼────────────┤
│ exit1_pass           │ 70cb6c0f          │ 1                    │ PASS       │
├──────────────────────┼───────────────────┼──────────────────────┼────────────┤
│ exit1_pass           │ e35f6895          │ 1                    │ PASS       │
├──────────────────────┼───────────────────┼──────────────────────┼────────────┤
│ exit1_pass           │ d4e668ef          │ 1                    │ PASS       │
├──────────────────────┼───────────────────┼──────────────────────┼────────────┤
│ exit1_pass           │ d3fb6c56          │ 1                    │ PASS       │
└──────────────────────┴───────────────────┴──────────────────────┴────────────┘

Filter by buildspec (--filter buildspec)

Likewise, we can filter results by buildspec file using buildspec attribute via --filter buildspec=<file>. The buildspec attribute must resolve to a file path which can be relative or absolute path. buildtest will resolve path (absolute path) and find the appropriate tests that belong to the buildspec file. If file doesn’t exist or is not found in cache it will raise an error.

buildtest rt --filter buildspec=tutorials/python-hello.yml --format=name,id,state,buildspec
$ buildtest rt --filter buildspec=tutorials/python-hello.yml --format=name,id,state,buildspec
                 Report File: /tmp/tmpjcb9dx2w/var/report.json                  
┏━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name         ┃ id       ┃ state ┃ buildspec                                  ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ python_hello │ ee489bf6 │ PASS  │ /home/docs/checkouts/readthedocs.org/user_ │
│              │          │       │ builds/buildtest/checkouts/stable/tutorial │
│              │          │       │ s/python-hello.yml                         │
├──────────────┼──────────┼───────┼────────────────────────────────────────────┤
│ python_hello │ e375ae0b │ PASS  │ /home/docs/checkouts/readthedocs.org/user_ │
│              │          │       │ builds/buildtest/checkouts/stable/tutorial │
│              │          │       │ s/python-hello.yml                         │
├──────────────┼──────────┼───────┼────────────────────────────────────────────┤
│ python_hello │ 20c19e20 │ PASS  │ /home/docs/checkouts/readthedocs.org/user_ │
│              │          │       │ builds/buildtest/checkouts/stable/tutorial │
│              │          │       │ s/python-hello.yml                         │
├──────────────┼──────────┼───────┼────────────────────────────────────────────┤
│ python_hello │ c1ae4e33 │ PASS  │ /home/docs/checkouts/readthedocs.org/user_ │
│              │          │       │ builds/buildtest/checkouts/stable/tutorial │
│              │          │       │ s/python-hello.yml                         │
└──────────────┴──────────┴───────┴────────────────────────────────────────────┘

Filter by test state (--filter state)

If you want to filter results by test state, use the state property. This can be useful if you want to know all pass or failed tests. The state property expects value of [PASS|FAIL] since these are the two recorded test states marked by buildtest. We can also pass multiple filter fields for instance if we want to find all FAIL tests for executor generic.local.sh we can do the following.

buildtest rt --filter state=FAIL,executor=generic.local.sh --format=name,id,state,executor
$ buildtest rt --filter state=FAIL,executor=generic.local.sh --format=name,id,state,executor
                 Report File: /tmp/tmpjcb9dx2w/var/report.json                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
┃ name                             ┃ id         ┃ state   ┃ executor           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
│ timelimit_max_fail               │ 70ed445f   │ FAIL    │ generic.local.sh   │
├──────────────────────────────────┼────────────┼─────────┼────────────────────┤
│ timelimit_min_fail               │ fbd61e45   │ FAIL    │ generic.local.sh   │
├──────────────────────────────────┼────────────┼─────────┼────────────────────┤
│ always_fail                      │ feab4fcf   │ FAIL    │ generic.local.sh   │
├──────────────────────────────────┼────────────┼─────────┼────────────────────┤
│ test_fail_returncode_match       │ 54cadb12   │ FAIL    │ generic.local.sh   │
└──────────────────────────────────┴────────────┴─────────┴────────────────────┘

Filter Exception Cases

The returncode filter field expects an integer value, so if you try a non-integer returncode you will get the following message

buildtest rt --filter returncode=1.5
$ buildtest rt --filter returncode=1.5
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ bin/buildtest:38 in <module>                                                 │
│                                                                              │
│   35                                                                         │
│   36 if __name__ == "__main__":                                              │
│   37                                                                         │
│ ❱ 38   sys.exit(main())                                                      │
│   39                                                                         │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ buildtest_file = '/home/docs/checkouts/readthedocs.org/user_builds/buil… │ │
│ │           main = <function main at 0x7f1369c6bee0>                       │ │
│ │             os = <module 'os' from                                       │ │
│ │                  '/home/docs/.asdf/installs/python/3.8.18/lib/python3.8… │ │
│ │         prefix = '/home/docs/checkouts/readthedocs.org/user_builds/buil… │ │
│ │            sys = <module 'sys' (built-in)>                               │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ buildtest/main.py:301 in main                                                │
│                                                                              │
│   298 │                                                                      │
│   299 │   # buildtest report                                                 │
│   300 │   elif args.subcommands in ["report", "rt"]:                         │
│ ❱ 301 │   │   report_cmd(args, configuration=configuration, report_file=repo │
│   302 │                                                                      │
│   303 │   elif args.subcommands == "path":                                   │
│   304 │   │   path_cmd(                                                      │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │             args = Namespace(color=None, configfile=None, count=None,    │ │
│ │                    debug=False, detailed=False, editor=None, end=None,   │ │
│ │                    fail=False, filter={'returncode': '1.5'},             │ │
│ │                    filterfields=False, format=None, formatfields=False,  │ │
│ │                    helpcolor=False, helpfilter=False, helpformat=False,  │ │
│ │                    latest=False, loglevel='DEBUG', logpath=False,        │ │
│ │                    no_color=False, no_header=False, oldest=False,        │ │
│ │                    pager=False, passed=False, print_log=False,           │ │
│ │                    report=None, report_subcommand=None, row_count=False, │ │
│ │                    start=None, subcommands='rt', terse=False,            │ │
│ │                    view_log=False)                                       │ │
│ │ buildtest_editor = '/usr/bin/vim'                                        │ │
│ │      config_file = None                                                  │ │
│ │    configuration = <buildtest.config.SiteConfiguration object at         │ │
│ │                    0x7f1367dff7f0>                                       │ │
│ │           logger = <Logger buildtest (DEBUG)>                            │ │
│ │         no_color = False                                                 │ │
│ │           parser = <buildtest.cli.BuildTestParser object at              │ │
│ │                    0x7f1369d0fa60>                                       │ │
│ │      report_file = None                                                  │ │
│ │           system = <buildtest.system.BuildTestSystem object at           │ │
│ │                    0x7f1367dff310>                                       │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ buildtest/cli/report.py:849 in report_cmd                                    │
│                                                                              │
│   846 │   │   list_report()                                                  │
│   847 │   │   return                                                         │
│   848 │                                                                      │
│ ❱ 849 │   results = Report(                                                  │
│   850 │   │   configuration=configuration,                                   │
│   851 │   │   filter_args=args.filter,                                       │
│   852 │   │   format_args=args.format,                                       │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │          args = Namespace(color=None, configfile=None, count=None,       │ │
│ │                 debug=False, detailed=False, editor=None, end=None,      │ │
│ │                 fail=False, filter={'returncode': '1.5'},                │ │
│ │                 filterfields=False, format=None, formatfields=False,     │ │
│ │                 helpcolor=False, helpfilter=False, helpformat=False,     │ │
│ │                 latest=False, loglevel='DEBUG', logpath=False,           │ │
│ │                 no_color=False, no_header=False, oldest=False,           │ │
│ │                 pager=False, passed=False, print_log=False, report=None, │ │
│ │                 report_subcommand=None, row_count=False, start=None,     │ │
│ │                 subcommands='rt', terse=False, view_log=False)           │ │
│ │ configuration = <buildtest.config.SiteConfiguration object at            │ │
│ │                 0x7f1367dff7f0>                                          │ │
│ │  consoleColor = 'default'                                                │ │
│ │         pager = False                                                    │ │
│ │   report_file = None                                                     │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ buildtest/cli/report.py:155 in __init__                                      │
│                                                                              │
│   152 │   │   │   self._reportfile = resolve_path(report_file)               │
│   153 │   │                                                                  │
│   154 │   │   self.report = self.load()                                      │
│ ❱ 155 │   │   self._check_filter_fields()                                    │
│   156 │   │   self._check_format_fields()                                    │
│   157 │   │   self._check_start_and_end_fields()                             │
│   158 │   │   self.filter_buildspecs_from_report()                           │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │           color = None                                                   │ │
│ │   configuration = <buildtest.config.SiteConfiguration object at          │ │
│ │                   0x7f1367dff7f0>                                        │ │
│ │           count = None                                                   │ │
│ │        detailed = None                                                   │ │
│ │             end = None                                                   │ │
│ │         failure = False                                                  │ │
│ │     filter_args = {'returncode': '1.5'}                                  │ │
│ │     format_args = None                                                   │ │
│ │ format_detailed = False                                                  │ │
│ │          latest = False                                                  │ │
│ │          oldest = False                                                  │ │
│ │           pager = None                                                   │ │
│ │          passed = False                                                  │ │
│ │     report_file = None                                                   │ │
│ │            self = <buildtest.cli.report.Report object at 0x7f1367e38550> │ │
│ │           start = None                                                   │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ buildtest/cli/report.py:191 in _check_filter_fields                          │
│                                                                              │
│   188 │   │   │   │   │   )                                                  │
│   189 │   │   │   │                                                          │
│   190 │   │   │   │   if key == "returncode" and not is_int(self.filter[key] │
│ ❱ 191 │   │   │   │   │   raise BuildTestError(                              │
│   192 │   │   │   │   │   │   f"Invalid returncode:{self.filter[key]} must b │
│   193 │   │   │   │   │   )                                                  │
│   194                                                                        │
│                                                                              │
│ ╭─────────────────────────── locals ────────────────────────────╮            │
│ │  key = 'returncode'                                           │            │
│ │ self = <buildtest.cli.report.Report object at 0x7f1367e38550> │            │
│ ╰───────────────────────────────────────────────────────────────╯            │
╰──────────────────────────────────────────────────────────────────────────────╯
BuildTestError: 'Invalid returncode:1.5 must be an integer'

The state filter field expects value of PASS or FAIL so if you specify an invalid state you will get an error as follows.

buildtest rt --filter state=UNKNOWN
$ buildtest rt --filter state=UNKNOWN
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ bin/buildtest:38 in <module>                                                 │
│                                                                              │
│   35                                                                         │
│   36 if __name__ == "__main__":                                              │
│   37                                                                         │
│ ❱ 38   sys.exit(main())                                                      │
│   39                                                                         │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ buildtest_file = '/home/docs/checkouts/readthedocs.org/user_builds/buil… │ │
│ │           main = <function main at 0x7f91a5badee0>                       │ │
│ │             os = <module 'os' from                                       │ │
│ │                  '/home/docs/.asdf/installs/python/3.8.18/lib/python3.8… │ │
│ │         prefix = '/home/docs/checkouts/readthedocs.org/user_builds/buil… │ │
│ │            sys = <module 'sys' (built-in)>                               │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ buildtest/main.py:301 in main                                                │
│                                                                              │
│   298 │                                                                      │
│   299 │   # buildtest report                                                 │
│   300 │   elif args.subcommands in ["report", "rt"]:                         │
│ ❱ 301 │   │   report_cmd(args, configuration=configuration, report_file=repo │
│   302 │                                                                      │
│   303 │   elif args.subcommands == "path":                                   │
│   304 │   │   path_cmd(                                                      │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │             args = Namespace(color=None, configfile=None, count=None,    │ │
│ │                    debug=False, detailed=False, editor=None, end=None,   │ │
│ │                    fail=False, filter={'state': 'UNKNOWN'},              │ │
│ │                    filterfields=False, format=None, formatfields=False,  │ │
│ │                    helpcolor=False, helpfilter=False, helpformat=False,  │ │
│ │                    latest=False, loglevel='DEBUG', logpath=False,        │ │
│ │                    no_color=False, no_header=False, oldest=False,        │ │
│ │                    pager=False, passed=False, print_log=False,           │ │
│ │                    report=None, report_subcommand=None, row_count=False, │ │
│ │                    start=None, subcommands='rt', terse=False,            │ │
│ │                    view_log=False)                                       │ │
│ │ buildtest_editor = '/usr/bin/vim'                                        │ │
│ │      config_file = None                                                  │ │
│ │    configuration = <buildtest.config.SiteConfiguration object at         │ │
│ │                    0x7f91a3d417f0>                                       │ │
│ │           logger = <Logger buildtest (DEBUG)>                            │ │
│ │         no_color = False                                                 │ │
│ │           parser = <buildtest.cli.BuildTestParser object at              │ │
│ │                    0x7f91a5c51a60>                                       │ │
│ │      report_file = None                                                  │ │
│ │           system = <buildtest.system.BuildTestSystem object at           │ │
│ │                    0x7f91a3d41310>                                       │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ buildtest/cli/report.py:849 in report_cmd                                    │
│                                                                              │
│   846 │   │   list_report()                                                  │
│   847 │   │   return                                                         │
│   848 │                                                                      │
│ ❱ 849 │   results = Report(                                                  │
│   850 │   │   configuration=configuration,                                   │
│   851 │   │   filter_args=args.filter,                                       │
│   852 │   │   format_args=args.format,                                       │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │          args = Namespace(color=None, configfile=None, count=None,       │ │
│ │                 debug=False, detailed=False, editor=None, end=None,      │ │
│ │                 fail=False, filter={'state': 'UNKNOWN'},                 │ │
│ │                 filterfields=False, format=None, formatfields=False,     │ │
│ │                 helpcolor=False, helpfilter=False, helpformat=False,     │ │
│ │                 latest=False, loglevel='DEBUG', logpath=False,           │ │
│ │                 no_color=False, no_header=False, oldest=False,           │ │
│ │                 pager=False, passed=False, print_log=False, report=None, │ │
│ │                 report_subcommand=None, row_count=False, start=None,     │ │
│ │                 subcommands='rt', terse=False, view_log=False)           │ │
│ │ configuration = <buildtest.config.SiteConfiguration object at            │ │
│ │                 0x7f91a3d417f0>                                          │ │
│ │  consoleColor = 'default'                                                │ │
│ │         pager = False                                                    │ │
│ │   report_file = None                                                     │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ buildtest/cli/report.py:158 in __init__                                      │
│                                                                              │
│   155 │   │   self._check_filter_fields()                                    │
│   156 │   │   self._check_format_fields()                                    │
│   157 │   │   self._check_start_and_end_fields()                             │
│ ❱ 158 │   │   self.filter_buildspecs_from_report()                           │
│   159 │   │                                                                  │
│   160 │   │   self.process_report()                                          │
│   161                                                                        │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │           color = None                                                   │ │
│ │   configuration = <buildtest.config.SiteConfiguration object at          │ │
│ │                   0x7f91a3d417f0>                                        │ │
│ │           count = None                                                   │ │
│ │        detailed = None                                                   │ │
│ │             end = None                                                   │ │
│ │         failure = False                                                  │ │
│ │     filter_args = {'state': 'UNKNOWN'}                                   │ │
│ │     format_args = None                                                   │ │
│ │ format_detailed = False                                                  │ │
│ │          latest = False                                                  │ │
│ │          oldest = False                                                  │ │
│ │           pager = None                                                   │ │
│ │          passed = False                                                  │ │
│ │     report_file = None                                                   │ │
│ │            self = <buildtest.cli.report.Report object at 0x7f91a3d7a3d0> │ │
│ │           start = None                                                   │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ buildtest/cli/report.py:323 in filter_buildspecs_from_report                 │
│                                                                              │
│   320 │   │   # ensure 'state' field in filter is either 'PASS' or 'FAIL', i │
│   321 │   │   if self.filter.get("state"):                                   │
│   322 │   │   │   if self.filter["state"] not in ["PASS", "FAIL"]:           │
│ ❱ 323 │   │   │   │   raise BuildTestError(                                  │
│   324 │   │   │   │   │   f"filter argument 'state' must be 'PASS' or 'FAIL' │
│   325 │   │   │   │   )                                                      │
│   326                                                                        │
│                                                                              │
│ ╭─────────────────────────── locals ────────────────────────────╮            │
│ │ self = <buildtest.cli.report.Report object at 0x7f91a3d7a3d0> │            │
│ ╰───────────────────────────────────────────────────────────────╯            │
╰──────────────────────────────────────────────────────────────────────────────╯
BuildTestError: "filter argument 'state' must be 'PASS' or 'FAIL' got value 
UNKNOWN"

The buildspec field expects a valid file path, it can be an absolute or relative path, buildtest will resolve absolute path and check if file exist and is in the report file. If it’s an invalid file we get an error such as

buildtest rt --filter buildspec=/path/to/invalid.yml
$ buildtest rt --filter buildspec=/path/to/invalid.yml
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ bin/buildtest:38 in <module>                                                 │
│                                                                              │
│   35                                                                         │
│   36 if __name__ == "__main__":                                              │
│   37                                                                         │
│ ❱ 38   sys.exit(main())                                                      │
│   39                                                                         │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ buildtest_file = '/home/docs/checkouts/readthedocs.org/user_builds/buil… │ │
│ │           main = <function main at 0x7f041d688ee0>                       │ │
│ │             os = <module 'os' from                                       │ │
│ │                  '/home/docs/.asdf/installs/python/3.8.18/lib/python3.8… │ │
│ │         prefix = '/home/docs/checkouts/readthedocs.org/user_builds/buil… │ │
│ │            sys = <module 'sys' (built-in)>                               │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ buildtest/main.py:301 in main                                                │
│                                                                              │
│   298 │                                                                      │
│   299 │   # buildtest report                                                 │
│   300 │   elif args.subcommands in ["report", "rt"]:                         │
│ ❱ 301 │   │   report_cmd(args, configuration=configuration, report_file=repo │
│   302 │                                                                      │
│   303 │   elif args.subcommands == "path":                                   │
│   304 │   │   path_cmd(                                                      │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │             args = Namespace(color=None, configfile=None, count=None,    │ │
│ │                    debug=False, detailed=False, editor=None, end=None,   │ │
│ │                    fail=False, filter={'buildspec':                      │ │
│ │                    '/path/to/invalid.yml'}, filterfields=False,          │ │
│ │                    format=None, formatfields=False, helpcolor=False,     │ │
│ │                    helpfilter=False, helpformat=False, latest=False,     │ │
│ │                    loglevel='DEBUG', logpath=False, no_color=False,      │ │
│ │                    no_header=False, oldest=False, pager=False,           │ │
│ │                    passed=False, print_log=False, report=None,           │ │
│ │                    report_subcommand=None, row_count=False, start=None,  │ │
│ │                    subcommands='rt', terse=False, view_log=False)        │ │
│ │ buildtest_editor = '/usr/bin/vim'                                        │ │
│ │      config_file = None                                                  │ │
│ │    configuration = <buildtest.config.SiteConfiguration object at         │ │
│ │                    0x7f041b81b820>                                       │ │
│ │           logger = <Logger buildtest (DEBUG)>                            │ │
│ │         no_color = False                                                 │ │
│ │           parser = <buildtest.cli.BuildTestParser object at              │ │
│ │                    0x7f041d72ca60>                                       │ │
│ │      report_file = None                                                  │ │
│ │           system = <buildtest.system.BuildTestSystem object at           │ │
│ │                    0x7f041b81b1f0>                                       │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ buildtest/cli/report.py:849 in report_cmd                                    │
│                                                                              │
│   846 │   │   list_report()                                                  │
│   847 │   │   return                                                         │
│   848 │                                                                      │
│ ❱ 849 │   results = Report(                                                  │
│   850 │   │   configuration=configuration,                                   │
│   851 │   │   filter_args=args.filter,                                       │
│   852 │   │   format_args=args.format,                                       │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │          args = Namespace(color=None, configfile=None, count=None,       │ │
│ │                 debug=False, detailed=False, editor=None, end=None,      │ │
│ │                 fail=False, filter={'buildspec':                         │ │
│ │                 '/path/to/invalid.yml'}, filterfields=False,             │ │
│ │                 format=None, formatfields=False, helpcolor=False,        │ │
│ │                 helpfilter=False, helpformat=False, latest=False,        │ │
│ │                 loglevel='DEBUG', logpath=False, no_color=False,         │ │
│ │                 no_header=False, oldest=False, pager=False,              │ │
│ │                 passed=False, print_log=False, report=None,              │ │
│ │                 report_subcommand=None, row_count=False, start=None,     │ │
│ │                 subcommands='rt', terse=False, view_log=False)           │ │
│ │ configuration = <buildtest.config.SiteConfiguration object at            │ │
│ │                 0x7f041b81b820>                                          │ │
│ │  consoleColor = 'default'                                                │ │
│ │         pager = False                                                    │ │
│ │   report_file = None                                                     │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ buildtest/cli/report.py:158 in __init__                                      │
│                                                                              │
│   155 │   │   self._check_filter_fields()                                    │
│   156 │   │   self._check_format_fields()                                    │
│   157 │   │   self._check_start_and_end_fields()                             │
│ ❱ 158 │   │   self.filter_buildspecs_from_report()                           │
│   159 │   │                                                                  │
│   160 │   │   self.process_report()                                          │
│   161                                                                        │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │           color = None                                                   │ │
│ │   configuration = <buildtest.config.SiteConfiguration object at          │ │
│ │                   0x7f041b81b820>                                        │ │
│ │           count = None                                                   │ │
│ │        detailed = None                                                   │ │
│ │             end = None                                                   │ │
│ │         failure = False                                                  │ │
│ │     filter_args = {'buildspec': '/path/to/invalid.yml'}                  │ │
│ │     format_args = None                                                   │ │
│ │ format_detailed = False                                                  │ │
│ │          latest = False                                                  │ │
│ │          oldest = False                                                  │ │
│ │           pager = None                                                   │ │
│ │          passed = False                                                  │ │
│ │     report_file = None                                                   │ │
│ │            self = <buildtest.cli.report.Report object at 0x7f041b855400> │ │
│ │           start = None                                                   │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ buildtest/cli/report.py:307 in filter_buildspecs_from_report                 │
│                                                                              │
│   304 │   │   │                                                              │
│   305 │   │   │   # if file doesn't exist we terminate with message          │
│   306 │   │   │   if not resolved_buildspecs:                                │
│ ❱ 307 │   │   │   │   raise BuildTestError(                                  │
│   308 │   │   │   │   │   f"Invalid File Path for filter field 'buildspec':  │
│   309 │   │   │   │   )                                                      │
│   310                                                                        │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ resolved_buildspecs = None                                               │ │
│ │                self = <buildtest.cli.report.Report object at             │ │
│ │                       0x7f041b855400>                                    │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
BuildTestError: "Invalid File Path for filter field 'buildspec': 
/path/to/invalid.yml"

You may have a valid filepath for buildspec filter field such as $BUILDTEST_ROOT/tutorials/invalid_executor.yml, but there is no record of a test in the report cache because this test wasn’t run. In this case you will get the following message.

buildtest rt --filter buildspec=$BUILDTEST_ROOT/tutorials/invalid_executor.yml
$ buildtest rt --filter buildspec=$BUILDTEST_ROOT/tutorials/invalid_executor.yml
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ bin/buildtest:38 in <module>                                                 │
│                                                                              │
│   35                                                                         │
│   36 if __name__ == "__main__":                                              │
│   37                                                                         │
│ ❱ 38   sys.exit(main())                                                      │
│   39                                                                         │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ buildtest_file = '/home/docs/checkouts/readthedocs.org/user_builds/buil… │ │
│ │           main = <function main at 0x7f4c92c38ee0>                       │ │
│ │             os = <module 'os' from                                       │ │
│ │                  '/home/docs/.asdf/installs/python/3.8.18/lib/python3.8… │ │
│ │         prefix = '/home/docs/checkouts/readthedocs.org/user_builds/buil… │ │
│ │            sys = <module 'sys' (built-in)>                               │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ buildtest/main.py:301 in main                                                │
│                                                                              │
│   298 │                                                                      │
│   299 │   # buildtest report                                                 │
│   300 │   elif args.subcommands in ["report", "rt"]:                         │
│ ❱ 301 │   │   report_cmd(args, configuration=configuration, report_file=repo │
│   302 │                                                                      │
│   303 │   elif args.subcommands == "path":                                   │
│   304 │   │   path_cmd(                                                      │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │             args = Namespace(color=None, configfile=None, count=None,    │ │
│ │                    debug=False, detailed=False, editor=None, end=None,   │ │
│ │                    fail=False, filter={'buildspec':                      │ │
│ │                    '$BUILDTEST_ROOT/tutorials/invalid_executor.yml'},    │ │
│ │                    filterfields=False, format=None, formatfields=False,  │ │
│ │                    helpcolor=False, helpfilter=False, helpformat=False,  │ │
│ │                    latest=False, loglevel='DEBUG', logpath=False,        │ │
│ │                    no_color=False, no_header=False, oldest=False,        │ │
│ │                    pager=False, passed=False, print_log=False,           │ │
│ │                    report=None, report_subcommand=None, row_count=False, │ │
│ │                    start=None, subcommands='rt', terse=False,            │ │
│ │                    view_log=False)                                       │ │
│ │ buildtest_editor = '/usr/bin/vim'                                        │ │
│ │      config_file = None                                                  │ │
│ │    configuration = <buildtest.config.SiteConfiguration object at         │ │
│ │                    0x7f4c90dcc790>                                       │ │
│ │           logger = <Logger buildtest (DEBUG)>                            │ │
│ │         no_color = False                                                 │ │
│ │           parser = <buildtest.cli.BuildTestParser object at              │ │
│ │                    0x7f4c92cdca60>                                       │ │
│ │      report_file = None                                                  │ │
│ │           system = <buildtest.system.BuildTestSystem object at           │ │
│ │                    0x7f4c90dcc2b0>                                       │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ buildtest/cli/report.py:849 in report_cmd                                    │
│                                                                              │
│   846 │   │   list_report()                                                  │
│   847 │   │   return                                                         │
│   848 │                                                                      │
│ ❱ 849 │   results = Report(                                                  │
│   850 │   │   configuration=configuration,                                   │
│   851 │   │   filter_args=args.filter,                                       │
│   852 │   │   format_args=args.format,                                       │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │          args = Namespace(color=None, configfile=None, count=None,       │ │
│ │                 debug=False, detailed=False, editor=None, end=None,      │ │
│ │                 fail=False, filter={'buildspec':                         │ │
│ │                 '$BUILDTEST_ROOT/tutorials/invalid_executor.yml'},       │ │
│ │                 filterfields=False, format=None, formatfields=False,     │ │
│ │                 helpcolor=False, helpfilter=False, helpformat=False,     │ │
│ │                 latest=False, loglevel='DEBUG', logpath=False,           │ │
│ │                 no_color=False, no_header=False, oldest=False,           │ │
│ │                 pager=False, passed=False, print_log=False, report=None, │ │
│ │                 report_subcommand=None, row_count=False, start=None,     │ │
│ │                 subcommands='rt', terse=False, view_log=False)           │ │
│ │ configuration = <buildtest.config.SiteConfiguration object at            │ │
│ │                 0x7f4c90dcc790>                                          │ │
│ │  consoleColor = 'default'                                                │ │
│ │         pager = False                                                    │ │
│ │   report_file = None                                                     │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ buildtest/cli/report.py:158 in __init__                                      │
│                                                                              │
│   155 │   │   self._check_filter_fields()                                    │
│   156 │   │   self._check_format_fields()                                    │
│   157 │   │   self._check_start_and_end_fields()                             │
│ ❱ 158 │   │   self.filter_buildspecs_from_report()                           │
│   159 │   │                                                                  │
│   160 │   │   self.process_report()                                          │
│   161                                                                        │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │           color = None                                                   │ │
│ │   configuration = <buildtest.config.SiteConfiguration object at          │ │
│ │                   0x7f4c90dcc790>                                        │ │
│ │           count = None                                                   │ │
│ │        detailed = None                                                   │ │
│ │             end = None                                                   │ │
│ │         failure = False                                                  │ │
│ │     filter_args = {                                                      │ │
│ │                   │   'buildspec':                                       │ │
│ │                   '$BUILDTEST_ROOT/tutorials/invalid_executor.yml'       │ │
│ │                   }                                                      │ │
│ │     format_args = None                                                   │ │
│ │ format_detailed = False                                                  │ │
│ │          latest = False                                                  │ │
│ │          oldest = False                                                  │ │
│ │           pager = None                                                   │ │
│ │          passed = False                                                  │ │
│ │     report_file = None                                                   │ │
│ │            self = <buildtest.cli.report.Report object at 0x7f4c90e064c0> │ │
│ │           start = None                                                   │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ buildtest/cli/report.py:313 in filter_buildspecs_from_report                 │
│                                                                              │
│   310 │   │   │                                                              │
│   311 │   │   │   # if file not found in cache we exit                       │
│   312 │   │   │   if not resolved_buildspecs in self.report.keys():          │
│ ❱ 313 │   │   │   │   raise BuildTestError(                                  │
│   314 │   │   │   │   │   f"buildspec file: {resolved_buildspecs} not found  │
│   315 │   │   │   │   )                                                      │
│   316                                                                        │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ resolved_buildspecs = '/home/docs/checkouts/readthedocs.org/user_builds… │ │
│ │                self = <buildtest.cli.report.Report object at             │ │
│ │                       0x7f4c90e064c0>                                    │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
BuildTestError: 'buildspec file: 
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tuto
rials/invalid_executor.yml not found in cache'

Find Latest or Oldest test

We can search for oldest or latest test for any given test. This can be useful if you want to see first or last test run for a particular test. If you want to retrieve the oldest test you can use --oldest option, likewise you can retrieve the latest run via --latest option.

Let’s take a look at this example, we filter by test name exit1_pass which retrieves all test runs. In subsequent example we filter by latest and oldest run.

Show oldest run
$ buildtest report --filter name=exit1_pass --format name,id,starttime
                 Report File: /tmp/tmpjcb9dx2w/var/report.json                  
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                 ┃ id               ┃ starttime                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ exit1_pass           │ 31743897         │ 2024/02/14 17:16:30                │
├──────────────────────┼──────────────────┼────────────────────────────────────┤
│ exit1_pass           │ 9167ac0e         │ 2024/02/14 17:17:45                │
├──────────────────────┼──────────────────┼────────────────────────────────────┤
│ exit1_pass           │ 72516f5e         │ 2024/02/14 17:17:49                │
├──────────────────────┼──────────────────┼────────────────────────────────────┤
│ exit1_pass           │ 1a292a6a         │ 2024/02/14 17:17:50                │
├──────────────────────┼──────────────────┼────────────────────────────────────┤
│ exit1_pass           │ fca13cf8         │ 2024/02/14 17:17:59                │
├──────────────────────┼──────────────────┼────────────────────────────────────┤
│ exit1_pass           │ 70cb6c0f         │ 2024/02/14 17:18:06                │
├──────────────────────┼──────────────────┼────────────────────────────────────┤
│ exit1_pass           │ e35f6895         │ 2024/02/14 17:18:06                │
├──────────────────────┼──────────────────┼────────────────────────────────────┤
│ exit1_pass           │ d4e668ef         │ 2024/02/14 17:18:08                │
├──────────────────────┼──────────────────┼────────────────────────────────────┤
│ exit1_pass           │ d3fb6c56         │ 2024/02/14 17:18:09                │
└──────────────────────┴──────────────────┴────────────────────────────────────┘
$ buildtest report --filter name=exit1_pass --format name,id,starttime --oldest
                 Report File: /tmp/tmpjcb9dx2w/var/report.json                  
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                 ┃ id               ┃ starttime                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ exit1_pass           │ 31743897         │ 2024/02/14 17:16:30                │
└──────────────────────┴──────────────────┴────────────────────────────────────┘
Show latest run
$ buildtest report --filter name=exit1_pass --format name,id,starttime --latest
                 Report File: /tmp/tmpjcb9dx2w/var/report.json                  
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                 ┃ id               ┃ starttime                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ exit1_pass           │ d3fb6c56         │ 2024/02/14 17:18:09                │
└──────────────────────┴──────────────────┴────────────────────────────────────┘

You may combine –oldest and –latest options in same command, in this case buildtest will retrieve the first and last record of every test.

Show oldest and latest run
$ buildtest report --filter name=exit1_pass --format name,id,starttime --oldest --latest
                 Report File: /tmp/tmpjcb9dx2w/var/report.json                  
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                 ┃ id               ┃ starttime                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ exit1_pass           │ 31743897         │ 2024/02/14 17:16:30                │
├──────────────────────┼──────────────────┼────────────────────────────────────┤
│ exit1_pass           │ d3fb6c56         │ 2024/02/14 17:18:09                │
└──────────────────────┴──────────────────┴────────────────────────────────────┘

Find all Failed Tests (buildtest report --fail)

The buildtest report --fail command can be used to retrieve all failed tests which is equivalent to filtering tests by state=FAIL since test state is determined by state property. This command can be useful to pin-point failures.

Let’s take a look at next two example, the first command will query report and limit output to 6 rows which can retrieve both pass and failed test. The second command will run same example with --fail and only retrieve failed tests. Take note of the state property in table, in second example buildtest will filter test and report all FAIL tests.

Show fail tests
$ buildtest report --format name,id,state --count=6
                 Report File: /tmp/tmpjcb9dx2w/var/report.json                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ name                                ┃ id                    ┃ state          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ variables_bash                      │ 17e5a942              │ PASS           │
├─────────────────────────────────────┼───────────────────────┼────────────────┤
│ variables_bash                      │ f0ba0142              │ PASS           │
├─────────────────────────────────────┼───────────────────────┼────────────────┤
│ variables_bash                      │ 14f5700c              │ PASS           │
├─────────────────────────────────────┼───────────────────────┼────────────────┤
│ variables_bash                      │ 141e90a4              │ PASS           │
├─────────────────────────────────────┼───────────────────────┼────────────────┤
│ variables_bash                      │ 71c7fbeb              │ PASS           │
├─────────────────────────────────────┼───────────────────────┼────────────────┤
│ exit1_pass                          │ 31743897              │ PASS           │
└─────────────────────────────────────┴───────────────────────┴────────────────┘
$ buildtest report --format name,id,state --fail --count=6
                 Report File: /tmp/tmpjcb9dx2w/var/report.json                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ name                                         ┃ id               ┃ state      ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ returncode_list_mismatch                     │ 84a02a2a         │ FAIL       │
├──────────────────────────────────────────────┼──────────────────┼────────────┤
│ returncode_list_mismatch                     │ c9a38d26         │ FAIL       │
├──────────────────────────────────────────────┼──────────────────┼────────────┤
│ returncode_list_mismatch                     │ 4b8846a2         │ FAIL       │
├──────────────────────────────────────────────┼──────────────────┼────────────┤
│ returncode_list_mismatch                     │ f5327c7d         │ FAIL       │
├──────────────────────────────────────────────┼──────────────────┼────────────┤
│ returncode_list_mismatch                     │ ee6d33c8         │ FAIL       │
├──────────────────────────────────────────────┼──────────────────┼────────────┤
│ returncode_list_mismatch                     │ a2a4cbfd         │ FAIL       │
└──────────────────────────────────────────────┴──────────────────┴────────────┘

Find all Passed Tests (buildtest report --pass)

The buildtest report --pass, works similar to buildtest report --fail where it will filter test by state=PASS which can be used to find all passed tests.

We can see in example below, that buildtest will retrieve only PASS tests which can be determined by the state property

Show pass tests
$ buildtest report --format name,id,state --pass --count=6
                 Report File: /tmp/tmpjcb9dx2w/var/report.json                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ name                                ┃ id                    ┃ state          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ variables_bash                      │ 17e5a942              │ PASS           │
├─────────────────────────────────────┼───────────────────────┼────────────────┤
│ variables_bash                      │ f0ba0142              │ PASS           │
├─────────────────────────────────────┼───────────────────────┼────────────────┤
│ variables_bash                      │ 14f5700c              │ PASS           │
├─────────────────────────────────────┼───────────────────────┼────────────────┤
│ variables_bash                      │ 141e90a4              │ PASS           │
├─────────────────────────────────────┼───────────────────────┼────────────────┤
│ variables_bash                      │ 71c7fbeb              │ PASS           │
├─────────────────────────────────────┼───────────────────────┼────────────────┤
│ exit1_pass                          │ 31743897              │ PASS           │
└─────────────────────────────────────┴───────────────────────┴────────────────┘

Note

The --pass and --fail are mutually exclusive option which will query all PASS or FAIL test from report file, if you try to specify both options on command line you will get an error

$ buildtest report --pass --fail
usage: buildtest [options] [COMMANDS] report [-h] [--pager] [--row-count]
                                             [--terse] [-n] [-c COUNT]
                                             [--filter FILTER] [--helpfilter]
                                             [--filterfields] [--helpformat]
                                             [--formatfields] [--latest]
                                             [--oldest] [-s START] [-e END]
                                             [-f | -p] [--format FORMAT | -d]
                                             ...
buildtest [options] [COMMANDS] report: error: argument -f/--fail: not allowed with argument -p/--pass

Find Tests by Start and End Date (buildtest report --start --end)

The buildtest report --start and buildtest report --end command can be used to retrieve test records based on start and end date.

Let’s take a look at these two example, the first test queries report by filtering by state and name. The second command will run same example with --start --end option. Take note of the starttime and endtime properties in table, in second example buildtest will filter test and only report tests in the range of [start, end] dates.

Filter by start and end date
$ buildtest report --filter state=FAIL,name=exit1_fail --format name,state,starttime,endtime
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkout
                             s/1082/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                                                   ┃ state        ┃ starttime                                ┃ endtime                                 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ exit1_fail                                             │ FAIL         │ 2022/06/09 17:51:50                      │ 2022/06/09 17:51:50                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/09 17:51:51                      │ 2022/06/09 17:51:51                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/09 17:51:53                      │ 2022/06/09 17:51:53                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/09 17:52:01                      │ 2022/06/09 17:52:01                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/10 17:51:50                      │ 2022/06/10 17:51:50                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/10 17:51:52                      │ 2022/06/10 17:51:52                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/10 17:51:52                      │ 2022/06/10 17:51:52                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/10 17:52:01                      │ 2022/06/10 17:52:01                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/11 17:51:57                      │ 2022/06/11 17:51:57                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/11 17:51:59                      │ 2022/06/11 17:51:59                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/11 17:51:57                      │ 2022/06/11 17:51:57                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/11 17:51:59                      │ 2022/06/11 17:51:59                     │
└────────────────────────────────────────────────────────┴──────────────┴──────────────────────────────────────────┴─────────────────────────────────────────┘
$ buildtest report --filter state=FAIL,name=exit1_fail --format name,state,starttime,endtime --start 2022-06-10 --end 2022-06-11
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkout
                             s/1082/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                                                   ┃ state        ┃ starttime                                ┃ endtime                                 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ exit1_fail                                             │ FAIL         │ 2022/06/10 17:51:50                      │ 2022/06/10 17:51:50                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/10 17:51:52                      │ 2022/06/10 17:51:52                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/10 17:51:52                      │ 2022/06/10 17:51:52                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/10 17:52:01                      │ 2022/06/10 17:52:01                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/11 17:51:57                      │ 2022/06/11 17:51:57                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/11 17:51:59                      │ 2022/06/11 17:51:59                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/11 17:51:57                      │ 2022/06/11 17:51:57                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/11 17:51:59                      │ 2022/06/11 17:51:59                     │
└────────────────────────────────────────────────────────┴──────────────┴──────────────────────────────────────────┴─────────────────────────────────────────┘

Limit number of test records (buildtest report --count)

The buildtest report command will show all test results from the report file and the output can be quite long. If you want to limit the number of records that get printed then you can use buildtest report --count where --count is number of rows that that get printed. Shown below is the output of 2 records in table format

buildtest report --count 2
$ buildtest report --count 2
                 Report File: /tmp/tmpjcb9dx2w/var/report.json                  
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ name                 ┃ id           ┃ state    ┃ runtime     ┃ returncode    ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ variables_bash       │ 17e5a942     │ PASS     │ 0.009769    │ 0             │
├──────────────────────┼──────────────┼──────────┼─────────────┼───────────────┤
│ variables_bash       │ f0ba0142     │ PASS     │ 0.009573    │ 0             │
└──────────────────────┴──────────────┴──────────┴─────────────┴───────────────┘

The --count option also works with terse mode --terse, shown below is the same output

buildtest report --terse --count 2
$ buildtest report --terse --count 2
name|id|state|runtime|returncode
variables_bash|17e5a942|PASS|0.009769|0
variables_bash|f0ba0142|PASS|0.009573|0

If you set --count=0 no records will be displayed; however, if you set any negative value for --count then all records will be displayed.

Query tests with --count at 0 and negative value
$ buildtest report --count=0 --no-header --terse | wc -l
25
$ buildtest report --count=-5 --no-header --terse | wc -l
133

Terse Output (buildtest report --terse)

If you would like to parse the result of buildtest report, you can use the --terse or -t option which will print the report in machine readable format that shows the name of each column followed by each entry. Each entry is delimited by PIPE symbol (|). The --terse option works with --format and --filter option. In this next example, we report all FAIL tests in terse output. The first line is the header of tables followed by output, if you want to disable output of header you can use --no-header option.

buildtest report --filter state=FAIL --format=name,id,state --terse
$ buildtest report --filter state=FAIL --format=name,id,state --terse
name|id|state
returncode_list_mismatch|84a02a2a|FAIL
returncode_list_mismatch|c9a38d26|FAIL
returncode_list_mismatch|4b8846a2|FAIL
returncode_list_mismatch|f5327c7d|FAIL
returncode_list_mismatch|ee6d33c8|FAIL
returncode_list_mismatch|a2a4cbfd|FAIL
returncode_list_mismatch|6187c299|FAIL
returncode_list_mismatch|b70d4e27|FAIL
exit1_fail|e1b85944|FAIL
exit1_fail|97aad2b9|FAIL
exit1_fail|f0f90e72|FAIL
exit1_fail|85ea05ea|FAIL
exit1_fail|d30ac0cc|FAIL
exit1_fail|80e27e9b|FAIL
exit1_fail|c9eb7edd|FAIL
exit1_fail|c3792104|FAIL
exit1_fail|e7b50826|FAIL
status_regex_stdout_fail|f7bc7c42|FAIL
status_regex_stderr_fail|ab079071|FAIL
regex_on_directory_not_supported|a36f76e0|FAIL
timelimit_max_fail|70ed445f|FAIL
timelimit_min_fail|fbd61e45|FAIL
always_fail|feab4fcf|FAIL
test_fail_returncode_match|54cadb12|FAIL
status_exists_failure|ceab01b7|FAIL

Report Summary (buildtest report summary)

The buildtest report summary command can be used to provide a short summary of the test report.

Shown below is an example output from the report summary.

buildtest report summary
$ buildtest report summary
Report File:  /tmp/tmpjcb9dx2w/var/report.json
Total Tests: 133
Total Tests by Names:  57
Number of buildspecs in report:  29

The buildtest report summary --detailed command can be used to provide a summary of the test report with breakdown statistics of tests including all fail tests, number of tests by name, test runs and buildspecs in report file. To see a short report use

Shown below is an example output from the report summary –detailed.

buildtest report summary --detailed
$ buildtest report summary --detailed
Report File:  /tmp/tmpjcb9dx2w/var/report.json
Total Tests: 133
Total Tests by Names:  57
Number of buildspecs in report:  29
                             Breakdown by test                             
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Name                             ┃ Total Pass ┃ Total Fail ┃ Total Runs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ variables_bash                   │ 5          │ 0          │ 5          │
│ exit1_pass                       │ 9          │ 0          │ 9          │
│ returncode_list_mismatch         │ 0          │ 8          │ 8          │
│ returncode_int_match             │ 8          │ 0          │ 8          │
│ exit1_fail                       │ 0          │ 9          │ 9          │
│ status_regex_stdout_fail         │ 0          │ 1          │ 1          │
│ status_regex_stderr_fail         │ 0          │ 1          │ 1          │
│ status_regex_stdout_pass         │ 1          │ 0          │ 1          │
│ status_regex_stderr_pass         │ 1          │ 0          │ 1          │
│ file_expansion_supported         │ 1          │ 0          │ 1          │
│ regex_on_multiple_files          │ 1          │ 0          │ 1          │
│ regex_on_directory_not_supported │ 0          │ 1          │ 1          │
│ timelimit_max                    │ 1          │ 0          │ 1          │
│ timelimit_max_fail               │ 0          │ 1          │ 1          │
│ timelimit_min                    │ 1          │ 0          │ 1          │
│ timelimit_min_max                │ 1          │ 0          │ 1          │
│ timelimit_min_fail               │ 0          │ 1          │ 1          │
│ always_pass                      │ 1          │ 0          │ 1          │
│ always_fail                      │ 0          │ 1          │ 1          │
│ test_pass_returncode_mismatch    │ 1          │ 0          │ 1          │
│ test_fail_returncode_match       │ 0          │ 1          │ 1          │
│ status_exists                    │ 1          │ 0          │ 1          │
│ status_exists_failure            │ 0          │ 1          │ 1          │
│ file_exists_pass                 │ 1          │ 0          │ 1          │
│ file_and_dir_checks              │ 0          │ 1          │ 1          │
│ combined_file_and_dir_checks     │ 1          │ 0          │ 1          │
│ symlink_test                     │ 1          │ 0          │ 1          │
│ file_count_by_extension          │ 1          │ 0          │ 1          │
│ file_count_on_directory          │ 1          │ 0          │ 1          │
│ file_extension_and_filepattern   │ 1          │ 0          │ 1          │
│ file_count_by_expression         │ 1          │ 0          │ 1          │
│ file_count_by_filetype           │ 0          │ 1          │ 1          │
│ file_traverse_limit              │ 0          │ 1          │ 1          │
│ status_logical_or                │ 1          │ 0          │ 1          │
│ status_logical_and               │ 1          │ 0          │ 1          │
│ unskipped                        │ 1          │ 0          │ 1          │
│ hello_world                      │ 9          │ 0          │ 9          │
│ ulimit_stacksize_unlimited       │ 0          │ 3          │ 3          │
│ ulimit_filedescriptor_4096       │ 0          │ 3          │ 3          │
│ ulimit_max_user_process_2048     │ 0          │ 3          │ 3          │
│ ulimit_cputime_unlimited         │ 3          │ 0          │ 3          │
│ ulimit_vmsize_unlimited          │ 3          │ 0          │ 3          │
│ ulimit_filelock_unlimited        │ 3          │ 0          │ 3          │
│ kernel_swapusage                 │ 0          │ 4          │ 4          │
│ root_disk_usage                  │ 4          │ 0          │ 4          │
│ systemd_default_target           │ 0          │ 4          │ 4          │
│ string_tag                       │ 3          │ 0          │ 3          │
│ list_of_strings_tags             │ 3          │ 0          │ 3          │
│ python_hello                     │ 4          │ 0          │ 4          │
│ circle_area                      │ 6          │ 0          │ 6          │
│ add_numbers                      │ 1          │ 0          │ 1          │
│ summary_example                  │ 1          │ 0          │ 1          │
│ tcsh_env_declaration             │ 1          │ 0          │ 1          │
│ csh_env_declaration              │ 1          │ 0          │ 1          │
│ bash_env_variables               │ 1          │ 0          │ 1          │
│ csh_shell                        │ 1          │ 0          │ 1          │
│ sleep                            │ 1          │ 1          │ 2          │
└──────────────────────────────────┴────────────┴────────────┴────────────┘
                                   PASS Tests                                   
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ name            ┃ id       ┃ executor        ┃ state ┃ returncode ┃ runtime  ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ variables_bash  │ 17e5a942 │ generic.local.b │ PASS  │ 0          │ 0.009769 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ variables_bash  │ f0ba0142 │ generic.local.b │ PASS  │ 0          │ 0.009573 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ variables_bash  │ 14f5700c │ generic.local.b │ PASS  │ 0          │ 0.009655 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ variables_bash  │ 141e90a4 │ generic.local.b │ PASS  │ 0          │ 0.009529 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ variables_bash  │ 71c7fbeb │ generic.local.b │ PASS  │ 0          │ 0.010044 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_pass      │ 31743897 │ generic.local.b │ PASS  │ 1          │ 0.014586 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_pass      │ 9167ac0e │ generic.local.b │ PASS  │ 1          │ 0.014208 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_pass      │ 72516f5e │ generic.local.b │ PASS  │ 1          │ 0.014722 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_pass      │ 1a292a6a │ generic.local.b │ PASS  │ 1          │ 0.014281 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_pass      │ fca13cf8 │ generic.local.b │ PASS  │ 1          │ 0.014001 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_pass      │ 70cb6c0f │ generic.local.b │ PASS  │ 1          │ 0.01468  │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_pass      │ e35f6895 │ generic.local.b │ PASS  │ 1          │ 0.014048 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_pass      │ d4e668ef │ generic.local.b │ PASS  │ 1          │ 0.014818 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_pass      │ d3fb6c56 │ generic.local.b │ PASS  │ 1          │ 0.014185 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_int_ │ fd42032f │ generic.local.b │ PASS  │ 128        │ 0.0145   │
│ match           │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_int_ │ 97233a17 │ generic.local.b │ PASS  │ 128        │ 0.014162 │
│ match           │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_int_ │ c23115dd │ generic.local.b │ PASS  │ 128        │ 0.014254 │
│ match           │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_int_ │ fd90fa34 │ generic.local.b │ PASS  │ 128        │ 0.014321 │
│ match           │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_int_ │ 7ff997f0 │ generic.local.b │ PASS  │ 128        │ 0.014951 │
│ match           │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_int_ │ 46895dd6 │ generic.local.b │ PASS  │ 128        │ 0.01434  │
│ match           │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_int_ │ e94b7475 │ generic.local.b │ PASS  │ 128        │ 0.014145 │
│ match           │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_int_ │ 879b33f7 │ generic.local.b │ PASS  │ 128        │ 0.014265 │
│ match           │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ status_regex_st │ a2573acc │ generic.local.b │ PASS  │ 0          │ 0.005598 │
│ dout_pass       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ status_regex_st │ 8bc6a3a2 │ generic.local.b │ PASS  │ 0          │ 0.005536 │
│ derr_pass       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ file_expansion_ │ d611b296 │ generic.local.b │ PASS  │ 0          │ 0.005944 │
│ supported       │          │ ash             │       │            │          │
└─────────────────┴──────────┴─────────────────┴───────┴────────────┴──────────┘
                                   FAIL Tests                                   
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ name            ┃ id       ┃ executor        ┃ state ┃ returncode ┃ runtime  ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ returncode_list │ 84a02a2a │ generic.local.b │ FAIL  │ 2          │ 0.015189 │
│ _mismatch       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_list │ c9a38d26 │ generic.local.b │ FAIL  │ 2          │ 0.01435  │
│ _mismatch       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_list │ 4b8846a2 │ generic.local.b │ FAIL  │ 2          │ 0.014249 │
│ _mismatch       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_list │ f5327c7d │ generic.local.b │ FAIL  │ 2          │ 0.014471 │
│ _mismatch       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_list │ ee6d33c8 │ generic.local.b │ FAIL  │ 2          │ 0.014902 │
│ _mismatch       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_list │ a2a4cbfd │ generic.local.b │ FAIL  │ 2          │ 0.014175 │
│ _mismatch       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_list │ 6187c299 │ generic.local.b │ FAIL  │ 2          │ 0.016105 │
│ _mismatch       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_list │ b70d4e27 │ generic.local.b │ FAIL  │ 2          │ 0.01453  │
│ _mismatch       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_fail      │ e1b85944 │ generic.local.b │ FAIL  │ 1          │ 0.014695 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_fail      │ 97aad2b9 │ generic.local.b │ FAIL  │ 1          │ 0.014328 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_fail      │ f0f90e72 │ generic.local.b │ FAIL  │ 1          │ 0.014044 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_fail      │ 85ea05ea │ generic.local.b │ FAIL  │ 1          │ 0.014086 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_fail      │ d30ac0cc │ generic.local.b │ FAIL  │ 1          │ 0.013955 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_fail      │ 80e27e9b │ generic.local.b │ FAIL  │ 1          │ 0.014956 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_fail      │ c9eb7edd │ generic.local.b │ FAIL  │ 1          │ 0.014039 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_fail      │ c3792104 │ generic.local.b │ FAIL  │ 1          │ 0.01402  │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_fail      │ e7b50826 │ generic.local.b │ FAIL  │ 1          │ 0.014746 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ status_regex_st │ f7bc7c42 │ generic.local.b │ FAIL  │ 0          │ 0.005988 │
│ dout_fail       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ status_regex_st │ ab079071 │ generic.local.b │ FAIL  │ 0          │ 0.005836 │
│ derr_fail       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ regex_on_direct │ a36f76e0 │ generic.local.b │ FAIL  │ 0          │ 0.006741 │
│ ory_not_support │          │ ash             │       │            │          │
│ ed              │          │                 │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ timelimit_max_f │ 70ed445f │ generic.local.s │ FAIL  │ 0          │ 3.006201 │
│ ail             │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ timelimit_min_f │ fbd61e45 │ generic.local.s │ FAIL  │ 0          │ 2.006525 │
│ ail             │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ always_fail     │ feab4fcf │ generic.local.s │ FAIL  │ 0          │ 0.004562 │
│                 │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ test_fail_retur │ 54cadb12 │ generic.local.s │ FAIL  │ 1          │ 0.012792 │
│ ncode_match     │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ status_exists_f │ ceab01b7 │ generic.local.b │ FAIL  │ 0          │ 0.006405 │
│ ailure          │          │ ash             │       │            │          │
└─────────────────┴──────────┴─────────────────┴───────┴────────────┴──────────┘

Inspect Tests Records via buildtest inspect

Note

buildtest it is an alias for buildtest inspect command.

In previous examples we saw how we can retrieve test records using buildtest report which is printed in table format. We have limited the output to a limited fields however, if you want to analyze a particular, we have a separate command called buildtest inspect that can be used for inspecting a test record based on name or id. Shown below is the command usage for buildtest inspect command.

buildtest inspect --help
$ buildtest inspect --help
usage: buildtest [options] [COMMANDS] inspect [-h]  ...

optional arguments:
  -h, --help     show this help message and exit

subcommands:
  Inspect Test result based on Test ID or Test Name

  
    buildspec (b)
                 Inspect a test based on buildspec
    name (n)     Specify name of test
    query (q)    Query fields from record
    list (ls)    List all test names, ids, and corresponding buildspecs

You can report all test names and corresponding ids using buildtest inspect list which will be used for querying tests by name or id.

buildtest inspect list
$ buildtest inspect list
                      Test Summary by id, name, buildspec                       
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ id                      ┃ name                     ┃ buildspec               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 17e5a942-9efe-478e-8c89 │ variables_bash           │ /home/docs/checkouts/re │
│ -9bc6e35ea7ec           │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ stable/tutorials/vars.y │
│                         │                          │ ml                      │
│ f0ba0142-0625-4e96-9378 │ variables_bash           │ /home/docs/checkouts/re │
│ -3d0a24d410ac           │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ stable/tutorials/vars.y │
│                         │                          │ ml                      │
│ 14f5700c-7f50-4852-975b │ variables_bash           │ /home/docs/checkouts/re │
│ -f35e514e853f           │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ stable/tutorials/vars.y │
│                         │                          │ ml                      │
│ 141e90a4-123f-4bd9-af29 │ variables_bash           │ /home/docs/checkouts/re │
...

You can fetch all builder names via buildtest inspect list --builder which is the format used for querying test records via buildtest inspect name or buildtest inspect query.

buildtest inspect list --builder
$ buildtest inspect list --builder
variables_bash/17e5a942-9efe-478e-8c89-9bc6e35ea7ec
variables_bash/f0ba0142-0625-4e96-9378-3d0a24d410ac
variables_bash/14f5700c-7f50-4852-975b-f35e514e853f
variables_bash/141e90a4-123f-4bd9-af29-61fb7fcdd1f1
variables_bash/71c7fbeb-3604-4073-be82-c44e7acd4c68
...

If you are interested in parsing output of buildtest inspect list, you can may find the --terse option useful. The output will show headers followed by entries, the headers can be omitted by specifying --no-header option.

buildtest inspect list --terse
$ buildtest inspect list --terse
id|name|buildspec
17e5a942-9efe-478e-8c89-9bc6e35ea7ec|variables_bash|/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/vars.yml
f0ba0142-0625-4e96-9378-3d0a24d410ac|variables_bash|/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/vars.yml
14f5700c-7f50-4852-975b-f35e514e853f|variables_bash|/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/vars.yml
141e90a4-123f-4bd9-af29-61fb7fcdd1f1|variables_bash|/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/vars.yml
...

Inspecting Test by Name via buildtest inspect name

The buildtest inspect name expects a list of positional argument that correspond to name of test you want to query and buildtest will fetch the last record for each named test. Let’s see an example to illustrate the point. We can see that each test is stored as a JSON format and buildtest keeps track of metadata for each test such as user, hostname, command, path to output and error file, content of test, state of test, returncode, etc… In this example, we will retrieve record for test name circle_area which will print the raw content of the test in JSON format.

buildtest inspect name circle_area
$ buildtest inspect name circle_area
We have detected 1 builders with the following names ['circle_area/813e3c6d-a62c-4b7d-b67b-86a0a39a831d']
{
    'circle_area': [
        {
            '813e3c6d-a62c-4b7d-b67b-86a0a39a831d': {
                'id': '813e3c6d',
                'full_id': '813e3c6d-a62c-4b7d-b67b-86a0a39a831d',
                'description': 'Calculate circle of area given a radius',
                'summary': '',
                'schemafile': 'script.schema.json',
                'executor': 'generic.local.bash',
                'compiler': None,
                'hostname': 'build-23444606-project-280831-buildtest',
                'user': 'docs',
                'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d',
                'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/circle_area.sh',
                'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/stage',
                'command': 'bash circle_area_build.sh',
                'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/circle_area.out',
                'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/circle_area.err',
                'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/build-env.txt',
                'buildspec_content': 'buildspecs:\n  circle_area:\n    executor: generic.local.bash\n    type: script\n    shell: python\n    description: "Calculate circle of area given a radius"\n    tags: [tutorials, python]\n    run: |\n      import math\n      radius = 2\n      area = math.pi * radius * radius\n      print("Circle Radius ", radius)\n      print("Area of circle ", area)\n',
                'test_content': '#!/bin/bash\n\npython /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/stage/circle_area.py',
                'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=circle_area\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\n/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/stage/circle_area.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
                'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_g9k3143t.log',
                'metrics': {},
                'check': {
                    'regex': None,
                    'returncode': None,
                    'runtime': None,
                    'file_regex': None,
                    'slurm_job_state': None,
                    'pbs_job_state': None,
                    'lsf_job_state': None,
                    'assert_ge': None,
                    'assert_gt': None,
                    'assert_le': None,
                    'assert_lt': None,
                    'assert_eq': None,
                    'assert_ne': None,
                    'contains': None,
                    'not_contains': None,
                    'is_symlink': None,
                    'exists': None,
                    'is_dir': None,
                    'is_file': None,
                    'file_count': None
                },
                'tags': 'tutorials python',
                'starttime': '2024/02/14 17:18:18',
                'endtime': '2024/02/14 17:18:18',
                'runtime': '0.031635',
                'state': 'PASS',
                'returncode': '0',
                'output': 'Circle Radius  2\nArea of circle  12.566370614359172\n',
                'error': '',
                'job': {},
                'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/circle_area_build.sh'
            }
        }
    ]
}

You can query multiple tests as positional arguments in the format: buildtest inspect name <test1> <test2> In this next example, we will retrieve test records for bash_shell and python_hello.

buildtest inspect name bash_shell python_hello
$ buildtest inspect name bash_shell python_hello
We have detected 1 builders with the following names ['python_hello/c1ae4e33-2c32-4d7a-831e-96251fdf9206']
{
    'python_hello': [
        {
            'c1ae4e33-2c32-4d7a-831e-96251fdf9206': {
                'id': 'c1ae4e33',
                'full_id': 'c1ae4e33-2c32-4d7a-831e-96251fdf9206',
                'description': 'Hello World python',
                'summary': '',
                'schemafile': 'script.schema.json',
                'executor': 'generic.local.bash',
                'compiler': None,
                'hostname': 'build-23444606-project-280831-buildtest',
                'user': 'docs',
                'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-hello/python_hello/c1ae4e33',
                'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-hello/python_hello/c1ae4e33/python_hello.sh',
                'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-hello/python_hello/c1ae4e33/stage',
                'command': 'bash python_hello_build.sh',
                'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-hello/python_hello/c1ae4e33/python_hello.out',
                'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-hello/python_hello/c1ae4e33/python_hello.err',
                'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-hello/python_hello/c1ae4e33/build-env.txt',
                'buildspec_content': 'buildspecs:\n  python_hello:\n    type: script\n    description: Hello World python\n    executor: generic.local.bash\n    tags: python\n    run: python hello.py\n\n',
                'test_content': '#!/usr/bin/bash \nset -eo pipefail\n# Content of run section\npython hello.py',
                'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=python_hello\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-hello/python_hello/c1ae4e33\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-hello/python_hello/c1ae4e33/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\nbash  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-hello/python_hello/c1ae4e33/stage/python_hello.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
                'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_g9k3143t.log',
                'metrics': {},
                'check': {
                    'regex': None,
                    'returncode': None,
                    'runtime': None,
                    'file_regex': None,
                    'slurm_job_state': None,
                    'pbs_job_state': None,
                    'lsf_job_state': None,
                    'assert_ge': None,
                    'assert_gt': None,
                    'assert_le': None,
                    'assert_lt': None,
                    'assert_eq': None,
                    'assert_ne': None,
                    'contains': None,
                    'not_contains': None,
                    'is_symlink': None,
                    'exists': None,
                    'is_dir': None,
                    'is_file': None,
                    'file_count': None
                },
                'tags': 'python',
                'starttime': '2024/02/14 17:18:18',
                'endtime': '2024/02/14 17:18:18',
                'runtime': '0.029861',
                'state': 'PASS',
                'returncode': '0',
                'output': 'Hello World\n',
                'error': '',
                'job': {},
                'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-hello/python_hello/c1ae4e33/python_hello_build.sh'
            }
        }
    ]
}

If you want to query all test records for a given name you can use the --all option which is applied to all positional arguments.

Inspect Test by buildspec via buildtest inspect buildspec

buildtest can fetch records based on buildspec via buildtest inspect buildspec which expects a list of buildspecs. By default, buildtest will fetch the latest record of each test, but if you want to fetch all records you can pass the --all option.

In example below we will fetch latest record for all tests in tutorials/vars.yml

buildtest it buildspec tutorials/vars.yml
$ buildtest it buildspec tutorials/vars.yml
{
│   '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/vars.yml': {
│   │   'variables_bash': {
│   │   │   'id': '71c7fbeb',
│   │   │   'full_id': '71c7fbeb-3604-4073-be82-c44e7acd4c68',
│   │   │   'description': 'Declare shell variables in bash',
│   │   │   'summary': '',
│   │   │   'schemafile': 'script.schema.json',
│   │   │   'executor': 'generic.local.bash',
│   │   │   'compiler': None,
│   │   │   'hostname': 'build-23444606-project-280831-buildtest',
│   │   │   'user': 'docs',
│   │   │   'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb',
│   │   │   'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb/variables_bash.sh',
│   │   │   'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb/stage',
│   │   │   'command': 'bash variables_bash_build.sh',
│   │   │   'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb/variables_bash.out',
│   │   │   'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb/variables_bash.err',
│   │   │   'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb/build-env.txt',
│   │   │   'buildspec_content': 'buildspecs:\n  variables_bash:\n    type: script\n    executor: generic.local.bash\n    description: Declare shell variables in bash\n    tags: [tutorials]\n    vars:\n      X: 1\n      Y: 2\n      literalstring: this is a literal string\n      singlequote: \\\'singlequote\\\'\n      doublequote: \\"doublequote\\"\n      current_user: "$(whoami)"\n      num_files: "`find $HOME -type f -maxdepth 1 | wc -l`"\n      multiline_string: |\n        Hello my name is Bob \\n\n        I am 30 years old\n\n\n    run: |\n      echo "$X+$Y="$(($X+$Y))\n      echo $literalstring\n      echo $singlequote\n      echo $doublequote\n      echo "current user:" $current_user\n      echo "number of files:" $num_files\n      echo -e $multiline_string\n',
│   │   │   'test_content': '#!/usr/bin/bash \nset -eo pipefail\nX="1"\nY="2"\nliteralstring="this is a literal string"\nsinglequote="\\\'singlequote\\\'"\ndoublequote="\\"doublequote\\""\ncurrent_user="$(whoami)"\nnum_files="`find $HOME -type f -maxdepth 1 | wc -l`"\nmultiline_string="Hello my name is Bob \\n\nI am 30 years old\n"\n# Content of run section\necho "$X+$Y="$(($X+$Y))\necho $literalstring\necho $singlequote\necho $doublequote\necho "current user:" $current_user\necho "number of files:" $num_files\necho -e $multiline_string\n',
│   │   │   'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=variables_bash\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\nbash  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb/stage/variables_bash.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_697b5uis.log',
│   │   │   'metrics': {},
│   │   │   'check': {
│   │   │   │   'regex': None,
│   │   │   │   'returncode': None,
│   │   │   │   'runtime': None,
│   │   │   │   'file_regex': None,
│   │   │   │   'slurm_job_state': None,
│   │   │   │   'pbs_job_state': None,
│   │   │   │   'lsf_job_state': None,
│   │   │   │   'assert_ge': None,
│   │   │   │   'assert_gt': None,
│   │   │   │   'assert_le': None,
│   │   │   │   'assert_lt': None,
│   │   │   │   'assert_eq': None,
│   │   │   │   'assert_ne': None,
│   │   │   │   'contains': None,
│   │   │   │   'not_contains': None,
│   │   │   │   'is_symlink': None,
│   │   │   │   'exists': None,
│   │   │   │   'is_dir': None,
│   │   │   │   'is_file': None,
│   │   │   │   'file_count': None
│   │   │   },
│   │   │   'tags': 'tutorials',
│   │   │   'starttime': '2024/02/14 17:18:12',
│   │   │   'endtime': '2024/02/14 17:18:12',
│   │   │   'runtime': '0.010044',
│   │   │   'state': 'PASS',
│   │   │   'returncode': '0',
│   │   │   'output': '1+2=3\nthis is a literal string\n\\\'singlequote\\\'\n"doublequote"\ncurrent user: docs\nnumber of files: 5\nHello my name is Bob \n I am 30 years old\n',
│   │   │   'error': '',
│   │   │   'job': {},
│   │   │   'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb/variables_bash_build.sh'
│   │   }
│   }
}

buildtest will report an error if an input buildspec is invalid filepath such as one below

$ buildtest it buildspec /tmp/buildspec.yml
buildspec: /tmp/buildspec.yml is not valid file
There are no buildspecs in cache based on input buildspecs: ['/tmp/buildspec.yml']

You can also pass multiple buildspes on the command line and fetch all records for a test. In example below we will fetch all records from buildspecs tutorials/vars.yml and tutorials/hello_world.yml

$ buildtest it buildspec --all tutorials/vars.yml tutorials/hello_world.yml
{
│   '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/vars.yml': {
│   │   'variables_bash': [
│   │   │   {
│   │   │   │   'id': '17e5a942',
│   │   │   │   'full_id': '17e5a942-9efe-478e-8c89-9bc6e35ea7ec',
│   │   │   │   'description': 'Declare shell variables in bash',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-23444606-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942',
│   │   │   │   'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942/variables_bash.sh',
│   │   │   │   'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942/stage',
│   │   │   │   'command': 'bash variables_bash_build.sh',
│   │   │   │   'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942/variables_bash.out',
│   │   │   │   'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942/variables_bash.err',
│   │   │   │   'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  variables_bash:\n    type: script\n    executor: generic.local.bash\n    description: Declare shell variables in bash\n    tags: [tutorials]\n    vars:\n      X: 1\n      Y: 2\n      literalstring: this is a literal string\n      singlequote: \\\'singlequote\\\'\n      doublequote: \\"doublequote\\"\n      current_user: "$(whoami)"\n      num_files: "`find $HOME -type f -maxdepth 1 | wc -l`"\n      multiline_string: |\n        Hello my name is Bob \\n\n        I am 30 years old\n\n\n    run: |\n      echo "$X+$Y="$(($X+$Y))\n      echo $literalstring\n      echo $singlequote\n      echo $doublequote\n      echo "current user:" $current_user\n      echo "number of files:" $num_files\n      echo -e $multiline_string\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nset -eo pipefail\nX="1"\nY="2"\nliteralstring="this is a literal string"\nsinglequote="\\\'singlequote\\\'"\ndoublequote="\\"doublequote\\""\ncurrent_user="$(whoami)"\nnum_files="`find $HOME -type f -maxdepth 1 | wc -l`"\nmultiline_string="Hello my name is Bob \\n\nI am 30 years old\n"\n# Content of run section\necho "$X+$Y="$(($X+$Y))\necho $literalstring\necho $singlequote\necho $doublequote\necho "current user:" $current_user\necho "number of files:" $num_files\necho -e $multiline_string\n',
│   │   │   │   'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=variables_bash\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\nbash  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942/stage/variables_bash.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_uaulb51y.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'regex': None,
│   │   │   │   │   'returncode': None,
│   │   │   │   │   'runtime': None,
│   │   │   │   │   'file_regex': None,
│   │   │   │   │   'slurm_job_state': None,
│   │   │   │   │   'pbs_job_state': None,
│   │   │   │   │   'lsf_job_state': None,
│   │   │   │   │   'assert_ge': None,
│   │   │   │   │   'assert_gt': None,
│   │   │   │   │   'assert_le': None,
│   │   │   │   │   'assert_lt': None,
│   │   │   │   │   'assert_eq': None,
│   │   │   │   │   'assert_ne': None,
│   │   │   │   │   'contains': None,
│   │   │   │   │   'not_contains': None,
│   │   │   │   │   'is_symlink': None,
│   │   │   │   │   'exists': None,
│   │   │   │   │   'is_dir': None,
│   │   │   │   │   'is_file': None,
│   │   │   │   │   'file_count': None
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2024/02/14 17:16:28',
│   │   │   │   'endtime': '2024/02/14 17:16:28',
│   │   │   │   'runtime': '0.009769',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': '1+2=3\nthis is a literal string\n\\\'singlequote\\\'\n"doublequote"\ncurrent user: docs\nnumber of files: 4\nHello my name is Bob \n I am 30 years old\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942/variables_bash_build.sh'
│   │   │   },
│   │   │   {
│   │   │   │   'id': 'f0ba0142',
│   │   │   │   'full_id': 'f0ba0142-0625-4e96-9378-3d0a24d410ac',
│   │   │   │   'description': 'Declare shell variables in bash',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-23444606-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/f0ba0142',
│   │   │   │   'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/f0ba0142/variables_bash.sh',
│   │   │   │   'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/f0ba0142/stage',
│   │   │   │   'command': 'bash variables_bash_build.sh',
│   │   │   │   'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/f0ba0142/variables_bash.out',
│   │   │   │   'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/f0ba0142/variables_bash.err',
│   │   │   │   'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/f0ba0142/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  variables_bash:\n    type: script\n    executor: generic.local.bash\n    description: Declare shell variables in bash\n    tags: [tutorials]\n    vars:\n      X: 1\n      Y: 2\n      literalstring: this is a literal string\n      singlequote: \\\'singlequote\\\'\n      doublequote: \\"doublequote\\"\n      current_user: "$(whoami)"\n      num_files: "`find $HOME -type f -maxdepth 1 | wc -l`"\n      multiline_string: |\n        Hello my name is Bob \\n\n        I am 30 years old\n\n\n    run: |\n      echo "$X+$Y="$(($X+$Y))\n      echo $literalstring\n      echo $singlequote\n      echo $doublequote\n      echo "current user:" $current_user\n      echo "number of files:" $num_files\n      echo -e $multiline_string\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nset -eo pipefail\nX="1"\nY="2"\nliteralstring="this is a literal string"\nsinglequote="\\\'singlequote\\\'"\ndoublequote="\\"doublequote\\""\ncurrent_user="$(whoami)"\nnum_files="`find $HOME -type f -maxdepth 1 | wc -l`"\nmultiline_string="Hello my name is Bob \\n\nI am 30 years old\n"\n# Content of run section\necho "$X+$Y="$(($X+$Y))\necho $literalstring\necho $singlequote\necho $doublequote\necho "current user:" $current_user\necho "number of files:" $num_files\necho -e $multiline_string\n',
│   │   │   │   'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=variables_bash\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/f0ba0142\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/f0ba0142/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\nbash  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/f0ba0142/stage/variables_bash.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_z8e0vlmo.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'regex': None,
│   │   │   │   │   'returncode': None,
│   │   │   │   │   'runtime': None,
│   │   │   │   │   'file_regex': None,
│   │   │   │   │   'slurm_job_state': None,
│   │   │   │   │   'pbs_job_state': None,
│   │   │   │   │   'lsf_job_state': None,
│   │   │   │   │   'assert_ge': None,
│   │   │   │   │   'assert_gt': None,
│   │   │   │   │   'assert_le': None,
│   │   │   │   │   'assert_lt': None,
│   │   │   │   │   'assert_eq': None,
│   │   │   │   │   'assert_ne': None,
│   │   │   │   │   'contains': None,
│   │   │   │   │   'not_contains': None,
│   │   │   │   │   'is_symlink': None,
│   │   │   │   │   'exists': None,
│   │   │   │   │   'is_dir': None,
│   │   │   │   │   'is_file': None,
│   │   │   │   │   'file_count': None
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2024/02/14 17:17:38',
│   │   │   │   'endtime': '2024/02/14 17:17:38',
│   │   │   │   'runtime': '0.009573',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': '1+2=3\nthis is a literal string\n\\\'singlequote\\\'\n"doublequote"\ncurrent user: docs\nnumber of files: 5\nHello my name is Bob \n I am 30 years old\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/f0ba0142/variables_bash_build.sh'
│   │   │   },
│   │   │   {
│   │   │   │   'id': '14f5700c',
│   │   │   │   'full_id': '14f5700c-7f50-4852-975b-f35e514e853f',
│   │   │   │   'description': 'Declare shell variables in bash',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-23444606-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/14f5700c',
│   │   │   │   'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/14f5700c/variables_bash.sh',
│   │   │   │   'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/14f5700c/stage',
│   │   │   │   'command': 'bash variables_bash_build.sh',
│   │   │   │   'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/14f5700c/variables_bash.out',
│   │   │   │   'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/14f5700c/variables_bash.err',
│   │   │   │   'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/14f5700c/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  variables_bash:\n    type: script\n    executor: generic.local.bash\n    description: Declare shell variables in bash\n    tags: [tutorials]\n    vars:\n      X: 1\n      Y: 2\n      literalstring: this is a literal string\n      singlequote: \\\'singlequote\\\'\n      doublequote: \\"doublequote\\"\n      current_user: "$(whoami)"\n      num_files: "`find $HOME -type f -maxdepth 1 | wc -l`"\n      multiline_string: |\n        Hello my name is Bob \\n\n        I am 30 years old\n\n\n    run: |\n      echo "$X+$Y="$(($X+$Y))\n      echo $literalstring\n      echo $singlequote\n      echo $doublequote\n      echo "current user:" $current_user\n      echo "number of files:" $num_files\n      echo -e $multiline_string\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nset -eo pipefail\nX="1"\nY="2"\nliteralstring="this is a literal string"\nsinglequote="\\\'singlequote\\\'"\ndoublequote="\\"doublequote\\""\ncurrent_user="$(whoami)"\nnum_files="`find $HOME -type f -maxdepth 1 | wc -l`"\nmultiline_string="Hello my name is Bob \\n\nI am 30 years old\n"\n# Content of run section\necho "$X+$Y="$(($X+$Y))\necho $literalstring\necho $singlequote\necho $doublequote\necho "current user:" $current_user\necho "number of files:" $num_files\necho -e $multiline_string\n',
│   │   │   │   'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=variables_bash\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/14f5700c\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/14f5700c/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\nbash  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/14f5700c/stage/variables_bash.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_khlo9yeh.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'regex': None,
│   │   │   │   │   'returncode': None,
│   │   │   │   │   'runtime': None,
│   │   │   │   │   'file_regex': None,
│   │   │   │   │   'slurm_job_state': None,
│   │   │   │   │   'pbs_job_state': None,
│   │   │   │   │   'lsf_job_state': None,
│   │   │   │   │   'assert_ge': None,
│   │   │   │   │   'assert_gt': None,
│   │   │   │   │   'assert_le': None,
│   │   │   │   │   'assert_lt': None,
│   │   │   │   │   'assert_eq': None,
│   │   │   │   │   'assert_ne': None,
│   │   │   │   │   'contains': None,
│   │   │   │   │   'not_contains': None,
│   │   │   │   │   'is_symlink': None,
│   │   │   │   │   'exists': None,
│   │   │   │   │   'is_dir': None,
│   │   │   │   │   'is_file': None,
│   │   │   │   │   'file_count': None
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2024/02/14 17:18:10',
│   │   │   │   'endtime': '2024/02/14 17:18:10',
│   │   │   │   'runtime': '0.009655',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': '1+2=3\nthis is a literal string\n\\\'singlequote\\\'\n"doublequote"\ncurrent user: docs\nnumber of files: 5\nHello my name is Bob \n I am 30 years old\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/14f5700c/variables_bash_build.sh'
│   │   │   },
│   │   │   {
│   │   │   │   'id': '141e90a4',
│   │   │   │   'full_id': '141e90a4-123f-4bd9-af29-61fb7fcdd1f1',
│   │   │   │   'description': 'Declare shell variables in bash',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-23444606-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/141e90a4',
│   │   │   │   'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/141e90a4/variables_bash.sh',
│   │   │   │   'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/141e90a4/stage',
│   │   │   │   'command': 'bash variables_bash_build.sh',
│   │   │   │   'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/141e90a4/variables_bash.out',
│   │   │   │   'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/141e90a4/variables_bash.err',
│   │   │   │   'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/141e90a4/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  variables_bash:\n    type: script\n    executor: generic.local.bash\n    description: Declare shell variables in bash\n    tags: [tutorials]\n    vars:\n      X: 1\n      Y: 2\n      literalstring: this is a literal string\n      singlequote: \\\'singlequote\\\'\n      doublequote: \\"doublequote\\"\n      current_user: "$(whoami)"\n      num_files: "`find $HOME -type f -maxdepth 1 | wc -l`"\n      multiline_string: |\n        Hello my name is Bob \\n\n        I am 30 years old\n\n\n    run: |\n      echo "$X+$Y="$(($X+$Y))\n      echo $literalstring\n      echo $singlequote\n      echo $doublequote\n      echo "current user:" $current_user\n      echo "number of files:" $num_files\n      echo -e $multiline_string\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nset -eo pipefail\nX="1"\nY="2"\nliteralstring="this is a literal string"\nsinglequote="\\\'singlequote\\\'"\ndoublequote="\\"doublequote\\""\ncurrent_user="$(whoami)"\nnum_files="`find $HOME -type f -maxdepth 1 | wc -l`"\nmultiline_string="Hello my name is Bob \\n\nI am 30 years old\n"\n# Content of run section\necho "$X+$Y="$(($X+$Y))\necho $literalstring\necho $singlequote\necho $doublequote\necho "current user:" $current_user\necho "number of files:" $num_files\necho -e $multiline_string\n',
│   │   │   │   'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=variables_bash\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/141e90a4\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/141e90a4/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\nbash  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/141e90a4/stage/variables_bash.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_tm36f1je.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'regex': None,
│   │   │   │   │   'returncode': None,
│   │   │   │   │   'runtime': None,
│   │   │   │   │   'file_regex': None,
│   │   │   │   │   'slurm_job_state': None,
│   │   │   │   │   'pbs_job_state': None,
│   │   │   │   │   'lsf_job_state': None,
│   │   │   │   │   'assert_ge': None,
│   │   │   │   │   'assert_gt': None,
│   │   │   │   │   'assert_le': None,
│   │   │   │   │   'assert_lt': None,
│   │   │   │   │   'assert_eq': None,
│   │   │   │   │   'assert_ne': None,
│   │   │   │   │   'contains': None,
│   │   │   │   │   'not_contains': None,
│   │   │   │   │   'is_symlink': None,
│   │   │   │   │   'exists': None,
│   │   │   │   │   'is_dir': None,
│   │   │   │   │   'is_file': None,
│   │   │   │   │   'file_count': None
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2024/02/14 17:18:11',
│   │   │   │   'endtime': '2024/02/14 17:18:11',
│   │   │   │   'runtime': '0.009529',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': '1+2=3\nthis is a literal string\n\\\'singlequote\\\'\n"doublequote"\ncurrent user: docs\nnumber of files: 5\nHello my name is Bob \n I am 30 years old\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/141e90a4/variables_bash_build.sh'
│   │   │   },
│   │   │   {
│   │   │   │   'id': '71c7fbeb',
│   │   │   │   'full_id': '71c7fbeb-3604-4073-be82-c44e7acd4c68',
│   │   │   │   'description': 'Declare shell variables in bash',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-23444606-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb',
│   │   │   │   'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb/variables_bash.sh',
│   │   │   │   'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb/stage',
│   │   │   │   'command': 'bash variables_bash_build.sh',
│   │   │   │   'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb/variables_bash.out',
│   │   │   │   'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb/variables_bash.err',
│   │   │   │   'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  variables_bash:\n    type: script\n    executor: generic.local.bash\n    description: Declare shell variables in bash\n    tags: [tutorials]\n    vars:\n      X: 1\n      Y: 2\n      literalstring: this is a literal string\n      singlequote: \\\'singlequote\\\'\n      doublequote: \\"doublequote\\"\n      current_user: "$(whoami)"\n      num_files: "`find $HOME -type f -maxdepth 1 | wc -l`"\n      multiline_string: |\n        Hello my name is Bob \\n\n        I am 30 years old\n\n\n    run: |\n      echo "$X+$Y="$(($X+$Y))\n      echo $literalstring\n      echo $singlequote\n      echo $doublequote\n      echo "current user:" $current_user\n      echo "number of files:" $num_files\n      echo -e $multiline_string\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nset -eo pipefail\nX="1"\nY="2"\nliteralstring="this is a literal string"\nsinglequote="\\\'singlequote\\\'"\ndoublequote="\\"doublequote\\""\ncurrent_user="$(whoami)"\nnum_files="`find $HOME -type f -maxdepth 1 | wc -l`"\nmultiline_string="Hello my name is Bob \\n\nI am 30 years old\n"\n# Content of run section\necho "$X+$Y="$(($X+$Y))\necho $literalstring\necho $singlequote\necho $doublequote\necho "current user:" $current_user\necho "number of files:" $num_files\necho -e $multiline_string\n',
│   │   │   │   'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=variables_bash\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\nbash  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb/stage/variables_bash.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_697b5uis.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'regex': None,
│   │   │   │   │   'returncode': None,
│   │   │   │   │   'runtime': None,
│   │   │   │   │   'file_regex': None,
│   │   │   │   │   'slurm_job_state': None,
│   │   │   │   │   'pbs_job_state': None,
│   │   │   │   │   'lsf_job_state': None,
│   │   │   │   │   'assert_ge': None,
│   │   │   │   │   'assert_gt': None,
│   │   │   │   │   'assert_le': None,
│   │   │   │   │   'assert_lt': None,
│   │   │   │   │   'assert_eq': None,
│   │   │   │   │   'assert_ne': None,
│   │   │   │   │   'contains': None,
│   │   │   │   │   'not_contains': None,
│   │   │   │   │   'is_symlink': None,
│   │   │   │   │   'exists': None,
│   │   │   │   │   'is_dir': None,
│   │   │   │   │   'is_file': None,
│   │   │   │   │   'file_count': None
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2024/02/14 17:18:12',
│   │   │   │   'endtime': '2024/02/14 17:18:12',
│   │   │   │   'runtime': '0.010044',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': '1+2=3\nthis is a literal string\n\\\'singlequote\\\'\n"doublequote"\ncurrent user: docs\nnumber of files: 5\nHello my name is Bob \n I am 30 years old\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/71c7fbeb/variables_bash_build.sh'
│   │   │   }
│   │   ]
│   },
│   '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/hello_world.yml': {
│   │   'hello_world': [
│   │   │   {
│   │   │   │   'id': '80d8edb6',
│   │   │   │   'full_id': '80d8edb6-47bc-4203-9694-dabdd119aa42',
│   │   │   │   'description': 'hello world example',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-23444606-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/80d8edb6',
│   │   │   │   'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/80d8edb6/hello_world.sh',
│   │   │   │   'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/80d8edb6/stage',
│   │   │   │   'command': 'bash hello_world_build.sh',
│   │   │   │   'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/80d8edb6/hello_world.out',
│   │   │   │   'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/80d8edb6/hello_world.err',
│   │   │   │   'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/80d8edb6/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  hello_world:\n    executor: generic.local.bash\n    type: script\n    tags: tutorials\n    description: "hello world example"\n    run: echo "hello world!"\nmaintainers:\n- "@shahzebsiddiqui"\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nset -eo pipefail\n# Content of run section\necho "hello world!"',
│   │   │   │   'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=hello_world\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/80d8edb6\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/80d8edb6/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\nbash  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/80d8edb6/stage/hello_world.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_3tta6_bb.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'regex': None,
│   │   │   │   │   'returncode': None,
│   │   │   │   │   'runtime': None,
│   │   │   │   │   'file_regex': None,
│   │   │   │   │   'slurm_job_state': None,
│   │   │   │   │   'pbs_job_state': None,
│   │   │   │   │   'lsf_job_state': None,
│   │   │   │   │   'assert_ge': None,
│   │   │   │   │   'assert_gt': None,
│   │   │   │   │   'assert_le': None,
│   │   │   │   │   'assert_lt': None,
│   │   │   │   │   'assert_eq': None,
│   │   │   │   │   'assert_ne': None,
│   │   │   │   │   'contains': None,
│   │   │   │   │   'not_contains': None,
│   │   │   │   │   'is_symlink': None,
│   │   │   │   │   'exists': None,
│   │   │   │   │   'is_dir': None,
│   │   │   │   │   'is_file': None,
│   │   │   │   │   'file_count': None
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2024/02/14 17:16:54',
│   │   │   │   'endtime': '2024/02/14 17:16:54',
│   │   │   │   'runtime': '0.005912',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': 'hello world!\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/80d8edb6/hello_world_build.sh'
│   │   │   },
│   │   │   {
│   │   │   │   'id': 'ba00d656',
│   │   │   │   'full_id': 'ba00d656-a5d2-4210-b2a6-2c72feff0010',
│   │   │   │   'description': 'hello world example',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-23444606-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ba00d656',
│   │   │   │   'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ba00d656/hello_world.sh',
│   │   │   │   'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ba00d656/stage',
│   │   │   │   'command': 'bash hello_world_build.sh',
│   │   │   │   'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ba00d656/hello_world.out',
│   │   │   │   'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ba00d656/hello_world.err',
│   │   │   │   'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ba00d656/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  hello_world:\n    executor: generic.local.bash\n    type: script\n    tags: tutorials\n    description: "hello world example"\n    run: echo "hello world!"\nmaintainers:\n- "@shahzebsiddiqui"\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nset -eo pipefail\n# Content of run section\necho "hello world!"',
│   │   │   │   'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=hello_world\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ba00d656\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ba00d656/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\nbash  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ba00d656/stage/hello_world.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_qdji02ww.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'regex': None,
│   │   │   │   │   'returncode': None,
│   │   │   │   │   'runtime': None,
│   │   │   │   │   'file_regex': None,
│   │   │   │   │   'slurm_job_state': None,
│   │   │   │   │   'pbs_job_state': None,
│   │   │   │   │   'lsf_job_state': None,
│   │   │   │   │   'assert_ge': None,
│   │   │   │   │   'assert_gt': None,
│   │   │   │   │   'assert_le': None,
│   │   │   │   │   'assert_lt': None,
│   │   │   │   │   'assert_eq': None,
│   │   │   │   │   'assert_ne': None,
│   │   │   │   │   'contains': None,
│   │   │   │   │   'not_contains': None,
│   │   │   │   │   'is_symlink': None,
│   │   │   │   │   'exists': None,
│   │   │   │   │   'is_dir': None,
│   │   │   │   │   'is_file': None,
│   │   │   │   │   'file_count': None
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2024/02/14 17:16:55',
│   │   │   │   'endtime': '2024/02/14 17:16:55',
│   │   │   │   'runtime': '0.006102',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': 'hello world!\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ba00d656/hello_world_build.sh'
│   │   │   },
│   │   │   {
│   │   │   │   'id': '3c38bc63',
│   │   │   │   'full_id': '3c38bc63-b6b3-4b8b-a5a0-6ca9054240bc',
│   │   │   │   'description': 'hello world example',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-23444606-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/3c38bc63',
│   │   │   │   'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/3c38bc63/hello_world.sh',
│   │   │   │   'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/3c38bc63/stage',
│   │   │   │   'command': 'bash hello_world_build.sh',
│   │   │   │   'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/3c38bc63/hello_world.out',
│   │   │   │   'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/3c38bc63/hello_world.err',
│   │   │   │   'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/3c38bc63/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  hello_world:\n    executor: generic.local.bash\n    type: script\n    tags: tutorials\n    description: "hello world example"\n    run: echo "hello world!"\nmaintainers:\n- "@shahzebsiddiqui"\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nset -eo pipefail\n# Content of run section\necho "hello world!"',
│   │   │   │   'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=hello_world\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/3c38bc63\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/3c38bc63/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\nbash  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/3c38bc63/stage/hello_world.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_umr4lynk.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'regex': None,
│   │   │   │   │   'returncode': None,
│   │   │   │   │   'runtime': None,
│   │   │   │   │   'file_regex': None,
│   │   │   │   │   'slurm_job_state': None,
│   │   │   │   │   'pbs_job_state': None,
│   │   │   │   │   'lsf_job_state': None,
│   │   │   │   │   'assert_ge': None,
│   │   │   │   │   'assert_gt': None,
│   │   │   │   │   'assert_le': None,
│   │   │   │   │   'assert_lt': None,
│   │   │   │   │   'assert_eq': None,
│   │   │   │   │   'assert_ne': None,
│   │   │   │   │   'contains': None,
│   │   │   │   │   'not_contains': None,
│   │   │   │   │   'is_symlink': None,
│   │   │   │   │   'exists': None,
│   │   │   │   │   'is_dir': None,
│   │   │   │   │   'is_file': None,
│   │   │   │   │   'file_count': None
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2024/02/14 17:17:52',
│   │   │   │   'endtime': '2024/02/14 17:17:52',
│   │   │   │   'runtime': '0.006236',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': 'hello world!\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/3c38bc63/hello_world_build.sh'
│   │   │   },
│   │   │   {
│   │   │   │   'id': '6cd32992',
│   │   │   │   'full_id': '6cd32992-b716-4911-b68f-df0ee46e247a',
│   │   │   │   'description': 'hello world example',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-23444606-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/6cd32992',
│   │   │   │   'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/6cd32992/hello_world.sh',
│   │   │   │   'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/6cd32992/stage',
│   │   │   │   'command': 'bash hello_world_build.sh',
│   │   │   │   'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/6cd32992/hello_world.out',
│   │   │   │   'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/6cd32992/hello_world.err',
│   │   │   │   'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/6cd32992/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  hello_world:\n    executor: generic.local.bash\n    type: script\n    tags: tutorials\n    description: "hello world example"\n    run: echo "hello world!"\nmaintainers:\n- "@shahzebsiddiqui"\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nset -eo pipefail\n# Content of run section\necho "hello world!"',
│   │   │   │   'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=hello_world\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/6cd32992\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/6cd32992/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\nbash  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/6cd32992/stage/hello_world.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_2zmvkh1r.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'regex': None,
│   │   │   │   │   'returncode': None,
│   │   │   │   │   'runtime': None,
│   │   │   │   │   'file_regex': None,
│   │   │   │   │   'slurm_job_state': None,
│   │   │   │   │   'pbs_job_state': None,
│   │   │   │   │   'lsf_job_state': None,
│   │   │   │   │   'assert_ge': None,
│   │   │   │   │   'assert_gt': None,
│   │   │   │   │   'assert_le': None,
│   │   │   │   │   'assert_lt': None,
│   │   │   │   │   'assert_eq': None,
│   │   │   │   │   'assert_ne': None,
│   │   │   │   │   'contains': None,
│   │   │   │   │   'not_contains': None,
│   │   │   │   │   'is_symlink': None,
│   │   │   │   │   'exists': None,
│   │   │   │   │   'is_dir': None,
│   │   │   │   │   'is_file': None,
│   │   │   │   │   'file_count': None
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2024/02/14 17:18:00',
│   │   │   │   'endtime': '2024/02/14 17:18:00',
│   │   │   │   'runtime': '0.006102',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': 'hello world!\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/6cd32992/hello_world_build.sh'
│   │   │   },
│   │   │   {
│   │   │   │   'id': 'dd2da8f2',
│   │   │   │   'full_id': 'dd2da8f2-a1fc-4659-aee6-e6fa8b374634',
│   │   │   │   'description': 'hello world example',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-23444606-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/dd2da8f2',
│   │   │   │   'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/dd2da8f2/hello_world.sh',
│   │   │   │   'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/dd2da8f2/stage',
│   │   │   │   'command': 'bash hello_world_build.sh',
│   │   │   │   'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/dd2da8f2/hello_world.out',
│   │   │   │   'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/dd2da8f2/hello_world.err',
│   │   │   │   'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/dd2da8f2/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  hello_world:\n    executor: generic.local.bash\n    type: script\n    tags: tutorials\n    description: "hello world example"\n    run: echo "hello world!"\nmaintainers:\n- "@shahzebsiddiqui"\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nset -eo pipefail\n# Content of run section\necho "hello world!"',
│   │   │   │   'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=hello_world\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/dd2da8f2\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/dd2da8f2/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\nbash  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/dd2da8f2/stage/hello_world.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_316aj550.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'regex': None,
│   │   │   │   │   'returncode': None,
│   │   │   │   │   'runtime': None,
│   │   │   │   │   'file_regex': None,
│   │   │   │   │   'slurm_job_state': None,
│   │   │   │   │   'pbs_job_state': None,
│   │   │   │   │   'lsf_job_state': None,
│   │   │   │   │   'assert_ge': None,
│   │   │   │   │   'assert_gt': None,
│   │   │   │   │   'assert_le': None,
│   │   │   │   │   'assert_lt': None,
│   │   │   │   │   'assert_eq': None,
│   │   │   │   │   'assert_ne': None,
│   │   │   │   │   'contains': None,
│   │   │   │   │   'not_contains': None,
│   │   │   │   │   'is_symlink': None,
│   │   │   │   │   'exists': None,
│   │   │   │   │   'is_dir': None,
│   │   │   │   │   'is_file': None,
│   │   │   │   │   'file_count': None
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2024/02/14 17:18:19',
│   │   │   │   'endtime': '2024/02/14 17:18:19',
│   │   │   │   'runtime': '0.005642',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': 'hello world!\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/dd2da8f2/hello_world_build.sh'
│   │   │   },
│   │   │   {
│   │   │   │   'id': 'ddf010d1',
│   │   │   │   'full_id': 'ddf010d1-2d57-4d38-9620-9e25252a70dd',
│   │   │   │   'description': 'hello world example',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-23444606-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ddf010d1',
│   │   │   │   'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ddf010d1/hello_world.sh',
│   │   │   │   'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ddf010d1/stage',
│   │   │   │   'command': 'bash hello_world_build.sh',
│   │   │   │   'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ddf010d1/hello_world.out',
│   │   │   │   'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ddf010d1/hello_world.err',
│   │   │   │   'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ddf010d1/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  hello_world:\n    executor: generic.local.bash\n    type: script\n    tags: tutorials\n    description: "hello world example"\n    run: echo "hello world!"\nmaintainers:\n- "@shahzebsiddiqui"\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nset -eo pipefail\n# Content of run section\necho "hello world!"',
│   │   │   │   'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=hello_world\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ddf010d1\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ddf010d1/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\nbash  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ddf010d1/stage/hello_world.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_316aj550.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'regex': None,
│   │   │   │   │   'returncode': None,
│   │   │   │   │   'runtime': None,
│   │   │   │   │   'file_regex': None,
│   │   │   │   │   'slurm_job_state': None,
│   │   │   │   │   'pbs_job_state': None,
│   │   │   │   │   'lsf_job_state': None,
│   │   │   │   │   'assert_ge': None,
│   │   │   │   │   'assert_gt': None,
│   │   │   │   │   'assert_le': None,
│   │   │   │   │   'assert_lt': None,
│   │   │   │   │   'assert_eq': None,
│   │   │   │   │   'assert_ne': None,
│   │   │   │   │   'contains': None,
│   │   │   │   │   'not_contains': None,
│   │   │   │   │   'is_symlink': None,
│   │   │   │   │   'exists': None,
│   │   │   │   │   'is_dir': None,
│   │   │   │   │   'is_file': None,
│   │   │   │   │   'file_count': None
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2024/02/14 17:18:19',
│   │   │   │   'endtime': '2024/02/14 17:18:19',
│   │   │   │   'runtime': '0.00601',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': 'hello world!\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/ddf010d1/hello_world_build.sh'
│   │   │   },
│   │   │   {
│   │   │   │   'id': 'd8f2b427',
│   │   │   │   'full_id': 'd8f2b427-95c3-4511-aa64-8eb500c5f309',
│   │   │   │   'description': 'hello world example',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-23444606-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/d8f2b427',
│   │   │   │   'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/d8f2b427/hello_world.sh',
│   │   │   │   'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/d8f2b427/stage',
│   │   │   │   'command': 'bash hello_world_build.sh',
│   │   │   │   'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/d8f2b427/hello_world.out',
│   │   │   │   'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/d8f2b427/hello_world.err',
│   │   │   │   'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/d8f2b427/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  hello_world:\n    executor: generic.local.bash\n    type: script\n    tags: tutorials\n    description: "hello world example"\n    run: echo "hello world!"\nmaintainers:\n- "@shahzebsiddiqui"\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nset -eo pipefail\n# Content of run section\necho "hello world!"',
│   │   │   │   'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=hello_world\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/d8f2b427\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/d8f2b427/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\nbash  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/d8f2b427/stage/hello_world.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_316aj550.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'regex': None,
│   │   │   │   │   'returncode': None,
│   │   │   │   │   'runtime': None,
│   │   │   │   │   'file_regex': None,
│   │   │   │   │   'slurm_job_state': None,
│   │   │   │   │   'pbs_job_state': None,
│   │   │   │   │   'lsf_job_state': None,
│   │   │   │   │   'assert_ge': None,
│   │   │   │   │   'assert_gt': None,
│   │   │   │   │   'assert_le': None,
│   │   │   │   │   'assert_lt': None,
│   │   │   │   │   'assert_eq': None,
│   │   │   │   │   'assert_ne': None,
│   │   │   │   │   'contains': None,
│   │   │   │   │   'not_contains': None,
│   │   │   │   │   'is_symlink': None,
│   │   │   │   │   'exists': None,
│   │   │   │   │   'is_dir': None,
│   │   │   │   │   'is_file': None,
│   │   │   │   │   'file_count': None
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2024/02/14 17:18:19',
│   │   │   │   'endtime': '2024/02/14 17:18:19',
│   │   │   │   'runtime': '0.005532',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': 'hello world!\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/d8f2b427/hello_world_build.sh'
│   │   │   },
│   │   │   {
│   │   │   │   'id': '66f6eef7',
│   │   │   │   'full_id': '66f6eef7-c3a7-4f97-a574-59feecaea345',
│   │   │   │   'description': 'hello world example',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-23444606-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/66f6eef7',
│   │   │   │   'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/66f6eef7/hello_world.sh',
│   │   │   │   'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/66f6eef7/stage',
│   │   │   │   'command': 'bash hello_world_build.sh',
│   │   │   │   'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/66f6eef7/hello_world.out',
│   │   │   │   'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/66f6eef7/hello_world.err',
│   │   │   │   'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/66f6eef7/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  hello_world:\n    executor: generic.local.bash\n    type: script\n    tags: tutorials\n    description: "hello world example"\n    run: echo "hello world!"\nmaintainers:\n- "@shahzebsiddiqui"\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nset -eo pipefail\n# Content of run section\necho "hello world!"',
│   │   │   │   'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=hello_world\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/66f6eef7\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/66f6eef7/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\nbash  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/66f6eef7/stage/hello_world.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_316aj550.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'regex': None,
│   │   │   │   │   'returncode': None,
│   │   │   │   │   'runtime': None,
│   │   │   │   │   'file_regex': None,
│   │   │   │   │   'slurm_job_state': None,
│   │   │   │   │   'pbs_job_state': None,
│   │   │   │   │   'lsf_job_state': None,
│   │   │   │   │   'assert_ge': None,
│   │   │   │   │   'assert_gt': None,
│   │   │   │   │   'assert_le': None,
│   │   │   │   │   'assert_lt': None,
│   │   │   │   │   'assert_eq': None,
│   │   │   │   │   'assert_ne': None,
│   │   │   │   │   'contains': None,
│   │   │   │   │   'not_contains': None,
│   │   │   │   │   'is_symlink': None,
│   │   │   │   │   'exists': None,
│   │   │   │   │   'is_dir': None,
│   │   │   │   │   'is_file': None,
│   │   │   │   │   'file_count': None
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2024/02/14 17:18:19',
│   │   │   │   'endtime': '2024/02/14 17:18:19',
│   │   │   │   'runtime': '0.005762',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': 'hello world!\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/66f6eef7/hello_world_build.sh'
│   │   │   },
│   │   │   {
│   │   │   │   'id': '7831db13',
│   │   │   │   'full_id': '7831db13-f20d-4043-81b7-edfb8f079da4',
│   │   │   │   'description': 'hello world example',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-23444606-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/7831db13',
│   │   │   │   'testpath': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/7831db13/hello_world.sh',
│   │   │   │   'stagedir': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/7831db13/stage',
│   │   │   │   'command': 'bash hello_world_build.sh',
│   │   │   │   'outfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/7831db13/hello_world.out',
│   │   │   │   'errfile': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/7831db13/hello_world.err',
│   │   │   │   'buildenv': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/7831db13/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  hello_world:\n    executor: generic.local.bash\n    type: script\n    tags: tutorials\n    description: "hello world example"\n    run: echo "hello world!"\nmaintainers:\n- "@shahzebsiddiqui"\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nset -eo pipefail\n# Content of run section\necho "hello world!"',
│   │   │   │   'buildscript_content': '#!/bin/bash\n\n# Function to handle all signals and perform cleanup\nfunction cleanup() {\n    echo "Signal trapped. Performing cleanup before exiting."\n    exitcode=$?\n    echo "buildtest: command \\`$BASH_COMMAND\' failed (exit code: $exitcode)"\n    exit $exitcode\n}\n\n# Trap all signals and call the cleanup function\ntrap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU\n\nexport BUILDTEST_TEST_NAME=hello_world\nexport BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/7831db13\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials\nexport BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/7831db13/stage\n# source executor startup script\nsource /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh\n# Run generated script\nbash  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/7831db13/stage/hello_world.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/tmp/tmpjcb9dx2w/var/logs/buildtest_316aj550.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'regex': None,
│   │   │   │   │   'returncode': None,
│   │   │   │   │   'runtime': None,
│   │   │   │   │   'file_regex': None,
│   │   │   │   │   'slurm_job_state': None,
│   │   │   │   │   'pbs_job_state': None,
│   │   │   │   │   'lsf_job_state': None,
│   │   │   │   │   'assert_ge': None,
│   │   │   │   │   'assert_gt': None,
│   │   │   │   │   'assert_le': None,
│   │   │   │   │   'assert_lt': None,
│   │   │   │   │   'assert_eq': None,
│   │   │   │   │   'assert_ne': None,
│   │   │   │   │   'contains': None,
│   │   │   │   │   'not_contains': None,
│   │   │   │   │   'is_symlink': None,
│   │   │   │   │   'exists': None,
│   │   │   │   │   'is_dir': None,
│   │   │   │   │   'is_file': None,
│   │   │   │   │   'file_count': None
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2024/02/14 17:18:19',
│   │   │   │   'endtime': '2024/02/14 17:18:19',
│   │   │   │   'runtime': '0.005507',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': 'hello world!\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/hello_world/hello_world/7831db13/hello_world_build.sh'
│   │   │   }
│   │   ]
│   }
}

Note

If you pass a valid filepath but file is not in cache you will get an error as follows

$ buildtest it buildspec $BUILDTEST_ROOT/README.rst
Unable to find any buildspecs in cache, please specify one of the following buildspecs: 
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/vars.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/pass_returncode.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/status_regex.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/regex_on_filename.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/runtime_status_test.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/explicit_state.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/exists.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/file_exists_with_number.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/file_and_dir_check.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/is_symlink.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/file_count.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/file_count_pattern.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/file_count_filetype.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/file_count_file_traverse_limit.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/mode.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/skip_tests.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/hello_world.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/general_tests/configuration/ulimits.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/general_tests/configuration/kernel_state.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/general_tests/configuration/disk_usage.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/general_tests/configuration/systemd-default-target.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/tags_example.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/python-hello.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/python-shell.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/add_numbers.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/summary_example.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/environment.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/csh_shell_examples.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/sleep.yml

Query Test Records via buildtest inspect query

The buildtest inspect query command can allow you to retrieve query certain fields from each test records that can be useful when you are inspecting a test. Currently, we can fetch content of output file, error file, testpath, and build script. Shown below are the list of available options for buildtest inspect query.

buildtest inspect --help
$ buildtest inspect query --help
usage: buildtest [options] [COMMANDS] inspect query [-h] [--pager]
                                                    [--theme Color Themes]
                                                    [-b] [-be] [-e] [-o] [-t]
                                                    [name [name ...]]

positional arguments:
  name                  Name of builder to query in report file

optional arguments:
  -h, --help            show this help message and exit
  --pager               Enable PAGING when viewing result
  --theme Color Themes  Specify a color theme, Pygments style to use when
                        displaying output. See
                        https://pygments.org/docs/styles/#getting-a-list-of-
                        available-styles for available themese
  -b, --buildscript     Print build script
  -be, --buildenv       Print content of build env
  -e, --error           Print error file
  -o, --output          Print output file
  -t, --testpath        Print content of testpath

The buildtest inspect query command expects positional arguments that are name of tests which you can get by running buildtest inspect list.

For instance, let’s query the test circle_area by running the following:

buildtest inspect query circle_area
$ buildtest inspect query circle_area
─────────────── circle_area/813e3c6d-a62c-4b7d-b67b-86a0a39a831d ───────────────
Executor: generic.local.bash
Description: Calculate circle of area given a radius
State: PASS
Returncode: 0
Runtime: 0.031635 sec
Starttime: 2024/02/14 17:18:18
Endtime: 2024/02/14 17:18:18
Command: bash circle_area_build.sh
Test Script: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/circle_area.sh
Build Script: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/circle_area_build.sh
Output File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/circle_area.out
Error File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/circle_area.err
Log File: /tmp/tmpjcb9dx2w/var/logs/buildtest_g9k3143t.log

buildtest will display metadata for each test. By default, buildtest will report the last run for each test that is specified as a positional argument.

You can retrieve content of output file via --output or short option -o. In this command, we retrieve the last run for circle_area and print content of output file

buildtest inspect query -o circle_area
$ buildtest inspect query -o circle_area
─────────────── circle_area/813e3c6d-a62c-4b7d-b67b-86a0a39a831d ───────────────
Executor: generic.local.bash
Description: Calculate circle of area given a radius
State: PASS
Returncode: 0
Runtime: 0.031635 sec
Starttime: 2024/02/14 17:18:18
Endtime: 2024/02/14 17:18:18
Command: bash circle_area_build.sh
Test Script: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/circle_area.sh
Build Script: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/circle_area_build.sh
Output File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/circle_area.out
Error File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/circle_area.err
Log File: /tmp/tmpjcb9dx2w/var/logs/buildtest_g9k3143t.log
─ Output File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/cir… ─
Circle Radius  2                                                                
Area of circle  12.566370614359172

If you want to see content of error file use the -e or --error flag. It would be useful to inspect content of build script and generated test, which can be retrieved using -t and --buildscript. Let’s query test circle_area and report all of the content fields

buildtest inspect query -o -e -t -b circle_area
$ buildtest inspect query -o -e -t -b circle_area
─────────────── circle_area/813e3c6d-a62c-4b7d-b67b-86a0a39a831d ───────────────
Executor: generic.local.bash
Description: Calculate circle of area given a radius
State: PASS
Returncode: 0
Runtime: 0.031635 sec
Starttime: 2024/02/14 17:18:18
Endtime: 2024/02/14 17:18:18
Command: bash circle_area_build.sh
Test Script: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/circle_area.sh
Build Script: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/circle_area_build.sh
Output File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/circle_area.out
Error File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/circle_area.err
Log File: /tmp/tmpjcb9dx2w/var/logs/buildtest_g9k3143t.log
─ Output File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/cir… ─
Circle Radius  2                                                                
Area of circle  12.566370614359172                                              
                                                                                
─ Error File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circ… ─
                                                                                
─ Test File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circl… ─
#!/bin/bash                                                                     
                                                                                
python /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/81
─ Test File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circl… ─
#!/bin/bash                                                                     
                                                                                
# Function to handle all signals and perform cleanup                            
function cleanup() {                                                            
    echo "Signal trapped. Performing cleanup before exiting."                   
    exitcode=$?                                                                 
    echo "buildtest: command \`$BASH_COMMAND' failed (exit code: $exitcode)"    
    exit $exitcode                                                              
}                                                                               
                                                                                
# Trap all signals and call the cleanup function                                
trap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTE
                                                                                
export BUILDTEST_TEST_NAME=circle_area                                          
export BUILDTEST_TEST_ROOT=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-
export BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/
export BUILDTEST_STAGE_DIR=/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-
# source executor startup script                                                
source /tmp/tmpjcb9dx2w/var/executor/generic.local.bash/before_script.sh        
# Run generated script                                                          
/tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-shell/circle_area/813e3c6d/
# Get return code                                                               
returncode=$?                                                                   
# Exit with return code                                                         
exit $returncode

We can query multiple tests using buildtest inspect query since each test is a positional argument. Any options specified to buildtest inspect query will be applied to all test. For instance, let’s fetch the output the of test names root_disk_usage and python_hello

buildtest inspect query -o root_disk_usage python_hello
$ buildtest inspect query -o root_disk_usage python_hello
────────────── python_hello/c1ae4e33-2c32-4d7a-831e-96251fdf9206 ───────────────
Executor: generic.local.bash
Description: Hello World python
State: PASS
Returncode: 0
Runtime: 0.029861 sec
Starttime: 2024/02/14 17:18:18
Endtime: 2024/02/14 17:18:18
Command: bash python_hello_build.sh
Test Script: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-hello/python_hello/c1ae4e33/python_hello.sh
Build Script: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-hello/python_hello/c1ae4e33/python_hello_build.sh
Output File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-hello/python_hello/c1ae4e33/python_hello.out
Error File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-hello/python_hello/c1ae4e33/python_hello.err
Log File: /tmp/tmpjcb9dx2w/var/logs/buildtest_g9k3143t.log
─ Output File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/python-hello/pyt… ─
Hello World                                                                     
                                                                                
───────────── root_disk_usage/429e7b9d-33d9-4a03-9888-13f94a5a48f8 ─────────────
Executor: generic.local.bash
Description: Check root disk usage and report if it exceeds threshold
State: PASS
Returncode: 0
Runtime: 0.009745 sec
Starttime: 2024/02/14 17:17:41
Endtime: 2024/02/14 17:17:41
Command: bash root_disk_usage_build.sh
Test Script: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/disk_usage/root_disk_usage/429e7b9d/root_disk_usage.sh
Build Script: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/disk_usage/root_disk_usage/429e7b9d/root_disk_usage_build.sh
Output File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/disk_usage/root_disk_usage/429e7b9d/root_disk_usage.out
Error File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/disk_usage/root_disk_usage/429e7b9d/root_disk_usage.err
Log File: /tmp/tmpjcb9dx2w/var/logs/buildtest_oqhi82ms.log
─ Output File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/disk_usage/root_… ─
[OK] Root disk is below threshold of 90%

If you want to query specific test ID, you can specify name of test followed by / and test ID. You don’t need to specify the full ID however tab completion is available to help fill in the names. For example if you want to query test record for circle_area/8edce927-2ecc-4991-ac40-e376c03394b4 shown in tab completion you can type a first few characters to query the record

$ buildtest inspect query circle_area/
circle_area/08f20b50-d2e2-41ab-a75e-a7df75e5afcc  circle_area/8edce927-2ecc-4991-ac40-e376c03394b4  circle_area/d47b6ba8-71b6-4531-b8cd-b6ba9b5f0c6c
circle_area/237c3a96-fad0-4ab7-ab1f-3e7ed1816955  circle_area/baea2e9b-a187-4f9f-bcea-75e768ccb0e0  circle_area/e6652700-4cdb-4f6b-80c5-261e4f448876
circle_area/2c279160-1abf-4c70-957f-d9e4608f521b  circle_area/bf8f1762-ebf9-458e-92e2-af3fc6e73eac  circle_area/e7cc7138-a650-4cd8-aca8-b904f901a0da

$ buildtest inspect query circle_area/8ed
──────────────────────────────────────────────────────────────────────────────────────────── circle_area/8edce927-2ecc-4991-ac40-e376c03394b4 ─────────────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: Calculate circle of area given a radius
State: PASS
Returncode: 0
Runtime: 0.360774 sec
Starttime: 2021/12/23 12:37:25
Endtime: 2021/12/23 12:37:25
Command: bash --norc --noprofile -eo pipefail circle_area_build.sh
Test Script: /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.bash/python-shell/circle_area/8edce927/circle_area.sh
Build Script: /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.bash/python-shell/circle_area/8edce927/circle_area_build.sh
Output File: /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.bash/python-shell/circle_area/8edce927/circle_area.out
Error File: /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.bash/python-shell/circle_area/8edce927/circle_area.err
Log File: /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/logs/buildtest_c5dnun2l.log

buildtest will search for test ID using re.match so it is possible to apply a regular expression to seek out multiple test records. The tests must be enclosed in quotes " in-order to have a valid regular expression. Here are few examples that can be useful

# retrieve all test records for name `circle_area`
buildtest inspect query circle_area/

# retrieve test records starting with ID `8a` and `bc` for test name `exit1`
buildtest inspect query "exit1/(8a|bc)"

Using Alternate Report File

The buildtest report and buildtest inspect command will read from the report file tracked by buildtest which is stored in $BUILDTEST_ROOT/var/report.json. This single file can became an issue if you are running jobs through CI where you can potentially overwrite same file or if you want separate report files for each set of builds. Luckily we have an option to handle this using the buildtest -r <report_path> build -b <buildspec_path> option which can be used to specify an alternate location to report file.

buildtest will write the report file in the desired location, then you can specify the path to report file via buildtest -r <report_path> report and buildtest -r <report_path> inspect to load the report file when reporting tests.

The report file must be valid JSON file that buildtest understands in order to use buildtest report and buildtest inspect command. Shown below are some examples using the alternate report file using buildtest report and buildtest inspect command.

$ buildtest -r $BUILDTEST_ROOT/python.json report --format name,id
                      Report File: /Users/siddiq90/Documents/GitHubDesktop/buildtest/python.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                                                               ┃ id                                            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ circle_area                                                        │ a2814554                                      │
├────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ python_hello                                                       │ dd447e43                                      │
└────────────────────────────────────────────────────────────────────┴───────────────────────────────────────────────┘

You can view path to all report files via buildtest report list which keeps track of any new report files created when using buildtest build -r option.

$ buildtest report list
/tmp/tmpjcb9dx2w/var/report.json
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/python.json