Query Test Report

buildtest keeps track of all tests and results in a JSON file. This file is read by buildtest report command to extract certain fields from JSON file and display them in table format. Shown below is command usage to query test reports.

buildtest report --help
$ buildtest report --help
usage: buildtest [options] [COMMANDS] report [-h] [--filter FILTER]
                                             [--format FORMAT] [--helpfilter]
                                             [--helpformat] [--filterfields]
                                             [--formatfields] [-f | -p]
                                             [-s START] [-e END] [--latest]
                                             [--oldest] [-c COUNT] [-n] [-t]
                                             [--row-count] [--color COLOR]
                                             [--pager]
                                             ...

optional arguments:
  -h, --help            show this help message and exit
  --filter FILTER       Filter report by filter fields. The filter fields must
                        be a key=value pair and multiple fields can be comma
                        separated in the following format: --filter
                        key1=val1,key2=val2 . For list of filter fields run:
                        --helpfilter.
  --format FORMAT       format field for printing purposes. For more details
                        see --helpformat for list of available fields. Fields
                        must be separated by comma (usage: --format
                        <field1>,<field2>,...)
  --helpfilter          List available filter fields to be used with --filter
                        option
  --helpformat          List of available format fields
  --filterfields        Print raw filter fields for --filter option to filter
                        the report
  --formatfields        Print raw format fields for --format option to format
                        the report
  -f, --fail            Retrieve all FAIL tests
  -p, --pass            Retrieve all PASS tests
  -s START, --start START
                        Retrieve tests by starttime
  -e END, --end END     Retrieve tests by endtime
  --latest              Retrieve latest record of particular test
  --oldest              Retrieve oldest record of particular test
  -c COUNT, --count COUNT
                        Retrieve limited number of rows that get printed
  -n, --no-header       Don't print headers column used with terse option
                        (--terse).
  -t, --terse           Print output in machine readable format
  --row-count           Print total count of records from the table.
  --color COLOR         Print output of report table with the selected color.
  --pager               Enable PAGING when viewing result

subcommands:
  Fetch test results from report file and print them in table format

  
    clear (c)           Remove all report file
    list (l)            List all report files
    path (p)            Print full path to the report file being used
    summary (sm)        Summarize test report

You may run buildtest report without any option, and buildtest will display all test results with default format fields. To see a list of all format fields, click here.

buildtest report
$ buildtest report
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
┏━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃        ┃        ┃       ┃ return ┃ startt ┃ endtim ┃ runtim ┃        ┃ build ┃
┃ name   ┃ id     ┃ state ┃ code   ┃ ime    ┃ e      ┃ e      ┃ tags   ┃ spec  ┃
┡━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ variab │ c70320 │ PASS  │ 0      │ 2022/1 │ 2022/1 │ 0.0106 │ tutori │ /home │
│ les_ba │ c7     │       │        │ 0/07   │ 0/07   │ 26     │ als    │ /docs │
│ sh     │        │       │        │ 18:35: │ 18:35: │        │        │ /chec │
│        │        │       │        │ 07     │ 07     │        │        │ kouts │
│        │        │       │        │        │        │        │        │ /read │
│        │        │       │        │        │        │        │        │ thedo │
│        │        │       │        │        │        │        │        │ cs.or │
│        │        │       │        │        │        │        │        │ g/use │
│        │        │       │        │        │        │        │        │ r_bui │
│        │        │       │        │        │        │        │        │ lds/b │
│        │        │       │        │        │        │        │        │ uildt │
│        │        │       │        │        │        │        │        │ est/c │
│        │        │       │        │        │        │        │        │ hecko │
│        │        │       │        │        │        │        │        │ uts/d │
│        │        │       │        │        │        │        │        │ evel/ │
...

You can run buildtest report path command to get the full path to the report file being used. This is helpful when the report file is not known in that case one can print the file by doing cat $(buildtest report path).

buildtest report path
$ buildtest report path
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json

The buildtest report command will fetch all records and display them in a table format, if you want to know how many records are displayed in the table you can use buildtest report --row-count which will display a raw count of records in the table. In the example below we will run buildtest report --row-count which will show number of tests in the report.

buildtest report --row-count
$ buildtest report --row-count
100

You may find it useful to fetch all failed records and determine pass/fail criteria as part of your shell script, this can be done by running buildtest report --row-count --fail as shown below.

buildtest report --row-count --fail
$ buildtest report --row-count --fail
36

Note

buildtest rt is an alias for buildtest report command.

Color output of buildtest report (buildtest report –color)

If you would like to change the output color for either buildtest report and buildtest report summary you can use the --color option. This will print the output of the previous two commands in the provided color. The color must be one of the supported colors by rich console:

buildtest report --color red
buildtest report --color red summary --detailed

Format Reports (buildtest report --format)

The buildtest report command displays a default format fields that can be changed using the --format option. The report file (JSON) contains many more fields and we expose some of the fields with the –format option. To see a list of available format fields you can run buildtest report --helpformat. This option will list all format fields with their description.

buildtest report --helpformat
$ buildtest report --helpformat
                                 Format Fields                                  
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Field      ┃ Description                                                     ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildspec  │ Buildspec File                                                  │
│ buildenv   │ Show build environment file for test                            │
│ command    │ Command executed                                                │
│ compiler   │ Retrieve compiler used for test (applicable for compiler        │
│            │ schema)                                                         │
│ endtime    │ End Time for test                                               │
│ errfile    │ Error File                                                      │
│ executor   │ Name of executor used for running test                          │
│ hostname   │ Hostname of machine where job was submitted from                │
│ full_id    │ Fully qualified build identifier                                │
│ id         │ Unique build identifier                                         │
│ metrics    │ List all metrics for test                                       │
│ name       │ Name of test                                                    │
│ outfile    │ Output file                                                     │
│ returncode │ Return Code                                                     │
│ runtime    │ Total runtime in seconds                                        │
│ schemafile │ Schema file used for validating test                            │
│ starttime  │ Start time of test                                              │
│ state      │ State of test (PASS/FAIL)                                       │
│ tags       │ Tag name                                                        │
│ testroot   │ Root of test directory                                          │
│ testpath   │ Path to test                                                    │
│ user       │ User who ran the test                                           │
└────────────┴─────────────────────────────────────────────────────────────────┘

To print just the raw format fields use the command buildtest report --formatfields.

buildtest report --formatfields
$ buildtest report --formatfields
buildspec
buildenv
command
compiler
endtime
errfile
executor
hostname
full_id
id
metrics
name
outfile
returncode
runtime
schemafile
starttime
state
tags
testroot
testpath
user

Format Field Usage

The --format field are specified in comma separated format (i.e --format <field1>,<field2>). In this example we format table by fields --format id,executor,state,returncode.

buildtest rt --format name,id,executor,state,returncode
$ buildtest rt --format name,id,executor,state,returncode
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓
┃ name                    ┃ id       ┃ executor           ┃ state ┃ returncode ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩
│ variables_bash          │ c70320c7 │ generic.local.bash │ PASS  │ 0          │
├─────────────────────────┼──────────┼────────────────────┼───────┼────────────┤
│ variables_bash          │ 532e3e6a │ generic.local.bash │ PASS  │ 0          │
├─────────────────────────┼──────────┼────────────────────┼───────┼────────────┤
│ variables_bash          │ b8a43f49 │ generic.local.bash │ PASS  │ 0          │
├─────────────────────────┼──────────┼────────────────────┼───────┼────────────┤
│ variables_bash          │ 7f5728b7 │ generic.local.bash │ PASS  │ 0          │
├─────────────────────────┼──────────┼────────────────────┼───────┼────────────┤
│ variables_bash          │ eb1dcd91 │ generic.local.bash │ PASS  │ 0          │
├─────────────────────────┼──────────┼────────────────────┼───────┼────────────┤
│ returncode_list_mismatc │ c5173a67 │ generic.local.bash │ FAIL  │ 2          │
│ h                       │          │                    │       │            │
├─────────────────────────┼──────────┼────────────────────┼───────┼────────────┤
│ returncode_list_mismatc │ 54a48aa7 │ generic.local.bash │ FAIL  │ 2          │
│ h                       │          │                    │       │            │
├─────────────────────────┼──────────┼────────────────────┼───────┼────────────┤
│ returncode_list_mismatc │ 96fc40ae │ generic.local.bash │ FAIL  │ 2          │
...

Filter Reports (buildtest report --filter)

The buildtest report command will display all tests results, which can be quite long depending on number of tests so therefore we need a mechanism to filter the test results. The --filter option can be used to filter out tests in the output based on filter fields. First, lets see the available filter fields by run buildtest report --helpfilter which shows a list of filter fields and their description.

buildtest report --helpfilter
$ buildtest report --helpfilter
                       Filter Fields                        
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Field      ┃ Description                ┃ Expected Value ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ buildspec  │ Filter by buildspec file   │ FILE           │
│ name       │ Filter by test name        │ STRING         │
│ executor   │ Filter by executor name    │ STRING         │
│ returncode │ Filter tests by returncode │ INT            │
│ state      │ Filter by test state       │ PASS/FAIL      │
│ tags       │ Filter tests by tag name   │ STRING         │
└────────────┴────────────────────────────┴────────────────┘

To print just the raw filter fields use the command buildtest report --filterfields.

buildtest report --filterfields
$ buildtest report --filterfields
buildspec
name
executor
returncode
state
tags

The --filter option expects arguments in key=value format. You can specify multiple filter delimited by comma. buildtest will treat multiple filters as logical AND operation. The filter option can be used with --format field. Let’s see some examples to illustrate the point.

Filter by returncode (--filter returncode)

If you want to retrieve all tests with a given returncode, we can use the returncode property. For instance, let’s retrieve all tests with returncode of 2 by setting --filter returncode=2.

buildtest rt --filter returncode=2 --format=name,id,returncode
$ buildtest rt --filter returncode=2 --format=name,id,returncode
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ name                                         ┃ id           ┃ returncode     ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ returncode_list_mismatch                     │ c5173a67     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ returncode_list_mismatch                     │ 54a48aa7     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ returncode_list_mismatch                     │ 96fc40ae     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ returncode_list_mismatch                     │ 120d7514     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ returncode_list_mismatch                     │ 9c53a81f     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ timelimit_min_fail                           │ 3ddcad13     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ timelimit_max_fail                           │ 02b32713     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ timelimit_max                                │ 55219611     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ timelimit_min                                │ 5deb36a4     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ timelimit_min_max                            │ f984dd75     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ always_pass                                  │ 013390a1     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ test_pass_returncode_mismatch                │ e818adcb     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ test_fail_returncode_match                   │ bc30b726     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ always_fail                                  │ 68750e2e     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ metric_regex_example                         │ 74311a1d     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ test2                                        │ 12cb5b6d     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ multiple_executors                           │ aa346ac6     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ executors_vars_env_declaration               │ 52156c51     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ executors_sbatch_declaration                 │ 4c5409ce     │ 2              │
├──────────────────────────────────────────────┼──────────────┼────────────────┤
│ status_returncode_by_executors               │ cdcb639e     │ 2              │
└──────────────────────────────────────────────┴──────────────┴────────────────┘

Note

buildtest automatically converts returncode to integer when matching returncode, so --filter returncode="2" will work too

Filter by test name (--filter name)

If you want to filter by test name, use the name attribute in filter option. Let’s assume we want to filter all tests by name exit1_pass, this can be achieved by setting filter field as follows: --filter name=exit1_pass. Shown below is an example using name filter field to filter test results.

buildtest rt --filter name=exit1_pass --format=name,id,returncode,state
$ buildtest rt --filter name=exit1_pass --format=name,id,returncode,state
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ name                 ┃ id                ┃ returncode           ┃ state      ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ exit1_pass           │ 1f716bb6          │ 1                    │ PASS       │
├──────────────────────┼───────────────────┼──────────────────────┼────────────┤
│ exit1_pass           │ f24c7458          │ 1                    │ PASS       │
├──────────────────────┼───────────────────┼──────────────────────┼────────────┤
│ exit1_pass           │ 36dc3774          │ 1                    │ PASS       │
├──────────────────────┼───────────────────┼──────────────────────┼────────────┤
│ exit1_pass           │ d71a7099          │ 1                    │ PASS       │
├──────────────────────┼───────────────────┼──────────────────────┼────────────┤
│ exit1_pass           │ 7187873d          │ 1                    │ PASS       │
├──────────────────────┼───────────────────┼──────────────────────┼────────────┤
│ exit1_pass           │ ce86fe2e          │ 1                    │ PASS       │
└──────────────────────┴───────────────────┴──────────────────────┴────────────┘

Filter by buildspec (--filter buildspec)

Likewise, we can filter results by buildspec file using buildspec attribute via --filter buildspec=<file>. The buildspec attribute must resolve to a file path which can be relative or absolute path. buildtest will resolve path (absolute path) and find the appropriate tests that belong to the buildspec file. If file doesn’t exist or is not found in cache it will raise an error.

buildtest rt --filter buildspec=tutorials/python-hello.yml --format=name,id,state,buildspec
$ buildtest rt --filter buildspec=tutorials/python-hello.yml --format=name,id,state,buildspec
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
┏━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name         ┃ id       ┃ state ┃ buildspec                                  ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ python_hello │ 0aa69331 │ PASS  │ /home/docs/checkouts/readthedocs.org/user_ │
│              │          │       │ builds/buildtest/checkouts/devel/tutorials │
│              │          │       │ /python-hello.yml                          │
├──────────────┼──────────┼───────┼────────────────────────────────────────────┤
│ python_hello │ 03af71fc │ PASS  │ /home/docs/checkouts/readthedocs.org/user_ │
│              │          │       │ builds/buildtest/checkouts/devel/tutorials │
│              │          │       │ /python-hello.yml                          │
├──────────────┼──────────┼───────┼────────────────────────────────────────────┤
│ python_hello │ 237968f1 │ PASS  │ /home/docs/checkouts/readthedocs.org/user_ │
│              │          │       │ builds/buildtest/checkouts/devel/tutorials │
│              │          │       │ /python-hello.yml                          │
└──────────────┴──────────┴───────┴────────────────────────────────────────────┘

Filter by test state (--filter state)

If you want to filter results by test state, use the state property. This can be useful if you want to know all pass or failed tests. The state property expects value of [PASS|FAIL] since these are the two recorded test states marked by buildtest. We can also pass multiple filter fields for instance if we want to find all FAIL tests for executor generic.local.sh we can do the following.

buildtest rt --filter state=FAIL,executor=generic.local.sh --format=name,id,state,executor
$ buildtest rt --filter state=FAIL,executor=generic.local.sh --format=name,id,state,executor
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
┃ name                               ┃ id         ┃ state  ┃ executor          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│ timelimit_min_fail                 │ 3ddcad13   │ FAIL   │ generic.local.sh  │
├────────────────────────────────────┼────────────┼────────┼───────────────────┤
│ timelimit_min                      │ 5deb36a4   │ FAIL   │ generic.local.sh  │
├────────────────────────────────────┼────────────┼────────┼───────────────────┤
│ timelimit_min_max                  │ f984dd75   │ FAIL   │ generic.local.sh  │
├────────────────────────────────────┼────────────┼────────┼───────────────────┤
│ test_fail_returncode_match         │ bc30b726   │ FAIL   │ generic.local.sh  │
├────────────────────────────────────┼────────────┼────────┼───────────────────┤
│ always_fail                        │ 68750e2e   │ FAIL   │ generic.local.sh  │
├────────────────────────────────────┼────────────┼────────┼───────────────────┤
│ metric_regex_example               │ 74311a1d   │ FAIL   │ generic.local.sh  │
├────────────────────────────────────┼────────────┼────────┼───────────────────┤
│ multiple_executors                 │ aa346ac6   │ FAIL   │ generic.local.sh  │
├────────────────────────────────────┼────────────┼────────┼───────────────────┤
│ executors_vars_env_declaration     │ 52156c51   │ FAIL   │ generic.local.sh  │
├────────────────────────────────────┼────────────┼────────┼───────────────────┤
│ executors_sbatch_declaration       │ 4c5409ce   │ FAIL   │ generic.local.sh  │
├────────────────────────────────────┼────────────┼────────┼───────────────────┤
│ status_returncode_by_executors     │ cdcb639e   │ FAIL   │ generic.local.sh  │
└────────────────────────────────────┴────────────┴────────┴───────────────────┘

Filter Exception Cases

The returncode filter field expects an integer value, so if you try a non-integer returncode you will get the following message

buildtest rt --filter returncode=1.5
$ buildtest rt --filter returncode=1.5
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ in/buildtest:36 in <module>                                                  │
│                                                                              │
│   33                                                                         │
│   34 import buildtest.main                                                   │
│   35                                                                         │
│ ❱ 36 buildtest.main.main()                                                   │
│   37                                                                         │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ __annotations__ = {}                                                     │ │
│ │    __builtins__ = <module 'builtins' (built-in)>                         │ │
│ │      __cached__ = None                                                   │ │
│ │         __doc__ = ':"\n# preferred pythons for running                   │ │
│ │                   buildtest\nPREFERRED_PYTHONS="python3 python"\n'+365   │ │
│ │        __file__ = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │      __loader__ = <_frozen_importlib_external.SourceFileLoader object at │ │
│ │                   0x7f9f658941d0>                                        │ │
│ │        __name__ = '__main__'                                             │ │
│ │     __package__ = None                                                   │ │
│ │        __spec__ = None                                                   │ │
│ │       buildtest = <module 'buildtest' from                               │ │
│ │                   '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │  buildtest_file = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │              os = <module 'os' from                                      │ │
│ │                   '/home/docs/.asdf/installs/python/3.7.13/lib/python3.… │ │
│ │          prefix = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │             sys = <module 'sys' (built-in)>                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/main.py:247 in main                                                 │
│                                                                              │
│   244 │                                                                      │
│   245 │   # buildtest report                                                 │
│   246 │   elif args.subcommands in ["report", "rt"]:                         │
│ ❱ 247 │   │   report_cmd(args, report_file)                                  │
│   248 │                                                                      │
│   249 │   elif args.subcommands == "path":                                   │
│   250 │   │   path_cmd(                                                      │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │               args = Namespace(color=None, configfile=None, count=None,  │ │
│ │                      debug=False, editor=None, end=None, fail=False,     │ │
│ │                      filter={'returncode': '1.5'}, filterfields=False,   │ │
│ │                      format=None, formatfields=False, helpfilter=False,  │ │
│ │                      helpformat=False, latest=False, logpath=False,      │ │
│ │                      no_color=False, no_header=False, oldest=False,      │ │
│ │                      pager=False, passed=False, print_log=False,         │ │
│ │                      report=None, report_subcommand=None,                │ │
│ │                      row_count=False, start=None, subcommands='rt',      │ │
│ │                      terse=False, view_log=False)                        │ │
│ │   buildtest_editor = '/usr/bin/vim'                                      │ │
│ │        config_file = None                                                │ │
│ │      configuration = <buildtest.config.SiteConfiguration object at       │ │
│ │                      0x7f9f63c24310>                                     │ │
│ │             logger = <Logger buildtest (DEBUG)>                          │ │
│ │           no_color = False                                               │ │
│ │             parser = ArgumentParser(prog='buildtest', usage='%(prog)s    │ │
│ │                      [options] [COMMANDS]', description='buildtest is a  │ │
│ │                      HPC testing framework for building and running      │ │
│ │                      tests.', formatter_class=<class                     │ │
│ │                      'argparse.RawDescriptionHelpFormatter'>,            │ │
│ │                      conflict_handler='error', add_help=True)            │ │
│ │        report_file = None                                                │ │
│ │             system = <buildtest.system.BuildTestSystem object at         │ │
│ │                      0x7f9f63865e10>                                     │ │
│ │ validate_executors = True                                                │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/cli/report.py:810 in report_cmd                                     │
│                                                                              │
│   807 │   │   latest=args.latest,                                            │
│   808 │   │   oldest=args.oldest,                                            │
│   809 │   │   report_file=report_file,                                       │
│ ❱ 810 │   │   count=args.count,                                              │
│   811 │   )                                                                  │
│   812 │                                                                      │
│   813 │   if args.report_subcommand in ["path", "p"]:                        │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │         args = Namespace(color=None, configfile=None, count=None,        │ │
│ │                debug=False, editor=None, end=None, fail=False,           │ │
│ │                filter={'returncode': '1.5'}, filterfields=False,         │ │
│ │                format=None, formatfields=False, helpfilter=False,        │ │
│ │                helpformat=False, latest=False, logpath=False,            │ │
│ │                no_color=False, no_header=False, oldest=False,            │ │
│ │                pager=False, passed=False, print_log=False, report=None,  │ │
│ │                report_subcommand=None, row_count=False, start=None,      │ │
│ │                subcommands='rt', terse=False, view_log=False)            │ │
│ │ consoleColor = 'default'                                                 │ │
│ │  report_file = None                                                      │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/cli/report.py:131 in __init__                                       │
│                                                                              │
│   128 │   │   │   self._reportfile = resolve_path(report_file)               │
│   129 │   │                                                                  │
│   130 │   │   self.report = self.load()                                      │
│ ❱ 131 │   │   self._check_filter_fields()                                    │
│   132 │   │   self._check_format_fields()                                    │
│   133 │   │   self._check_start_and_end_fields()                             │
│   134 │   │   self.filter_buildspecs_from_report()                           │
│                                                                              │
│ ╭─────────────────────────────── locals ───────────────────────────────╮     │
│ │       color = None                                                   │     │
│ │       count = None                                                   │     │
│ │    detailed = None                                                   │     │
│ │         end = None                                                   │     │
│ │     failure = False                                                  │     │
│ │ filter_args = {'returncode': '1.5'}                                  │     │
│ │ format_args = None                                                   │     │
│ │      latest = False                                                  │     │
│ │      oldest = False                                                  │     │
│ │       pager = None                                                   │     │
│ │      passed = False                                                  │     │
│ │ report_file = None                                                   │     │
│ │        self = <buildtest.cli.report.Report object at 0x7f9f637b4dd0> │     │
│ │       start = None                                                   │     │
│ ╰──────────────────────────────────────────────────────────────────────╯     │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/cli/report.py:169 in _check_filter_fields                           │
│                                                                              │
│   166 │   │   │   │                                                          │
│   167 │   │   │   │   if key == "returncode" and not is_int(self.filter[key] │
│   168 │   │   │   │   │   raise BuildTestError(                              │
│ ❱ 169 │   │   │   │   │   │   f"Invalid returncode:{self.filter[key]} must b │
│   170 │   │   │   │   │   )                                                  │
│   171 │   │   │   │                                                          │
│   172 │   │   │   │   logger.debug(f"filter field: {key} is valid")          │
│                                                                              │
│ ╭─────────────────────────── locals ────────────────────────────╮            │
│ │  key = 'returncode'                                           │            │
│ │ self = <buildtest.cli.report.Report object at 0x7f9f637b4dd0> │            │
│ ╰───────────────────────────────────────────────────────────────╯            │
╰──────────────────────────────────────────────────────────────────────────────╯
BuildTestError: 'Invalid returncode:1.5 must be an integer'

The state filter field expects value of PASS or FAIL so if you specify an invalid state you will get an error as follows.

buildtest rt --filter state=UNKNOWN
$ buildtest rt --filter state=UNKNOWN
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ in/buildtest:36 in <module>                                                  │
│                                                                              │
│   33                                                                         │
│   34 import buildtest.main                                                   │
│   35                                                                         │
│ ❱ 36 buildtest.main.main()                                                   │
│   37                                                                         │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ __annotations__ = {}                                                     │ │
│ │    __builtins__ = <module 'builtins' (built-in)>                         │ │
│ │      __cached__ = None                                                   │ │
│ │         __doc__ = ':"\n# preferred pythons for running                   │ │
│ │                   buildtest\nPREFERRED_PYTHONS="python3 python"\n'+365   │ │
│ │        __file__ = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │      __loader__ = <_frozen_importlib_external.SourceFileLoader object at │ │
│ │                   0x7f34790351d0>                                        │ │
│ │        __name__ = '__main__'                                             │ │
│ │     __package__ = None                                                   │ │
│ │        __spec__ = None                                                   │ │
│ │       buildtest = <module 'buildtest' from                               │ │
│ │                   '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │  buildtest_file = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │              os = <module 'os' from                                      │ │
│ │                   '/home/docs/.asdf/installs/python/3.7.13/lib/python3.… │ │
│ │          prefix = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │             sys = <module 'sys' (built-in)>                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/main.py:247 in main                                                 │
│                                                                              │
│   244 │                                                                      │
│   245 │   # buildtest report                                                 │
│   246 │   elif args.subcommands in ["report", "rt"]:                         │
│ ❱ 247 │   │   report_cmd(args, report_file)                                  │
│   248 │                                                                      │
│   249 │   elif args.subcommands == "path":                                   │
│   250 │   │   path_cmd(                                                      │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │               args = Namespace(color=None, configfile=None, count=None,  │ │
│ │                      debug=False, editor=None, end=None, fail=False,     │ │
│ │                      filter={'state': 'UNKNOWN'}, filterfields=False,    │ │
│ │                      format=None, formatfields=False, helpfilter=False,  │ │
│ │                      helpformat=False, latest=False, logpath=False,      │ │
│ │                      no_color=False, no_header=False, oldest=False,      │ │
│ │                      pager=False, passed=False, print_log=False,         │ │
│ │                      report=None, report_subcommand=None,                │ │
│ │                      row_count=False, start=None, subcommands='rt',      │ │
│ │                      terse=False, view_log=False)                        │ │
│ │   buildtest_editor = '/usr/bin/vim'                                      │ │
│ │        config_file = None                                                │ │
│ │      configuration = <buildtest.config.SiteConfiguration object at       │ │
│ │                      0x7f347700f9d0>                                     │ │
│ │             logger = <Logger buildtest (DEBUG)>                          │ │
│ │           no_color = False                                               │ │
│ │             parser = ArgumentParser(prog='buildtest', usage='%(prog)s    │ │
│ │                      [options] [COMMANDS]', description='buildtest is a  │ │
│ │                      HPC testing framework for building and running      │ │
│ │                      tests.', formatter_class=<class                     │ │
│ │                      'argparse.RawDescriptionHelpFormatter'>,            │ │
│ │                      conflict_handler='error', add_help=True)            │ │
│ │        report_file = None                                                │ │
│ │             system = <buildtest.system.BuildTestSystem object at         │ │
│ │                      0x7f3477000d10>                                     │ │
│ │ validate_executors = True                                                │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/cli/report.py:810 in report_cmd                                     │
│                                                                              │
│   807 │   │   latest=args.latest,                                            │
│   808 │   │   oldest=args.oldest,                                            │
│   809 │   │   report_file=report_file,                                       │
│ ❱ 810 │   │   count=args.count,                                              │
│   811 │   )                                                                  │
│   812 │                                                                      │
│   813 │   if args.report_subcommand in ["path", "p"]:                        │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │         args = Namespace(color=None, configfile=None, count=None,        │ │
│ │                debug=False, editor=None, end=None, fail=False,           │ │
│ │                filter={'state': 'UNKNOWN'}, filterfields=False,          │ │
│ │                format=None, formatfields=False, helpfilter=False,        │ │
│ │                helpformat=False, latest=False, logpath=False,            │ │
│ │                no_color=False, no_header=False, oldest=False,            │ │
│ │                pager=False, passed=False, print_log=False, report=None,  │ │
│ │                report_subcommand=None, row_count=False, start=None,      │ │
│ │                subcommands='rt', terse=False, view_log=False)            │ │
│ │ consoleColor = 'default'                                                 │ │
│ │  report_file = None                                                      │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/cli/report.py:134 in __init__                                       │
│                                                                              │
│   131 │   │   self._check_filter_fields()                                    │
│   132 │   │   self._check_format_fields()                                    │
│   133 │   │   self._check_start_and_end_fields()                             │
│ ❱ 134 │   │   self.filter_buildspecs_from_report()                           │
│   135 │   │                                                                  │
│   136 │   │   self.process_report()                                          │
│   137                                                                        │
│                                                                              │
│ ╭─────────────────────────────── locals ───────────────────────────────╮     │
│ │       color = None                                                   │     │
│ │       count = None                                                   │     │
│ │    detailed = None                                                   │     │
│ │         end = None                                                   │     │
│ │     failure = False                                                  │     │
│ │ filter_args = {'state': 'UNKNOWN'}                                   │     │
│ │ format_args = None                                                   │     │
│ │      latest = False                                                  │     │
│ │      oldest = False                                                  │     │
│ │       pager = None                                                   │     │
│ │      passed = False                                                  │     │
│ │ report_file = None                                                   │     │
│ │        self = <buildtest.cli.report.Report object at 0x7f3477bf1ed0> │     │
│ │       start = None                                                   │     │
│ ╰──────────────────────────────────────────────────────────────────────╯     │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/cli/report.py:303 in filter_buildspecs_from_report                  │
│                                                                              │
│   300 │   │   if self.filter.get("state"):                                   │
│   301 │   │   │   if self.filter["state"] not in ["PASS", "FAIL"]:           │
│   302 │   │   │   │   raise BuildTestError(                                  │
│ ❱ 303 │   │   │   │   │   f"filter argument 'state' must be 'PASS' or 'FAIL' │
│   304 │   │   │   │   )                                                      │
│   305 │                                                                      │
│   306 │   def filter_by_start_end(self, test):                               │
│                                                                              │
│ ╭─────────────────────────── locals ────────────────────────────╮            │
│ │ self = <buildtest.cli.report.Report object at 0x7f3477bf1ed0> │            │
│ ╰───────────────────────────────────────────────────────────────╯            │
╰──────────────────────────────────────────────────────────────────────────────╯
BuildTestError: "filter argument 'state' must be 'PASS' or 'FAIL' got value 
UNKNOWN"

The buildspec field expects a valid file path, it can be an absolute or relative path, buildtest will resolve absolute path and check if file exist and is in the report file. If it’s an invalid file we get an error such as

buildtest rt --filter buildspec=/path/to/invalid.yml
$ buildtest rt --filter buildspec=/path/to/invalid.yml
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ in/buildtest:36 in <module>                                                  │
│                                                                              │
│   33                                                                         │
│   34 import buildtest.main                                                   │
│   35                                                                         │
│ ❱ 36 buildtest.main.main()                                                   │
│   37                                                                         │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ __annotations__ = {}                                                     │ │
│ │    __builtins__ = <module 'builtins' (built-in)>                         │ │
│ │      __cached__ = None                                                   │ │
│ │         __doc__ = ':"\n# preferred pythons for running                   │ │
│ │                   buildtest\nPREFERRED_PYTHONS="python3 python"\n'+365   │ │
│ │        __file__ = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │      __loader__ = <_frozen_importlib_external.SourceFileLoader object at │ │
│ │                   0x7f13051fd190>                                        │ │
│ │        __name__ = '__main__'                                             │ │
│ │     __package__ = None                                                   │ │
│ │        __spec__ = None                                                   │ │
│ │       buildtest = <module 'buildtest' from                               │ │
│ │                   '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │  buildtest_file = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │              os = <module 'os' from                                      │ │
│ │                   '/home/docs/.asdf/installs/python/3.7.13/lib/python3.… │ │
│ │          prefix = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │             sys = <module 'sys' (built-in)>                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/main.py:247 in main                                                 │
│                                                                              │
│   244 │                                                                      │
│   245 │   # buildtest report                                                 │
│   246 │   elif args.subcommands in ["report", "rt"]:                         │
│ ❱ 247 │   │   report_cmd(args, report_file)                                  │
│   248 │                                                                      │
│   249 │   elif args.subcommands == "path":                                   │
│   250 │   │   path_cmd(                                                      │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │               args = Namespace(color=None, configfile=None, count=None,  │ │
│ │                      debug=False, editor=None, end=None, fail=False,     │ │
│ │                      filter={'buildspec': '/path/to/invalid.yml'},       │ │
│ │                      filterfields=False, format=None,                    │ │
│ │                      formatfields=False, helpfilter=False,               │ │
│ │                      helpformat=False, latest=False, logpath=False,      │ │
│ │                      no_color=False, no_header=False, oldest=False,      │ │
│ │                      pager=False, passed=False, print_log=False,         │ │
│ │                      report=None, report_subcommand=None,                │ │
│ │                      row_count=False, start=None, subcommands='rt',      │ │
│ │                      terse=False, view_log=False)                        │ │
│ │   buildtest_editor = '/usr/bin/vim'                                      │ │
│ │        config_file = None                                                │ │
│ │      configuration = <buildtest.config.SiteConfiguration object at       │ │
│ │                      0x7f13031c77d0>                                     │ │
│ │             logger = <Logger buildtest (DEBUG)>                          │ │
│ │           no_color = False                                               │ │
│ │             parser = ArgumentParser(prog='buildtest', usage='%(prog)s    │ │
│ │                      [options] [COMMANDS]', description='buildtest is a  │ │
│ │                      HPC testing framework for building and running      │ │
│ │                      tests.', formatter_class=<class                     │ │
│ │                      'argparse.RawDescriptionHelpFormatter'>,            │ │
│ │                      conflict_handler='error', add_help=True)            │ │
│ │        report_file = None                                                │ │
│ │             system = <buildtest.system.BuildTestSystem object at         │ │
│ │                      0x7f13034f9810>                                     │ │
│ │ validate_executors = True                                                │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/cli/report.py:810 in report_cmd                                     │
│                                                                              │
│   807 │   │   latest=args.latest,                                            │
│   808 │   │   oldest=args.oldest,                                            │
│   809 │   │   report_file=report_file,                                       │
│ ❱ 810 │   │   count=args.count,                                              │
│   811 │   )                                                                  │
│   812 │                                                                      │
│   813 │   if args.report_subcommand in ["path", "p"]:                        │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │         args = Namespace(color=None, configfile=None, count=None,        │ │
│ │                debug=False, editor=None, end=None, fail=False,           │ │
│ │                filter={'buildspec': '/path/to/invalid.yml'},             │ │
│ │                filterfields=False, format=None, formatfields=False,      │ │
│ │                helpfilter=False, helpformat=False, latest=False,         │ │
│ │                logpath=False, no_color=False, no_header=False,           │ │
│ │                oldest=False, pager=False, passed=False, print_log=False, │ │
│ │                report=None, report_subcommand=None, row_count=False,     │ │
│ │                start=None, subcommands='rt', terse=False,                │ │
│ │                view_log=False)                                           │ │
│ │ consoleColor = 'default'                                                 │ │
│ │  report_file = None                                                      │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/cli/report.py:134 in __init__                                       │
│                                                                              │
│   131 │   │   self._check_filter_fields()                                    │
│   132 │   │   self._check_format_fields()                                    │
│   133 │   │   self._check_start_and_end_fields()                             │
│ ❱ 134 │   │   self.filter_buildspecs_from_report()                           │
│   135 │   │                                                                  │
│   136 │   │   self.process_report()                                          │
│   137                                                                        │
│                                                                              │
│ ╭─────────────────────────────── locals ───────────────────────────────╮     │
│ │       color = None                                                   │     │
│ │       count = None                                                   │     │
│ │    detailed = None                                                   │     │
│ │         end = None                                                   │     │
│ │     failure = False                                                  │     │
│ │ filter_args = {'buildspec': '/path/to/invalid.yml'}                  │     │
│ │ format_args = None                                                   │     │
│ │      latest = False                                                  │     │
│ │      oldest = False                                                  │     │
│ │       pager = None                                                   │     │
│ │      passed = False                                                  │     │
│ │ report_file = None                                                   │     │
│ │        self = <buildtest.cli.report.Report object at 0x7f1303598390> │     │
│ │       start = None                                                   │     │
│ ╰──────────────────────────────────────────────────────────────────────╯     │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/cli/report.py:287 in filter_buildspecs_from_report                  │
│                                                                              │
│   284 │   │   │   # if file doesn't exist we terminate with message          │
│   285 │   │   │   if not resolved_buildspecs:                                │
│   286 │   │   │   │   raise BuildTestError(                                  │
│ ❱ 287 │   │   │   │   │   f"Invalid File Path for filter field 'buildspec':  │
│   288 │   │   │   │   )                                                      │
│   289 │   │   │                                                              │
│   290 │   │   │   # if file not found in cache we exit                       │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ resolved_buildspecs = None                                               │ │
│ │                self = <buildtest.cli.report.Report object at             │ │
│ │                       0x7f1303598390>                                    │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
BuildTestError: "Invalid File Path for filter field 'buildspec': 
/path/to/invalid.yml"

You may have a valid filepath for buildspec filter field such as $BUILDTEST_ROOT/tutorials/invalid_executor.yml, but there is no record of a test in the report cache because this test wasn’t run. In this case you will get the following message.

buildtest rt --filter buildspec=$BUILDTEST_ROOT/tutorials/invalid_executor.yml
$ buildtest rt --filter buildspec=$BUILDTEST_ROOT/tutorials/invalid_executor.yml
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ in/buildtest:36 in <module>                                                  │
│                                                                              │
│   33                                                                         │
│   34 import buildtest.main                                                   │
│   35                                                                         │
│ ❱ 36 buildtest.main.main()                                                   │
│   37                                                                         │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ __annotations__ = {}                                                     │ │
│ │    __builtins__ = <module 'builtins' (built-in)>                         │ │
│ │      __cached__ = None                                                   │ │
│ │         __doc__ = ':"\n# preferred pythons for running                   │ │
│ │                   buildtest\nPREFERRED_PYTHONS="python3 python"\n'+365   │ │
│ │        __file__ = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │      __loader__ = <_frozen_importlib_external.SourceFileLoader object at │ │
│ │                   0x7f5a1d427110>                                        │ │
│ │        __name__ = '__main__'                                             │ │
│ │     __package__ = None                                                   │ │
│ │        __spec__ = None                                                   │ │
│ │       buildtest = <module 'buildtest' from                               │ │
│ │                   '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │  buildtest_file = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │              os = <module 'os' from                                      │ │
│ │                   '/home/docs/.asdf/installs/python/3.7.13/lib/python3.… │ │
│ │          prefix = '/home/docs/checkouts/readthedocs.org/user_builds/bui… │ │
│ │             sys = <module 'sys' (built-in)>                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/main.py:247 in main                                                 │
│                                                                              │
│   244 │                                                                      │
│   245 │   # buildtest report                                                 │
│   246 │   elif args.subcommands in ["report", "rt"]:                         │
│ ❱ 247 │   │   report_cmd(args, report_file)                                  │
│   248 │                                                                      │
│   249 │   elif args.subcommands == "path":                                   │
│   250 │   │   path_cmd(                                                      │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │               args = Namespace(color=None, configfile=None, count=None,  │ │
│ │                      debug=False, editor=None, end=None, fail=False,     │ │
│ │                      filter={'buildspec':                                │ │
│ │                      '$BUILDTEST_ROOT/tutorials/invalid_executor.yml'},  │ │
│ │                      filterfields=False, format=None,                    │ │
│ │                      formatfields=False, helpfilter=False,               │ │
│ │                      helpformat=False, latest=False, logpath=False,      │ │
│ │                      no_color=False, no_header=False, oldest=False,      │ │
│ │                      pager=False, passed=False, print_log=False,         │ │
│ │                      report=None, report_subcommand=None,                │ │
│ │                      row_count=False, start=None, subcommands='rt',      │ │
│ │                      terse=False, view_log=False)                        │ │
│ │   buildtest_editor = '/usr/bin/vim'                                      │ │
│ │        config_file = None                                                │ │
│ │      configuration = <buildtest.config.SiteConfiguration object at       │ │
│ │                      0x7f5a1b7b7510>                                     │ │
│ │             logger = <Logger buildtest (DEBUG)>                          │ │
│ │           no_color = False                                               │ │
│ │             parser = ArgumentParser(prog='buildtest', usage='%(prog)s    │ │
│ │                      [options] [COMMANDS]', description='buildtest is a  │ │
│ │                      HPC testing framework for building and running      │ │
│ │                      tests.', formatter_class=<class                     │ │
│ │                      'argparse.RawDescriptionHelpFormatter'>,            │ │
│ │                      conflict_handler='error', add_help=True)            │ │
│ │        report_file = None                                                │ │
│ │             system = <buildtest.system.BuildTestSystem object at         │ │
│ │                      0x7f5a1b3915d0>                                     │ │
│ │ validate_executors = True                                                │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/cli/report.py:810 in report_cmd                                     │
│                                                                              │
│   807 │   │   latest=args.latest,                                            │
│   808 │   │   oldest=args.oldest,                                            │
│   809 │   │   report_file=report_file,                                       │
│ ❱ 810 │   │   count=args.count,                                              │
│   811 │   )                                                                  │
│   812 │                                                                      │
│   813 │   if args.report_subcommand in ["path", "p"]:                        │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │         args = Namespace(color=None, configfile=None, count=None,        │ │
│ │                debug=False, editor=None, end=None, fail=False,           │ │
│ │                filter={'buildspec':                                      │ │
│ │                '$BUILDTEST_ROOT/tutorials/invalid_executor.yml'},        │ │
│ │                filterfields=False, format=None, formatfields=False,      │ │
│ │                helpfilter=False, helpformat=False, latest=False,         │ │
│ │                logpath=False, no_color=False, no_header=False,           │ │
│ │                oldest=False, pager=False, passed=False, print_log=False, │ │
│ │                report=None, report_subcommand=None, row_count=False,     │ │
│ │                start=None, subcommands='rt', terse=False,                │ │
│ │                view_log=False)                                           │ │
│ │ consoleColor = 'default'                                                 │ │
│ │  report_file = None                                                      │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/cli/report.py:134 in __init__                                       │
│                                                                              │
│   131 │   │   self._check_filter_fields()                                    │
│   132 │   │   self._check_format_fields()                                    │
│   133 │   │   self._check_start_and_end_fields()                             │
│ ❱ 134 │   │   self.filter_buildspecs_from_report()                           │
│   135 │   │                                                                  │
│   136 │   │   self.process_report()                                          │
│   137                                                                        │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │       color = None                                                       │ │
│ │       count = None                                                       │ │
│ │    detailed = None                                                       │ │
│ │         end = None                                                       │ │
│ │     failure = False                                                      │ │
│ │ filter_args = {                                                          │ │
│ │               │   'buildspec':                                           │ │
│ │               '$BUILDTEST_ROOT/tutorials/invalid_executor.yml'           │ │
│ │               }                                                          │ │
│ │ format_args = None                                                       │ │
│ │      latest = False                                                      │ │
│ │      oldest = False                                                      │ │
│ │       pager = None                                                       │ │
│ │      passed = False                                                      │ │
│ │ report_file = None                                                       │ │
│ │        self = <buildtest.cli.report.Report object at 0x7f5a1b723410>     │ │
│ │       start = None                                                       │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/b │
│ uildtest/cli/report.py:293 in filter_buildspecs_from_report                  │
│                                                                              │
│   290 │   │   │   # if file not found in cache we exit                       │
│   291 │   │   │   if not resolved_buildspecs in self.report.keys():          │
│   292 │   │   │   │   raise BuildTestError(                                  │
│ ❱ 293 │   │   │   │   │   f"buildspec file: {resolved_buildspecs} not found  │
│   294 │   │   │   │   )                                                      │
│   295 │   │   │                                                              │
│   296 │   │   │   # need to set as a list since we will loop over all tests  │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ resolved_buildspecs = '/home/docs/checkouts/readthedocs.org/user_builds… │ │
│ │                self = <buildtest.cli.report.Report object at             │ │
│ │                       0x7f5a1b723410>                                    │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
BuildTestError: 'buildspec file: 
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutor
ials/invalid_executor.yml not found in cache'

Find Latest or Oldest test

We can search for oldest or latest test for any given test. This can be useful if you want to see first or last test run for a particular test. If you want to retrieve the oldest test you can use --oldest option, likewise you can retrieve the latest run via --latest option.

Let’s take a look at this example, we filter by test name exit1_pass which retrieves all test runs. In subsequent example we filter by latest and oldest run.

Show oldest run
$ buildtest report --filter name=exit1_pass --format name,id,starttime
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                 ┃ id               ┃ starttime                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ exit1_pass           │ 1f716bb6         │ 2022/10/07 18:35:08                │
├──────────────────────┼──────────────────┼────────────────────────────────────┤
│ exit1_pass           │ f24c7458         │ 2022/10/07 18:36:12                │
├──────────────────────┼──────────────────┼────────────────────────────────────┤
│ exit1_pass           │ 36dc3774         │ 2022/10/07 18:36:12                │
├──────────────────────┼──────────────────┼────────────────────────────────────┤
│ exit1_pass           │ d71a7099         │ 2022/10/07 18:36:25                │
├──────────────────────┼──────────────────┼────────────────────────────────────┤
│ exit1_pass           │ 7187873d         │ 2022/10/07 18:36:32                │
├──────────────────────┼──────────────────┼────────────────────────────────────┤
│ exit1_pass           │ ce86fe2e         │ 2022/10/07 18:36:32                │
└──────────────────────┴──────────────────┴────────────────────────────────────┘
$ buildtest report --filter name=exit1_pass --format name,id,starttime --oldest
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                 ┃ id               ┃ starttime                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ exit1_pass           │ 1f716bb6         │ 2022/10/07 18:35:08                │
└──────────────────────┴──────────────────┴────────────────────────────────────┘
Show latest run
$ buildtest report --filter name=exit1_pass --format name,id,starttime --latest
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                 ┃ id               ┃ starttime                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ exit1_pass           │ ce86fe2e         │ 2022/10/07 18:36:32                │
└──────────────────────┴──────────────────┴────────────────────────────────────┘

You may combine –oldest and –latest options in same command, in this case buildtest will retrieve the first and last record of every test.

Show oldest and latest run
$ buildtest report --filter name=exit1_pass --format name,id,starttime --oldest --latest
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                 ┃ id               ┃ starttime                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ exit1_pass           │ 1f716bb6         │ 2022/10/07 18:35:08                │
├──────────────────────┼──────────────────┼────────────────────────────────────┤
│ exit1_pass           │ ce86fe2e         │ 2022/10/07 18:36:32                │
└──────────────────────┴──────────────────┴────────────────────────────────────┘

Find all Failed Tests (buildtest report --fail)

The buildtest report --fail command can be used to retrieve all failed tests which is equivalent to filtering tests by state=FAIL since test state is determined by state property. This command can be useful to pin-point failures.

Let’s take a look at next two example, the first command will query report and limit output to 6 rows which can retrieve both pass and failed test. The second command will run same example with --fail and only retrieve failed tests. Take note of the state property in table, in second example buildtest will filter test and report all FAIL tests.

Show fail tests
$ buildtest report --format name,id,state --count=6
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ name                                         ┃ id               ┃ state      ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ variables_bash                               │ c70320c7         │ PASS       │
├──────────────────────────────────────────────┼──────────────────┼────────────┤
│ variables_bash                               │ 532e3e6a         │ PASS       │
├──────────────────────────────────────────────┼──────────────────┼────────────┤
│ variables_bash                               │ b8a43f49         │ PASS       │
├──────────────────────────────────────────────┼──────────────────┼────────────┤
│ variables_bash                               │ 7f5728b7         │ PASS       │
├──────────────────────────────────────────────┼──────────────────┼────────────┤
│ variables_bash                               │ eb1dcd91         │ PASS       │
├──────────────────────────────────────────────┼──────────────────┼────────────┤
│ returncode_list_mismatch                     │ c5173a67         │ FAIL       │
└──────────────────────────────────────────────┴──────────────────┴────────────┘
$ buildtest report --format name,id,state --fail --count=6
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ name                                         ┃ id               ┃ state      ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ returncode_list_mismatch                     │ c5173a67         │ FAIL       │
├──────────────────────────────────────────────┼──────────────────┼────────────┤
│ returncode_list_mismatch                     │ 54a48aa7         │ FAIL       │
├──────────────────────────────────────────────┼──────────────────┼────────────┤
│ returncode_list_mismatch                     │ 96fc40ae         │ FAIL       │
├──────────────────────────────────────────────┼──────────────────┼────────────┤
│ returncode_list_mismatch                     │ 120d7514         │ FAIL       │
├──────────────────────────────────────────────┼──────────────────┼────────────┤
│ returncode_list_mismatch                     │ 9c53a81f         │ FAIL       │
├──────────────────────────────────────────────┼──────────────────┼────────────┤
│ exit1_fail                                   │ bf2d9f82         │ FAIL       │
└──────────────────────────────────────────────┴──────────────────┴────────────┘

Find all Passed Tests (buildtest report --pass)

The buildtest report --pass, works similar to buildtest report --fail where it will filter test by state=PASS which can be used to find all passed tests.

We can see in example below, that buildtest will retrieve only PASS tests which can be determined by the state property

Show pass tests
$ buildtest report --format name,id,state --pass --count=6
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ name                                      ┃ id                 ┃ state       ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ variables_bash                            │ c70320c7           │ PASS        │
├───────────────────────────────────────────┼────────────────────┼─────────────┤
│ variables_bash                            │ 532e3e6a           │ PASS        │
├───────────────────────────────────────────┼────────────────────┼─────────────┤
│ variables_bash                            │ b8a43f49           │ PASS        │
├───────────────────────────────────────────┼────────────────────┼─────────────┤
│ variables_bash                            │ 7f5728b7           │ PASS        │
├───────────────────────────────────────────┼────────────────────┼─────────────┤
│ variables_bash                            │ eb1dcd91           │ PASS        │
├───────────────────────────────────────────┼────────────────────┼─────────────┤
│ returncode_int_match                      │ dbc1df78           │ PASS        │
└───────────────────────────────────────────┴────────────────────┴─────────────┘

Note

The --pass and --fail are mutually exclusive option which will query all PASS or FAIL test from report file, if you try to specify both options on command line you will get an error

$ buildtest report --pass --fail
usage: buildtest [options] [COMMANDS] report [-h] [--filter FILTER]
                                             [--format FORMAT] [--helpfilter]
                                             [--helpformat] [--filterfields]
                                             [--formatfields] [-f | -p]
                                             [-s START] [-e END] [--latest]
                                             [--oldest] [-c COUNT] [-n] [-t]
                                             [--row-count] [--color COLOR]
                                             [--pager]
                                             ...
buildtest [options] [COMMANDS] report: error: argument -f/--fail: not allowed with argument -p/--pass

Find Tests by Start and End Date (buildtest report --start --end)

The buildtest report --start and buildtest report --end command can be used to retrieve test records based on start and end date.

Let’s take a look at these two example, the first test queries report by filtering by state and name. The second command will run same example with --start --end option. Take note of the starttime and endtime properties in table, in second example buildtest will filter test and only report tests in the range of [start, end] dates.

Filter by start and end date
$ buildtest report --filter state=FAIL,name=exit1_fail --format name,state,starttime,endtime
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkout
                             s/1082/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                                                   ┃ state        ┃ starttime                                ┃ endtime                                 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ exit1_fail                                             │ FAIL         │ 2022/06/09 17:51:50                      │ 2022/06/09 17:51:50                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/09 17:51:51                      │ 2022/06/09 17:51:51                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/09 17:51:53                      │ 2022/06/09 17:51:53                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/09 17:52:01                      │ 2022/06/09 17:52:01                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/10 17:51:50                      │ 2022/06/10 17:51:50                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/10 17:51:52                      │ 2022/06/10 17:51:52                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/10 17:51:52                      │ 2022/06/10 17:51:52                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/10 17:52:01                      │ 2022/06/10 17:52:01                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/11 17:51:57                      │ 2022/06/11 17:51:57                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/11 17:51:59                      │ 2022/06/11 17:51:59                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/11 17:51:57                      │ 2022/06/11 17:51:57                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/11 17:51:59                      │ 2022/06/11 17:51:59                     │
└────────────────────────────────────────────────────────┴──────────────┴──────────────────────────────────────────┴─────────────────────────────────────────┘
$ buildtest report --filter state=FAIL,name=exit1_fail --format name,state,starttime,endtime --start 2022-06-10 --end 2022-06-11
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkout
                             s/1082/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                                                   ┃ state        ┃ starttime                                ┃ endtime                                 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ exit1_fail                                             │ FAIL         │ 2022/06/10 17:51:50                      │ 2022/06/10 17:51:50                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/10 17:51:52                      │ 2022/06/10 17:51:52                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/10 17:51:52                      │ 2022/06/10 17:51:52                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/10 17:52:01                      │ 2022/06/10 17:52:01                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/11 17:51:57                      │ 2022/06/11 17:51:57                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/11 17:51:59                      │ 2022/06/11 17:51:59                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/11 17:51:57                      │ 2022/06/11 17:51:57                     │
├────────────────────────────────────────────────────────┼──────────────┼──────────────────────────────────────────┼─────────────────────────────────────────┤
│ exit1_fail                                             │ FAIL         │ 2022/06/11 17:51:59                      │ 2022/06/11 17:51:59                     │
└────────────────────────────────────────────────────────┴──────────────┴──────────────────────────────────────────┴─────────────────────────────────────────┘

Find limited Tests (buildtest report --count)

The buildtest report command will show all test results from the report file and the output can be quite long. If you want to limit the number of records that get printed then you can use buildtest report --count where --count is number of rows that that get printed. Shown below is the output of 2 records in table format

$ buildtest report --count 2
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
┏━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃        ┃        ┃       ┃ return ┃ startt ┃ endtim ┃ runtim ┃        ┃ build ┃
┃ name   ┃ id     ┃ state ┃ code   ┃ ime    ┃ e      ┃ e      ┃ tags   ┃ spec  ┃
┡━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ variab │ c70320 │ PASS  │ 0      │ 2022/1 │ 2022/1 │ 0.0106 │ tutori │ /home │
│ les_ba │ c7     │       │        │ 0/07   │ 0/07   │ 26     │ als    │ /docs │
│ sh     │        │       │        │ 18:35: │ 18:35: │        │        │ /chec │
│        │        │       │        │ 07     │ 07     │        │        │ kouts │
│        │        │       │        │        │        │        │        │ /read │
│        │        │       │        │        │        │        │        │ thedo │
│        │        │       │        │        │        │        │        │ cs.or │
│        │        │       │        │        │        │        │        │ g/use │
│        │        │       │        │        │        │        │        │ r_bui │
│        │        │       │        │        │        │        │        │ lds/b │
│        │        │       │        │        │        │        │        │ uildt │
│        │        │       │        │        │        │        │        │ est/c │
│        │        │       │        │        │        │        │        │ hecko │
│        │        │       │        │        │        │        │        │ uts/d │
│        │        │       │        │        │        │        │        │ evel/ │
│        │        │       │        │        │        │        │        │ tutor │
│        │        │       │        │        │        │        │        │ ials/ │
│        │        │       │        │        │        │        │        │ vars. │
│        │        │       │        │        │        │        │        │ yml   │
├────────┼────────┼───────┼────────┼────────┼────────┼────────┼────────┼───────┤
│ variab │ 532e3e │ PASS  │ 0      │ 2022/1 │ 2022/1 │ 0.0086 │ tutori │ /home │
│ les_ba │ 6a     │       │        │ 0/07   │ 0/07   │ 06     │ als    │ /docs │
│ sh     │        │       │        │ 18:36: │ 18:36: │        │        │ /chec │
│        │        │       │        │ 06     │ 06     │        │        │ kouts │
│        │        │       │        │        │        │        │        │ /read │
│        │        │       │        │        │        │        │        │ thedo │
│        │        │       │        │        │        │        │        │ cs.or │
│        │        │       │        │        │        │        │        │ g/use │
│        │        │       │        │        │        │        │        │ r_bui │
│        │        │       │        │        │        │        │        │ lds/b │
│        │        │       │        │        │        │        │        │ uildt │
│        │        │       │        │        │        │        │        │ est/c │
│        │        │       │        │        │        │        │        │ hecko │
│        │        │       │        │        │        │        │        │ uts/d │
│        │        │       │        │        │        │        │        │ evel/ │
│        │        │       │        │        │        │        │        │ tutor │
│        │        │       │        │        │        │        │        │ ials/ │
│        │        │       │        │        │        │        │        │ vars. │
│        │        │       │        │        │        │        │        │ yml   │
└────────┴────────┴───────┴────────┴────────┴────────┴────────┴────────┴───────┘

The --count option also works with terse mode --terse, shown below is the same output

$ buildtest report --terse --count 2
name|id|state|returncode|starttime|endtime|runtime|tags|buildspec
variables_bash|c70320c7|PASS|0|2022/10/07 18:35:07|2022/10/07 18:35:07|0.010626|tutorials|/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/vars.yml
variables_bash|532e3e6a|PASS|0|2022/10/07 18:36:06|2022/10/07 18:36:06|0.008606|tutorials|/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/vars.yml

Terse Output

If you would like to parse the result of buildtest report, you can use the --terse or -t option which will print the report in machine readable format that shows the name of each column followed by each entry. Each entry is delimited by PIPE symbol (|). The --terse option works with --format and --filter option. In this next example, we report all FAIL tests in terse output. The first line is the header of tables followed by output, if you want to disable output of header you can use --no-header option.

buildtest report --filter state=FAIL --format=name,id,state -t
$ buildtest report --filter state=FAIL --format=name,id,state -t
name|id|state
returncode_list_mismatch|c5173a67|FAIL
returncode_list_mismatch|54a48aa7|FAIL
returncode_list_mismatch|96fc40ae|FAIL
returncode_list_mismatch|120d7514|FAIL
returncode_list_mismatch|9c53a81f|FAIL
exit1_fail|bf2d9f82|FAIL
exit1_fail|f92a208f|FAIL
exit1_fail|c6a25b66|FAIL
exit1_fail|b3806d5b|FAIL
exit1_fail|2076aac2|FAIL
status_regex_fail|3773058a|FAIL
timelimit_min_fail|3ddcad13|FAIL
timelimit_min|5deb36a4|FAIL
timelimit_min_max|f984dd75|FAIL
test_fail_returncode_match|bc30b726|FAIL
always_fail|68750e2e|FAIL
metric_regex_example|74311a1d|FAIL
fail_test|d1230063|FAIL
multiple_executors|aa346ac6|FAIL
executors_vars_env_declaration|52156c51|FAIL
executors_sbatch_declaration|4c5409ce|FAIL
status_returncode_by_executors|cdcb639e|FAIL
ulimit_max_user_process_2048|c0665cfc|FAIL
ulimit_max_user_process_2048|3d06dabf|FAIL
ulimit_stacksize_unlimited|430d08f3|FAIL
ulimit_stacksize_unlimited|3ad463a6|FAIL
ulimit_filedescriptor_4096|8cd1c835|FAIL
ulimit_filedescriptor_4096|a6843235|FAIL
kernel_swapusage|1ffe842d|FAIL
kernel_swapusage|ec2da2e5|FAIL
kernel_swapusage|318f2a8e|FAIL
systemd_default_target|a5022534|FAIL
systemd_default_target|ed72918c|FAIL
systemd_default_target|dbea98d0|FAIL
tcsh_env_declaration|f298dcea|FAIL
sleep|757fba1a|FAIL

Report Summary (buildtest report summary)

The buildtest report summary command can be used to provide a short summary of the test report.

Shown below is an example output from the report summary.

buildtest report summary
$ buildtest report summary
Report File:  /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
Total Tests: 100
Total Tests by Names:  55
Number of buildspecs in report:  27

The buildtest report summary --detailed command can be used to provide a summary of the test report with breakdown statistics of tests including all fail tests, number of tests by name, test runs and buildspecs in report file. To see a short report use

Shown below is an example output from the report summary –detailed.

buildtest report summary --detailed
$ buildtest report summary --detailed
Report File:  /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
Total Tests: 100
Total Tests by Names:  55
Number of buildspecs in report:  27
                            Breakdown by test                            
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Name                           ┃ Total Pass ┃ Total Fail ┃ Total Runs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ variables_bash                 │ 5          │ 0          │ 5          │
│ returncode_list_mismatch       │ 0          │ 5          │ 5          │
│ returncode_int_match           │ 6          │ 0          │ 6          │
│ exit1_pass                     │ 6          │ 0          │ 6          │
│ exit1_fail                     │ 0          │ 5          │ 5          │
│ status_regex_pass              │ 1          │ 0          │ 1          │
│ status_regex_fail              │ 0          │ 1          │ 1          │
│ timelimit_min_fail             │ 0          │ 1          │ 1          │
│ timelimit_max_fail             │ 1          │ 0          │ 1          │
│ timelimit_max                  │ 1          │ 0          │ 1          │
│ timelimit_min                  │ 0          │ 1          │ 1          │
│ timelimit_min_max              │ 0          │ 1          │ 1          │
│ always_pass                    │ 1          │ 0          │ 1          │
│ test_pass_returncode_mismatch  │ 1          │ 0          │ 1          │
│ test_fail_returncode_match     │ 0          │ 1          │ 1          │
│ always_fail                    │ 0          │ 1          │ 1          │
│ bash_login_shebang             │ 1          │ 0          │ 1          │
│ bash_nonlogin_shebang          │ 1          │ 0          │ 1          │
│ unskipped                      │ 1          │ 0          │ 1          │
│ metric_regex_example           │ 0          │ 1          │ 1          │
│ jobC                           │ 1          │ 0          │ 1          │
│ jobA                           │ 1          │ 0          │ 1          │
│ jobB                           │ 1          │ 0          │ 1          │
│ test1                          │ 1          │ 0          │ 1          │
│ test2                          │ 1          │ 0          │ 1          │
│ test3                          │ 1          │ 0          │ 1          │
│ pass_and_fail_test             │ 1          │ 0          │ 1          │
│ fail_test                      │ 0          │ 1          │ 1          │
│ pass_test                      │ 1          │ 0          │ 1          │
│ final_test                     │ 1          │ 0          │ 1          │
│ runtime_test                   │ 1          │ 0          │ 1          │
│ runtime_test_pass              │ 1          │ 0          │ 1          │
│ multiple_executors             │ 1          │ 1          │ 2          │
│ executors_vars_env_declaration │ 1          │ 1          │ 2          │
│ executors_sbatch_declaration   │ 1          │ 1          │ 2          │
│ status_returncode_by_executors │ 1          │ 1          │ 2          │
│ ulimit_cputime_unlimited       │ 2          │ 0          │ 2          │
│ ulimit_vmsize_unlimited        │ 2          │ 0          │ 2          │
│ ulimit_max_user_process_2048   │ 0          │ 2          │ 2          │
│ ulimit_filelock_unlimited      │ 2          │ 0          │ 2          │
│ ulimit_stacksize_unlimited     │ 0          │ 2          │ 2          │
│ ulimit_filedescriptor_4096     │ 0          │ 2          │ 2          │
│ kernel_swapusage               │ 0          │ 3          │ 3          │
│ root_disk_usage                │ 3          │ 0          │ 3          │
│ systemd_default_target         │ 0          │ 3          │ 3          │
│ string_tag                     │ 1          │ 0          │ 1          │
│ list_of_strings_tags           │ 1          │ 0          │ 1          │
│ python_hello                   │ 3          │ 0          │ 3          │
│ circle_area                    │ 5          │ 0          │ 5          │
│ csh_env_declaration            │ 1          │ 0          │ 1          │
│ tcsh_env_declaration           │ 0          │ 1          │ 1          │
│ bash_env_variables             │ 1          │ 0          │ 1          │
│ csh_shell                      │ 1          │ 0          │ 1          │
│ hello_world                    │ 1          │ 0          │ 1          │
│ sleep                          │ 1          │ 1          │ 2          │
└────────────────────────────────┴────────────┴────────────┴────────────┘
                                   PASS Tests                                   
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ name            ┃ id       ┃ executor        ┃ state ┃ returncode ┃ runtime  ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ variables_bash  │ c70320c7 │ generic.local.b │ PASS  │ 0          │ 0.010626 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ variables_bash  │ 532e3e6a │ generic.local.b │ PASS  │ 0          │ 0.008606 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ variables_bash  │ b8a43f49 │ generic.local.b │ PASS  │ 0          │ 0.009286 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ variables_bash  │ 7f5728b7 │ generic.local.b │ PASS  │ 0          │ 0.009327 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ variables_bash  │ eb1dcd91 │ generic.local.b │ PASS  │ 0          │ 0.009432 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_int_ │ dbc1df78 │ generic.local.b │ PASS  │ 128        │ 0.005294 │
│ match           │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_int_ │ 42a739cd │ generic.local.b │ PASS  │ 128        │ 0.005706 │
│ match           │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_int_ │ f9b918bf │ generic.local.b │ PASS  │ 128        │ 0.005587 │
│ match           │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_int_ │ f41444c8 │ generic.local.b │ PASS  │ 128        │ 0.005293 │
│ match           │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_int_ │ 4a65a42e │ generic.local.b │ PASS  │ 128        │ 0.0055   │
│ match           │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_int_ │ 7a88518a │ generic.local.b │ PASS  │ 128        │ 0.005284 │
│ match           │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_pass      │ 1f716bb6 │ generic.local.b │ PASS  │ 1          │ 0.005596 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_pass      │ f24c7458 │ generic.local.b │ PASS  │ 1          │ 0.005601 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_pass      │ 36dc3774 │ generic.local.b │ PASS  │ 1          │ 0.005511 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_pass      │ d71a7099 │ generic.local.b │ PASS  │ 1          │ 0.005546 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_pass      │ 7187873d │ generic.local.b │ PASS  │ 1          │ 0.005345 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_pass      │ ce86fe2e │ generic.local.b │ PASS  │ 1          │ 0.005386 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ status_regex_pa │ 18ae38ce │ generic.local.b │ PASS  │ 0          │ 0.005551 │
│ ss              │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ timelimit_max_f │ 02b32713 │ generic.local.s │ PASS  │ 2          │ 0.004803 │
│ ail             │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ timelimit_max   │ 55219611 │ generic.local.s │ PASS  │ 2          │ 0.003243 │
│                 │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ always_pass     │ 013390a1 │ generic.local.s │ PASS  │ 2          │ 0.002934 │
│                 │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ test_pass_retur │ e818adcb │ generic.local.s │ PASS  │ 2          │ 0.002929 │
│ ncode_mismatch  │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ bash_login_sheb │ 68e1acaf │ generic.local.b │ PASS  │ 0          │ 0.009129 │
│ ang             │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ bash_nonlogin_s │ 823de30f │ generic.local.b │ PASS  │ 0          │ 0.00539  │
│ hebang          │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ unskipped       │ 679b086a │ generic.local.b │ PASS  │ 0          │ 0.00636  │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ jobC            │ 62c38b67 │ generic.local.b │ PASS  │ 0          │ 2.007196 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ jobA            │ 92e67f34 │ generic.local.b │ PASS  │ 0          │ 5.007371 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ jobB            │ 7ddc51dc │ generic.local.b │ PASS  │ 0          │ 2.0071   │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ test1           │ 85cef532 │ generic.local.b │ PASS  │ 1          │ 0.00572  │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ test2           │ 12cb5b6d │ generic.local.b │ PASS  │ 2          │ 0.005252 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ test3           │ 5b8e087b │ generic.local.b │ PASS  │ 1          │ 0.005064 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ pass_and_fail_t │ 526aa15f │ generic.local.b │ PASS  │ 1          │ 2.007383 │
│ est             │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ pass_test       │ 6d44d7fb │ generic.local.b │ PASS  │ 0          │ 2.006913 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ final_test      │ b97c4bfd │ generic.local.b │ PASS  │ 0          │ 2.007123 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ runtime_test    │ b2fb210d │ generic.local.b │ PASS  │ 0          │ 5.006813 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ runtime_test_pa │ 53a16a39 │ generic.local.b │ PASS  │ 0          │ 0.005411 │
│ ss              │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ multiple_execut │ 0e22449e │ generic.local.b │ PASS  │ 0          │ 0.007684 │
│ ors             │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ executors_vars_ │ e7d0eb41 │ generic.local.b │ PASS  │ 0          │ 0.005596 │
│ env_declaration │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ executors_sbatc │ ef7117bf │ generic.local.b │ PASS  │ 0          │ 0.006247 │
│ h_declaration   │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ status_returnco │ dbbf39a8 │ generic.local.b │ PASS  │ 0          │ 0.005267 │
│ de_by_executors │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ ulimit_cputime_ │ b890008c │ generic.local.b │ PASS  │ 0          │ 0.005551 │
│ unlimited       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ ulimit_cputime_ │ 2f2ddcfb │ generic.local.b │ PASS  │ 0          │ 0.005442 │
│ unlimited       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ ulimit_vmsize_u │ 2948d368 │ generic.local.b │ PASS  │ 0          │ 0.005832 │
│ nlimited        │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ ulimit_vmsize_u │ 569236eb │ generic.local.b │ PASS  │ 0          │ 0.00566  │
│ nlimited        │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ ulimit_filelock │ 20ac7f02 │ generic.local.b │ PASS  │ 0          │ 0.00546  │
│ _unlimited      │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ ulimit_filelock │ 2447aa35 │ generic.local.b │ PASS  │ 0          │ 0.005451 │
│ _unlimited      │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ root_disk_usage │ d8a8aa22 │ generic.local.b │ PASS  │ 0          │ 0.117554 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ root_disk_usage │ de555e22 │ generic.local.b │ PASS  │ 0          │ 0.009156 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ root_disk_usage │ 9aca320b │ generic.local.b │ PASS  │ 0          │ 0.008687 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ string_tag      │ 96d3cb9a │ generic.local.b │ PASS  │ 0          │ 0.006638 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ list_of_strings │ 021bf164 │ generic.local.b │ PASS  │ 0          │ 3.031318 │
│ _tags           │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ python_hello    │ 0aa69331 │ generic.local.b │ PASS  │ 0          │ 0.030863 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ python_hello    │ 03af71fc │ generic.local.b │ PASS  │ 0          │ 0.030067 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ python_hello    │ 237968f1 │ generic.local.b │ PASS  │ 0          │ 0.030242 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ circle_area     │ 1f815f1e │ generic.local.b │ PASS  │ 0          │ 0.031002 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ circle_area     │ 6c9572d5 │ generic.local.b │ PASS  │ 0          │ 0.03094  │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ circle_area     │ 6372b390 │ generic.local.b │ PASS  │ 0          │ 0.031279 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ circle_area     │ 105b0f6e │ generic.local.b │ PASS  │ 0          │ 0.031864 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ circle_area     │ bc92b1d2 │ generic.local.b │ PASS  │ 0          │ 0.031735 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ csh_env_declara │ 5b87dcea │ generic.local.c │ PASS  │ 0          │ 4.391376 │
│ tion            │          │ sh              │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ bash_env_variab │ 4612209b │ generic.local.b │ PASS  │ 0          │ 0.007337 │
│ les             │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ csh_shell       │ abf7fe2f │ generic.local.c │ PASS  │ 0          │ 4.386763 │
│                 │          │ sh              │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ hello_world     │ a2a3b259 │ generic.local.b │ PASS  │ 0          │ 0.005629 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ sleep           │ 681e0989 │ generic.local.b │ PASS  │ 0          │ 2.020545 │
│                 │          │ ash             │       │            │          │
└─────────────────┴──────────┴─────────────────┴───────┴────────────┴──────────┘
                                   FAIL Tests                                   
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ name            ┃ id       ┃ executor        ┃ state ┃ returncode ┃ runtime  ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ returncode_list │ c5173a67 │ generic.local.b │ FAIL  │ 2          │ 0.005064 │
│ _mismatch       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_list │ 54a48aa7 │ generic.local.b │ FAIL  │ 2          │ 0.005748 │
│ _mismatch       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_list │ 96fc40ae │ generic.local.b │ FAIL  │ 2          │ 0.005471 │
│ _mismatch       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_list │ 120d7514 │ generic.local.b │ FAIL  │ 2          │ 0.005564 │
│ _mismatch       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ returncode_list │ 9c53a81f │ generic.local.b │ FAIL  │ 2          │ 0.005645 │
│ _mismatch       │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_fail      │ bf2d9f82 │ generic.local.b │ FAIL  │ 1          │ 0.004991 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_fail      │ f92a208f │ generic.local.b │ FAIL  │ 1          │ 0.005386 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_fail      │ c6a25b66 │ generic.local.b │ FAIL  │ 1          │ 0.005395 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_fail      │ b3806d5b │ generic.local.b │ FAIL  │ 1          │ 0.00548  │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ exit1_fail      │ 2076aac2 │ generic.local.b │ FAIL  │ 1          │ 0.005634 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ status_regex_fa │ 3773058a │ generic.local.b │ FAIL  │ 0          │ 0.005264 │
│ il              │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ timelimit_min_f │ 3ddcad13 │ generic.local.s │ FAIL  │ 2          │ 0.003583 │
│ ail             │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ timelimit_min   │ 5deb36a4 │ generic.local.s │ FAIL  │ 2          │ 0.004004 │
│                 │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ timelimit_min_m │ f984dd75 │ generic.local.s │ FAIL  │ 2          │ 0.003627 │
│ ax              │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ test_fail_retur │ bc30b726 │ generic.local.s │ FAIL  │ 2          │ 0.003062 │
│ ncode_match     │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ always_fail     │ 68750e2e │ generic.local.s │ FAIL  │ 2          │ 0.00311  │
│                 │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ metric_regex_ex │ 74311a1d │ generic.local.s │ FAIL  │ 2          │ 0.003431 │
│ ample           │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ fail_test       │ d1230063 │ generic.local.b │ FAIL  │ 0          │ 2.007187 │
│                 │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ multiple_execut │ aa346ac6 │ generic.local.s │ FAIL  │ 2          │ 0.003017 │
│ ors             │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ executors_vars_ │ 52156c51 │ generic.local.s │ FAIL  │ 2          │ 0.003562 │
│ env_declaration │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ executors_sbatc │ 4c5409ce │ generic.local.s │ FAIL  │ 2          │ 0.003    │
│ h_declaration   │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ status_returnco │ cdcb639e │ generic.local.s │ FAIL  │ 2          │ 0.003111 │
│ de_by_executors │          │ h               │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ ulimit_max_user │ c0665cfc │ generic.local.b │ FAIL  │ 0          │ 0.005493 │
│ _process_2048   │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ ulimit_max_user │ 3d06dabf │ generic.local.b │ FAIL  │ 0          │ 0.005327 │
│ _process_2048   │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ ulimit_stacksiz │ 430d08f3 │ generic.local.b │ FAIL  │ 0          │ 0.005774 │
│ e_unlimited     │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ ulimit_stacksiz │ 3ad463a6 │ generic.local.b │ FAIL  │ 0          │ 0.005451 │
│ e_unlimited     │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ ulimit_filedesc │ 8cd1c835 │ generic.local.b │ FAIL  │ 0          │ 0.005377 │
│ riptor_4096     │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ ulimit_filedesc │ a6843235 │ generic.local.b │ FAIL  │ 0          │ 0.005172 │
│ riptor_4096     │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ kernel_swapusag │ 1ffe842d │ generic.local.b │ FAIL  │ 255        │ 0.138695 │
│ e               │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ kernel_swapusag │ ec2da2e5 │ generic.local.b │ FAIL  │ 255        │ 0.006661 │
│ e               │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ kernel_swapusag │ 318f2a8e │ generic.local.b │ FAIL  │ 255        │ 0.006252 │
│ e               │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ systemd_default │ a5022534 │ generic.local.b │ FAIL  │ 1          │ 0.084561 │
│ _target         │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ systemd_default │ ed72918c │ generic.local.b │ FAIL  │ 1          │ 0.008166 │
│ _target         │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ systemd_default │ dbea98d0 │ generic.local.b │ FAIL  │ 1          │ 0.008523 │
│ _target         │          │ ash             │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ tcsh_env_declar │ f298dcea │ generic.local.c │ FAIL  │ 1          │ 2.193499 │
│ ation           │          │ sh              │       │            │          │
├─────────────────┼──────────┼─────────────────┼───────┼────────────┼──────────┤
│ sleep           │ 757fba1a │ generic.local.b │ FAIL  │ -9         │ 1.00356  │
│                 │          │ ash             │       │            │          │
└─────────────────┴──────────┴─────────────────┴───────┴────────────┴──────────┘

Inspect Tests Records via buildtest inspect

Note

buildtest it is an alias for buildtest inspect command.

In previous examples we saw how we can retrieve test records using buildtest report which is printed in table format. We have limited the output to a limited fields however, if you want to analyze a particular, we have a separate command called buildtest inspect that can be used for inspecting a test record based on name or id. Shown below is the command usage for buildtest inspect command.

buildtest inspect --help
$ buildtest inspect --help
usage: buildtest [options] [COMMANDS] inspect [-h]  ...

optional arguments:
  -h, --help     show this help message and exit

subcommands:
  Inspect Test result based on Test ID or Test Name

  
    buildspec (b)
                 Inspect a test based on buildspec
    name (n)     Specify name of test
    query (q)    Query fields from record
    list (l)     List all test names, ids, and corresponding buildspecs

You can report all test names and corresponding ids using buildtest inspect list which will be used for querying tests by name or id.

buildtest inspect list
$ buildtest inspect list
                      Test Summary by name, id, buildspec                       
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ id                      ┃ name                     ┃ buildspec               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ c70320c7-feb1-44c4-a26… │ variables_bash           │ /home/docs/checkouts/re │
│                         │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/vars.ym │
│                         │                          │ l                       │
│ 532e3e6a-8823-4ae3-a14… │ variables_bash           │ /home/docs/checkouts/re │
│                         │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/vars.ym │
│                         │                          │ l                       │
│ b8a43f49-61dc-489b-b9f… │ variables_bash           │ /home/docs/checkouts/re │
│                         │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/vars.ym │
│                         │                          │ l                       │
│ 7f5728b7-e747-4ac2-a22… │ variables_bash           │ /home/docs/checkouts/re │
...

You can fetch all builder names via buildtest inspect list --builder which is the format used for querying test records via buildtest inspect name or buildtest inspect query.

buildtest inspect list --builder
$ buildtest inspect list --builder
variables_bash/c70320c7-feb1-44c4-a26c-e944ed22ceeb
variables_bash/532e3e6a-8823-4ae3-a14c-d2686157c952
variables_bash/b8a43f49-61dc-489b-b9fe-51e571893ba8
variables_bash/7f5728b7-e747-4ac2-a226-d3a7574edb87
variables_bash/eb1dcd91-9d53-4641-855e-e242c5854b42
...

If you are interested in parsing output of buildtest inspect list, you can may find the --terse option useful. The output will show headers followed by entries, the headers can be omitted by specifying --no-header option.

buildtest inspect list -t
$ buildtest inspect list -t
id|name|buildspec
c70320c7-feb1-44c4-a26c-e944ed22ceeb|variables_bash|/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/vars.yml
532e3e6a-8823-4ae3-a14c-d2686157c952|variables_bash|/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/vars.yml
b8a43f49-61dc-489b-b9fe-51e571893ba8|variables_bash|/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/vars.yml
7f5728b7-e747-4ac2-a226-d3a7574edb87|variables_bash|/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/vars.yml
...

Inspecting Test by Name via buildtest inspect name

The buildtest inspect name expects a list of positional argument that correspond to name of test you want to query and buildtest will fetch the last record for each named test. Let’s see an example to illustrate the point. We can see that each test is stored as a JSON format and buildtest keeps track of metadata for each test such as user, hostname, command, path to output and error file, content of test, state of test, returncode, etc… In this example, we will retrieve record for test name circle_area which will print the raw content of the test in JSON format.

buildtest inspect name circle_area
$ buildtest inspect name circle_area
We have detected 1 builders with the following names ['circle_area/bc92b1d2-d548-4189-b82d-9b161fee3a01']
{
│   'circle_area': [
│   │   {
│   │   │   'bc92b1d2-d548-4189-b82d-9b161fee3a01': {
│   │   │   │   'id': 'bc92b1d2',
│   │   │   │   'full_id': 'bc92b1d2-d548-4189-b82d-9b161fee3a01',
│   │   │   │   'description': 'Calculate circle of area given a radius',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-18279078-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2',
│   │   │   │   'testpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/circle_area.sh',
│   │   │   │   'stagedir': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/stage',
│   │   │   │   'command': 'bash --norc --noprofile -eo pipefail circle_area_build.sh',
│   │   │   │   'outfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/circle_area.out',
│   │   │   │   'errfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/circle_area.err',
│   │   │   │   'buildenv': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  circle_area:\n    executor: generic.local.bash\n    type: script\n    shell: python\n    description: "Calculate circle of area given a radius"\n    tags: [tutorials, python]\n    run: |\n      import math\n      radius = 2\n      area = math.pi * radius * radius\n      print("Circle Radius ", radius)\n      print("Area of circle ", area)\n',
│   │   │   │   'test_content': '#!/bin/bash\npython /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/stage/circle_area.py',
│   │   │   │   'buildscript_content': '#!/bin/bash\nexport BUILDTEST_TEST_NAME=circle_area\nexport BUILDTEST_TEST_ROOT=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials\nexport BUILDTEST_STAGE_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/stage\n# source executor startup script\nsource /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/executor/generic.local.bash/before_script.sh\n# Run generated script\n/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/stage/circle_area.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/logs/buildtest_29l2f9eg.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'returncode': 'N/A',
│   │   │   │   │   'regex': 'N/A',
│   │   │   │   │   'runtime': 'N/A'
│   │   │   │   },
│   │   │   │   'tags': 'tutorials python',
│   │   │   │   'starttime': '2022/10/07 18:36:31',
│   │   │   │   'endtime': '2022/10/07 18:36:31',
│   │   │   │   'runtime': '0.031735',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': 'Circle Radius  2\nArea of circle  12.566370614359172\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/circle_area_build.sh'
│   │   │   }
│   │   }
│   ]
}

You can query multiple tests as positional arguments in the format: buildtest inspect name <test1> <test2> In this next example, we will retrieve test records for bash_shell and python_hello.

buildtest inspect name bash_shell python_hello
$ buildtest inspect name bash_shell python_hello
We have detected 1 builders with the following names ['python_hello/237968f1-e611-4fc9-bf12-fec7396ad585']
{
│   'python_hello': [
│   │   {
│   │   │   '237968f1-e611-4fc9-bf12-fec7396ad585': {
│   │   │   │   'id': '237968f1',
│   │   │   │   'full_id': '237968f1-e611-4fc9-bf12-fec7396ad585',
│   │   │   │   'description': 'Hello World python',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-18279078-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-hello/python_hello/237968f1',
│   │   │   │   'testpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-hello/python_hello/237968f1/python_hello.sh',
│   │   │   │   'stagedir': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-hello/python_hello/237968f1/stage',
│   │   │   │   'command': 'bash --norc --noprofile -eo pipefail python_hello_build.sh',
│   │   │   │   'outfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-hello/python_hello/237968f1/python_hello.out',
│   │   │   │   'errfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-hello/python_hello/237968f1/python_hello.err',
│   │   │   │   'buildenv': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-hello/python_hello/237968f1/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  python_hello:\n    type: script\n    description: Hello World python\n    executor: generic.local.bash\n    tags: python\n    run: python hello.py\n\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \n# Content of run section\npython hello.py',
│   │   │   │   'buildscript_content': '#!/bin/bash\nexport BUILDTEST_TEST_NAME=python_hello\nexport BUILDTEST_TEST_ROOT=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-hello/python_hello/237968f1\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials\nexport BUILDTEST_STAGE_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-hello/python_hello/237968f1/stage\n# source executor startup script\nsource /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/executor/generic.local.bash/before_script.sh\n# Run generated script\n/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-hello/python_hello/237968f1/stage/python_hello.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/logs/buildtest_i5k9h68i.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'returncode': 'N/A',
│   │   │   │   │   'regex': 'N/A',
│   │   │   │   │   'runtime': 'N/A'
│   │   │   │   },
│   │   │   │   'tags': 'python',
│   │   │   │   'starttime': '2022/10/07 18:36:28',
│   │   │   │   'endtime': '2022/10/07 18:36:28',
│   │   │   │   'runtime': '0.030242',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': 'Hello World\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-hello/python_hello/237968f1/python_hello_build.sh'
│   │   │   }
│   │   }
│   ]
}

If you want to query all test records for a given name you can use the --all option which is applied to all positional arguments.

Inspect Test by buildspec via buildtest inspect buildspec

buildtest can fetch records based on buildspec via buildtest inspect buildspec which expects a list of buildspecs. By default, buildtest will fetch the latest record of each test, but if you want to fetch all records you can pass the --all option.

In example below we will fetch latest record for all tests in tutorials/vars.yml

buildtest it buildspec tutorials/vars.yml
$ buildtest it buildspec tutorials/vars.yml
{
│   '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/vars.yml': {
│   │   'variables_bash': {
│   │   │   'id': 'eb1dcd91',
│   │   │   'full_id': 'eb1dcd91-9d53-4641-855e-e242c5854b42',
│   │   │   'description': 'Declare shell variables in bash',
│   │   │   'summary': '',
│   │   │   'schemafile': 'script.schema.json',
│   │   │   'executor': 'generic.local.bash',
│   │   │   'compiler': None,
│   │   │   'hostname': 'build-18279078-project-280831-buildtest',
│   │   │   'user': 'docs',
│   │   │   'testroot': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91',
│   │   │   'testpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91/variables_bash.sh',
│   │   │   'stagedir': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91/stage',
│   │   │   'command': 'bash --norc --noprofile -eo pipefail variables_bash_build.sh',
│   │   │   'outfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91/variables_bash.out',
│   │   │   'errfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91/variables_bash.err',
│   │   │   'buildenv': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91/build-env.txt',
│   │   │   'buildspec_content': 'buildspecs:\n  variables_bash:\n    type: script\n    executor: generic.local.bash\n    description: Declare shell variables in bash\n    tags: [tutorials]\n    vars:\n      X: 1\n      Y: 2\n      literalstring: this is a literal string\n      singlequote: \\\'singlequote\\\'\n      doublequote: \\"doublequote\\"\n      current_user: "$(whoami)"\n      num_files: "`find $HOME -type f -maxdepth 1 | wc -l`"\n      multiline_string: |\n        Hello my name is Bob \\n\n        I am 30 years old\n\n\n    run: |\n      echo "$X+$Y="$(($X+$Y))\n      echo $literalstring\n      echo $singlequote\n      echo $doublequote\n      echo "current user:" $current_user\n      echo "number of files:" $num_files\n      echo -e $multiline_string\n',
│   │   │   'test_content': '#!/usr/bin/bash \nX="1"\nY="2"\nliteralstring="this is a literal string"\nsinglequote="\\\'singlequote\\\'"\ndoublequote="\\"doublequote\\""\ncurrent_user="$(whoami)"\nnum_files="`find $HOME -type f -maxdepth 1 | wc -l`"\nmultiline_string="Hello my name is Bob \\n\nI am 30 years old\n"\n# Content of run section\necho "$X+$Y="$(($X+$Y))\necho $literalstring\necho $singlequote\necho $doublequote\necho "current user:" $current_user\necho "number of files:" $num_files\necho -e $multiline_string\n',
│   │   │   'buildscript_content': '#!/bin/bash\nexport BUILDTEST_TEST_NAME=variables_bash\nexport BUILDTEST_TEST_ROOT=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials\nexport BUILDTEST_STAGE_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91/stage\n# source executor startup script\nsource /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/executor/generic.local.bash/before_script.sh\n# Run generated script\n/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91/stage/variables_bash.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   'logpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/logs/buildtest_p8bmw9qi.log',
│   │   │   'metrics': {},
│   │   │   'check': {'returncode': 'N/A', 'regex': 'N/A', 'runtime': 'N/A'},
│   │   │   'tags': 'tutorials',
│   │   │   'starttime': '2022/10/07 18:36:34',
│   │   │   'endtime': '2022/10/07 18:36:34',
│   │   │   'runtime': '0.009432',
│   │   │   'state': 'PASS',
│   │   │   'returncode': '0',
│   │   │   'output': '1+2=3\nthis is a literal string\n\\\'singlequote\\\'\n"doublequote"\ncurrent user: docs\nnumber of files: 4\nHello my name is Bob \n I am 30 years old\n',
│   │   │   'error': '',
│   │   │   'job': {},
│   │   │   'build_script': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91/variables_bash_build.sh'
│   │   }
│   }
}

buildtest will report an error if an input buildspec is invalid filepath such as one below

$ buildtest it buildspec /tmp/buildspec.yml
buildspec: /tmp/buildspec.yml is not valid file
There are no buildspecs in cache based on input buildspecs: ['/tmp/buildspec.yml']

You can also pass multiple buildspes on the command line and fetch all records for a test. In example below we will fetch all records from buildspecs tutorials/vars.yml and tutorials/hello_world.yml

$ buildtest it buildspec --all tutorials/vars.yml tutorials/hello_world.yml
{
│   '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/vars.yml': {
│   │   'variables_bash': [
│   │   │   {
│   │   │   │   'id': 'c70320c7',
│   │   │   │   'full_id': 'c70320c7-feb1-44c4-a26c-e944ed22ceeb',
│   │   │   │   'description': 'Declare shell variables in bash',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-18279078-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/c70320c7',
│   │   │   │   'testpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/c70320c7/variables_bash.sh',
│   │   │   │   'stagedir': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/c70320c7/stage',
│   │   │   │   'command': 'bash --norc --noprofile -eo pipefail variables_bash_build.sh',
│   │   │   │   'outfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/c70320c7/variables_bash.out',
│   │   │   │   'errfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/c70320c7/variables_bash.err',
│   │   │   │   'buildenv': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/c70320c7/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  variables_bash:\n    type: script\n    executor: generic.local.bash\n    description: Declare shell variables in bash\n    tags: [tutorials]\n    vars:\n      X: 1\n      Y: 2\n      literalstring: this is a literal string\n      singlequote: \\\'singlequote\\\'\n      doublequote: \\"doublequote\\"\n      current_user: "$(whoami)"\n      num_files: "`find $HOME -type f -maxdepth 1 | wc -l`"\n      multiline_string: |\n        Hello my name is Bob \\n\n        I am 30 years old\n\n\n    run: |\n      echo "$X+$Y="$(($X+$Y))\n      echo $literalstring\n      echo $singlequote\n      echo $doublequote\n      echo "current user:" $current_user\n      echo "number of files:" $num_files\n      echo -e $multiline_string\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nX="1"\nY="2"\nliteralstring="this is a literal string"\nsinglequote="\\\'singlequote\\\'"\ndoublequote="\\"doublequote\\""\ncurrent_user="$(whoami)"\nnum_files="`find $HOME -type f -maxdepth 1 | wc -l`"\nmultiline_string="Hello my name is Bob \\n\nI am 30 years old\n"\n# Content of run section\necho "$X+$Y="$(($X+$Y))\necho $literalstring\necho $singlequote\necho $doublequote\necho "current user:" $current_user\necho "number of files:" $num_files\necho -e $multiline_string\n',
│   │   │   │   'buildscript_content': '#!/bin/bash\nexport BUILDTEST_TEST_NAME=variables_bash\nexport BUILDTEST_TEST_ROOT=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/c70320c7\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials\nexport BUILDTEST_STAGE_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/c70320c7/stage\n# source executor startup script\nsource /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/executor/generic.local.bash/before_script.sh\n# Run generated script\n/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/c70320c7/stage/variables_bash.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/logs/buildtest_5tom4dty.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'returncode': 'N/A',
│   │   │   │   │   'regex': 'N/A',
│   │   │   │   │   'runtime': 'N/A'
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2022/10/07 18:35:07',
│   │   │   │   'endtime': '2022/10/07 18:35:07',
│   │   │   │   'runtime': '0.010626',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': '1+2=3\nthis is a literal string\n\\\'singlequote\\\'\n"doublequote"\ncurrent user: docs\nnumber of files: 4\nHello my name is Bob \n I am 30 years old\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/c70320c7/variables_bash_build.sh'
│   │   │   },
│   │   │   {
│   │   │   │   'id': '532e3e6a',
│   │   │   │   'full_id': '532e3e6a-8823-4ae3-a14c-d2686157c952',
│   │   │   │   'description': 'Declare shell variables in bash',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-18279078-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/532e3e6a',
│   │   │   │   'testpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/532e3e6a/variables_bash.sh',
│   │   │   │   'stagedir': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/532e3e6a/stage',
│   │   │   │   'command': 'bash --norc --noprofile -eo pipefail variables_bash_build.sh',
│   │   │   │   'outfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/532e3e6a/variables_bash.out',
│   │   │   │   'errfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/532e3e6a/variables_bash.err',
│   │   │   │   'buildenv': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/532e3e6a/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  variables_bash:\n    type: script\n    executor: generic.local.bash\n    description: Declare shell variables in bash\n    tags: [tutorials]\n    vars:\n      X: 1\n      Y: 2\n      literalstring: this is a literal string\n      singlequote: \\\'singlequote\\\'\n      doublequote: \\"doublequote\\"\n      current_user: "$(whoami)"\n      num_files: "`find $HOME -type f -maxdepth 1 | wc -l`"\n      multiline_string: |\n        Hello my name is Bob \\n\n        I am 30 years old\n\n\n    run: |\n      echo "$X+$Y="$(($X+$Y))\n      echo $literalstring\n      echo $singlequote\n      echo $doublequote\n      echo "current user:" $current_user\n      echo "number of files:" $num_files\n      echo -e $multiline_string\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nX="1"\nY="2"\nliteralstring="this is a literal string"\nsinglequote="\\\'singlequote\\\'"\ndoublequote="\\"doublequote\\""\ncurrent_user="$(whoami)"\nnum_files="`find $HOME -type f -maxdepth 1 | wc -l`"\nmultiline_string="Hello my name is Bob \\n\nI am 30 years old\n"\n# Content of run section\necho "$X+$Y="$(($X+$Y))\necho $literalstring\necho $singlequote\necho $doublequote\necho "current user:" $current_user\necho "number of files:" $num_files\necho -e $multiline_string\n',
│   │   │   │   'buildscript_content': '#!/bin/bash\nexport BUILDTEST_TEST_NAME=variables_bash\nexport BUILDTEST_TEST_ROOT=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/532e3e6a\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials\nexport BUILDTEST_STAGE_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/532e3e6a/stage\n# source executor startup script\nsource /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/executor/generic.local.bash/before_script.sh\n# Run generated script\n/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/532e3e6a/stage/variables_bash.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/logs/buildtest_mxuw3s_v.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'returncode': 'N/A',
│   │   │   │   │   'regex': 'N/A',
│   │   │   │   │   'runtime': 'N/A'
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2022/10/07 18:36:06',
│   │   │   │   'endtime': '2022/10/07 18:36:06',
│   │   │   │   'runtime': '0.008606',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': '1+2=3\nthis is a literal string\n\\\'singlequote\\\'\n"doublequote"\ncurrent user: docs\nnumber of files: 4\nHello my name is Bob \n I am 30 years old\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/532e3e6a/variables_bash_build.sh'
│   │   │   },
│   │   │   {
│   │   │   │   'id': 'b8a43f49',
│   │   │   │   'full_id': 'b8a43f49-61dc-489b-b9fe-51e571893ba8',
│   │   │   │   'description': 'Declare shell variables in bash',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-18279078-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/b8a43f49',
│   │   │   │   'testpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/b8a43f49/variables_bash.sh',
│   │   │   │   'stagedir': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/b8a43f49/stage',
│   │   │   │   'command': 'bash --norc --noprofile -eo pipefail variables_bash_build.sh',
│   │   │   │   'outfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/b8a43f49/variables_bash.out',
│   │   │   │   'errfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/b8a43f49/variables_bash.err',
│   │   │   │   'buildenv': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/b8a43f49/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  variables_bash:\n    type: script\n    executor: generic.local.bash\n    description: Declare shell variables in bash\n    tags: [tutorials]\n    vars:\n      X: 1\n      Y: 2\n      literalstring: this is a literal string\n      singlequote: \\\'singlequote\\\'\n      doublequote: \\"doublequote\\"\n      current_user: "$(whoami)"\n      num_files: "`find $HOME -type f -maxdepth 1 | wc -l`"\n      multiline_string: |\n        Hello my name is Bob \\n\n        I am 30 years old\n\n\n    run: |\n      echo "$X+$Y="$(($X+$Y))\n      echo $literalstring\n      echo $singlequote\n      echo $doublequote\n      echo "current user:" $current_user\n      echo "number of files:" $num_files\n      echo -e $multiline_string\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nX="1"\nY="2"\nliteralstring="this is a literal string"\nsinglequote="\\\'singlequote\\\'"\ndoublequote="\\"doublequote\\""\ncurrent_user="$(whoami)"\nnum_files="`find $HOME -type f -maxdepth 1 | wc -l`"\nmultiline_string="Hello my name is Bob \\n\nI am 30 years old\n"\n# Content of run section\necho "$X+$Y="$(($X+$Y))\necho $literalstring\necho $singlequote\necho $doublequote\necho "current user:" $current_user\necho "number of files:" $num_files\necho -e $multiline_string\n',
│   │   │   │   'buildscript_content': '#!/bin/bash\nexport BUILDTEST_TEST_NAME=variables_bash\nexport BUILDTEST_TEST_ROOT=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/b8a43f49\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials\nexport BUILDTEST_STAGE_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/b8a43f49/stage\n# source executor startup script\nsource /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/executor/generic.local.bash/before_script.sh\n# Run generated script\n/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/b8a43f49/stage/variables_bash.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/logs/buildtest_w0lqecl4.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'returncode': 'N/A',
│   │   │   │   │   'regex': 'N/A',
│   │   │   │   │   'runtime': 'N/A'
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2022/10/07 18:36:33',
│   │   │   │   'endtime': '2022/10/07 18:36:33',
│   │   │   │   'runtime': '0.009286',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': '1+2=3\nthis is a literal string\n\\\'singlequote\\\'\n"doublequote"\ncurrent user: docs\nnumber of files: 4\nHello my name is Bob \n I am 30 years old\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/b8a43f49/variables_bash_build.sh'
│   │   │   },
│   │   │   {
│   │   │   │   'id': '7f5728b7',
│   │   │   │   'full_id': '7f5728b7-e747-4ac2-a226-d3a7574edb87',
│   │   │   │   'description': 'Declare shell variables in bash',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-18279078-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/7f5728b7',
│   │   │   │   'testpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/7f5728b7/variables_bash.sh',
│   │   │   │   'stagedir': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/7f5728b7/stage',
│   │   │   │   'command': 'bash --norc --noprofile -eo pipefail variables_bash_build.sh',
│   │   │   │   'outfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/7f5728b7/variables_bash.out',
│   │   │   │   'errfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/7f5728b7/variables_bash.err',
│   │   │   │   'buildenv': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/7f5728b7/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  variables_bash:\n    type: script\n    executor: generic.local.bash\n    description: Declare shell variables in bash\n    tags: [tutorials]\n    vars:\n      X: 1\n      Y: 2\n      literalstring: this is a literal string\n      singlequote: \\\'singlequote\\\'\n      doublequote: \\"doublequote\\"\n      current_user: "$(whoami)"\n      num_files: "`find $HOME -type f -maxdepth 1 | wc -l`"\n      multiline_string: |\n        Hello my name is Bob \\n\n        I am 30 years old\n\n\n    run: |\n      echo "$X+$Y="$(($X+$Y))\n      echo $literalstring\n      echo $singlequote\n      echo $doublequote\n      echo "current user:" $current_user\n      echo "number of files:" $num_files\n      echo -e $multiline_string\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nX="1"\nY="2"\nliteralstring="this is a literal string"\nsinglequote="\\\'singlequote\\\'"\ndoublequote="\\"doublequote\\""\ncurrent_user="$(whoami)"\nnum_files="`find $HOME -type f -maxdepth 1 | wc -l`"\nmultiline_string="Hello my name is Bob \\n\nI am 30 years old\n"\n# Content of run section\necho "$X+$Y="$(($X+$Y))\necho $literalstring\necho $singlequote\necho $doublequote\necho "current user:" $current_user\necho "number of files:" $num_files\necho -e $multiline_string\n',
│   │   │   │   'buildscript_content': '#!/bin/bash\nexport BUILDTEST_TEST_NAME=variables_bash\nexport BUILDTEST_TEST_ROOT=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/7f5728b7\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials\nexport BUILDTEST_STAGE_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/7f5728b7/stage\n# source executor startup script\nsource /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/executor/generic.local.bash/before_script.sh\n# Run generated script\n/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/7f5728b7/stage/variables_bash.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/logs/buildtest_avwtp0g_.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'returncode': 'N/A',
│   │   │   │   │   'regex': 'N/A',
│   │   │   │   │   'runtime': 'N/A'
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2022/10/07 18:36:34',
│   │   │   │   'endtime': '2022/10/07 18:36:34',
│   │   │   │   'runtime': '0.009327',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': '1+2=3\nthis is a literal string\n\\\'singlequote\\\'\n"doublequote"\ncurrent user: docs\nnumber of files: 4\nHello my name is Bob \n I am 30 years old\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/7f5728b7/variables_bash_build.sh'
│   │   │   },
│   │   │   {
│   │   │   │   'id': 'eb1dcd91',
│   │   │   │   'full_id': 'eb1dcd91-9d53-4641-855e-e242c5854b42',
│   │   │   │   'description': 'Declare shell variables in bash',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-18279078-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91',
│   │   │   │   'testpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91/variables_bash.sh',
│   │   │   │   'stagedir': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91/stage',
│   │   │   │   'command': 'bash --norc --noprofile -eo pipefail variables_bash_build.sh',
│   │   │   │   'outfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91/variables_bash.out',
│   │   │   │   'errfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91/variables_bash.err',
│   │   │   │   'buildenv': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  variables_bash:\n    type: script\n    executor: generic.local.bash\n    description: Declare shell variables in bash\n    tags: [tutorials]\n    vars:\n      X: 1\n      Y: 2\n      literalstring: this is a literal string\n      singlequote: \\\'singlequote\\\'\n      doublequote: \\"doublequote\\"\n      current_user: "$(whoami)"\n      num_files: "`find $HOME -type f -maxdepth 1 | wc -l`"\n      multiline_string: |\n        Hello my name is Bob \\n\n        I am 30 years old\n\n\n    run: |\n      echo "$X+$Y="$(($X+$Y))\n      echo $literalstring\n      echo $singlequote\n      echo $doublequote\n      echo "current user:" $current_user\n      echo "number of files:" $num_files\n      echo -e $multiline_string\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \nX="1"\nY="2"\nliteralstring="this is a literal string"\nsinglequote="\\\'singlequote\\\'"\ndoublequote="\\"doublequote\\""\ncurrent_user="$(whoami)"\nnum_files="`find $HOME -type f -maxdepth 1 | wc -l`"\nmultiline_string="Hello my name is Bob \\n\nI am 30 years old\n"\n# Content of run section\necho "$X+$Y="$(($X+$Y))\necho $literalstring\necho $singlequote\necho $doublequote\necho "current user:" $current_user\necho "number of files:" $num_files\necho -e $multiline_string\n',
│   │   │   │   'buildscript_content': '#!/bin/bash\nexport BUILDTEST_TEST_NAME=variables_bash\nexport BUILDTEST_TEST_ROOT=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials\nexport BUILDTEST_STAGE_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91/stage\n# source executor startup script\nsource /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/executor/generic.local.bash/before_script.sh\n# Run generated script\n/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91/stage/variables_bash.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/logs/buildtest_p8bmw9qi.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'returncode': 'N/A',
│   │   │   │   │   'regex': 'N/A',
│   │   │   │   │   'runtime': 'N/A'
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2022/10/07 18:36:34',
│   │   │   │   'endtime': '2022/10/07 18:36:34',
│   │   │   │   'runtime': '0.009432',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': '1+2=3\nthis is a literal string\n\\\'singlequote\\\'\n"doublequote"\ncurrent user: docs\nnumber of files: 4\nHello my name is Bob \n I am 30 years old\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/vars/variables_bash/eb1dcd91/variables_bash_build.sh'
│   │   │   }
│   │   ]
│   },
│   '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/hello_world.yml': {
│   │   'hello_world': [
│   │   │   {
│   │   │   │   'id': 'a2a3b259',
│   │   │   │   'full_id': 'a2a3b259-a2a3-4173-8e01-cd8e26141a92',
│   │   │   │   'description': 'hello world example',
│   │   │   │   'summary': '',
│   │   │   │   'schemafile': 'script.schema.json',
│   │   │   │   'executor': 'generic.local.bash',
│   │   │   │   'compiler': None,
│   │   │   │   'hostname': 'build-18279078-project-280831-buildtest',
│   │   │   │   'user': 'docs',
│   │   │   │   'testroot': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/hello_world/hello_world/a2a3b259',
│   │   │   │   'testpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/hello_world/hello_world/a2a3b259/hello_world.sh',
│   │   │   │   'stagedir': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/hello_world/hello_world/a2a3b259/stage',
│   │   │   │   'command': 'bash --norc --noprofile -eo pipefail hello_world_build.sh',
│   │   │   │   'outfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/hello_world/hello_world/a2a3b259/hello_world.out',
│   │   │   │   'errfile': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/hello_world/hello_world/a2a3b259/hello_world.err',
│   │   │   │   'buildenv': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/hello_world/hello_world/a2a3b259/build-env.txt',
│   │   │   │   'buildspec_content': 'buildspecs:\n  hello_world:\n    executor: generic.local.bash\n    type: script\n    tags: tutorials\n    description: "hello world example"\n    run: echo "hello world!"\nmaintainers:\n- "@shahzebsiddiqui"\n',
│   │   │   │   'test_content': '#!/usr/bin/bash \n# Content of run section\necho "hello world!"',
│   │   │   │   'buildscript_content': '#!/bin/bash\nexport BUILDTEST_TEST_NAME=hello_world\nexport BUILDTEST_TEST_ROOT=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/hello_world/hello_world/a2a3b259\nexport BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials\nexport BUILDTEST_STAGE_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/hello_world/hello_world/a2a3b259/stage\n# source executor startup script\nsource /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/executor/generic.local.bash/before_script.sh\n# Run generated script\n/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/hello_world/hello_world/a2a3b259/stage/hello_world.sh\n# Get return code\nreturncode=$?\n# Exit with return code\nexit $returncode',
│   │   │   │   'logpath': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/logs/buildtest_53_vsb5h.log',
│   │   │   │   'metrics': {},
│   │   │   │   'check': {
│   │   │   │   │   'returncode': 'N/A',
│   │   │   │   │   'regex': 'N/A',
│   │   │   │   │   'runtime': 'N/A'
│   │   │   │   },
│   │   │   │   'tags': 'tutorials',
│   │   │   │   'starttime': '2022/10/07 18:36:26',
│   │   │   │   'endtime': '2022/10/07 18:36:26',
│   │   │   │   'runtime': '0.005629',
│   │   │   │   'state': 'PASS',
│   │   │   │   'returncode': '0',
│   │   │   │   'output': 'hello world!\n',
│   │   │   │   'error': '',
│   │   │   │   'job': {},
│   │   │   │   'build_script': '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/hello_world/hello_world/a2a3b259/hello_world_build.sh'
│   │   │   }
│   │   ]
│   }
}

Note

If you pass a valid filepath but file is not in cache you will get an error as follows

$ buildtest it buildspec $BUILDTEST_ROOT/README.rst
Unable to find any buildspecs in cache, please specify one of the following buildspecs: 
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/vars.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/pass_returncode.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/status_regex.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/runtime_status_test.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/explicit_state.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/shebang.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/skip_tests.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/metrics_regex.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/job_dependency/ex1.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/job_dependency/ex2.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/job_dependency/ex3.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/job_dependency/ex4.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/multi_executors/executor_regex_script.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/multi_executors/executors_var_env_declaration.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/multi_executors/executor_scheduler.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/multi_executors/status_by_executors.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/general_tests/configuration/ulimits.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/general_tests/configuration/kernel_state.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/general_tests/configuration/disk_usage.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/general_tests/configuration/systemd-default-target.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/tags_example.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/python-hello.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/python-shell.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/environment.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/csh_shell_examples.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/hello_world.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/sleep.yml

Query Test Records via buildtest inspect query

The buildtest inspect query command can allow you to retrieve query certain fields from each test records that can be useful when you are inspecting a test. Currently, we can fetch content of output file, error file, testpath, and build script. Shown below are the list of available options for buildtest inspect query.

buildtest inspect --help
$ buildtest inspect query --help
usage: buildtest [options] [COMMANDS] inspect query [-h] [-b] [-be] [-e] [-o]
                                                    [-t]
                                                    [name [name ...]]

positional arguments:
  name               Name of builder to query in report file

optional arguments:
  -h, --help         show this help message and exit
  -b, --buildscript  Print build script
  -be, --buildenv    Print content of build env
  -e, --error        Print error file
  -o, --output       Print output file
  -t, --testpath     Print content of testpath

The buildtest inspect query command expects positional arguments that are name of tests which you can get by running buildtest inspect list.

For instance, let’s query the test circle_area by running the following:

buildtest inspect query circle_area
$ buildtest inspect query circle_area
─────────────── circle_area/bc92b1d2-d548-4189-b82d-9b161fee3a01 ───────────────
Executor: generic.local.bash
Description: Calculate circle of area given a radius
State: PASS
Returncode: 0
Runtime: 0.031735 sec
Starttime: 2022/10/07 18:36:31
Endtime: 2022/10/07 18:36:31
Command: bash --norc --noprofile -eo pipefail circle_area_build.sh
Test Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/circle_area.sh
Build Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/circle_area_build.sh
Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/circle_area.out
Error File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/circle_area.err
Log File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/logs/buildtest_29l2f9eg.log

buildtest will display metadata for each test. By default, buildtest will report the last run for each test that is specified as a positional argument.

You can retrieve content of output file via --output or short option -o. In this command, we retrieve the last run for circle_area and print content of output file

buildtest inspect query -o circle_area
$ buildtest inspect query -o circle_area
─────────────── circle_area/bc92b1d2-d548-4189-b82d-9b161fee3a01 ───────────────
Executor: generic.local.bash
Description: Calculate circle of area given a radius
State: PASS
Returncode: 0
Runtime: 0.031735 sec
Starttime: 2022/10/07 18:36:31
Endtime: 2022/10/07 18:36:31
Command: bash --norc --noprofile -eo pipefail circle_area_build.sh
Test Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/circle_area.sh
Build Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/circle_area_build.sh
Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/circle_area.out
Error File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/circle_area.err
Log File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/logs/buildtest_29l2f9eg.log
─ Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/che… ─
Circle Radius  2                                                                
Area of circle  12.566370614359172

If you want to see content of error file use the -e or --error flag. It would be useful to inspect content of build script and generated test, which can be retrieved using --testpath and --buildscript. Let’s query test circle_area and report all of the content fields

buildtest inspect query -o -e -t -b circle_area
$ buildtest inspect query -o -e -t -b circle_area
─────────────── circle_area/bc92b1d2-d548-4189-b82d-9b161fee3a01 ───────────────
Executor: generic.local.bash
Description: Calculate circle of area given a radius
State: PASS
Returncode: 0
Runtime: 0.031735 sec
Starttime: 2022/10/07 18:36:31
Endtime: 2022/10/07 18:36:31
Command: bash --norc --noprofile -eo pipefail circle_area_build.sh
Test Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/circle_area.sh
Build Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/circle_area_build.sh
Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/circle_area.out
Error File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-shell/circle_area/bc92b1d2/circle_area.err
Log File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/logs/buildtest_29l2f9eg.log
─ Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/che… ─
Circle Radius  2                                                                
Area of circle  12.566370614359172                                              
                                                                                
─ Error File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/chec… ─
                                                                                
─ Test File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/check… ─
#!/bin/bash                                                                     
python /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/deve
─ Test File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/check… ─
#!/bin/bash                                                                     
export BUILDTEST_TEST_NAME=circle_area                                          
export BUILDTEST_TEST_ROOT=/home/docs/checkouts/readthedocs.org/user_builds/buil
export BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/
export BUILDTEST_STAGE_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buil
# source executor startup script                                                
source /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/deve
# Run generated script                                                          
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/t
# Get return code                                                               
returncode=$?                                                                   
# Exit with return code                                                         
exit $returncode

We can query multiple tests using buildtest inspect query since each test is a positional argument. Any options specified to buildtest inspect query will be applied to all test. For instance, let’s fetch the output the of test names root_disk_usage and python_hello

buildtest inspect query -o root_disk_usage python_hello
$ buildtest inspect query -o root_disk_usage python_hello
───────────── root_disk_usage/9aca320b-07d4-43e5-bfd8-92ef698a3cb5 ─────────────
Executor: generic.local.bash
Description: Check root disk usage and report if it exceeds threshold
State: PASS
Returncode: 0
Runtime: 0.008687 sec
Starttime: 2022/10/07 18:36:07
Endtime: 2022/10/07 18:36:07
Command: bash --norc --noprofile -eo pipefail root_disk_usage_build.sh
Test Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/disk_usage/root_disk_usage/9aca320b/root_disk_usage.sh
Build Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/disk_usage/root_disk_usage/9aca320b/root_disk_usage_build.sh
Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/disk_usage/root_disk_usage/9aca320b/root_disk_usage.out
Error File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/disk_usage/root_disk_usage/9aca320b/root_disk_usage.err
Log File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/logs/buildtest_w9mqrhr4.log
─ Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/che… ─
[OK] Root disk is below threshold of 90%                                        
                                                                                
────────────── python_hello/237968f1-e611-4fc9-bf12-fec7396ad585 ───────────────
Executor: generic.local.bash
Description: Hello World python
State: PASS
Returncode: 0
Runtime: 0.030242 sec
Starttime: 2022/10/07 18:36:28
Endtime: 2022/10/07 18:36:28
Command: bash --norc --noprofile -eo pipefail python_hello_build.sh
Test Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-hello/python_hello/237968f1/python_hello.sh
Build Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-hello/python_hello/237968f1/python_hello_build.sh
Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-hello/python_hello/237968f1/python_hello.out
Error File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/tests/generic.local.bash/python-hello/python_hello/237968f1/python_hello.err
Log File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/logs/buildtest_i5k9h68i.log
─ Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/che… ─
Hello World

If you want to query specific test ID, you can specify name of test followed by / and test ID. You don’t need to specify the full ID however tab completion is available to help fill in the names. For example if you want to query test record for circle_area/8edce927-2ecc-4991-ac40-e376c03394b4 shown in tab completion you can type a first few characters to query the record

$ buildtest inspect query circle_area/
circle_area/08f20b50-d2e2-41ab-a75e-a7df75e5afcc  circle_area/8edce927-2ecc-4991-ac40-e376c03394b4  circle_area/d47b6ba8-71b6-4531-b8cd-b6ba9b5f0c6c
circle_area/237c3a96-fad0-4ab7-ab1f-3e7ed1816955  circle_area/baea2e9b-a187-4f9f-bcea-75e768ccb0e0  circle_area/e6652700-4cdb-4f6b-80c5-261e4f448876
circle_area/2c279160-1abf-4c70-957f-d9e4608f521b  circle_area/bf8f1762-ebf9-458e-92e2-af3fc6e73eac  circle_area/e7cc7138-a650-4cd8-aca8-b904f901a0da

$ buildtest inspect query circle_area/8ed
──────────────────────────────────────────────────────────────────────────────────────────── circle_area/8edce927-2ecc-4991-ac40-e376c03394b4 ─────────────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: Calculate circle of area given a radius
State: PASS
Returncode: 0
Runtime: 0.360774 sec
Starttime: 2021/12/23 12:37:25
Endtime: 2021/12/23 12:37:25
Command: bash --norc --noprofile -eo pipefail circle_area_build.sh
Test Script: /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.bash/python-shell/circle_area/8edce927/circle_area.sh
Build Script: /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.bash/python-shell/circle_area/8edce927/circle_area_build.sh
Output File: /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.bash/python-shell/circle_area/8edce927/circle_area.out
Error File: /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.bash/python-shell/circle_area/8edce927/circle_area.err
Log File: /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/logs/buildtest_c5dnun2l.log

buildtest will search for test ID using re.match so it is possible to apply a regular expression to seek out multiple test records. The tests must be enclosed in quotes " in-order to have a valid regular expression. Here are few examples that can be useful

# retrieve all test records for name `circle_area`
buildtest inspect query circle_area/

# retrieve test records starting with ID `8a` and `bc` for test name `exit1`
buildtest inspect query "exit1/(8a|bc)"

Using Alternate Report File

The buildtest report and buildtest inspect command will read from the report file tracked by buildtest which is stored in $BUILDTEST_ROOT/var/report.json. This single file can became an issue if you are running jobs through CI where you can potentially overwrite same file or if you want separate report files for each set of builds. Luckily we have an option to handle this using the buildtest -r <report_path> build -b <buildspec_path> option which can be used to specify an alternate location to report file.

buildtest will write the report file in the desired location, then you can specify the path to report file via buildtest -r <report_path> report and buildtest -r <report_path> inspect to load the report file when reporting tests.

The report file must be valid JSON file that buildtest understands in order to use buildtest report and buildtest inspect command. Shown below are some examples using the alternate report file using buildtest report and buildtest inspect command.

$ buildtest -r $BUILDTEST_ROOT/python.json report --format name,id
                      Report File: /Users/siddiq90/Documents/GitHubDesktop/buildtest/python.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                                                               ┃ id                                            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ circle_area                                                        │ a2814554                                      │
├────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────┤
│ python_hello                                                       │ dd447e43                                      │
└────────────────────────────────────────────────────────────────────┴───────────────────────────────────────────────┘

You can view path to all report files via buildtest report list which keeps track of any new report files created when using buildtest build -r option.

$ buildtest report list
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/var/report.json
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/python.json