Query Test Report

buildtest keeps track of all tests and results in a JSON file. This file is read by buildtest report command to extract certain fields from JSON file and display them in table format. We use python tabulate library for pretty print data in tables. Shown below is command usage to query test reports.

$ buildtest report --help
usage: buildtest [options] [COMMANDS] report [-h] [--helpformat] [--helpfilter] [--format FORMAT] [--filter FILTER]
                                             [--latest] [--oldest] [-r REPORT] [-t]
                                             ...

optional arguments:
  -h, --help            show this help message and exit
  --helpformat          List of available format fields
  --helpfilter          List available filter fields to be used with --filter option
  --format FORMAT       format field for printing purposes. For more details see --helpformat for list of available
                        fields. Fields must be separated by comma (usage: --format <field1>,<field2>,...)
  --filter FILTER       Filter report by filter fields. The filter fields must be a key=value pair and multiple fields
                        can be comma separated in the following format: --filter key1=val1,key2=val2 . For list of
                        filter fields run: --helpfilter.
  --latest              Retrieve latest record of particular test
  --oldest              Retrieve oldest record of particular test
  -r REPORT, --report REPORT
                        Specify a report file to read
  -t, --terse           Print output in machine readable format

subcommands:
  Fetch test results from report file and print them in table format

  
    clear               delete report file
    list                List all report files

You may run buildtest report without any option, and buildtest will display all test results with default format fields. To see a list of all format fields, click here.

$ buildtest report
Reading report file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/report.json 

+------------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| name                         | id       | state   |   returncode | starttime           | endtime             |   runtime | tags               | buildspec                                                                                                                           |
+==============================+==========+=========+==============+=====================+=====================+===========+====================+=====================================================================================================================================+
| exit1_fail                   | 2bc66cba | FAIL    |            1 | 2021/07/30 22:01:35 | 2021/07/30 22:01:35 |  0.00575  | tutorials fail     | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/pass_returncode.yml                          |
+------------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| exit1_fail                   | cad7aa63 | FAIL    |            1 | 2021/07/30 22:02:02 | 2021/07/30 22:02:02 |  0.005078 | tutorials fail     | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/pass_returncode.yml                          |
+------------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| exit1_fail                   | 4a9f6ee2 | FAIL    |            1 | 2021/07/30 22:02:02 | 2021/07/30 22:02:02 |  0.005236 | tutorials fail     | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/pass_returncode.yml                          |
+------------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| exit1_fail                   | 16429a3c | FAIL    |            1 | 2021/07/30 22:02:06 | 2021/07/30 22:02:06 |  0.005043 | tutorials fail     | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/pass_returncode.yml                          |
+------------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| exit1_fail                   | 7714d388 | FAIL    |            1 | 2021/07/30 22:02:06 | 2021/07/30 22:02:06 |  0.004243 | tutorials fail     | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/pass_returncode.yml                          |
+------------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| exit1_pass                   | 56a7caa5 | PASS    |            1 | 2021/07/30 22:01:35 | 2021/07/30 22:01:35 |  0.00448  | tutorials pass     | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/pass_returncode.yml                          |
+------------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| exit1_pass                   | cbd241fa | PASS    |            1 | 2021/07/30 22:02:02 | 2021/07/30 22:02:02 |  0.004605 | tutorials pass     | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/pass_returncode.yml                          |
+------------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| exit1_pass                   | 0190d85b | PASS    |            1 | 2021/07/30 22:02:02 | 2021/07/30 22:02:02 |  0.007753 | tutorials pass     | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/pass_returncode.yml                          |
...

Format Reports

Available Format Fields

The buildtest report command displays a default format fields that can be changed using the --format option. The report file (JSON) contains many more fields and we expose some of the fields with the –format option. To see a list of available format fields you can run buildtest report --helpformat. This option will list all format fields with their description.

$ buildtest report --helpformat
Fields      Description
----------  ----------------------------------------------------------------
buildspec   Buildspec file
command     Command executed
compiler    Retrieve compiler used for test (applicable for compiler schema)
endtime     End Time for Test in date format
errfile     Error File
executor    Executor name
hostname    Retrieve hostname of machine where job was submitted from
full_id     Full qualified unique build identifier
id          Unique Build Identifier (abbreviated)
metrics     List all metrics if applicable
name        Name of test defined in buildspec
outfile     Output file
returncode  Return Code from Test Execution
runtime     Total runtime in seconds
schemafile  Schema file used for validation
starttime   Start Time of test in date format
state       Test State reported by buildtest (PASS/FAIL)
tags        Tag name
testroot    Root of test directory
testpath    Path to test
user        Get user who submitted job

Format Field Usage

The --format field are specified in comma separated format (i.e --format <field1>,<field2>). In this example we format table by fields --format id,executor,state,returncode.

$ buildtest report --format name,id,executor,state,returncode
Reading report file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/report.json 

+------------------------------+----------+----------------------+---------+--------------+
| name                         | id       | executor             | state   |   returncode |
+==============================+==========+======================+=========+==============+
| exit1_fail                   | 2bc66cba | generic.local.sh     | FAIL    |            1 |
+------------------------------+----------+----------------------+---------+--------------+
| exit1_fail                   | cad7aa63 | generic.local.sh     | FAIL    |            1 |
+------------------------------+----------+----------------------+---------+--------------+
| exit1_fail                   | 4a9f6ee2 | generic.local.sh     | FAIL    |            1 |
+------------------------------+----------+----------------------+---------+--------------+
| exit1_fail                   | 16429a3c | generic.local.sh     | FAIL    |            1 |
+------------------------------+----------+----------------------+---------+--------------+
| exit1_fail                   | 7714d388 | generic.local.sh     | FAIL    |            1 |
+------------------------------+----------+----------------------+---------+--------------+
| exit1_pass                   | 56a7caa5 | generic.local.sh     | PASS    |            1 |
+------------------------------+----------+----------------------+---------+--------------+
| exit1_pass                   | cbd241fa | generic.local.sh     | PASS    |            1 |
+------------------------------+----------+----------------------+---------+--------------+
| exit1_pass                   | 0190d85b | generic.local.sh     | PASS    |            1 |
+------------------------------+----------+----------------------+---------+--------------+
...

Filter Reports

The buildtest report command will display all tests results, which can be quite long depending on number of tests so therefore we need a mechanism to filter the test results. The --filter option can be used to filter out tests in the output based on filter fields. First, lets see the available filter fields by run buildtest report --helpfilter which shows a list of filter fields and their description.

$ buildtest report --helpfilter
Filter Fields    Description                 Expected Value
---------------  --------------------------  ----------------
buildspec        Filter by buildspec file    FILE
name             Filter by test name         STRING
executor         Filter by executor name     STRING
state            Filter by test state        PASS/FAIL
tags             Filter tests by tag name    STRING
returncode       Filter tests by returncode  INT

The --filter option expects arguments in key=value format. You can specify multiple filter delimited by comma. buildtest will treat multiple filters as logical AND operation. The filter option can be used with --format field. Let’s see some examples to illustrate the point.

Filter by returncode

If you want to retrieve all tests with a given returncode, we can use the returncode property. For instance, let’s retrieve all tests with returncode of 2 by setting --filter returncode=2.

$ buildtest report --filter returncode=2 --format=name,id,returncode
Reading report file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/report.json 

+--------------------------+----------+--------------+
| name                     | id       |   returncode |
+==========================+==========+==============+
| returncode_list_mismatch | 61b70f13 |            2 |
+--------------------------+----------+--------------+
| returncode_list_mismatch | 99ee050f |            2 |
+--------------------------+----------+--------------+
| returncode_list_mismatch | e36655e4 |            2 |
+--------------------------+----------+--------------+
| returncode_list_mismatch | a22198e9 |            2 |
+--------------------------+----------+--------------+
| returncode_list_mismatch | eca6beab |            2 |
+--------------------------+----------+--------------+

Note

buildtest automatically converts returncode to integer when matching returncode, so --filter returncode="2" will work too

Filter by test name

If you want to filter by test name, use the name attribute in filter option. Let’s assume we want to filter all tests by name exit1_pass, this can be achieved by setting filter field as follows: --filter name=exit1_pass. Shown below is an example using name filter field to filter test results.

$ buildtest report --filter name=exit1_pass --format=name,id,returncode,state
Reading report file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/report.json 

+------------+----------+--------------+---------+
| name       | id       |   returncode | state   |
+============+==========+==============+=========+
| exit1_pass | 56a7caa5 |            1 | PASS    |
+------------+----------+--------------+---------+
| exit1_pass | cbd241fa |            1 | PASS    |
+------------+----------+--------------+---------+
| exit1_pass | 0190d85b |            1 | PASS    |
+------------+----------+--------------+---------+
| exit1_pass | 2c1a87d2 |            1 | PASS    |
+------------+----------+--------------+---------+
| exit1_pass | 1438fb0e |            1 | PASS    |
+------------+----------+--------------+---------+
| exit1_pass | fa9cc65e |            1 | PASS    |
+------------+----------+--------------+---------+

Filter by buildspec

Likewise, we can filter results by buildspec file using buildspec attribute via --filter buildspec=<file>. The buildspec attribute must resolve to a file path which can be relative or absolute path. buildtest will resolve path (absolute path) and find the appropriate tests that belong to the buildspec file. If file doesn’t exist or is not found in cache it will raise an error.

$ buildtest report --filter buildspec=tutorials/python-hello.yml --format=name,id,state,buildspec
Reading report file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/report.json 

+--------------+----------+---------+---------------------------------------------------------------------------------------------------------+
| name         | id       | state   | buildspec                                                                                               |
+==============+==========+=========+=========================================================================================================+
| python_hello | aa2fbe0b | PASS    | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/python-hello.yml |
+--------------+----------+---------+---------------------------------------------------------------------------------------------------------+
| python_hello | 82bc0254 | PASS    | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/python-hello.yml |
+--------------+----------+---------+---------------------------------------------------------------------------------------------------------+

Filter by test state

If you want to filter results by test state, use the state property. This can be useful if you want to know all pass or failed tests. The state property expects value of [PASS|FAIL] since these are the two recorded test states marked by buildtest. We can also pass multiple filter fields for instance if we want to find all FAIL tests for executor generic.local.sh we can do the following.

$ buildtest report --filter state=FAIL,executor=generic.local.sh --format=name,id,state,executor
Reading report file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/report.json 

+--------------------------+----------+---------+------------------+
| name                     | id       | state   | executor         |
+==========================+==========+=========+==================+
| exit1_fail               | 2bc66cba | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| exit1_fail               | cad7aa63 | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| exit1_fail               | 4a9f6ee2 | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| exit1_fail               | 16429a3c | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| exit1_fail               | 7714d388 | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| returncode_list_mismatch | 61b70f13 | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| returncode_list_mismatch | 99ee050f | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| returncode_list_mismatch | e36655e4 | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| returncode_list_mismatch | a22198e9 | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| returncode_list_mismatch | eca6beab | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| timelimit_min_fail       | b51bc30b | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| timelimit_max_fail       | b1ad9cb2 | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+

Filter Exception Cases

The returncode filter field expects an integer value, so if you try a non-integer returncode you will get the following message

$ buildtest report --filter returncode=1.5
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/bin/buildtest", line 17, in <module>
    buildtest.main.main()
  File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/buildtest/main.py", line 125, in main
    report_cmd(args)
  File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/buildtest/cli/report.py", line 525, in report_cmd
    report_file=args.report,
  File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/buildtest/cli/report.py", line 85, in __init__
    self._check_filter_fields()
  File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/buildtest/cli/report.py", line 119, in _check_filter_fields
    f"Invalid returncode:{self.filter[key]} must be an integer"
buildtest.exceptions.BuildTestError: 'Invalid returncode:1.5 must be an integer'

The state filter field expects value of PASS or FAIL so if you specify an invalid state you will get an error as follows.

$ buildtest report --filter state=UNKNOWN
filter argument 'state' must be 'PASS' or 'FAIL' got value UNKNOWN

The buildspec field expects a valid file path, it can be an absolute or relative path, buildtest will resolve absolute path and check if file exist and is in the report file. If it’s an invalid file we get an error such as

$ buildtest report --filter buildspec=/path/to/invalid.yml
Invalid File Path for filter field 'buildspec': /path/to/invalid.yml

You may have a valid filepath for buildspec filter field such as $BUILDTEST_ROOT/tutorials/invalid_executor.yml, but there is no record of a test in the report cache because this test wasn’t run. In this case you will get the following message.

$ buildtest report --filter buildspec=$BUILDTEST_ROOT/tutorials/invalid_executor.yml
buildspec file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/invalid_executor.yml not found in cache

Find Latest or Oldest test

We can search for oldest or latest test for any given test. This can be useful if you want to see first or last test run. If you want to retrieve the oldest test you can use --oldest option. buildtest will append tests, therefore last record in dictionary will be latest record, similarly first record is the oldest record.

Let’s take a look at this example, we filter by test name hello_f which retrieves three entries. Now let’s filter by oldest record by specifying –oldest option and it will retrieve the first record which is test id 349f3ada.

$ buildtest report --filter name=hello_f --format name,id,starttime
 Reading Report File: /Users/siddiq90/.buildtest/report.json

 +---------+----------+---------------------+
 | name    | id       | starttime           |
 +=========+==========+=====================+
 | hello_f | 349f3ada | 2021/02/11 18:13:08 |
 +---------+----------+---------------------+
 | hello_f | ecd4a3f2 | 2021/02/11 18:13:18 |
 +---------+----------+---------------------+
 | hello_f | 5c87978b | 2021/02/11 18:13:33 |
 +---------+----------+---------------------+

 $ buildtest report --filter name=hello_f --format name,id,starttime --oldest
 Reading Report File: /Users/siddiq90/.buildtest/report.json

 +---------+----------+---------------------+
 | name    | id       | starttime           |
 +=========+==========+=====================+
 | hello_f | 349f3ada | 2021/02/11 18:13:08 |
 +---------+----------+---------------------+

If you want to retrieve the latest test result you can use --latest option which will retrieve the last record, in the same example we will retrieve test id 5c87978b.

$ buildtest report --filter name=hello_f --format name,id,starttime --latest
Reading Report File: /Users/siddiq90/.buildtest/report.json

+---------+----------+---------------------+
| name    | id       | starttime           |
+=========+==========+=====================+
| hello_f | 5c87978b | 2021/02/11 18:13:33 |
+---------+----------+---------------------+

You may combine –oldest and –latest options in same command, in this case buildtest will retrieve the first and last record of every test.

$ buildtest report --format name,id,starttime --oldest --latest | more
Reading Report File: /Users/siddiq90/.buildtest/report.json

+------------------------------+----------+---------------------+
| name                         | id       | starttime           |
+==============================+==========+=====================+
| variables_bash               | 750f48bc | 2021/02/11 18:13:03 |
+------------------------------+----------+---------------------+
| variables_bash               | 1bdfd403 | 2021/02/11 18:13:32 |
+------------------------------+----------+---------------------+
| ulimit_filelock_unlimited    | b7b852e4 | 2021/02/11 18:13:03 |
+------------------------------+----------+---------------------+
| ulimit_filelock_unlimited    | 56345a43 | 2021/02/11 18:13:18 |
+------------------------------+----------+---------------------+

Terse Output

If you would like to parse the result of buildtest report, you can use the --terse or -t option which will print the report in machine readable format that shows the name of each column followed by each entry. Each entry is delimited by PIPE symbol (|). The --terse option works with --format and --filter option. In this next example, we report all FAIL tests in terse output.

$ buildtest report --filter state=FAIL --format=name,id,state -t
name|id|state
exit1_fail|2bc66cba|FAIL
exit1_fail|cad7aa63|FAIL
exit1_fail|4a9f6ee2|FAIL
exit1_fail|16429a3c|FAIL
exit1_fail|7714d388|FAIL
returncode_list_mismatch|61b70f13|FAIL
returncode_list_mismatch|99ee050f|FAIL
returncode_list_mismatch|e36655e4|FAIL
returncode_list_mismatch|a22198e9|FAIL
returncode_list_mismatch|eca6beab|FAIL
timelimit_min_fail|b51bc30b|FAIL
timelimit_max_fail|b1ad9cb2|FAIL
kernel_swapusage|78385aa3|FAIL
kernel_swapusage|ac8de77a|FAIL
kernel_swapusage|4892fa48|FAIL
ulimit_stacksize_unlimited|f41c7eba|FAIL
ulimit_stacksize_unlimited|7600c5e0|FAIL
ulimit_filedescriptor_4096|725bf02b|FAIL
ulimit_filedescriptor_4096|467c31bb|FAIL
ulimit_max_user_process_2048|f5fc40c0|FAIL
ulimit_max_user_process_2048|9d024d0b|FAIL
systemd_default_target|5e0428e3|FAIL
systemd_default_target|dfa421fa|FAIL
systemd_default_target|b38db031|FAIL
ssh_localhost_remotecommand|87badcae|FAIL
ssh_localhost_remotecommand|2b361527|FAIL
ssh_localhost_remotecommand|2aef4455|FAIL
list_of_strings_tags|3e32ff68|FAIL

Inspect Tests Records via buildtest inspect

In previous examples we saw how we can retrieve test records using buildtest report which is printed in table format. We have limited the output to a limited fields however, if you want to analyze a particular, we have a separate command called buildtest inspect that can be used for inspecting a test record based on name or id. Shown below is the command usage for buildtest inspect command.

$ buildtest inspect --help
usage: buildtest [options] [COMMANDS] inspect [-h] [-r REPORT]  ...

optional arguments:
  -h, --help            show this help message and exit
  -r REPORT, --report REPORT
                        Specify a report file to load when inspecting test

subcommands:
  Inspect Test result based on Test ID or Test Name

  
    name                Specify name of test
    id                  Specify a Test ID
    list                List all test ids
    query               Query fields from record

You can report all test names and corresponding ids using buildtest inspect list which will be used for querying tests by name or id.

$ buildtest inspect list
+------------------------------+--------------------------------------+
| name                         | id                                   |
+==============================+======================================+
| exit1_fail                   | 2bc66cba-4356-4ace-b20b-d8ecff3734a3 |
+------------------------------+--------------------------------------+
| exit1_fail                   | cad7aa63-91f3-474f-be10-79c5ee6fa589 |
+------------------------------+--------------------------------------+
| exit1_fail                   | 4a9f6ee2-7e59-4fee-ab33-73ec21ba60e2 |
+------------------------------+--------------------------------------+
| exit1_fail                   | 16429a3c-0bfc-484c-bdd5-6e4b3841558e |
+------------------------------+--------------------------------------+
| exit1_fail                   | 7714d388-1a5e-41ec-981e-f6448c501b8d |
+------------------------------+--------------------------------------+
| exit1_pass                   | 56a7caa5-b235-4d45-91b3-97f977f5ad6a |
+------------------------------+--------------------------------------+
| exit1_pass                   | cbd241fa-4145-42b7-9c46-86dcbf368b7a |
+------------------------------+--------------------------------------+
| exit1_pass                   | 0190d85b-157f-415d-ba3b-38b5bf17e88b |
+------------------------------+--------------------------------------+
| exit1_pass                   | 2c1a87d2-3f85-4da0-bb2b-3cd196524090 |
...

Inspecting Test by Name via buildtest inspect name

The buildtest inspect name expects a list of positional argument that correspond to name of test you want to query and buildtest will fetch the last record for each named test. Let’s see an example to illustrate the point. We can see that each test is stored as a JSON format and buildtest keeps track of metadata for each test such as user, hostname, command, path to output and error file, content of test, state of test, returncode, etc… In this example, we will retrieve record for test name circle_area which will print the raw content of the test in JSON format.

$ buildtest inspect name circle_area
{
  "circle_area": {
    "id": "12a030f6",
    "full_id": "12a030f6-fd09-48cb-a270-b7cd6fa49c42",
    "description": "Calculate circle of area given a radius",
    "schemafile": "script-v1.0.schema.json",
    "executor": "generic.local.python",
    "compiler": null,
    "hostname": "build-14364669-project-280831-buildtest",
    "user": "docs",
    "testroot": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/5",
    "testpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/5/circle_area.sh",
    "stagedir": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/5/stage",
    "command": "sh circle_area_build.sh",
    "outfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/5/circle_area.out",
    "errfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/5/circle_area.err",
    "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  circle_area:\n    executor: generic.local.python\n    type: script\n    shell: python\n    description: \"Calculate circle of area given a radius\"\n    tags: [tutorials, python]\n    run: |\n      import math\n      radius = 2\n      area = math.pi * radius * radius\n      print(\"Circle Radius \", radius)\n      print(\"Area of circle \", area)\n",
    "test_content": "#!/bin/bash\npython /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/5/stage/circle_area.py",
    "logpath": "/tmp/buildtest_3rjt4orh.log",
    "metrics": {},
    "tags": "tutorials python",
    "starttime": "2021/07/30 22:02:05",
    "endtime": "2021/07/30 22:02:05",
    "runtime": "0.043701",
    "state": "PASS",
    "returncode": "0",
    "output": "Circle Radius  2\nArea of circle  12.566370614359172\n",
    "error": "circle_area_build.sh: 14: circle_area_build.sh: source: not found\n",
    "job": null,
    "build_script": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/5/circle_area_build.sh"
  }
}

You can query multiple tests as positional arguments in the format: buildtest inspect name <test1> <test2> In this next example, we will retrieve test records for bash_shell and python_hello.

$ buildtest inspect name bash_shell python_hello
{
  "python_hello": {
    "id": "82bc0254",
    "full_id": "82bc0254-5dd2-4852-b13d-d286fbd2ef33",
    "description": "Hello World python",
    "schemafile": "script-v1.0.schema.json",
    "executor": "generic.local.bash",
    "compiler": null,
    "hostname": "build-14364669-project-280831-buildtest",
    "user": "docs",
    "testroot": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.bash/python-hello/python_hello/2",
    "testpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.bash/python-hello/python_hello/2/python_hello.sh",
    "stagedir": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.bash/python-hello/python_hello/2/stage",
    "command": "sh python_hello_build.sh",
    "outfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.bash/python-hello/python_hello/2/python_hello.out",
    "errfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.bash/python-hello/python_hello/2/python_hello.err",
    "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  python_hello:\n    type: script\n    description: Hello World python\n    executor: generic.local.bash\n    tags: python\n    run: python hello.py\n\n",
    "test_content": "#!/bin/bash \n# Content of run section\npython hello.py",
    "logpath": "/tmp/buildtest_lbmm9q3d.log",
    "metrics": {},
    "tags": "python",
    "starttime": "2021/07/30 22:02:02",
    "endtime": "2021/07/30 22:02:02",
    "runtime": "0.045624",
    "state": "PASS",
    "returncode": "0",
    "output": "Hello World\n",
    "error": "python_hello_build.sh: 14: python_hello_build.sh: source: not found\n",
    "job": null,
    "build_script": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.bash/python-hello/python_hello/2/python_hello_build.sh"
  }
}

If you want to query all test records for a given name you can use the --all option which is applied to all positional arguments.

Inspecting Test by ID via buildtest inspect id

The buildtest inspect id works similar to buildtest inspect name except that it operates on test id. This can be useful if you want to extract a particular test record and not see all test records at once.

You only need to specify a few characters and buildtest will resolve full test id if there is a match. The buildtest inspect id can operate on single or multiple ids if you want to specify multiple ids in single command you can do buildtest inspect id <identifier1> <identifier2>.

Let’s see an example where we query a single test record. Notice, that we only specify a few characters fee and buildtest found a matching record fee66c67-db4e-4d35-8c6d-28ac5cbbaba0

$ buildtest inspect id fee
Reading Report File: /Users/siddiq90/.buildtest/report.json

{
  "fee66c67-db4e-4d35-8c6d-28ac5cbbaba0": {
    "id": "fee66c67",
    "full_id": "fee66c67-db4e-4d35-8c6d-28ac5cbbaba0",
    "schemafile": "script-v1.0.schema.json",
    "executor": "generic.local.bash",
    "compiler": null,
    "hostname": "DOE-7086392.local",
    "user": "siddiq90",
    "testroot": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/2",
    "testpath": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/2/stage/generate.sh",
    "stagedir": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/2/stage",
    "rundir": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/2/run",
    "command": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/2/stage/generate.sh",
    "outfile": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/2/run/python_hello.out",
    "errfile": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/2/run/python_hello.err",
    "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  python_hello:\n    type: script\n    description: Hello World python\n    executor: generic.local.bash\n    tags: python\n    run: python hello.py\n\n",
    "test_content": "#!/bin/bash \nsource /Users/siddiq90/Documents/github/buildtest/var/executors/generic.local.bash/before_script.sh\npython hello.py\nsource /Users/siddiq90/Documents/github/buildtest/var/executors/generic.local.bash/after_script.sh",
    "tags": "python",
    "starttime": "2021/03/31 11:18:21",
    "endtime": "2021/03/31 11:18:21",
    "runtime": 0.104714,
    "state": "PASS",
    "returncode": 0,
    "output": "Hello World\n",
    "error": "",
    "job": null
  }
}

We can pass multiple IDs to buildtest inspect id and buildtest will retrieve test record if there is a match. You only need to specify a few characters to ensure we have a unique test ID and buildtest will retrieve the record.

$ buildtest inspect id 944 a76
 Reading Report File: /Users/siddiq90/.buildtest/report.json

 {
   "a76799db-f11e-4050-8dcb-8b147092c536": {
     "id": "a76799db",
     "full_id": "a76799db-f11e-4050-8dcb-8b147092c536",
     "schemafile": "script-v1.0.schema.json",
     "executor": "generic.local.bash",
     "compiler": null,
     "hostname": "DOE-7086392.local",
     "user": "siddiq90",
     "testroot": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/disk_usage/root_disk_usage/0",
     "testpath": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/disk_usage/root_disk_usage/0/stage/generate.sh",
     "stagedir": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/disk_usage/root_disk_usage/0/stage",
     "rundir": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/disk_usage/root_disk_usage/0/run",
     "command": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/disk_usage/root_disk_usage/0/stage/generate.sh",
     "outfile": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/disk_usage/root_disk_usage/0/run/root_disk_usage.out",
     "errfile": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/disk_usage/root_disk_usage/0/run/root_disk_usage.err",
     "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  root_disk_usage:\n    executor: generic.local.bash\n    type: script\n    tags: [filesystem, storage]\n    description: Check root disk usage and report if it exceeds threshold\n    env:\n      threshold: 90\n    run: |\n      root_disk_usage=`df -a / | tail -n 1 |  awk '{print $5'} | sed 's/[^0-9]*//g'`\n      # if root exceeds threshold\n      if [ \"$root_disk_usage\" -gt \"$threshold\" ]; then\n        echo \"[WARNING] Root Disk Usage: $root_disk_usage% exceeded threshold of $threshold%\"\n        exit 1\n      fi\n      echo \"[OK] Root disk is below threshold of $threshold%\"\n",
     "test_content": "#!/bin/bash \nsource /Users/siddiq90/Documents/github/buildtest/var/executors/generic.local.bash/before_script.sh\nexport threshold=90\nroot_disk_usage=`df -a / | tail -n 1 |  awk '{print $5'} | sed 's/[^0-9]*//g'`\n# if root exceeds threshold\nif [ \"$root_disk_usage\" -gt \"$threshold\" ]; then\n  echo \"[WARNING] Root Disk Usage: $root_disk_usage% exceeded threshold of $threshold%\"\n  exit 1\nfi\necho \"[OK] Root disk is below threshold of $threshold%\"\n\nsource /Users/siddiq90/Documents/github/buildtest/var/executors/generic.local.bash/after_script.sh",
     "tags": "filesystem storage",
     "starttime": "2021/03/31 11:17:50",
     "endtime": "2021/03/31 11:17:50",
     "runtime": 0.114321,
     "state": "PASS",
     "returncode": 0,
     "output": "[OK] Root disk is below threshold of 90%\n",
     "error": "",
     "job": null
   },
   "944f6399-b82b-47f9-bb15-8f529dedd4e6": {
     "id": "944f6399",
     "full_id": "944f6399-b82b-47f9-bb15-8f529dedd4e6",
     "schemafile": "script-v1.0.schema.json",
     "executor": "generic.local.python",
     "compiler": null,
     "hostname": "DOE-7086392.local",
     "user": "siddiq90",
     "testroot": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.python/python-shell/circle_area/0",
     "testpath": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.python/python-shell/circle_area/0/stage/generate.sh",
     "stagedir": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.python/python-shell/circle_area/0/stage",
     "rundir": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.python/python-shell/circle_area/0/run",
     "command": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.python/python-shell/circle_area/0/stage/generate.sh",
     "outfile": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.python/python-shell/circle_area/0/run/circle_area.out",
     "errfile": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.python/python-shell/circle_area/0/run/circle_area.err",
     "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  circle_area:\n    executor: generic.local.python\n    type: script\n    shell: python\n    description: \"Calculate circle of area given a radius\"\n    tags: [tutorials, python]\n    run: |\n      import math\n      radius = 2\n      area = math.pi * radius * radius\n      print(\"Circle Radius \", radius)\n      print(\"Area of circle \", area)\n",
     "test_content": "#!/bin/bash\nsource /Users/siddiq90/Documents/github/buildtest/var/executors/generic.local.python/before_script.sh\npython /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.python/python-shell/circle_area/0/stage/circle_area.py\nsource /Users/siddiq90/Documents/github/buildtest/var/executors/generic.local.python/after_script.sh",
     "tags": "tutorials python",
     "starttime": "2021/03/31 11:18:00",
     "endtime": "2021/03/31 11:18:00",
     "runtime": 0.144171,
     "state": "PASS",
     "returncode": 0,
     "output": "Circle Radius  2\nArea of circle  12.566370614359172\n",
     "error": "",
     "job": null
   }
 }

If you specify an invalid test id using buildtest inspect id you will get an error message as follows.

$ buildtest inspect id lad

Unable to find any test records based on id: ['lad'], please run 'buildtest inspect list' to see list of ids.

You will see similar message if you specify an invalid test name using buildtest inspect name command.

Query Test Records via buildtest inspect query

The buildtest inspect query command can allow you to retrieve query certain fields from each test records that can be useful when you are inspecting a test. Currently, we can fetch content of output file, error file, testpath, and build script. Shown below are the list of available options for buildtest inspect query.

$ buildtest inspect query --help
usage: buildtest [options] [COMMANDS] inspect query [-h] [-t] [-o] [-e] [-b] [-d {first,last,all}] [name [name ...]]

positional arguments:
  name                  Name of test

optional arguments:
  -h, --help            show this help message and exit
  -t, --testpath        Print content of testpath
  -o, --output          Print output file
  -e, --error           Print error file
  -b, --buildscript     Print build script
  -d {first,last,all}, --display {first,last,all}
                        Determine how records are fetched, by default it will report the last record of the test.

The buildtest inspect query command expects positional arguments that are name of tests which you can get by running “buildtest inspect name”.

For instance, let’s query the test circle_area by running the following:

$ buildtest inspect query circle_area
______________________________ circle_area (ID: 12a030f6-fd09-48cb-a270-b7cd6fa49c42) ______________________________
description:  Calculate circle of area given a radius
state:  PASS
returncode:  0
runtime:  0.043701
starttime:  2021/07/30 22:02:05
endtime:  2021/07/30 22:02:05

buildtest will display metadata for each test. By default, buildtest will report the latest record for each test that is specified as a positional argument. If you want to see all runs for a particular test you can use -d all or --display all which will report all records. By default, it will use -d last which reports the last record. You can retrieve the first record by running -d first which is the oldest record.

Now as you run test, you want to inspect the output file, this can be done by passing -o or --output. Let’s take what we learned and see the following. In this command, we retrieve all records for circle_area and print content of output file

$ buildtest inspect query -d all -o circle_area
______________________________ circle_area (ID: 1f60989b-5ce4-457c-8532-2655a3447e3a) ______________________________
description:  Calculate circle of area given a radius
state:  PASS
returncode:  0
runtime:  0.043358
starttime:  2021/07/30 22:02:02
endtime:  2021/07/30 22:02:02
************************* Start of Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/1/circle_area.out *************************
Circle Radius  2
Area of circle  12.566370614359172

************************* End of Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/1/circle_area.out *************************

______________________________ circle_area (ID: 0bbd21fa-d186-4924-9ad0-f30d9e7454c1) ______________________________
description:  Calculate circle of area given a radius
state:  PASS
returncode:  0
runtime:  0.043108
starttime:  2021/07/30 22:02:02
endtime:  2021/07/30 22:02:02
************************* Start of Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/2/circle_area.out *************************
Circle Radius  2
Area of circle  12.566370614359172

************************* End of Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/2/circle_area.out *************************

______________________________ circle_area (ID: f1a17ac0-9586-4cb7-9557-5c2e225b7711) ______________________________
description:  Calculate circle of area given a radius
state:  PASS
returncode:  0
runtime:  0.043972
starttime:  2021/07/30 22:02:05
endtime:  2021/07/30 22:02:05
************************* Start of Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/3/circle_area.out *************************
Circle Radius  2
Area of circle  12.566370614359172

************************* End of Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/3/circle_area.out *************************

______________________________ circle_area (ID: d14e4082-64db-4314-8553-8ce752898435) ______________________________
description:  Calculate circle of area given a radius
state:  PASS
returncode:  0
runtime:  0.043493
starttime:  2021/07/30 22:02:05
endtime:  2021/07/30 22:02:05
************************* Start of Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/4/circle_area.out *************************
Circle Radius  2
Area of circle  12.566370614359172

************************* End of Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/4/circle_area.out *************************

______________________________ circle_area (ID: 12a030f6-fd09-48cb-a270-b7cd6fa49c42) ______________________________
description:  Calculate circle of area given a radius
state:  PASS
returncode:  0
runtime:  0.043701
starttime:  2021/07/30 22:02:05
endtime:  2021/07/30 22:02:05
************************* Start of Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/5/circle_area.out *************************
Circle Radius  2
Area of circle  12.566370614359172

************************* End of Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/5/circle_area.out *************************

If you want to see content of error file use the -e or --error flag. It would be useful to inspect content of build script and generated test, which can be retrieved using --testpath and --buildscript. Let’s see query the first record of circle_area and report all of the content fields

$ buildtest inspect query -d first -o -e -t -b circle_area
______________________________ circle_area (ID: 1f60989b-5ce4-457c-8532-2655a3447e3a) ______________________________
description:  Calculate circle of area given a radius
state:  PASS
returncode:  0
runtime:  0.043358
starttime:  2021/07/30 22:02:02
endtime:  2021/07/30 22:02:02
************************* Start of Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/1/circle_area.out *************************
Circle Radius  2
Area of circle  12.566370614359172

************************* End of Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/1/circle_area.out *************************

************************* Start of Error File:  /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/1/circle_area.err *************************
circle_area_build.sh: 14: circle_area_build.sh: source: not found

************************* End of Error File:  /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/1/circle_area.err *************************

************************* Start of Test Path:  /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/1/circle_area.sh *************************
#!/bin/bash
python /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/1/stage/circle_area.py
************************* End of Test Path:  /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/1/circle_area.sh *************************

************************* Start of Build Script:  /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/1/circle_area_build.sh *************************
#!/bin/bash


############# START VARIABLE DECLARATION ########################
export BUILDTEST_TEST_NAME=circle_area
export BUILDTEST_TEST_ROOT=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/1
export BUILDTEST_BUILDSPEC_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials
export BUILDTEST_STAGE_DIR=/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/1/stage
export BUILDTEST_TEST_ID=1f60989b-5ce4-457c-8532-2655a3447e3a
############# END VARIABLE DECLARATION   ########################


# source executor startup script
source /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/executor/generic.local.python/before_script.sh
# Run generated script
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/1/stage/circle_area.sh
# Get return code
returncode=$?
# Exit with return code
exit $returncode
************************* End of Build Script:  /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/1/circle_area_build.sh *************************

We can query multiple tests using buildtest inspect query since each test is a positional argument. Any options specified to buildtest inspect query will be applied to all test. For instance, let’s fetch the output the of test names root_disk_usage and python_hello

$ buildtest inspect query -o  root_disk_usage python_hello
______________________________ root_disk_usage (ID: a7303c75-2411-4388-98ec-a8af6935b812) ______________________________
description:  Check root disk usage and report if it exceeds threshold
state:  PASS
returncode:  0
runtime:  0.010837
starttime:  2021/07/30 22:02:01
endtime:  2021/07/30 22:02:01
************************* Start of Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.bash/disk_usage/root_disk_usage/2/root_disk_usage.out *************************
[OK] Root disk is below threshold of 90%

************************* End of Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.bash/disk_usage/root_disk_usage/2/root_disk_usage.out *************************

______________________________ python_hello (ID: 82bc0254-5dd2-4852-b13d-d286fbd2ef33) ______________________________
description:  Hello World python
state:  PASS
returncode:  0
runtime:  0.045624
starttime:  2021/07/30 22:02:02
endtime:  2021/07/30 22:02:02
************************* Start of Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.bash/python-hello/python_hello/2/python_hello.out *************************
Hello World

************************* End of Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.bash/python-hello/python_hello/2/python_hello.out *************************

Using Alternate Report File

The buildtest report and buildtest inspect command will read from the report file tracked by buildtest which is stored in $BUILDTEST_ROOT/var/report.json. This single file can became an issue if you are running jobs through CI where you can potentially overwrite same file or if you want separate report files for each set of builds. Luckily we have an option to handle this using the buildtest build -r /path/to/report option which can be used to specify an alternate location to report file.

buildtest will write the report file in the desired location, then you can specify the path to report file via buildtest report -r /path/to/report and buildtest inspect -r /path/to/report to load the report file when reporting tests.

The report file must be valid JSON file that buildtest understands in order to use buildtest report and buildtest inspect command. Shown below are some examples using the alternate report file using buildtest report and buildtest inspect command.

$ buildtest report -r python.json --format name,id
Reading report file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/docs/python.json

+--------------+----------+
| name         | id       |
+==============+==========+
| circle_area  | 6be6c404 |
+--------------+----------+
| python_hello | f21ba744 |
+--------------+----------+
$ buildtest inspect -r test.json name variables_bash
Reading Report File: /Users/siddiq90/Documents/GitHubDesktop/buildtest/test.json

{
  "variables_bash": [
    {
      "id": "cd0511ce",
      "full_id": "cd0511ce-377e-4ed2-95f4-f244e5518732",
      "schemafile": "script-v1.0.schema.json",
      "executor": "generic.local.bash",
      "compiler": null,
      "hostname": "DOE-7086392.local",
      "user": "siddiq90",
      "testroot": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/1",
      "testpath": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/1/stage/generate.sh",
      "stagedir": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/1/stage",
      "rundir": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/1/run",
      "command": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/1/stage/generate.sh",
      "outfile": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/1/run/variables_bash.out",
      "errfile": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/1/run/variables_bash.err",
      "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  variables_bash:\n    type: script\n    executor: generic.local.bash\n    description: Declare shell variables in bash\n    tags: [tutorials]\n    vars:\n      X: 1\n      Y: 2\n      literalstring: |\n        \"this is a literal string ':' \"\n      singlequote: \"'singlequote'\"\n      doublequote: \"\\\"doublequote\\\"\"\n      current_user: \"$(whoami)\"\n      files_homedir: \"`find $HOME -type f -maxdepth 1`\"\n\n    run: |\n      echo \"$X+$Y=\" $(($X+$Y))\n      echo $literalstring\n      echo $singlequote\n      echo $doublequote\n\n      echo $current_user\n      echo $files_homedir",
      "test_content": "#!/bin/bash \nsource /Users/siddiq90/.buildtest/executor/generic.local.bash/before_script.sh\nX=1\nY=2\nliteralstring=\"this is a literal string ':' \"\n\nsinglequote='singlequote'\ndoublequote=\"doublequote\"\ncurrent_user=$(whoami)\nfiles_homedir=`find $HOME -type f -maxdepth 1`\necho \"$X+$Y=\" $(($X+$Y))\necho $literalstring\necho $singlequote\necho $doublequote\n\necho $current_user\necho $files_homedir\nsource /Users/siddiq90/.buildtest/executor/generic.local.bash/after_script.sh",
      "tags": "tutorials",
      "starttime": "2021/04/16 14:29:25",
      "endtime": "2021/04/16 14:29:25",
      "runtime": 0.213196,
      "state": "PASS",
      "returncode": 0,
      "output": "1+2= 3\nthis is a literal string ':'\nsinglequote\ndoublequote\nsiddiq90\n/Users/siddiq90/buildtest_e7yxgttm.log /Users/siddiq90/.anyconnect /Users/siddiq90/buildtest_utwigb8w.log /Users/siddiq90/.DS_Store /Users/siddiq90/.serverauth.555 /Users/siddiq90/.CFUserTextEncoding /Users/siddiq90/.wget-hsts /Users/siddiq90/.bashrc /Users/siddiq90/.zshrc /Users/siddiq90/.coverage /Users/siddiq90/.serverauth.87055 /Users/siddiq90/buildtest_r7bck5zh.log /Users/siddiq90/.zsh_history /Users/siddiq90/.lesshst /Users/siddiq90/calltracker.py /Users/siddiq90/.git-completion.bash /Users/siddiq90/buildtest_wvjaaztp.log /Users/siddiq90/buildtest.log /Users/siddiq90/darhan.log /Users/siddiq90/ascent.yml /Users/siddiq90/.cshrc /Users/siddiq90/buildtest_nyq22whj.log /Users/siddiq90/github-tokens /Users/siddiq90/buildtest_ozb8b52z.log /Users/siddiq90/.zcompdump /Users/siddiq90/buildtest_nab_ckph.log /Users/siddiq90/.serverauth.543 /Users/siddiq90/.s.PGSQL.15007.lock /Users/siddiq90/.bash_profile /Users/siddiq90/.Xauthority /Users/siddiq90/.python_history /Users/siddiq90/.gitconfig /Users/siddiq90/output.txt /Users/siddiq90/.bash_history /Users/siddiq90/.viminfo\n",
      "error": "",
      "job": null
    },
    {
      "id": "e0901505",
      "full_id": "e0901505-a66b-4c91-9b29-d027cb6fabb6",
      "schemafile": "script-v1.0.schema.json",
      "executor": "generic.local.bash",
      "compiler": null,
      "hostname": "DOE-7086392.local",
      "user": "siddiq90",
      "testroot": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/2",
      "testpath": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/2/stage/generate.sh",
      "stagedir": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/2/stage",
      "rundir": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/2/run",
      "command": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/2/stage/generate.sh",
      "outfile": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/2/run/variables_bash.out",
      "errfile": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/2/run/variables_bash.err",
      "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  variables_bash:\n    type: script\n    executor: generic.local.bash\n    description: Declare shell variables in bash\n    tags: [tutorials]\n    vars:\n      X: 1\n      Y: 2\n      literalstring: |\n        \"this is a literal string ':' \"\n      singlequote: \"'singlequote'\"\n      doublequote: \"\\\"doublequote\\\"\"\n      current_user: \"$(whoami)\"\n      files_homedir: \"`find $HOME -type f -maxdepth 1`\"\n\n    run: |\n      echo \"$X+$Y=\" $(($X+$Y))\n      echo $literalstring\n      echo $singlequote\n      echo $doublequote\n\n      echo $current_user\n      echo $files_homedir",
      "test_content": "#!/bin/bash \nsource /Users/siddiq90/.buildtest/executor/generic.local.bash/before_script.sh\nX=1\nY=2\nliteralstring=\"this is a literal string ':' \"\n\nsinglequote='singlequote'\ndoublequote=\"doublequote\"\ncurrent_user=$(whoami)\nfiles_homedir=`find $HOME -type f -maxdepth 1`\necho \"$X+$Y=\" $(($X+$Y))\necho $literalstring\necho $singlequote\necho $doublequote\n\necho $current_user\necho $files_homedir\nsource /Users/siddiq90/.buildtest/executor/generic.local.bash/after_script.sh",
      "tags": "tutorials",
      "starttime": "2021/04/16 14:29:58",
      "endtime": "2021/04/16 14:29:58",
      "runtime": 0.075224,
      "state": "PASS",
      "returncode": 0,
      "output": "1+2= 3\nthis is a literal string ':'\nsinglequote\ndoublequote\nsiddiq90\n/Users/siddiq90/buildtest_e7yxgttm.log /Users/siddiq90/.anyconnect /Users/siddiq90/buildtest_utwigb8w.log /Users/siddiq90/.DS_Store /Users/siddiq90/.serverauth.555 /Users/siddiq90/.CFUserTextEncoding /Users/siddiq90/.wget-hsts /Users/siddiq90/.bashrc /Users/siddiq90/.zshrc /Users/siddiq90/.coverage /Users/siddiq90/.serverauth.87055 /Users/siddiq90/buildtest_r7bck5zh.log /Users/siddiq90/.zsh_history /Users/siddiq90/.lesshst /Users/siddiq90/calltracker.py /Users/siddiq90/.git-completion.bash /Users/siddiq90/buildtest_wvjaaztp.log /Users/siddiq90/buildtest.log /Users/siddiq90/darhan.log /Users/siddiq90/ascent.yml /Users/siddiq90/.cshrc /Users/siddiq90/buildtest_nyq22whj.log /Users/siddiq90/github-tokens /Users/siddiq90/buildtest_ozb8b52z.log /Users/siddiq90/.zcompdump /Users/siddiq90/buildtest_nab_ckph.log /Users/siddiq90/.serverauth.543 /Users/siddiq90/.s.PGSQL.15007.lock /Users/siddiq90/.bash_profile /Users/siddiq90/.Xauthority /Users/siddiq90/.python_history /Users/siddiq90/.gitconfig /Users/siddiq90/output.txt /Users/siddiq90/.bash_history /Users/siddiq90/.viminfo\n",
      "error": "",
      "job": null
    }
  ]
}