Query Test Report

buildtest keeps track of all tests and results in a JSON file. This file is read by buildtest report command to extract certain fields from JSON file and display them in table format. We use python tabulate library for pretty print data in tables. Shown below is command usage to query test reports.

$ buildtest report --help
usage: buildtest [options] [COMMANDS] report [-h] [--helpformat] [--helpfilter] [--format FORMAT] [--filter FILTER]
                                             [--latest] [--oldest] [-r REPORT]
                                             ...

optional arguments:
  -h, --help            show this help message and exit
  --helpformat          List of available format fields
  --helpfilter          List available filter fields to be used with --filter option
  --format FORMAT       format field for printing purposes. For more details see --helpformat for list of available
                        fields. Fields must be separated by comma (usage: --format <field1>,<field2>,...)
  --filter FILTER       Filter report by filter fields. The filter fields must be a key=value pair and multiple fields
                        can be comma separated in the following format: --filter key1=val1,key2=val2 . For list of
                        filter fields run: --helpfilter.
  --latest              Retrieve latest record of particular test
  --oldest              Retrieve oldest record of particular test
  -r REPORT, --report REPORT
                        Specify a report file to read

subcommands:
  Fetch test results from report file and print them in table format

  
    clear               delete report file
    list                List all report files

You may run buildtest report without any option, and buildtest will display all test results with default format fields. To see a list of all format fields, click here.

$ buildtest report
Reading report file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/report.json 

+------------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| name                         | id       | state   |   returncode | starttime           | endtime             |   runtime | tags               | buildspec                                                                                                                           |
+==============================+==========+=========+==============+=====================+=====================+===========+====================+=====================================================================================================================================+
| exit1_fail                   | ae3ff33a | FAIL    |            1 | 2021/07/30 15:08:30 | 2021/07/30 15:08:30 |  0.004566 | tutorials fail     | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/pass_returncode.yml                          |
+------------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| exit1_fail                   | 84c02c94 | FAIL    |            1 | 2021/07/30 15:08:53 | 2021/07/30 15:08:53 |  0.004414 | tutorials fail     | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/pass_returncode.yml                          |
+------------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| exit1_fail                   | 93ad2bb3 | FAIL    |            1 | 2021/07/30 15:08:54 | 2021/07/30 15:08:54 |  0.004397 | tutorials fail     | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/pass_returncode.yml                          |
+------------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| exit1_fail                   | e1052582 | FAIL    |            1 | 2021/07/30 15:08:56 | 2021/07/30 15:08:56 |  0.00506  | tutorials fail     | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/pass_returncode.yml                          |
+------------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| exit1_fail                   | 65444282 | FAIL    |            1 | 2021/07/30 15:08:56 | 2021/07/30 15:08:56 |  0.004771 | tutorials fail     | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/pass_returncode.yml                          |
+------------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| exit1_pass                   | f8323253 | PASS    |            1 | 2021/07/30 15:08:30 | 2021/07/30 15:08:30 |  0.004372 | tutorials pass     | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/pass_returncode.yml                          |
+------------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| exit1_pass                   | 3e06bf49 | PASS    |            1 | 2021/07/30 15:08:53 | 2021/07/30 15:08:53 |  0.004803 | tutorials pass     | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/pass_returncode.yml                          |
+------------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| exit1_pass                   | dec97567 | PASS    |            1 | 2021/07/30 15:08:54 | 2021/07/30 15:08:54 |  0.004444 | tutorials pass     | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/pass_returncode.yml                          |
...

Format Reports

Available Format Fields

The buildtest report command displays a default format fields that can be changed using the --format option. The report file (JSON) contains many more fields and we expose some of the fields with the –format option. To see a list of available format fields you can run buildtest report --helpformat. This option will list all format fields with their description.

$ buildtest report --helpformat
Fields      Description
----------  ----------------------------------------------------------------
buildspec   Buildspec file
command     Command executed
compiler    Retrieve compiler used for test (applicable for compiler schema)
endtime     End Time for Test in date format
errfile     Error File
executor    Executor name
hostname    Retrieve hostname of machine where job was submitted from
full_id     Full qualified unique build identifier
id          Unique Build Identifier (abbreviated)
name        Name of test defined in buildspec
outfile     Output file
returncode  Return Code from Test Execution
runtime     Total runtime in seconds
schemafile  Schema file used for validation
starttime   Start Time of test in date format
state       Test State reported by buildtest (PASS/FAIL)
tags        Tag name
testroot    Root of test directory
testpath    Path to test
user        Get user who submitted job

Format Field Usage

The --format field are specified in comma separated format (i.e --format <field1>,<field2>). In this example we format table by fields --format id,executor,state,returncode.

$ buildtest report --format name,id,executor,state,returncode
Reading report file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/report.json 

+------------------------------+----------+----------------------+---------+--------------+
| name                         | id       | executor             | state   |   returncode |
+==============================+==========+======================+=========+==============+
| exit1_fail                   | ae3ff33a | generic.local.sh     | FAIL    |            1 |
+------------------------------+----------+----------------------+---------+--------------+
| exit1_fail                   | 84c02c94 | generic.local.sh     | FAIL    |            1 |
+------------------------------+----------+----------------------+---------+--------------+
| exit1_fail                   | 93ad2bb3 | generic.local.sh     | FAIL    |            1 |
+------------------------------+----------+----------------------+---------+--------------+
| exit1_fail                   | e1052582 | generic.local.sh     | FAIL    |            1 |
+------------------------------+----------+----------------------+---------+--------------+
| exit1_fail                   | 65444282 | generic.local.sh     | FAIL    |            1 |
+------------------------------+----------+----------------------+---------+--------------+
| exit1_pass                   | f8323253 | generic.local.sh     | PASS    |            1 |
+------------------------------+----------+----------------------+---------+--------------+
| exit1_pass                   | 3e06bf49 | generic.local.sh     | PASS    |            1 |
+------------------------------+----------+----------------------+---------+--------------+
| exit1_pass                   | dec97567 | generic.local.sh     | PASS    |            1 |
+------------------------------+----------+----------------------+---------+--------------+
...

Filter Reports

The buildtest report command will display all tests results, which can be quite long depending on number of tests so therefore we need a mechanism to filter the test results. The --filter option can be used to filter out tests in the output based on filter fields. First, lets see the available filter fields by run buildtest report --helpfilter which shows a list of filter fields and their description.

$ buildtest report --helpfilter
Filter Fields    Description                 Expected Value
---------------  --------------------------  ----------------
buildspec        Filter by buildspec file    FILE
name             Filter by test name         STRING
executor         Filter by executor name     STRING
state            Filter by test state        PASS/FAIL
tags             Filter tests by tag name    STRING
returncode       Filter tests by returncode  INT

The --filter option expects arguments in key=value format. You can specify multiple filter delimited by comma. buildtest will treat multiple filters as logical AND operation. The filter option can be used with --format field. Let’s see some examples to illustrate the point.

Filter by returncode

If you want to retrieve all tests with a given returncode, we can use the returncode property. For instance, let’s retrieve all tests with returncode of 2 by setting --filter returncode=2.

$ buildtest report --filter returncode=2 --format=name,id,returncode
Reading report file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/report.json 

+--------------------------+----------+--------------+
| name                     | id       |   returncode |
+==========================+==========+==============+
| returncode_list_mismatch | a2c8f242 |            2 |
+--------------------------+----------+--------------+
| returncode_list_mismatch | ed0a8187 |            2 |
+--------------------------+----------+--------------+
| returncode_list_mismatch | 8a13233b |            2 |
+--------------------------+----------+--------------+
| returncode_list_mismatch | 21d2f5cb |            2 |
+--------------------------+----------+--------------+
| returncode_list_mismatch | c13bc1ff |            2 |
+--------------------------+----------+--------------+

Note

buildtest automatically converts returncode to integer when matching returncode, so --filter returncode="2" will work too

Filter by test name

If you want to filter by test name, use the name attribute in filter option. Let’s assume we want to filter all tests by name exit1_pass, this can be achieved by setting filter field as follows: --filter name=exit1_pass. Shown below is an example using name filter field to filter test results.

$ buildtest report --filter name=exit1_pass --format=name,id,returncode,state
Reading report file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/report.json 

+------------+----------+--------------+---------+
| name       | id       |   returncode | state   |
+============+==========+==============+=========+
| exit1_pass | f8323253 |            1 | PASS    |
+------------+----------+--------------+---------+
| exit1_pass | 3e06bf49 |            1 | PASS    |
+------------+----------+--------------+---------+
| exit1_pass | dec97567 |            1 | PASS    |
+------------+----------+--------------+---------+
| exit1_pass | 7c74495b |            1 | PASS    |
+------------+----------+--------------+---------+
| exit1_pass | 74ba735f |            1 | PASS    |
+------------+----------+--------------+---------+
| exit1_pass | 68a9fdf0 |            1 | PASS    |
+------------+----------+--------------+---------+

Filter by buildspec

Likewise, we can filter results by buildspec file using buildspec attribute via --filter buildspec=<file>. The buildspec attribute must resolve to a file path which can be relative or absolute path. buildtest will resolve path (absolute path) and find the appropriate tests that belong to the buildspec file. If file doesn’t exist or is not found in cache it will raise an error.

$ buildtest report --filter buildspec=tutorials/python-hello.yml --format=name,id,state,buildspec
Reading report file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/report.json 

+--------------+----------+---------+---------------------------------------------------------------------------------------------------------+
| name         | id       | state   | buildspec                                                                                               |
+==============+==========+=========+=========================================================================================================+
| python_hello | 04a87317 | PASS    | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/python-hello.yml |
+--------------+----------+---------+---------------------------------------------------------------------------------------------------------+
| python_hello | de751b51 | PASS    | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/python-hello.yml |
+--------------+----------+---------+---------------------------------------------------------------------------------------------------------+

Filter by test state

If you want to filter results by test state, use the state property. This can be useful if you want to know all pass or failed tests. The state property expects value of [PASS|FAIL] since these are the two recorded test states marked by buildtest. We can also pass multiple filter fields for instance if we want to find all FAIL tests for executor generic.local.sh we can do the following.

$ buildtest report --filter state=FAIL,executor=generic.local.sh --format=name,id,state,executor
Reading report file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/report.json 

+--------------------------+----------+---------+------------------+
| name                     | id       | state   | executor         |
+==========================+==========+=========+==================+
| exit1_fail               | ae3ff33a | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| exit1_fail               | 84c02c94 | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| exit1_fail               | 93ad2bb3 | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| exit1_fail               | e1052582 | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| exit1_fail               | 65444282 | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| returncode_list_mismatch | a2c8f242 | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| returncode_list_mismatch | ed0a8187 | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| returncode_list_mismatch | 8a13233b | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| returncode_list_mismatch | 21d2f5cb | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| returncode_list_mismatch | c13bc1ff | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| timelimit_min_fail       | d6e452ef | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+
| timelimit_max_fail       | 9d91d66e | FAIL    | generic.local.sh |
+--------------------------+----------+---------+------------------+

Filter Exception Cases

The returncode filter field expects an integer value, so if you try a non-integer returncode you will get the following message

$ buildtest report --filter returncode=1.5
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/bin/buildtest", line 17, in <module>
    buildtest.main.main()
  File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/buildtest/main.py", line 124, in main
    report_cmd(args)
  File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/buildtest/cli/report.py", line 439, in report_cmd
    results = Report(args.filter, args.format, args.latest, args.oldest, args.report)
  File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/buildtest/cli/report.py", line 72, in __init__
    self._check_filter_fields()
  File "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/buildtest/cli/report.py", line 97, in _check_filter_fields
    f"Invalid returncode:{self.filter[key]} must be an integer"
buildtest.exceptions.BuildTestError: 'Invalid returncode:1.5 must be an integer'

The state filter field expects value of PASS or FAIL so if you specify an invalid state you will get an error as follows.

$ buildtest report --filter state=UNKNOWN
filter argument 'state' must be 'PASS' or 'FAIL' got value UNKNOWN

The buildspec field expects a valid file path, it can be an absolute or relative path, buildtest will resolve absolute path and check if file exist and is in the report file. If it’s an invalid file we get an error such as

$ buildtest report --filter buildspec=/path/to/invalid.yml
Invalid File Path for filter field 'buildspec': /path/to/invalid.yml

You may have a valid filepath for buildspec filter field such as $BUILDTEST_ROOT/tutorials/invalid_executor.yml, but there is no record of a test in the report cache because this test wasn’t run. In this case you will get the following message.

$ buildtest report --filter buildspec=$BUILDTEST_ROOT/tutorials/invalid_executor.yml
buildspec file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/tutorials/invalid_executor.yml not found in cache

Find Latest or Oldest test

We can search for oldest or latest test for any given test. This can be useful if you want to see first or last test run. If you want to retrieve the oldest test you can use --oldest option. buildtest will append tests, therefore last record in dictionary will be latest record, similarly first record is the oldest record.

Let’s take a look at this example, we filter by test name hello_f which retrieves three entries. Now let’s filter by oldest record by specifying –oldest option and it will retrieve the first record which is test id 349f3ada.

$ buildtest report --filter name=hello_f --format name,id,starttime
 Reading Report File: /Users/siddiq90/.buildtest/report.json

 +---------+----------+---------------------+
 | name    | id       | starttime           |
 +=========+==========+=====================+
 | hello_f | 349f3ada | 2021/02/11 18:13:08 |
 +---------+----------+---------------------+
 | hello_f | ecd4a3f2 | 2021/02/11 18:13:18 |
 +---------+----------+---------------------+
 | hello_f | 5c87978b | 2021/02/11 18:13:33 |
 +---------+----------+---------------------+

 $ buildtest report --filter name=hello_f --format name,id,starttime --oldest
 Reading Report File: /Users/siddiq90/.buildtest/report.json

 +---------+----------+---------------------+
 | name    | id       | starttime           |
 +=========+==========+=====================+
 | hello_f | 349f3ada | 2021/02/11 18:13:08 |
 +---------+----------+---------------------+

If you want to retrieve the latest test result you can use --latest option which will retrieve the last record, in the same example we will retrieve test id 5c87978b.

$ buildtest report --filter name=hello_f --format name,id,starttime --latest
Reading Report File: /Users/siddiq90/.buildtest/report.json

+---------+----------+---------------------+
| name    | id       | starttime           |
+=========+==========+=====================+
| hello_f | 5c87978b | 2021/02/11 18:13:33 |
+---------+----------+---------------------+

You may combine –oldest and –latest options in same command, in this case buildtest will retrieve the first and last record of every test.

$ buildtest report --format name,id,starttime --oldest --latest | more
Reading Report File: /Users/siddiq90/.buildtest/report.json

+------------------------------+----------+---------------------+
| name                         | id       | starttime           |
+==============================+==========+=====================+
| variables_bash               | 750f48bc | 2021/02/11 18:13:03 |
+------------------------------+----------+---------------------+
| variables_bash               | 1bdfd403 | 2021/02/11 18:13:32 |
+------------------------------+----------+---------------------+
| ulimit_filelock_unlimited    | b7b852e4 | 2021/02/11 18:13:03 |
+------------------------------+----------+---------------------+
| ulimit_filelock_unlimited    | 56345a43 | 2021/02/11 18:13:18 |
+------------------------------+----------+---------------------+

Inspect Tests Records

In previous examples we saw how we can retrieve test records using buildtest report which is printed in table format. We have limited the output to a limited fields however, if you want to analyze a particular, we have a separate command called buildtest inspect that can be used for inspecting a test record based on name or id. Shown below is the command usage for buildtest inspect command.

$ buildtest inspect --help
usage: buildtest [options] [COMMANDS] inspect [-h] [-r REPORT]  ...

optional arguments:
  -h, --help            show this help message and exit
  -r REPORT, --report REPORT
                        Specify a report file to load when inspecting test

subcommands:
  Inspect Test result based on Test ID or Test Name

  
    name                Specify name of test
    id                  Specify a Test ID
    list                List all test ids

You can report all test names and corresponding ids using buildtest inspect list which will be used for querying tests by name or id.

$ buildtest inspect list
+------------------------------+--------------------------------------+
| name                         | id                                   |
+==============================+======================================+
| exit1_fail                   | ae3ff33a-161a-4ca0-b3a1-eafbd1dedf5e |
+------------------------------+--------------------------------------+
| exit1_fail                   | 84c02c94-9890-4e9f-9e22-393d70ea7436 |
+------------------------------+--------------------------------------+
| exit1_fail                   | 93ad2bb3-b779-47a7-b589-1e10a8c7fdf4 |
+------------------------------+--------------------------------------+
| exit1_fail                   | e1052582-0c75-4bfa-82d0-5d8b55b07f6f |
+------------------------------+--------------------------------------+
| exit1_fail                   | 65444282-0e03-45d6-a050-0825dcee3d61 |
+------------------------------+--------------------------------------+
| exit1_pass                   | f8323253-0578-425c-877e-db50756a07de |
+------------------------------+--------------------------------------+
| exit1_pass                   | 3e06bf49-230a-4b16-9804-cdbeda798ce4 |
+------------------------------+--------------------------------------+
| exit1_pass                   | dec97567-03b8-4af0-8c76-2c9a4da48e37 |
+------------------------------+--------------------------------------+
| exit1_pass                   | 7c74495b-825b-4300-a702-514ae61bb5d4 |
...

Inspecting Test by Name

The buildtest inspect name expects a list of positional argument that correspond to name of test you want to query and buildtest will fetch all records for each named test. Let’s see an example to illustrate the point. We can see that each test is stored as a JSON format and buildtest keeps track of metadata for each test such as user, hostname, command, path to output and error file, content of test, state of test, returncode, etc… In this example, we will retrieve record for test name circle_area which will print the raw content of the test in JSON format.

$ buildtest inspect name circle_area
{
  "circle_area": [
    {
      "id": "178ef5ce",
      "full_id": "178ef5ce-35b4-40bb-bc40-a619b8b52ac9",
      "description": "Calculate circle of area given a radius",
      "schemafile": "script-v1.0.schema.json",
      "executor": "generic.local.python",
      "compiler": null,
      "hostname": "build-14361642-project-280831-buildtest",
      "user": "docs",
      "testroot": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/1",
      "testpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/1/stage/circle_area.sh",
      "stagedir": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/1/stage",
      "command": "sh circle_area_build.sh",
      "outfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/1/circle_area.out",
      "errfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/1/circle_area.err",
      "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  circle_area:\n    executor: generic.local.python\n    type: script\n    shell: python\n    description: \"Calculate circle of area given a radius\"\n    tags: [tutorials, python]\n    run: |\n      import math\n      radius = 2\n      area = math.pi * radius * radius\n      print(\"Circle Radius \", radius)\n      print(\"Area of circle \", area)\n",
      "test_content": "#!/bin/bash\npython /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/1/stage/circle_area.py",
      "logpath": "/tmp/buildtest_u8hjw81_.log",
      "tags": "tutorials python",
      "starttime": "2021/07/30 15:08:53",
      "endtime": "2021/07/30 15:08:53",
      "runtime": 0.044649,
      "state": "PASS",
      "returncode": 0,
      "output": "Circle Radius  2\nArea of circle  12.566370614359172\n",
      "error": "circle_area_build.sh: 14: circle_area_build.sh: source: not found\n",
      "job": null,
      "build_script": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/1/circle_area_build.sh"
    },
    {
      "id": "3475676d",
      "full_id": "3475676d-2ed6-48e0-be38-f53e1ac2d8b7",
      "description": "Calculate circle of area given a radius",
      "schemafile": "script-v1.0.schema.json",
      "executor": "generic.local.python",
      "compiler": null,
      "hostname": "build-14361642-project-280831-buildtest",
      "user": "docs",
      "testroot": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/2",
      "testpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/2/stage/circle_area.sh",
      "stagedir": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/2/stage",
      "command": "sh circle_area_build.sh",
      "outfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/2/circle_area.out",
      "errfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/2/circle_area.err",
      "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  circle_area:\n    executor: generic.local.python\n    type: script\n    shell: python\n    description: \"Calculate circle of area given a radius\"\n    tags: [tutorials, python]\n    run: |\n      import math\n      radius = 2\n      area = math.pi * radius * radius\n      print(\"Circle Radius \", radius)\n      print(\"Area of circle \", area)\n",
      "test_content": "#!/bin/bash\npython /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/2/stage/circle_area.py",
      "logpath": "/tmp/buildtest_3p596gw_.log",
      "tags": "tutorials python",
      "starttime": "2021/07/30 15:08:54",
      "endtime": "2021/07/30 15:08:54",
      "runtime": 0.042053,
      "state": "PASS",
      "returncode": 0,
      "output": "Circle Radius  2\nArea of circle  12.566370614359172\n",
      "error": "circle_area_build.sh: 14: circle_area_build.sh: source: not found\n",
      "job": null,
      "build_script": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/2/circle_area_build.sh"
    },
    {
      "id": "a1f47ba1",
      "full_id": "a1f47ba1-caa5-4eeb-a1a4-dfed987ef2b5",
      "description": "Calculate circle of area given a radius",
      "schemafile": "script-v1.0.schema.json",
      "executor": "generic.local.python",
      "compiler": null,
      "hostname": "build-14361642-project-280831-buildtest",
      "user": "docs",
      "testroot": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/3",
      "testpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/3/stage/circle_area.sh",
      "stagedir": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/3/stage",
      "command": "sh circle_area_build.sh",
      "outfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/3/circle_area.out",
      "errfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/3/circle_area.err",
      "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  circle_area:\n    executor: generic.local.python\n    type: script\n    shell: python\n    description: \"Calculate circle of area given a radius\"\n    tags: [tutorials, python]\n    run: |\n      import math\n      radius = 2\n      area = math.pi * radius * radius\n      print(\"Circle Radius \", radius)\n      print(\"Area of circle \", area)\n",
      "test_content": "#!/bin/bash\npython /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/3/stage/circle_area.py",
      "logpath": "/tmp/buildtest_krchtpxi.log",
      "tags": "tutorials python",
      "starttime": "2021/07/30 15:08:55",
      "endtime": "2021/07/30 15:08:56",
      "runtime": 0.04379,
      "state": "PASS",
      "returncode": 0,
      "output": "Circle Radius  2\nArea of circle  12.566370614359172\n",
      "error": "circle_area_build.sh: 14: circle_area_build.sh: source: not found\n",
      "job": null,
      "build_script": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/3/circle_area_build.sh"
    },
    {
      "id": "66d02fbd",
      "full_id": "66d02fbd-fd86-4710-95b9-d7c33f252523",
      "description": "Calculate circle of area given a radius",
      "schemafile": "script-v1.0.schema.json",
      "executor": "generic.local.python",
      "compiler": null,
      "hostname": "build-14361642-project-280831-buildtest",
      "user": "docs",
      "testroot": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/4",
      "testpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/4/stage/circle_area.sh",
      "stagedir": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/4/stage",
      "command": "sh circle_area_build.sh",
      "outfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/4/circle_area.out",
      "errfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/4/circle_area.err",
      "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  circle_area:\n    executor: generic.local.python\n    type: script\n    shell: python\n    description: \"Calculate circle of area given a radius\"\n    tags: [tutorials, python]\n    run: |\n      import math\n      radius = 2\n      area = math.pi * radius * radius\n      print(\"Circle Radius \", radius)\n      print(\"Area of circle \", area)\n",
      "test_content": "#!/bin/bash\npython /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/4/stage/circle_area.py",
      "logpath": "/tmp/buildtest_krchtpxi.log",
      "tags": "tutorials python",
      "starttime": "2021/07/30 15:08:56",
      "endtime": "2021/07/30 15:08:56",
      "runtime": 0.042957,
      "state": "PASS",
      "returncode": 0,
      "output": "Circle Radius  2\nArea of circle  12.566370614359172\n",
      "error": "circle_area_build.sh: 14: circle_area_build.sh: source: not found\n",
      "job": null,
      "build_script": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/4/circle_area_build.sh"
    },
    {
      "id": "020608ed",
      "full_id": "020608ed-c2cd-486f-9ac8-1146edf20c25",
      "description": "Calculate circle of area given a radius",
      "schemafile": "script-v1.0.schema.json",
      "executor": "generic.local.python",
      "compiler": null,
      "hostname": "build-14361642-project-280831-buildtest",
      "user": "docs",
      "testroot": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/5",
      "testpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/5/stage/circle_area.sh",
      "stagedir": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/5/stage",
      "command": "sh circle_area_build.sh",
      "outfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/5/circle_area.out",
      "errfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/5/circle_area.err",
      "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  circle_area:\n    executor: generic.local.python\n    type: script\n    shell: python\n    description: \"Calculate circle of area given a radius\"\n    tags: [tutorials, python]\n    run: |\n      import math\n      radius = 2\n      area = math.pi * radius * radius\n      print(\"Circle Radius \", radius)\n      print(\"Area of circle \", area)\n",
      "test_content": "#!/bin/bash\npython /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/5/stage/circle_area.py",
      "logpath": "/tmp/buildtest_krchtpxi.log",
      "tags": "tutorials python",
      "starttime": "2021/07/30 15:08:56",
      "endtime": "2021/07/30 15:08:56",
      "runtime": 0.042338,
      "state": "PASS",
      "returncode": 0,
      "output": "Circle Radius  2\nArea of circle  12.566370614359172\n",
      "error": "circle_area_build.sh: 14: circle_area_build.sh: source: not found\n",
      "job": null,
      "build_script": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.python/python-shell/circle_area/5/circle_area_build.sh"
    }
  ]
}

You can query multiple tests as positional arguments in the format: buildtest inspect name <test1> <test2> In this next example, we will retrieve test records for bash_shell and python_hello.

$ buildtest inspect name bash_shell python_hello
{
  "python_hello": [
    {
      "id": "04a87317",
      "full_id": "04a87317-adf9-419b-8ade-863d58f0fe72",
      "description": "Hello World python",
      "schemafile": "script-v1.0.schema.json",
      "executor": "generic.local.bash",
      "compiler": null,
      "hostname": "build-14361642-project-280831-buildtest",
      "user": "docs",
      "testroot": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.bash/python-hello/python_hello/1",
      "testpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.bash/python-hello/python_hello/1/stage/python_hello.sh",
      "stagedir": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.bash/python-hello/python_hello/1/stage",
      "command": "sh python_hello_build.sh",
      "outfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.bash/python-hello/python_hello/1/python_hello.out",
      "errfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.bash/python-hello/python_hello/1/python_hello.err",
      "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  python_hello:\n    type: script\n    description: Hello World python\n    executor: generic.local.bash\n    tags: python\n    run: python hello.py\n\n",
      "test_content": "#!/bin/bash \n# Content of run section\npython hello.py",
      "logpath": "/tmp/buildtest_u8hjw81_.log",
      "tags": "python",
      "starttime": "2021/07/30 15:08:53",
      "endtime": "2021/07/30 15:08:53",
      "runtime": 0.04198,
      "state": "PASS",
      "returncode": 0,
      "output": "Hello World\n",
      "error": "python_hello_build.sh: 14: python_hello_build.sh: source: not found\n",
      "job": null,
      "build_script": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.bash/python-hello/python_hello/1/python_hello_build.sh"
    },
    {
      "id": "de751b51",
      "full_id": "de751b51-171e-4ce8-b202-213762071814",
      "description": "Hello World python",
      "schemafile": "script-v1.0.schema.json",
      "executor": "generic.local.bash",
      "compiler": null,
      "hostname": "build-14361642-project-280831-buildtest",
      "user": "docs",
      "testroot": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.bash/python-hello/python_hello/2",
      "testpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.bash/python-hello/python_hello/2/stage/python_hello.sh",
      "stagedir": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.bash/python-hello/python_hello/2/stage",
      "command": "sh python_hello_build.sh",
      "outfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.bash/python-hello/python_hello/2/python_hello.out",
      "errfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.bash/python-hello/python_hello/2/python_hello.err",
      "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  python_hello:\n    type: script\n    description: Hello World python\n    executor: generic.local.bash\n    tags: python\n    run: python hello.py\n\n",
      "test_content": "#!/bin/bash \n# Content of run section\npython hello.py",
      "logpath": "/tmp/buildtest_5ra6r44a.log",
      "tags": "python",
      "starttime": "2021/07/30 15:08:54",
      "endtime": "2021/07/30 15:08:54",
      "runtime": 0.042309,
      "state": "PASS",
      "returncode": 0,
      "output": "Hello World\n",
      "error": "python_hello_build.sh: 14: python_hello_build.sh: source: not found\n",
      "job": null,
      "build_script": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.0/var/tests/generic.local.bash/python-hello/python_hello/2/python_hello_build.sh"
    }
  ]
}

Inspecting Test by ID

The buildtest inspect id works similar to buildtest inspect name except that it operates on test id. This can be useful if you want to extract a particular test record and not see all test records at once.

You only need to specify a few characters and buildtest will resolve full test id if there is a match. The buildtest inspect id can operate on single or multiple ids if you want to specify multiple ids in single command you can do buildtest inspect id <identifier1> <identifier2>.

Let’s see an example where we query a single test record. Notice, that we only specify a few characters fee and buildtest found a matching record fee66c67-db4e-4d35-8c6d-28ac5cbbaba0

$ buildtest inspect id fee
Reading Report File: /Users/siddiq90/.buildtest/report.json

{
  "fee66c67-db4e-4d35-8c6d-28ac5cbbaba0": {
    "id": "fee66c67",
    "full_id": "fee66c67-db4e-4d35-8c6d-28ac5cbbaba0",
    "schemafile": "script-v1.0.schema.json",
    "executor": "generic.local.bash",
    "compiler": null,
    "hostname": "DOE-7086392.local",
    "user": "siddiq90",
    "testroot": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/2",
    "testpath": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/2/stage/generate.sh",
    "stagedir": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/2/stage",
    "rundir": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/2/run",
    "command": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/2/stage/generate.sh",
    "outfile": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/2/run/python_hello.out",
    "errfile": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/2/run/python_hello.err",
    "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  python_hello:\n    type: script\n    description: Hello World python\n    executor: generic.local.bash\n    tags: python\n    run: python hello.py\n\n",
    "test_content": "#!/bin/bash \nsource /Users/siddiq90/Documents/github/buildtest/var/executors/generic.local.bash/before_script.sh\npython hello.py\nsource /Users/siddiq90/Documents/github/buildtest/var/executors/generic.local.bash/after_script.sh",
    "tags": "python",
    "starttime": "2021/03/31 11:18:21",
    "endtime": "2021/03/31 11:18:21",
    "runtime": 0.104714,
    "state": "PASS",
    "returncode": 0,
    "output": "Hello World\n",
    "error": "",
    "job": null
  }
}

We can pass multiple IDs to buildtest inspect id and buildtest will retrieve test record if there is a match. You only need to specify a few characters to ensure we have a unique test ID and buildtest will retrieve the record.

$ buildtest inspect id 944 a76
 Reading Report File: /Users/siddiq90/.buildtest/report.json

 {
   "a76799db-f11e-4050-8dcb-8b147092c536": {
     "id": "a76799db",
     "full_id": "a76799db-f11e-4050-8dcb-8b147092c536",
     "schemafile": "script-v1.0.schema.json",
     "executor": "generic.local.bash",
     "compiler": null,
     "hostname": "DOE-7086392.local",
     "user": "siddiq90",
     "testroot": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/disk_usage/root_disk_usage/0",
     "testpath": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/disk_usage/root_disk_usage/0/stage/generate.sh",
     "stagedir": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/disk_usage/root_disk_usage/0/stage",
     "rundir": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/disk_usage/root_disk_usage/0/run",
     "command": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/disk_usage/root_disk_usage/0/stage/generate.sh",
     "outfile": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/disk_usage/root_disk_usage/0/run/root_disk_usage.out",
     "errfile": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/disk_usage/root_disk_usage/0/run/root_disk_usage.err",
     "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  root_disk_usage:\n    executor: generic.local.bash\n    type: script\n    tags: [filesystem, storage]\n    description: Check root disk usage and report if it exceeds threshold\n    env:\n      threshold: 90\n    run: |\n      root_disk_usage=`df -a / | tail -n 1 |  awk '{print $5'} | sed 's/[^0-9]*//g'`\n      # if root exceeds threshold\n      if [ \"$root_disk_usage\" -gt \"$threshold\" ]; then\n        echo \"[WARNING] Root Disk Usage: $root_disk_usage% exceeded threshold of $threshold%\"\n        exit 1\n      fi\n      echo \"[OK] Root disk is below threshold of $threshold%\"\n",
     "test_content": "#!/bin/bash \nsource /Users/siddiq90/Documents/github/buildtest/var/executors/generic.local.bash/before_script.sh\nexport threshold=90\nroot_disk_usage=`df -a / | tail -n 1 |  awk '{print $5'} | sed 's/[^0-9]*//g'`\n# if root exceeds threshold\nif [ \"$root_disk_usage\" -gt \"$threshold\" ]; then\n  echo \"[WARNING] Root Disk Usage: $root_disk_usage% exceeded threshold of $threshold%\"\n  exit 1\nfi\necho \"[OK] Root disk is below threshold of $threshold%\"\n\nsource /Users/siddiq90/Documents/github/buildtest/var/executors/generic.local.bash/after_script.sh",
     "tags": "filesystem storage",
     "starttime": "2021/03/31 11:17:50",
     "endtime": "2021/03/31 11:17:50",
     "runtime": 0.114321,
     "state": "PASS",
     "returncode": 0,
     "output": "[OK] Root disk is below threshold of 90%\n",
     "error": "",
     "job": null
   },
   "944f6399-b82b-47f9-bb15-8f529dedd4e6": {
     "id": "944f6399",
     "full_id": "944f6399-b82b-47f9-bb15-8f529dedd4e6",
     "schemafile": "script-v1.0.schema.json",
     "executor": "generic.local.python",
     "compiler": null,
     "hostname": "DOE-7086392.local",
     "user": "siddiq90",
     "testroot": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.python/python-shell/circle_area/0",
     "testpath": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.python/python-shell/circle_area/0/stage/generate.sh",
     "stagedir": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.python/python-shell/circle_area/0/stage",
     "rundir": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.python/python-shell/circle_area/0/run",
     "command": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.python/python-shell/circle_area/0/stage/generate.sh",
     "outfile": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.python/python-shell/circle_area/0/run/circle_area.out",
     "errfile": "/Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.python/python-shell/circle_area/0/run/circle_area.err",
     "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  circle_area:\n    executor: generic.local.python\n    type: script\n    shell: python\n    description: \"Calculate circle of area given a radius\"\n    tags: [tutorials, python]\n    run: |\n      import math\n      radius = 2\n      area = math.pi * radius * radius\n      print(\"Circle Radius \", radius)\n      print(\"Area of circle \", area)\n",
     "test_content": "#!/bin/bash\nsource /Users/siddiq90/Documents/github/buildtest/var/executors/generic.local.python/before_script.sh\npython /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.python/python-shell/circle_area/0/stage/circle_area.py\nsource /Users/siddiq90/Documents/github/buildtest/var/executors/generic.local.python/after_script.sh",
     "tags": "tutorials python",
     "starttime": "2021/03/31 11:18:00",
     "endtime": "2021/03/31 11:18:00",
     "runtime": 0.144171,
     "state": "PASS",
     "returncode": 0,
     "output": "Circle Radius  2\nArea of circle  12.566370614359172\n",
     "error": "",
     "job": null
   }
 }

If you specify an invalid test id using buildtest inspect id you will get an error message as follows.

$ buildtest inspect id lad

Unable to find any test records based on id: ['lad'], please run 'buildtest inspect list' to see list of ids.

You will see similar message if you specify an invalid test name using buildtest inspect name command.

Using Alternate Report File

The buildtest report and buildtest inspect command will read from the report file tracked by buildtest which is stored in $BUILDTEST_ROOT/var/report.json. This single file can became an issue if you are running jobs through CI where you can potentially overwrite same file or if you want separate report files for each set of builds. Luckily we have an option to handle this using the buildtest build -r /path/to/report option which can be used to specify an alternate location to report file.

buildtest will write the report file in the desired location, then you can specify the path to report file via buildtest report -r /path/to/report and buildtest inspect -r /path/to/report to load the report file when reporting tests.

The report file must be valid JSON file that buildtest understands in order to use buildtest report and buildtest inspect command. Shown below are some examples using the alternate report file using buildtest report and buildtest inspect command.

$ buildtest report -r python.json --format name,id
Reading report file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/docs/python.json

+--------------+----------+
| name         | id       |
+==============+==========+
| circle_area  | 6be6c404 |
+--------------+----------+
| python_hello | f21ba744 |
+--------------+----------+

$ buildtest inspect -r test.json name variables_bash
Reading Report File: /Users/siddiq90/Documents/GitHubDesktop/buildtest/test.json

{
  "variables_bash": [
    {
      "id": "cd0511ce",
      "full_id": "cd0511ce-377e-4ed2-95f4-f244e5518732",
      "schemafile": "script-v1.0.schema.json",
      "executor": "generic.local.bash",
      "compiler": null,
      "hostname": "DOE-7086392.local",
      "user": "siddiq90",
      "testroot": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/1",
      "testpath": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/1/stage/generate.sh",
      "stagedir": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/1/stage",
      "rundir": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/1/run",
      "command": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/1/stage/generate.sh",
      "outfile": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/1/run/variables_bash.out",
      "errfile": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/1/run/variables_bash.err",
      "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  variables_bash:\n    type: script\n    executor: generic.local.bash\n    description: Declare shell variables in bash\n    tags: [tutorials]\n    vars:\n      X: 1\n      Y: 2\n      literalstring: |\n        \"this is a literal string ':' \"\n      singlequote: \"'singlequote'\"\n      doublequote: \"\\\"doublequote\\\"\"\n      current_user: \"$(whoami)\"\n      files_homedir: \"`find $HOME -type f -maxdepth 1`\"\n\n    run: |\n      echo \"$X+$Y=\" $(($X+$Y))\n      echo $literalstring\n      echo $singlequote\n      echo $doublequote\n\n      echo $current_user\n      echo $files_homedir",
      "test_content": "#!/bin/bash \nsource /Users/siddiq90/.buildtest/executor/generic.local.bash/before_script.sh\nX=1\nY=2\nliteralstring=\"this is a literal string ':' \"\n\nsinglequote='singlequote'\ndoublequote=\"doublequote\"\ncurrent_user=$(whoami)\nfiles_homedir=`find $HOME -type f -maxdepth 1`\necho \"$X+$Y=\" $(($X+$Y))\necho $literalstring\necho $singlequote\necho $doublequote\n\necho $current_user\necho $files_homedir\nsource /Users/siddiq90/.buildtest/executor/generic.local.bash/after_script.sh",
      "tags": "tutorials",
      "starttime": "2021/04/16 14:29:25",
      "endtime": "2021/04/16 14:29:25",
      "runtime": 0.213196,
      "state": "PASS",
      "returncode": 0,
      "output": "1+2= 3\nthis is a literal string ':'\nsinglequote\ndoublequote\nsiddiq90\n/Users/siddiq90/buildtest_e7yxgttm.log /Users/siddiq90/.anyconnect /Users/siddiq90/buildtest_utwigb8w.log /Users/siddiq90/.DS_Store /Users/siddiq90/.serverauth.555 /Users/siddiq90/.CFUserTextEncoding /Users/siddiq90/.wget-hsts /Users/siddiq90/.bashrc /Users/siddiq90/.zshrc /Users/siddiq90/.coverage /Users/siddiq90/.serverauth.87055 /Users/siddiq90/buildtest_r7bck5zh.log /Users/siddiq90/.zsh_history /Users/siddiq90/.lesshst /Users/siddiq90/calltracker.py /Users/siddiq90/.git-completion.bash /Users/siddiq90/buildtest_wvjaaztp.log /Users/siddiq90/buildtest.log /Users/siddiq90/darhan.log /Users/siddiq90/ascent.yml /Users/siddiq90/.cshrc /Users/siddiq90/buildtest_nyq22whj.log /Users/siddiq90/github-tokens /Users/siddiq90/buildtest_ozb8b52z.log /Users/siddiq90/.zcompdump /Users/siddiq90/buildtest_nab_ckph.log /Users/siddiq90/.serverauth.543 /Users/siddiq90/.s.PGSQL.15007.lock /Users/siddiq90/.bash_profile /Users/siddiq90/.Xauthority /Users/siddiq90/.python_history /Users/siddiq90/.gitconfig /Users/siddiq90/output.txt /Users/siddiq90/.bash_history /Users/siddiq90/.viminfo\n",
      "error": "",
      "job": null
    },
    {
      "id": "e0901505",
      "full_id": "e0901505-a66b-4c91-9b29-d027cb6fabb6",
      "schemafile": "script-v1.0.schema.json",
      "executor": "generic.local.bash",
      "compiler": null,
      "hostname": "DOE-7086392.local",
      "user": "siddiq90",
      "testroot": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/2",
      "testpath": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/2/stage/generate.sh",
      "stagedir": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/2/stage",
      "rundir": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/2/run",
      "command": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/2/stage/generate.sh",
      "outfile": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/2/run/variables_bash.out",
      "errfile": "/Users/siddiq90/.buildtest/var/tests/generic.local.bash/vars/variables_bash/2/run/variables_bash.err",
      "buildspec_content": "version: \"1.0\"\nbuildspecs:\n  variables_bash:\n    type: script\n    executor: generic.local.bash\n    description: Declare shell variables in bash\n    tags: [tutorials]\n    vars:\n      X: 1\n      Y: 2\n      literalstring: |\n        \"this is a literal string ':' \"\n      singlequote: \"'singlequote'\"\n      doublequote: \"\\\"doublequote\\\"\"\n      current_user: \"$(whoami)\"\n      files_homedir: \"`find $HOME -type f -maxdepth 1`\"\n\n    run: |\n      echo \"$X+$Y=\" $(($X+$Y))\n      echo $literalstring\n      echo $singlequote\n      echo $doublequote\n\n      echo $current_user\n      echo $files_homedir",
      "test_content": "#!/bin/bash \nsource /Users/siddiq90/.buildtest/executor/generic.local.bash/before_script.sh\nX=1\nY=2\nliteralstring=\"this is a literal string ':' \"\n\nsinglequote='singlequote'\ndoublequote=\"doublequote\"\ncurrent_user=$(whoami)\nfiles_homedir=`find $HOME -type f -maxdepth 1`\necho \"$X+$Y=\" $(($X+$Y))\necho $literalstring\necho $singlequote\necho $doublequote\n\necho $current_user\necho $files_homedir\nsource /Users/siddiq90/.buildtest/executor/generic.local.bash/after_script.sh",
      "tags": "tutorials",
      "starttime": "2021/04/16 14:29:58",
      "endtime": "2021/04/16 14:29:58",
      "runtime": 0.075224,
      "state": "PASS",
      "returncode": 0,
      "output": "1+2= 3\nthis is a literal string ':'\nsinglequote\ndoublequote\nsiddiq90\n/Users/siddiq90/buildtest_e7yxgttm.log /Users/siddiq90/.anyconnect /Users/siddiq90/buildtest_utwigb8w.log /Users/siddiq90/.DS_Store /Users/siddiq90/.serverauth.555 /Users/siddiq90/.CFUserTextEncoding /Users/siddiq90/.wget-hsts /Users/siddiq90/.bashrc /Users/siddiq90/.zshrc /Users/siddiq90/.coverage /Users/siddiq90/.serverauth.87055 /Users/siddiq90/buildtest_r7bck5zh.log /Users/siddiq90/.zsh_history /Users/siddiq90/.lesshst /Users/siddiq90/calltracker.py /Users/siddiq90/.git-completion.bash /Users/siddiq90/buildtest_wvjaaztp.log /Users/siddiq90/buildtest.log /Users/siddiq90/darhan.log /Users/siddiq90/ascent.yml /Users/siddiq90/.cshrc /Users/siddiq90/buildtest_nyq22whj.log /Users/siddiq90/github-tokens /Users/siddiq90/buildtest_ozb8b52z.log /Users/siddiq90/.zcompdump /Users/siddiq90/buildtest_nab_ckph.log /Users/siddiq90/.serverauth.543 /Users/siddiq90/.s.PGSQL.15007.lock /Users/siddiq90/.bash_profile /Users/siddiq90/.Xauthority /Users/siddiq90/.python_history /Users/siddiq90/.gitconfig /Users/siddiq90/output.txt /Users/siddiq90/.bash_history /Users/siddiq90/.viminfo\n",
      "error": "",
      "job": null
    }
  ]
}