Additional Features

Accessing build history

buildtest keeps track of all builds (buildtest build) that can be retrieved using buildtest history command which can be useful when you want to analyze or troubleshoot past builds. The buildtest history command comes with two subcommands buildtest history list and buildtest history query.

If you want to list all builds you should run buildtest history list which will report a table style format of all builds with corresponding build ID to differentiate each build. Shown below is an example output. The build IDs start at 0 and increment as you run buildtest build command.

$ buildtest history list
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|   id | hostname                                | user   | system   | date                |   pass_tests |   fail_tests |   total_tests |   pass_rate |   fail_rate | command                                                                                                                                                                                             |
+======+=========================================+========+==========+=====================+==============+==============+===============+=============+=============+=====================================================================================================================================================================================================+
|    0 | build-14364669-project-280831-buildtest | docs   | generic  | 2021/07/30 22:01:35 |            2 |            2 |             4 |          50 |          50 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/bin/buildtest build -b tutorials/pass_returncode.yml                                                                   |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|    1 | build-14364669-project-280831-buildtest | docs   | generic  | 2021/07/30 22:01:35 |            1 |            0 |             1 |         100 |           0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/bin/buildtest build -b /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/vars.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|    2 | build-14364669-project-280831-buildtest | docs   | generic  | 2021/07/30 22:01:36 |            2 |            0 |             2 |         100 |           0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/bin/buildtest build -b tutorials/shebang.yml                                                                           |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|    3 | build-14364669-project-280831-buildtest | docs   | generic  | 2021/07/30 22:01:36 |            1 |            0 |             1 |         100 |           0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/bin/buildtest build -b tutorials/skip_tests.yml                                                                        |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|    4 | build-14364669-project-280831-buildtest | docs   | generic  | 2021/07/30 22:01:36 |            1 |            0 |             1 |         100 |           0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/bin/buildtest build -b tutorials/metrics_regex.yml                                                                     |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|    5 | build-14364669-project-280831-buildtest | docs   | generic  | 2021/07/30 22:01:37 |            1 |            0 |             1 |         100 |           0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/bin/buildtest build -b tutorials/metrics_variable.yml                                                                  |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|    6 | build-14364669-project-280831-buildtest | docs   | generic  | 2021/07/30 22:01:49 |            3 |            2 |             5 |          60 |          40 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/bin/buildtest build -b tutorials/runtime_status_test.yml                                                               |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

The buildtest history query command is particularly useful when you want to inspect a particular build. This command expects a Build Identifier which can be found by inspecting output column id in buildtest history list.

Shown below is an output of build ID 0 which reports relevant detail for the build such as input command, username, hostname, platform, date, etc…

$ buildtest history query 0
{
  "command": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/bin/buildtest build -b tutorials/pass_returncode.yml",
  "user": "docs",
  "hostname": "build-14364669-project-280831-buildtest",
  "platform": "Linux",
  "date": "2021/07/30 22:01:35",
  "buildtest": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/bin/buildtest",
  "python": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/envs/v0.10.1/bin/python",
  "python_version": "3.6.12",
  "testdir": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests",
  "configuration": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/buildtest/settings/config.yml",
  "system": "generic",
  "logpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/.history/0/buildtest_86x145su.log",
  "invalid_buildspecs": [],
  "buildspecs": {
    "detected": [
      "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/pass_returncode.yml"
    ],
    "included": [
      "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/pass_returncode.yml"
    ],
    "excluded": []
  },
  "test_summary": {
    "pass": "2",
    "fail": "2",
    "total": "4",
    "pass_rate": "50.000",
    "fail_rate": "50.000"
  },
  "builders": {
    "2bc66cba-4356-4ace-b20b-d8ecff3734a3": {
      "name": "exit1_fail",
      "buildspec": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/pass_returncode.yml",
      "tags": [
        "tutorials",
        "fail"
      ],
      "executors": "generic.local.sh",
      "state": "FAIL",
      "returncode": 1,
      "runtime": 0.00575,
      "testpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.sh/pass_returncode/exit1_fail/0/exit1_fail.sh",
      "errfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.sh/pass_returncode/exit1_fail/0/exit1_fail.err",
      "outfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.sh/pass_returncode/exit1_fail/0/exit1_fail.out"
    },
    "56a7caa5-b235-4d45-91b3-97f977f5ad6a": {
      "name": "exit1_pass",
      "buildspec": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/pass_returncode.yml",
      "tags": [
        "tutorials",
        "pass"
      ],
      "executors": "generic.local.sh",
      "state": "PASS",
      "returncode": 1,
      "runtime": 0.00448,
      "testpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.sh/pass_returncode/exit1_pass/0/exit1_pass.sh",
      "errfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.sh/pass_returncode/exit1_pass/0/exit1_pass.err",
      "outfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.sh/pass_returncode/exit1_pass/0/exit1_pass.out"
    },
    "61b70f13-292a-4a46-bcb0-a24dfc5c1d63": {
      "name": "returncode_list_mismatch",
      "buildspec": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/pass_returncode.yml",
      "tags": [
        "tutorials",
        "fail"
      ],
      "executors": "generic.local.sh",
      "state": "FAIL",
      "returncode": 2,
      "runtime": 0.004318,
      "testpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.sh/pass_returncode/returncode_list_mismatch/0/returncode_list_mismatch.sh",
      "errfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.sh/pass_returncode/returncode_list_mismatch/0/returncode_list_mismatch.err",
      "outfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.sh/pass_returncode/returncode_list_mismatch/0/returncode_list_mismatch.out"
    },
    "193eb872-f7ae-41d3-a0f2-8f0af1fc347b": {
      "name": "returncode_int_match",
      "buildspec": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/pass_returncode.yml",
      "tags": [
        "tutorials",
        "pass"
      ],
      "executors": "generic.local.sh",
      "state": "PASS",
      "returncode": 128,
      "runtime": 0.004264,
      "testpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.sh/pass_returncode/returncode_int_match/0/returncode_int_match.sh",
      "errfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.sh/pass_returncode/returncode_int_match/0/returncode_int_match.err",
      "outfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.sh/pass_returncode/returncode_int_match/0/returncode_int_match.out"
    }
  }
}

buildtest schemas

buildtest uses JSON Schema for validating buildspecs and buildtest configuration file. You can use buildtest schema command to see the list of schemas supported by buildtest. The schema files are denoted by .schema.json file extension.

$ buildtest schema
global.schema.json
definitions.schema.json
settings.schema.json
compiler-v1.0.schema.json
spack-v1.0.schema.json
script-v1.0.schema.json

Shown below is the command usage of buildtest schema

$ buildtest schema --help
usage: buildtest [options] [COMMANDS] schema [-h] [-n Schema Name] [-e] [-j]

optional arguments:
  -h, --help            show this help message and exit
  -n Schema Name, --name Schema Name
                        show schema by name (e.g., script)
  -e, --example         Show schema examples
  -j, --json            Display json schema file

The json schemas are published at https://buildtesters.github.io/buildtest/ and we provide a command line interface to view schema files and examples. You must use the --name option to select a schema, for instance if you want to view the JSON Schema for script-v1.0.schema.json you can run the following:

buildtest schema --name script-v1.0.schema.json --json

Similarly, if you want to view example buildspecs for a schema use the --example option with a schema. For example to view all example schemas for compiler-v1.0.schema.json run the following:

buildtest schema --name compiler-v1.0.schema.json --example

To learn more about schema files and and examples click here.

Accessing buildtest documentation

We provide two command line options to access main documentation and schema docs. This will open a browser on your machine.

To access buildtest docs you can run:

buildtest docs

To access schema docs you can run:

buildtest schemadocs

Color Mode

buildtest will display output in color, if you want to disable color you can set environment variable BUILDTEST_COLOR to False and buildtest will not display the ANSI codes. This can be useful if you don’t want to see ANSI color codes in the text output.

CDASH Integration

The buildtest cdash command is responsible for uploading tests to CDASH server. You will need to specify CDASH Configuration in your configuration file. Shown below is the command usage.

$ buildtest cdash --help
usage: buildtest [options] [COMMANDS] cdash [-h]  ...

optional arguments:
  -h, --help  show this help message and exit

subcommands:
  buildtest CDASH integeration

  
    view      Open CDASH project in webbrowser
    upload    Upload Test to CDASH server

The buildtest cdash upload command is responsible for uploading all tests in report.json into CDASH. You must specify a buildname when using buildtest cdash upload in this example we will specify a buildname called tutorials:

$ buildtest cdash upload tutorials
Reading configuration file:  /Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/settings/config.yml
Reading report file:  /Users/siddiq90/.buildtest/report.json
build name:  tutorials
site:  generic
stamp:  20210428-1512-Experimental
MD5SUM: d7651cb3fbdd19298b0188c441704c3a
PUT STATUS: 200
You can view the results at: https://my.cdash.org//viewTest.php?buildid=2004360

We can see the output of these tests in CDASH if we go to url https://my.cdash.org//viewTest.php?buildid=2004360

../_images/CDASH.png

By default buildtest will read the report file in your $HOME/.buildtest/report.json, we can specify an alternate report file. First let’s see the available help options for buildtest cdash upload.

$ buildtest cdash upload --help
usage: buildtest [options] [COMMANDS] cdash upload [-h] [--site SITE] [-r REPORT] buildname

positional arguments:
  buildname             Specify Build Name reported in CDASH

optional arguments:
  -h, --help            show this help message and exit
  --site SITE           Specify site name reported in CDASH
  -r REPORT, --report REPORT
                        Path to report file to upload test results

We can pass an alternate report file using -r option when uploading tests to CDASH. This can be useful if you want to map test results to different buildnames in CDASH perhaps running a different subset of tests via buildtest build --tags and upload the test results with different buildname assuming you have different paths to report file.

Let’s say we want to build all python tests using tags and store them in a report file which we want to push to CDASH with buildgroup name python we can do that as follows

$ buildtest build --tags python -r $BUILDTEST_ROOT/python.json


User:  docs
Hostname:  build-14364669-project-280831-buildtest
Platform:  Linux
Current Time:  2021/07/30 22:01:58
buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/bin/buildtest
buildtest version:  0.10.1
python path: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/envs/v0.10.1/bin/python
python version:  3.6.12
Test Directory:  /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests
Configuration File:  /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/buildtest/settings/config.yml
Command: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/bin/buildtest build --tags python -r /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/python.json

+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+ 

+---------------------------------------------------------------------------------------------------------+
| Discovered Buildspecs                                                                                   |
+=========================================================================================================+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/python-shell.yml |
+---------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/python-hello.yml |
+---------------------------------------------------------------------------------------------------------+
Discovered Buildspecs:  2
Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  2

BREAKDOWN OF BUILDSPECS BY TAGS
----------------------------------
Detected Tag Names: ['python']
+---------------------------------------------------------------------------------------------------------+
| python                                                                                                  |
+=========================================================================================================+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/python-shell.yml |
+---------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/python-hello.yml |
+---------------------------------------------------------------------------------------------------------+



+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+ 

 schemafile              | validstate   | buildspec
-------------------------+--------------+---------------------------------------------------------------------------------------------------------
 script-v1.0.schema.json | True         | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/python-shell.yml
 script-v1.0.schema.json | True         | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/tutorials/python-hello.yml



name          description
------------  ---------------------------------------
circle_area   Calculate circle of area given a radius
python_hello  Hello World python

+----------------------+
| Stage: Building Test |
+----------------------+ 

 name         | id       | type   | executor             | tags                    | testpath
--------------+----------+--------+----------------------+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------
 circle_area  | ccf119a5 | script | generic.local.python | ['tutorials', 'python'] | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.python/python-shell/circle_area/0/circle_area_build.sh
 python_hello | 1f0ea278 | script | generic.local.bash   | python                  | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/var/tests/generic.local.bash/python-hello/python_hello/0/python_hello_build.sh





+---------------------+
| Stage: Running Test |
+---------------------+ 

 name         | id       | executor             | status   |   returncode
--------------+----------+----------------------+----------+--------------
 circle_area  | ccf119a5 | generic.local.python | PASS     |            0
 python_hello | 1f0ea278 | generic.local.bash   | PASS     |            0

+----------------------+
| Stage: Test Summary  |
+----------------------+ 
    
Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%


Writing Logfile to: /tmp/buildtest_q81act4z.log
A copy of logfile can be found at $BUILDTEST_ROOT/buildtest.log -  /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/buildtest.log

Next we upload the tests using the -r option to specify the report file

$ buildtest cdash upload -r $BUILDTEST_ROOT/python.json python
Reading configuration file:  /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/buildtest/settings/config.yml
Reading report file:  /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.1/python.json
build name:  python
site:  generic
stamp:  20210730-2201-Experimental
MD5SUM: 1ffd30e0b5da434f0f5cc6a52eb0dcac
PUT STATUS: 200
You can view the results at: https://my.cdash.org//viewTest.php?buildid=2045669

The buildtest cdash view command can be used to open CDASH project in a web browser using the command line. This feature assumes you have set the CDASH setting in your configuration file.