Additional Features¶
Accessing build history¶
buildtest keeps track of all builds (buildtest build
) that can be retrieved using buildtest history
command
which can be useful when you want to analyze or troubleshoot past builds. The buildtest history command comes with two
subcommands buildtest history list
and buildtest history query
.
If you want to list all builds you should run buildtest history list which will report a table style format of all builds with corresponding build ID to differentiate each build. Shown below is an example output. The build IDs start at 0 and increment as you run buildtest build command.
$ buildtest history list
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id | hostname | user | system | date | pass_tests | fail_tests | total_tests | pass_rate | fail_rate | command |
+======+=========================================+========+==========+=====================+==============+==============+===============+=============+=============+=====================================================================================================================================================================================================+
| 0 | build-14488818-project-280831-buildtest | docs | generic | 2021/08/16 22:11:15 | 1 | 0 | 1 | 100 | 0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest build -b /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/vars.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 1 | build-14488818-project-280831-buildtest | docs | generic | 2021/08/16 22:11:16 | 2 | 2 | 4 | 50 | 50 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest build -b tutorials/pass_returncode.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 2 | build-14488818-project-280831-buildtest | docs | generic | 2021/08/16 22:11:16 | 1 | 1 | 2 | 50 | 50 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest build -b tutorials/status_regex.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 3 | build-14488818-project-280831-buildtest | docs | generic | 2021/08/16 22:11:27 | 3 | 2 | 5 | 60 | 40 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest build -b tutorials/runtime_status_test.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 4 | build-14488818-project-280831-buildtest | docs | generic | 2021/08/16 22:11:28 | 2 | 0 | 2 | 100 | 0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest build -b tutorials/shebang.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 5 | build-14488818-project-280831-buildtest | docs | generic | 2021/08/16 22:11:28 | 1 | 0 | 1 | 100 | 0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest build -b tutorials/skip_tests.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 6 | build-14488818-project-280831-buildtest | docs | generic | 2021/08/16 22:11:29 | 1 | 0 | 1 | 100 | 0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest build -b tutorials/metrics_regex.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 7 | build-14488818-project-280831-buildtest | docs | generic | 2021/08/16 22:11:29 | 1 | 0 | 1 | 100 | 0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest build -b tutorials/metrics_variable.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 8 | build-14488818-project-280831-buildtest | docs | generic | 2021/08/16 22:11:30 | 2 | 0 | 2 | 100 | 0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest build -b tutorials/executor_regex_script.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 9 | build-14488818-project-280831-buildtest | docs | generic | 2021/08/16 22:11:30 | 2 | 0 | 2 | 100 | 0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest build -b tutorials/script/multiple_executors.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 10 | build-14488818-project-280831-buildtest | docs | generic | 2021/08/16 22:11:31 | 2 | 0 | 2 | 100 | 0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest build -b tutorials/script/executor_scheduler.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 11 | build-14488818-project-280831-buildtest | docs | generic | 2021/08/16 22:11:31 | 1 | 1 | 2 | 50 | 50 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest build -b tutorials/script/status_by_executors.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
The buildtest history query
command is particularly useful when you want to inspect a particular build. This command
expects a Build Identifier which can be found by inspecting output column id in buildtest history list.
Shown below is an output of build ID 0 which reports relevant detail for the build such as input command, username, hostname, platform, date, etc…
$ buildtest history query 0
{
"command": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest build -b /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/vars.yml",
"user": "docs",
"hostname": "build-14488818-project-280831-buildtest",
"platform": "Linux",
"date": "2021/08/16 22:11:15",
"buildtest": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest",
"python": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/envs/v0.10.2/bin/python",
"python_version": "3.6.12",
"testdir": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/var/tests",
"configuration": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/buildtest/settings/config.yml",
"system": "generic",
"logpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/var/.history/0/buildtest_hr_5xctx.log",
"invalid_buildspecs": [],
"buildspecs": {
"detected": [
"/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/vars.yml"
],
"included": [
"/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/vars.yml"
],
"excluded": []
},
"test_summary": {
"pass": "1",
"fail": "0",
"total": "1",
"pass_rate": "100.000",
"fail_rate": "0.000"
},
"builders": {
"1c4ba849-bc6a-4989-8e43-06d38a332531": {
"name": "variables_bash",
"buildspec": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/vars.yml",
"tags": [
"tutorials"
],
"executors": "generic.local.bash",
"state": "PASS",
"returncode": 0,
"runtime": 0.010175,
"testpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/var/tests/generic.local.bash/vars/variables_bash/1c4ba849/variables_bash.sh",
"errfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/var/tests/generic.local.bash/vars/variables_bash/1c4ba849/variables_bash.err",
"outfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/var/tests/generic.local.bash/vars/variables_bash/1c4ba849/variables_bash.out"
}
}
}
buildtest schemas¶
buildtest uses JSON Schema for validating buildspecs and buildtest configuration file.
You can use buildtest schema
command to see the list of schemas
supported by buildtest. The schema files are denoted by .schema.json
file extension.
$ buildtest schema
global.schema.json
definitions.schema.json
settings.schema.json
compiler-v1.0.schema.json
spack-v1.0.schema.json
script-v1.0.schema.json
Shown below is the command usage of buildtest schema
$ buildtest schema --help
usage: buildtest [options] [COMMANDS] schema [-h] [-e] [-j] [-n Schema Name]
optional arguments:
-h, --help show this help message and exit
-e, --example Show schema examples
-j, --json Display json schema file
-n Schema Name, --name Schema Name
show schema by name (e.g., script)
The json schemas are published at https://buildtesters.github.io/buildtest/ and we
provide a command line interface to view schema files and examples. You must use the
--name
option to select a schema, for instance if you want to view the JSON Schema for
script-v1.0.schema.json you can run the following:
buildtest schema --name script-v1.0.schema.json --json
Similarly, if you want to view example buildspecs for a schema use the --example
option with a schema. For example to view all example schemas for
compiler-v1.0.schema.json run the following:
buildtest schema --name compiler-v1.0.schema.json --example
To learn more about schema files and and examples click here.
Accessing buildtest documentation¶
We provide two command line options to access main documentation and schema docs. This will open a browser on your machine.
To access buildtest docs you can run:
buildtest docs
To access schema docs you can run:
buildtest schemadocs
Color Mode¶
buildtest will display output in color by default which can be configured on command line
via buildtest --color [on|off]
or via environment variable BUILDTEST_COLOR. You can disable
color output via command argument``–color off`` or environment BUILDTEST_COLOR=False
.
If --color on
is set with BUILDTEST_COLOR=False
, the value of environment variable will be honored.
CDASH Integration¶
The buildtest cdash
command is responsible for uploading tests to CDASH server. You will
need to specify CDASH Configuration in your configuration file. Shown below is the command
usage.
$ buildtest cdash --help
usage: buildtest [options] [COMMANDS] cdash [-h] ...
optional arguments:
-h, --help show this help message and exit
subcommands:
buildtest CDASH integeration
view Open CDASH project in webbrowser
upload Upload Test to CDASH server
The buildtest cdash upload
command is responsible for uploading all tests in report.json
into CDASH. You must specify a buildname when using buildtest cdash upload in this example we will
specify a buildname called tutorials:
$ buildtest cdash upload tutorials
Reading configuration file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/settings/config.yml
Reading report file: /Users/siddiq90/.buildtest/report.json
build name: tutorials
site: generic
stamp: 20210428-1512-Experimental
MD5SUM: d7651cb3fbdd19298b0188c441704c3a
PUT STATUS: 200
You can view the results at: https://my.cdash.org//viewTest.php?buildid=2004360
We can see the output of these tests in CDASH if we go to url https://my.cdash.org//viewTest.php?buildid=2004360
By default buildtest will read the report file in your $HOME/.buildtest/report.json, we can
specify an alternate report file. First let’s see the available help options for
buildtest cdash upload
.
$ buildtest cdash upload --help
usage: buildtest [options] [COMMANDS] cdash upload [-h] [-r REPORT] [--site SITE] buildname
positional arguments:
buildname Specify Build Name reported in CDASH
optional arguments:
-h, --help show this help message and exit
-r REPORT, --report REPORT
Path to report file to upload test results
--site SITE Specify site name reported in CDASH
We can pass an alternate report file using -r
option when uploading tests
to CDASH. This can be useful if you want to map test results to different buildnames in CDASH
perhaps running a different subset of tests via buildtest build --tags
and upload
the test results with different buildname assuming you have different paths to report file.
Let’s say we want to build all python tests using tags and store them in a report file which we
want to push to CDASH with buildgroup name python
we can do that as follows
$ buildtest build --tags python -r $BUILDTEST_ROOT/python.json
User: docs
Hostname: build-14488818-project-280831-buildtest
Platform: Linux
Current Time: 2021/08/16 22:11:41
buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest
buildtest version: 0.10.2
python path: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/envs/v0.10.2/bin/python
python version: 3.6.12
Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/var/tests
Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/buildtest/settings/config.yml
Command: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/bin/buildtest build --tags python -r /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/python.json
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
+---------------------------------------------------------------------------------------------------------+
| Discovered Buildspecs |
+=========================================================================================================+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/python-shell.yml |
+---------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/python-hello.yml |
+---------------------------------------------------------------------------------------------------------+
Discovered Buildspecs: 2
Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 2
BREAKDOWN OF BUILDSPECS BY TAGS
----------------------------------
Detected Tag Names: ['python']
+---------------------------------------------------------------------------------------------------------+
| python |
+=========================================================================================================+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/python-shell.yml |
+---------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/python-hello.yml |
+---------------------------------------------------------------------------------------------------------+
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+---------------------------------------------------------------------------------------------------------
script-v1.0.schema.json | True | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/python-shell.yml
script-v1.0.schema.json | True | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/tutorials/python-hello.yml
name description
------------ ---------------------------------------
circle_area Calculate circle of area given a radius
python_hello Hello World python
+----------------------+
| Stage: Building Test |
+----------------------+
name | id | type | executor | tags | testpath
--------------+----------+--------+----------------------+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------
circle_area | a689fc7b | script | generic.local.python | ['tutorials', 'python'] | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/var/tests/generic.local.python/python-shell/circle_area/a689fc7b/circle_area_build.sh
python_hello | c4fa9083 | script | generic.local.bash | python | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/var/tests/generic.local.bash/python-hello/python_hello/c4fa9083/python_hello_build.sh
+---------------------+
| Stage: Running Test |
+---------------------+
name | id | executor | status | returncode
--------------+----------+----------------------+----------+--------------
circle_area | a689fc7b | generic.local.python | PASS | 0
python_hello | c4fa9083 | generic.local.bash | PASS | 0
+----------------------+
| Stage: Test Summary |
+----------------------+
Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%
Writing Logfile to: /tmp/buildtest__ykoxdes.log
A copy of logfile can be found at $BUILDTEST_ROOT/buildtest.log - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/buildtest.log
Next we upload the tests using the -r
option to specify the report file
$ buildtest cdash upload -r $BUILDTEST_ROOT/python.json python
Reading configuration file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/buildtest/settings/config.yml
Reading report file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.10.2/python.json
build name: python
site: generic
stamp: 20210816-2211-Experimental
MD5SUM: e1bf5fd1954f1e37dbc930d5817e4a4d
PUT STATUS: 200
You can view the results at: https://my.cdash.org//viewTest.php?buildid=2053302
The buildtest cdash view
command can be used to open CDASH project in a web browser
using the command line. This feature assumes you have set the CDASH setting in your
configuration file.