Additional Features¶
Accessing build history¶
buildtest keeps track of all builds (buildtest build) that can be retrieved using buildtest history command
which can be useful when you want to analyze or troubleshoot past builds. The buildtest history command comes with two
subcommands buildtest history list and buildtest history query.
If you want to list all builds you should run buildtest history list which will report a table style format of all builds with corresponding build ID to differentiate each build. Shown below is an example output. The build IDs start at 0 and increment as you run buildtest build command.
$ buildtest history list
['/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/0/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/1/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/10/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/11/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/12/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/13/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/14/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/15/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/16/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/17/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/18/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/19/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/2/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/20/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/3/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/4/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/5/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/6/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/7/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/8/build.json', '/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/9/build.json']
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id | hostname | user | system | date | pass_tests | fail_tests | total_tests | pass_rate | fail_rate | command |
+======+===================+==========+==========+=====================+==============+==============+===============+=============+=============+===================================================================================================================================================================================================+
| 20 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:07:45 | 1 | 0 | 1 | 100 | 0 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build -b tutorials/run_only_distro.yml |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 18 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:07:43 | 1 | 0 | 1 | 100 | 0 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build -b tutorials/skip_tests.yml |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 9 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:07:18 | 30 | 20 | 50 | 60 | 40 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build --executor generic.local.sh --executor generic.local.bash |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 0 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:06:11 | 1 | 0 | 1 | 100 | 0 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build -b /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/vars.yml |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 11 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:07:22 | 3 | 0 | 3 | 100 | 0 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build -b tutorials/python-shell.yml --rebuild=3 |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 7 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:06:38 | 2 | 0 | 2 | 100 | 0 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build --tags pass --filter-tags pass -b tutorials/python-hello.yml |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 16 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:07:42 | 0 | 1 | 1 | 0 | 100 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build -b tutorials/compilers/gnu_hello_fortran.yml |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 6 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:06:37 | 3 | 2 | 5 | 60 | 40 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build --tags pass --buildspec tutorials/python-hello.yml |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 17 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:07:42 | 2 | 2 | 4 | 50 | 50 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build -b tutorials/pass_returncode.yml |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 1 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:06:12 | 4 | 5 | 9 | 44.444 | 55.556 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build -b general_tests/configuration/ |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 10 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:07:20 | 2 | 0 | 2 | 100 | 0 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build -b tutorials/shebang.yml |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 19 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:07:44 | 1 | 0 | 1 | 100 | 0 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build -b tutorials/run_only_platform.yml |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 8 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:06:50 | 10 | 4 | 14 | 71.429 | 28.571 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build --executor generic.local.sh |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 4 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:06:35 | 27 | 7 | 34 | 79.412 | 20.588 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build --tags tutorials |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 15 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:07:41 | 1 | 0 | 1 | 100 | 0 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build -b tutorials/vars.yml |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 3 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:06:15 | 1 | 2 | 3 | 33.333 | 66.667 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build -b general_tests/configuration/ -x general_tests/configuration/ulimits.yml -x general_tests/configuration/ssh_localhost.yml |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 12 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:07:24 | 4 | 4 | 8 | 50 | 50 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build --tags fail --rebuild=2 |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 2 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:06:14 | 5 | 5 | 10 | 50 | 50 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build -b general_tests/configuration/ -b tutorials/vars.yml |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 13 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:07:25 | 2 | 0 | 2 | 100 | 0 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build -b tutorials/executor_regex_script.yml |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 5 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:06:36 | 4 | 2 | 6 | 66.667 | 33.333 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build --tags pass --tags python |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 14 | DOE-7086392.local | siddiq90 | generic | 2021/06/08 18:07:36 | 3 | 2 | 5 | 60 | 40 | /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build -b tutorials/runtime_status_test.yml |
+------+-------------------+----------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
The buildtest history query command is particularly useful when you want to inspect a particular build. This command
expects a Build Identifier which can be found by inspecting output column id in buildtest history list.
Shown below is an output of build ID 0 which reports relevant detail for the build such as input command, username, hostname, platform, date, etc…
$ buildtest history query 0
{
"command": "/Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest build -b /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/vars.yml",
"user": "siddiq90",
"hostname": "DOE-7086392.local",
"platform": "Darwin",
"date": "2021/06/08 18:06:11",
"buildtest": "/Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest",
"python": "/Users/siddiq90/.local/share/virtualenvs/buildtest-KLOcDrW0/bin/python",
"python-ver": "3.7.3",
"testdir": "/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests",
"configfile": "/Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/settings/config.yml",
"system": "generic",
"logpath": "/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history/0/buildtest_khwo_zrb.log",
"invalid_buildspecs": [],
"buildspecs": {
"detected": [
"/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/vars.yml"
],
"included": [
"/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/vars.yml"
],
"excluded": []
},
"test_summary": {
"pass": "1",
"fail": "0",
"total": "1",
"pass_rate": "100.000",
"fail_rate": "0.000"
}
}
buildtest schemas¶
buildtest uses JSON Schema for validating buildspecs and buildtest configuration file.
You can use buildtest schema command to see the list of schemas
supported by buildtest. The schema files are denoted by .schema.json file extension.
$ buildtest schema
global.schema.json
definitions.schema.json
settings.schema.json
compiler-v1.0.schema.json
script-v1.0.schema.json
Shown below is the command usage of buildtest schema
$ buildtest schema --help
usage: buildtest [options] [COMMANDS] schema [-h] [-n Schema Name] [-e] [-j]
optional arguments:
-h, --help show this help message and exit
-n Schema Name, --name Schema Name
show schema by name (e.g., script)
-e, --example Show schema examples
-j, --json Display json schema file
The json schemas are published at https://buildtesters.github.io/buildtest/ and we provide a command line interface to view schema files and examples.
To select a JSON schema use the --name option to select a schema, for example
to view a JSON Schema for script-v1.0.schema.json run the following:
$ buildtest schema --name script-v1.0.schema.json --json
Similarly, if you want to view example buildspecs for a schema use the --example
option with a schema. For example to view all example schemas for
compiler-v1.0.schema.json run the following:
$ buildtest schema --name compiler-v1.0.schema.json --example
To learn more about schema files and and examples click here.
Accessing buildtest documentation¶
We provide two command line options to access main documentation and schema docs. This will open a browser on your machine.
To access buildtest docs you can run:
$ buildtest docs
To access schema docs you can run:
$ buildtest schemadocs
Color Mode¶
buildtest will display output in color, if you want to disable color you can set
environment variable BUILDTEST_COLOR to False and buildtest will not display
the ANSI codes. This can be useful if you don’t want to see ANSI color codes in the
text output.
CDASH Integration¶
The buildtest cdash command is responsible for uploading tests to CDASH server. You will
need to specify CDASH Configuration in your configuration file. Shown below is the command
usage.
$ buildtest cdash --help
usage: buildtest [options] [COMMANDS] cdash [-h] ...
optional arguments:
-h, --help show this help message and exit
subcommands:
buildtest CDASH integeration
view Open CDASH project in webbrowser
upload Upload Test to CDASH server
The buildtest cdash upload command is responsible for uploading all tests in report.json
into CDASH. You must specify a buildname when using buildtest cdash upload in this example we will
specify a buildname called tutorials:
$ buildtest cdash upload tutorials
Reading configuration file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/settings/config.yml
Reading report file: /Users/siddiq90/.buildtest/report.json
build name: tutorials
site: generic
stamp: 20210428-1512-Experimental
MD5SUM: d7651cb3fbdd19298b0188c441704c3a
PUT STATUS: 200
You can view the results at: https://my.cdash.org//viewTest.php?buildid=2004360
We can see the output of these tests in CDASH if we go to url https://my.cdash.org//viewTest.php?buildid=2004360
By default buildtest will read the report file in your $HOME/.buildtest/report.json, we can
specify an alternate report file. First let’s see the available help options for
buildtest cdash upload.
$ buildtest cdash upload --help
usage: buildtest [options] [COMMANDS] cdash upload [-h] [--site SITE] [-r REPORT_FILE] buildname
positional arguments:
buildname Specify Build Name reported in CDASH
optional arguments:
-h, --help show this help message and exit
--site SITE Specify site name reported in CDASH
-r REPORT_FILE, --report-file REPORT_FILE
Path to report file to upload test results
We can pass an alternate report file using --report-file option when uploading tests
to CDASH. This can be useful if you want to map test results to different buildnames in CDASH
perhaps running a different subset of tests via buildtest build --tags and upload
the test results with different buildname assuming you have different paths to report file.
Let’s say we want to build all python tests using tags and store them in a report file which we
want to push to CDASH with buildgroup name python we can do that as follows:
$ buildtest build --tags python -r python.json
User: siddiq90
Hostname: DOE-7086392.local
Platform: Darwin
Current Time: 2021/04/28 15:36:06
buildtest path: /Users/siddiq90/Documents/GitHubDesktop/buildtest/bin/buildtest
buildtest version: 0.9.5
python path: /Users/siddiq90/.local/share/virtualenvs/buildtest-KLOcDrW0/bin/python
python version: 3.7.3
Test Directory: /Users/siddiq90/.buildtest/var/tests
Configuration File: /Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/settings/config.yml
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/python-shell.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/python-hello.yml
BREAKDOWN OF BUILDSPECS BY TAGS
python
----------------------------------------------------------------------------
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/python-shell.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/python-hello.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+------------------------------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/python-shell.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/python-hello.yml
name description
------------ ---------------------------------------
circle_area Calculate circle of area given a radius
python_hello Hello World python
+----------------------+
| Stage: Building Test |
+----------------------+
name | id | type | executor | tags | testpath
--------------+----------+--------+----------------------+-------------------------+--------------------------------------------------------------------------------------------------------
circle_area | 6be6c404 | script | generic.local.python | ['tutorials', 'python'] | /Users/siddiq90/.buildtest/var/tests/generic.local.python/python-shell/circle_area/5/stage/generate.sh
python_hello | f21ba744 | script | generic.local.bash | python | /Users/siddiq90/.buildtest/var/tests/generic.local.bash/python-hello/python_hello/3/stage/generate.sh
+---------------------+
| Stage: Running Test |
+---------------------+
name | id | executor | status | returncode
--------------+----------+----------------------+----------+--------------
circle_area | 6be6c404 | generic.local.python | PASS | 0
python_hello | f21ba744 | generic.local.bash | PASS | 0
+----------------------+
| Stage: Test Summary |
+----------------------+
Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%
Writing Logfile to: /var/folders/1m/_jjv09h17k37mkktwnmbkmj0002t_q/T/buildtest_k6swspn5.log
A copy of logfile can be found at $BUILDTEST_ROOT/buildtest.log - /Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest.log
Next we upload the tests using the -r option to specify the report file:
(buildtest) bash-3.2$ buildtest cdash upload -r python.json python
Reading configuration file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/settings/config.yml
Reading report file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/docs/python.json
build name: python
site: generic
stamp: 20210428-1536-Experimental
MD5SUM: d1c467eaf166999fd6e12b311e767cf9
PUT STATUS: 200
You can view the results at: https://my.cdash.org//viewTest.php?buildid=2004362
The buildtest cdash view command can be used to open CDASH project in a web browser
using the command line. This feature assumes you have set the CDASH setting in your
configuration file.