Additional Features¶
Accessing build history (buildtest history
)¶
Note
buildtest hy
is an alias for buildtest history
command.
buildtest keeps track of all builds (buildtest build
) that can be retrieved using buildtest history
command
which can be useful when you want to analyze or troubleshoot past builds. The buildtest history command comes with two
subcommands buildtest history list
and buildtest history query
.
If you want to list all builds you should run buildtest history list which will report a table style format of all builds with corresponding build ID to differentiate each build. Shown below is an example output. The build IDs start at 0 and increment as you run buildtest build command.
$ buildtest history list
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id | hostname | user | system | date | pass_tests | fail_tests | total_tests | pass_rate | fail_rate | command |
+======+=========================================+========+==========+=====================+==============+==============+===============+=============+=============+=====================================================================================================================================================================================================+
| 0 | build-14673784-project-280831-buildtest | docs | generic | 2021/09/09 15:54:50 | 1 | 0 | 1 | 100 | 0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest build -b /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/tutorials/vars.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 1 | build-14673784-project-280831-buildtest | docs | generic | 2021/09/09 15:54:51 | 2 | 2 | 4 | 50 | 50 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest build -b tutorials/pass_returncode.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 2 | build-14673784-project-280831-buildtest | docs | generic | 2021/09/09 15:54:51 | 1 | 1 | 2 | 50 | 50 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest build -b tutorials/status_regex.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 3 | build-14673784-project-280831-buildtest | docs | generic | 2021/09/09 15:54:59 | 3 | 2 | 5 | 60 | 40 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest build -b tutorials/runtime_status_test.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 4 | build-14673784-project-280831-buildtest | docs | generic | 2021/09/09 15:54:59 | 2 | 0 | 2 | 100 | 0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest build -b tutorials/shebang.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 5 | build-14673784-project-280831-buildtest | docs | generic | 2021/09/09 15:55:00 | 1 | 0 | 1 | 100 | 0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest build -b tutorials/skip_tests.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 6 | build-14673784-project-280831-buildtest | docs | generic | 2021/09/09 15:55:00 | 1 | 0 | 1 | 100 | 0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest build -b tutorials/metrics_regex.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 7 | build-14673784-project-280831-buildtest | docs | generic | 2021/09/09 15:55:01 | 1 | 0 | 1 | 100 | 0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest build -b tutorials/metrics_variable.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 8 | build-14673784-project-280831-buildtest | docs | generic | 2021/09/09 15:55:01 | 2 | 0 | 2 | 100 | 0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest build -b tutorials/executor_regex_script.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 9 | build-14673784-project-280831-buildtest | docs | generic | 2021/09/09 15:55:02 | 2 | 0 | 2 | 100 | 0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest build -b tutorials/script/multiple_executors.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 10 | build-14673784-project-280831-buildtest | docs | generic | 2021/09/09 15:55:02 | 2 | 0 | 2 | 100 | 0 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest build -b tutorials/script/executor_scheduler.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 11 | build-14673784-project-280831-buildtest | docs | generic | 2021/09/09 15:55:03 | 1 | 1 | 2 | 50 | 50 | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest build -b tutorials/script/status_by_executors.yml |
+------+-----------------------------------------+--------+----------+---------------------+--------------+--------------+---------------+-------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
The buildtest history query
command is particularly useful when you want to inspect a particular build. This command
expects a Build Identifier which can be found by inspecting output column id in buildtest history list.
Shown below is an output of build ID 0 which reports relevant detail for the build such as input command, username, hostname, platform, date, etc…
$ buildtest history query 0
{
"command": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest build -b /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/tutorials/vars.yml",
"user": "docs",
"hostname": "build-14673784-project-280831-buildtest",
"platform": "Linux",
"date": "2021/09/09 15:54:50",
"buildtest": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest",
"python": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/envs/v0.11.0/bin/python",
"python_version": "3.7.9",
"testdir": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/tests",
"configuration": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/buildtest/settings/config.yml",
"system": "generic",
"logpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/.history/0/buildtest_8xn_louq.log",
"invalid_buildspecs": [],
"buildspecs": {
"detected": [
"/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/tutorials/vars.yml"
],
"included": [
"/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/tutorials/vars.yml"
],
"excluded": []
},
"test_summary": {
"pass": "1",
"fail": "0",
"total": "1",
"pass_rate": "100.000",
"fail_rate": "0.000"
},
"builders": {
"98b07db8-9b3e-4ff0-b526-1ebcebbc3a1e": {
"name": "variables_bash",
"buildspec": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/tutorials/vars.yml",
"tags": [
"tutorials"
],
"executors": "generic.local.bash",
"state": "PASS",
"returncode": 0,
"runtime": 0.012987,
"testpath": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/tests/generic.local.bash/vars/variables_bash/98b07db8/variables_bash.sh",
"errfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/tests/generic.local.bash/vars/variables_bash/98b07db8/variables_bash.err",
"outfile": "/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/tests/generic.local.bash/vars/variables_bash/98b07db8/variables_bash.out"
}
}
}
If you want to see all available build IDs, you can use the following command. The -t
is terse format and --no-header
will
omit the headers for each column and pipe the output to cut to extract the first column which corresponds to build IDs.
$ buildtest hy list -t --no-header | cut -f 1 -d '|'
0
1
2
3
4
5
6
7
8
9
10
11
buildtest has tab complete on buildtest history query
which reports a list of build IDs which is another way to
see available IDs to query.
$ buildtest history query
0 1 10 11 12 13 14 15 16 17 18 19 2 20 21 22 23 3 4 5 6 7 8 9
If you want to see logfile for build ID 0 you can use --log
option to see logfile in an editor as follows:
buildtest history query 0 --log
Accessing buildtest documentation¶
We provide two command line options to access main documentation and schema docs. This will open a browser on your machine.
To access buildtest docs you can run:
buildtest docs
To access schema docs you can run:
buildtest schemadocs
Color Mode¶
buildtest will display output in color by default which can be configured on command line
via buildtest --color [on|off]
or via environment variable BUILDTEST_COLOR. You can disable
color output via command argument``–color off`` or environment BUILDTEST_COLOR=False
.
If --color on
is set with BUILDTEST_COLOR=False
, the value of environment variable will be honored.
CDASH Integration (buildtest cdash
)¶
The buildtest cdash
command is responsible for uploading tests to CDASH server. You will
need to specify Configuring test directory in your configuration file. Shown below is the command
usage.
$ buildtest cdash --help
usage: buildtest [options] [COMMANDS] cdash [-h] ...
optional arguments:
-h, --help show this help message and exit
subcommands:
buildtest CDASH integeration
view Open CDASH project in webbrowser
upload Upload Test to CDASH server
The buildtest cdash upload
command is responsible for uploading all tests in report.json
into CDASH. You must specify a buildname when using buildtest cdash upload in this example we will
specify a buildname called tutorials:
$ buildtest cdash upload tutorials
Reading configuration file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/settings/config.yml
Reading report file: /Users/siddiq90/.buildtest/report.json
build name: tutorials
site: generic
stamp: 20210428-1512-Experimental
MD5SUM: d7651cb3fbdd19298b0188c441704c3a
PUT STATUS: 200
You can view the results at: https://my.cdash.org//viewTest.php?buildid=2004360
We can see the output of these tests in CDASH if we go to url https://my.cdash.org//viewTest.php?buildid=2004360
By default buildtest will read the report file in your $HOME/.buildtest/report.json, we can
specify an alternate report file. First let’s see the available help options for
buildtest cdash upload
.
$ buildtest cdash upload --help
usage: buildtest [options] [COMMANDS] cdash upload [-h] [-r REPORT] [--site SITE] buildname
positional arguments:
buildname Specify Build Name reported in CDASH
optional arguments:
-h, --help show this help message and exit
-r REPORT, --report REPORT
Path to report file to upload test results
--site SITE Specify site name reported in CDASH
We can pass an alternate report file using -r
option when uploading tests
to CDASH. This can be useful if you want to map test results to different buildnames in CDASH
perhaps running a different subset of tests via buildtest build --tags
and upload
the test results with different buildname assuming you have different paths to report file.
Let’s say we want to build all python tests using tags and store them in a report file which we
want to push to CDASH with buildgroup name python
we can do that as follows
$ buildtest build --tags python -r $BUILDTEST_ROOT/python.json
User: docs
Hostname: build-14673784-project-280831-buildtest
Platform: Linux
Current Time: 2021/09/09 15:55:07
buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest
buildtest version: 0.11.0
python path: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/envs/v0.11.0/bin/python
python version: 3.7.9
Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/tests
Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/buildtest/settings/config.yml
Command: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest build --tags python -r /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/python.json
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
+---------------------------------------------------------------------------------------------------------+
| Discovered Buildspecs |
+=========================================================================================================+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/tutorials/python-hello.yml |
+---------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/tutorials/python-shell.yml |
+---------------------------------------------------------------------------------------------------------+
Discovered Buildspecs: 2
Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 2
BREAKDOWN OF BUILDSPECS BY TAGS
----------------------------------
Detected Tag Names: ['python']
+---------------------------------------------------------------------------------------------------------+
| python |
+=========================================================================================================+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/tutorials/python-hello.yml |
+---------------------------------------------------------------------------------------------------------+
| /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/tutorials/python-shell.yml |
+---------------------------------------------------------------------------------------------------------+
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
Valid Buildspecs: 2
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/tutorials/python-hello.yml: VALID
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/tutorials/python-shell.yml: VALID
Total builder objects created: 2
builders: [python_hello/66936186, circle_area/8f494a0c]
name id description buildspecs
------------ -------- --------------------------------------- -------------------------------------------------------------------------------------------------------
python_hello 66936186 Hello World python /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/tutorials/python-hello.yml
circle_area 8f494a0c Calculate circle of area given a radius /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/tutorials/python-shell.yml
+----------------------+
| Stage: Building Test |
+----------------------+
name | id | type | executor | tags | testpath
--------------+----------+--------+----------------------+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------
python_hello | 66936186 | script | generic.local.bash | python | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/tests/generic.local.bash/python-hello/python_hello/66936186/python_hello_build.sh
circle_area | 8f494a0c | script | generic.local.python | ['tutorials', 'python'] | /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/tests/generic.local.python/python-shell/circle_area/8f494a0c/circle_area_build.sh
+---------------------+
| Stage: Running Test |
+---------------------+
circle_area/8f494a0c: completed with returncode: 0
circle_area/8f494a0c: Writing output file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/tests/generic.local.python/python-shell/circle_area/8f494a0c/circle_area.out
circle_area/8f494a0c: Writing error file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/tests/generic.local.python/python-shell/circle_area/8f494a0c/circle_area.err
python_hello/66936186: completed with returncode: 0
python_hello/66936186: Writing output file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/tests/generic.local.bash/python-hello/python_hello/66936186/python_hello.out
python_hello/66936186: Writing error file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/tests/generic.local.bash/python-hello/python_hello/66936186/python_hello.err
______________________________
Launching test: python_hello
Test ID: 66936186-97cf-49ff-9f4b-25e6d4a7cb82
Executor Name: generic.local.bash
Running Test: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/tests/generic.local.bash/python-hello/python_hello/66936186/python_hello_build.sh
______________________________
Launching test: circle_area
Test ID: 8f494a0c-c597-4a6b-93e9-07a9f7234ec9
Executor Name: generic.local.python
Running Test: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/tests/generic.local.python/python-shell/circle_area/8f494a0c/circle_area_build.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
name | id | executor | status | returncode_match | regex_match | runtime_match | returncode | runtime
--------------+----------+----------------------+----------+--------------------+---------------+-----------------+--------------+-----------
circle_area | 8f494a0c | generic.local.python | PASS | N/A | N/A | N/A | 0 | 0.11377
python_hello | 66936186 | generic.local.bash | PASS | N/A | N/A | N/A | 0 | 0.122556
Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%
Writing Logfile to: /tmp/buildtest_k8x48zrz.log
A copy of logfile can be found at $BUILDTEST_ROOT/buildtest.log - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/buildtest.log
Next we upload the tests using the -r
option to specify the report file
$ buildtest cdash upload -r $BUILDTEST_ROOT/python.json python
Reading configuration file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/buildtest/settings/config.yml
Reading report file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/python.json
build name: python
site: generic
stamp: 20210909-1555-Experimental
MD5SUM: afc72529503d490df26855e3012a78c3
PUT STATUS: 200
You can view the results at: https://my.cdash.org//viewTest.php?buildid=2064112
The buildtest cdash view
command can be used to open CDASH project in a web browser
using the command line. This feature assumes you have set the CDASH setting in your
configuration file.
Cleaning buildtest files (buildtest clean
)¶
The buildtest clean
command can be used to remove files generated by buildtest such
as test files, report files, buildspec cache, and history files. You will be prompted for
response to clean up files for confirmation. If you want to avoid user response you can use buildtest clean -y
to accept confirmation for all prompts and buildtest will remove the files.
$ buildtest clean
Remove Test Directory /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests (y/n) [default: y]
Remove Report File /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/report.json (y/n) [default: y]
Remove History Directory /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/.history (y/n) [default: y]
Remove Buildspec Cache /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/buildspecs/cache.json (y/n) [default: y]
======> Remove Test Directory
======> Removing Report File
======> Removing History Directory
======> Removing buildspec cache
Changing Directories (buildtest cd
)¶
The buildtest cd
command can be used to change directory to root of test given
a test name. The change will be applied to your shell upon completion of
command. Let’s assume we want to change directory to root of test exit1_pass
we can do this as
follows:
$ buildtest cd exit1_pass
Changing directory to root of test: exit1_pass/8c4b6ac9-e94e-40d9-8d96-7aaa3a5d3723
$ pwd
/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.sh/pass_returncode/exit1_pass/8c4b6ac9
In this previous example, buildtest will use the latest run for test exit1_pass
and switch directory to root of test.
We can confirm this directory is from the latest run by running the following command. The testroot
is a property
in the report table that can be fetch via --format
field. The --latest
option will fetch
the latest run for the test.
$ buildtest report --latest --filter name=exit1_pass --format testroot --terse --no-header
/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.sh/pass_returncode/exit1_pass/8c4b6ac9
If you switch cd into a particular build you can specify the name followed by backslash and name of test ID. In this example below,
we will specify test name kernel_swapusage/1fa
and buildtest will attempt to find first record that starts with the test ID and switch
directory to root of test.
$ buildtest cd kernel_swapusage/1fa
Changing directory to root of test: kernel_swapusage/1fa21875-b099-41b6-8bc7-30e0d2dcc13b
$ pwd
/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.bash/kernel_state/kernel_swapusage/1fa21875
Get Path for Test (buildtest path
)¶
The buildtest path
command is used to display path attributes for a test that is available in the test report.
Shown below are available options for buildtest path
$ buildtest path -h
usage: buildtest [options] [COMMANDS] path [-h] [-t | -o | -e | -b | --stagedir] name
positional arguments:
name Name of test
optional arguments:
-h, --help show this help message and exit
-t, --testpath Show path to test script
-o, --outfile Show path to output file
-e, --errfile Show path to error file
-b, --buildscript Show path to build script
--stagedir Show path to stage directory
If you want to fetch the last run for any given test you can specify the name of the test as follows: buildtest path <name>
.
We can specify a test ID for a test by separating the name and test ID with backslash character (/
) as follows: buildtest path <name>/<ID>
If you don’t specify any option you will get root of test. In this example, we will retrieve testroot
for test variables_bash which is a property of the test found in the report file.
$ buildtest path variables_bash
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/tests/generic.local.bash/vars/variables_bash/98b07db8
You can get path to testscript via -t
option as show below
$ buildtest path -t variables_bash
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/tests/generic.local.bash/vars/variables_bash/98b07db8/variables_bash.sh
If you want to see content of output file, you can use -o
option with cat command as follows:
$ cat $(buildtest path -o variables_bash)
1+2=3
this is a literal string ':'
'singlequote'
"doublequote"
current user: docs
number of files: 4
In this next example we will query test circle_area with build ID aaa and buildtest will find the first match record that starts with this record and resolves to aaaa622d which is the short ID of test.
# query testroot for circle_area with build ID "aaa"
$ buildtest path circle_area/aaa
/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.python/python-shell/circle_area/aaaa622d
# query testroot for latest run of circle_area
$ buildtest path circle_area
/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.python/python-shell/circle_area/fc221b84
We have setup mutual exclusion to avoid user from passing two option at same time. If you do run into this situation you will get the following error.
$ buildtest path -o -e variables_bash
usage: buildtest [options] [COMMANDS] path [-h] [-t | -o | -e | -b | --stagedir] name
buildtest [options] [COMMANDS] path: error: argument -e/--errfile: not allowed with argument -o/--outfile
If you specify an invalid test name or buildtest can’t find the test id, then buildtest will print list of available test names with IDs.
Command Line Interface to buildtest configuration¶
Once you have implemented your buildtest configuration, you can query the configuration
details using buildtest config
command. Shown below is the command usage.
$ buildtest config --help
usage: buildtest [options] [COMMANDS] config [-h] ...
optional arguments:
-h, --help show this help message and exit
subcommands:
Query information from buildtest configuration file
compilers
Search compilers
executors
Query executors from buildtest configuration
summary Provide summary of buildtest settings.
systems List all available systems
validate Validate buildtest settings file with schema.
view View Buildtest Configuration File
Note
buildtest cg
is an alias for buildtest config
command.
Validate buildtest configuration (buildtest config validate
)¶
First thing you should do once you implement your configuration file is to make sure your configuration is valid with the schema.
This can be achieved by running buildtest config validate
. When you invoke this
command, buildtest will load the configuration and attempt to validate the file with
schema settings.schema.json. If validation is successful you will get the following message:
$ buildtest config validate
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/buildtest/settings/config.yml is valid
Note
If you defined a user setting (~/.buildtest/config.yml
) buildtest will validate this file instead of default one.
If there is an error during validation, the output from jsonschema.exceptions.ValidationError
will be displayed in terminal. For example the error below indicates that
moduletool
property was expecting one of the values
[environment-modules
, lmod
, N/A
] but it recieved a value of none
:
$ buildtest config validate
Traceback (most recent call last):
File "/Users/siddiq90/Documents/buildtest/bin/buildtest", line 17, in <module>
buildtest.main.main()
File "/Users/siddiq90/Documents/buildtest/buildtest/main.py", line 39, in main
buildtest_configuration = check_settings(settings_file, retrieve_settings=True)
File "/Users/siddiq90/Documents/buildtest/buildtest/config.py", line 41, in check_settings
validate(instance=user_schema, schema=config_schema)
File "/Users/siddiq90/.local/share/virtualenvs/buildtest-1gHVG2Pd/lib/python3.7/site-packages/jsonschema/validators.py", line 934, in validate
raise error
jsonschema.exceptions.ValidationError: 'none' is not one of ['environment-modules', 'lmod', 'N/A']
Failed validating 'enum' in schema['properties']['moduletool']:
{'description': 'Specify modules tool used for interacting with '
'``module`` command. ',
'enum': ['environment-modules', 'lmod', 'N/A'],
'type': 'string'}
On instance['moduletool']:
'none'
View buildtest configuration (buildtest config view
)¶
If you want to view buildtest configuration you can run buildtest config view
which will print content of buildtest configuration.
$ buildtest config view
hostnames:
- .*
description: Generic System
moduletool: N/A
load_default_buildspecs: true
executors:
local:
bash:
description: submit jobs on local machine using bash shell
shell: bash
sh:
description: submit jobs on local machine using sh shell
shell: sh
csh:
description: submit jobs on local machine using csh shell
shell: csh
zsh:
description: submit jobs on local machine using zsh shell
shell: zsh
disable: true
python:
description: submit jobs on local machine using python shell
shell: python
compilers:
compiler:
gcc:
builtin_gcc:
cc: gcc
fc: gfortran
cxx: g++
cdash:
url: https://my.cdash.org/
project: buildtest
site: generic
buildname: tutorials
________________________________________________________________________________
Settings File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/buildtest/settings/config.yml
Note
buildtest config view
will display contents of user buildtest settings ~/.buildtest/config.yml
if found, otherwise it will display the default configuration
View Executors (buildtest config executors
)¶
You can use the command buildtest config executors
to view executors from buildtest
configuration file. Shown below is the command usage
$ buildtest config executors --help
usage: buildtest [options] [COMMANDS] config executors [-h] [-j | -y | -d | -i]
optional arguments:
-h, --help show this help message and exit
-j, --json View executor in JSON format
-y, --yaml View executors in YAML format
-d, --disabled Show disabled executors
-i, --invalid Show invalid executors
You can run buildtest config executors
without any options and it will report a list of named executors that
you would reference in buildspec using the executor
property. If you prefer json or yaml format you can use --json
or --yaml
option.
$ buildtest config executors
generic.local.bash
generic.local.sh
generic.local.csh
generic.local.python
View Registered Systems¶
Your buildtest configuration may compose of one or more systems since you can define multiple systems
in a single configuration file to run buildtest for different HPC clusters. You can use
buildtest config systems
to report all system details defined in your configuration file.
In this example below we should the generic
system. If you have multiple entries, you will see one
entry per system record.
$ buildtest config systems
+----------+----------------+-------------+--------------+
| system | description | hostnames | moduletool |
+==========+================+=============+==============+
| generic | Generic System | ['.*'] | N/A |
+----------+----------------+-------------+--------------+
Configuration Summary¶
You can get a summary of buildtest using buildtest config summary
, this will
display information from several sources into one single command along.
$ buildtest config summary
buildtest version: 0.11.0
buildtest Path: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/bin/buildtest
Machine Details
______________________________
Operating System: ubuntu
Hostname: build-14673784-project-280831-buildtest
Machine: x86_64
Processor: x86_64
Python Path /home/docs/checkouts/readthedocs.org/user_builds/buildtest/envs/v0.11.0/bin/python
Python Version: 3.7.9
User: docs
Buildtest Settings
________________________________________________________________________________
Buildtest Settings: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/buildtest/settings/config.yml
Executors: ['local.bash', 'local.sh', 'local.csh', 'local.zsh', 'local.python']
Buildspec Cache File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/var/buildspecs/cache.json
Buildtest Schemas
________________________________________________________________________________
Available Schemas: ['script-v1.0.schema.json', 'compiler-v1.0.schema.json', 'global.schema.json', 'settings.schema.json']
Example Configurations¶
buildtest provides a few example configurations for configuring buildtest this
can be retrieved by running buildtest schema -n settings.schema.json --examples
or short option (-e
), which will validate each example with schema file
settings.schema.json
.
$ buildtest schema -n settings.schema.json -e
File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/buildtest/schemas/examples/settings.schema.json/valid/slurm-example.yml
________________________________________________________________________________
system:
generic:
hostnames: ['.*']
moduletool: lmod
load_default_buildspecs: True
buildspec_roots:
- $HOME/buildtest-cori
testdir: /tmp/buildtest
executors:
defaults:
pollinterval: 20
launcher: sbatch
max_pend_time: 30
account: admin
slurm:
normal:
options: ["-C haswell"]
qos: normal
before_script: |
time
echo "commands run before job"
compilers:
compiler:
gcc:
default:
cc: /usr/bin/gcc
cxx: /usr/bin/g++
fc: /usr/bin/gfortran
File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/buildtest/schemas/examples/settings.schema.json/valid/cobalt-example.yml
________________________________________________________________________________
system:
generic:
hostnames: ['.*']
moduletool: lmod
load_default_buildspecs: True
executors:
defaults:
launcher: qsub
max_pend_time: 30
cobalt:
knl:
queue: knl
haswell:
queue: haswell
compilers:
compiler:
gcc:
default:
cc: /usr/bin/gcc
cxx: /usr/bin/g++
fc: /usr/bin/gfortran
File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/buildtest/schemas/examples/settings.schema.json/valid/pbs-example.yml
________________________________________________________________________________
system:
generic:
hostnames: ['.*']
moduletool: N/A
load_default_buildspecs: True
executors:
defaults:
pollinterval: 10
launcher: qsub
max_pend_time: 30
pbs:
workq:
queue: workq
compilers:
compiler:
gcc:
default:
cc: /usr/bin/gcc
cxx: /usr/bin/g++
fc: /usr/bin/gfortran
File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/buildtest/schemas/examples/settings.schema.json/valid/lsf-example.yml
________________________________________________________________________________
system:
generic:
hostnames: ['.*']
moduletool: lmod
load_default_buildspecs: False
executors:
defaults:
pollinterval: 10
launcher: bsub
max_pend_time: 45
lsf:
batch:
description: "LSF Executor name 'batch' that submits jobs to 'batch' queue"
queue: batch
account: developer
options: ["-W 20"]
before_script: |
time
echo "commands run before job"
test:
description: "LSF Executor name 'test' that submits jobs to 'test' queue"
launcher: bsub
queue: test
account: qa
options: ["-W 20"]
compilers:
compiler:
gcc:
default:
cc: /usr/bin/gcc
cxx: /usr/bin/g++
fc: /usr/bin/gfortran
File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/buildtest/schemas/examples/settings.schema.json/valid/local-executor.yml
________________________________________________________________________________
system:
generic:
hostnames: ['.*']
logdir: $BUILDTEST_ROOT/logs
testdir: $BUILDTEST_ROOT/tests
moduletool: N/A
load_default_buildspecs: False
cdash:
url: https://my.cdash.org
project: buildtest
site: laptop
processor:
numcpus: 8
cores: 4
threads_per_core: 2
sockets: 1
model: "Intel(R) Core(TM) i7-8569U CPU @ 2.80GHz"
executors:
local:
bash:
description: submit jobs on local machine using bash shell
shell: bash
before_script: |
time
echo "commands run before job"
sh:
description: submit jobs on local machine using sh shell
shell: sh
csh:
description: submit jobs on local machine using csh shell
shell: csh -x
tcsh:
description: submit jobs on local machine using tcsh shell
shell: /bin/tcsh
zsh:
description: submit jobs on local machine using zsh shell
shell: /bin/zsh
python:
description: submit jobs on local machine using python shell
shell: python
compilers:
find:
gcc: "^(gcc|GCC|PrgEnv-gnu)"
intel: "^(intel|Intel|PrgEnv-intel)"
cray: "^(cray|PrgEnv-cray)"
clang: "^(clang|Clang)"
cuda: "^(cuda|CUDA)"
pgi: "^(pgi|PGI|PrgEnv-pgi)"
compiler:
gcc:
default:
cc: /usr/bin/gcc
cxx: /usr/bin/g++
fc: /usr/bin/gfortran
gcc@7.2.0:
cc: 'cc'
cxx: 'cxx'
fc: 'fc'
module:
load:
- gcc/7.2.0
intel:
intel@2019:
cc: 'icc'
cxx: 'icpc'
fc: 'ifort'
module:
purge: True
load:
- gcc/7.2.0
- intel/2019
cray:
craype@2.6.2:
cc: 'cc'
cxx: 'CC'
fc: 'fc'
module:
load: [craype/2.6.2]
swap: [PrgEnv-gnu, PrgEnv-cray]
clang:
clang@12.0.0:
cc: 'clang'
cxx: 'clang++'
fc: 'None'
module:
load: [clang/12.0]
cuda:
cuda@11.0:
cc: 'nvcc'
cxx: 'nvcc'
fc: 'None'
module:
load: [cuda/11.0]
pgi:
pgi@18.0:
cc: 'pgcc'
cxx: 'pgc++'
fc: 'pgfortran'
module:
load: [pgi/18.0]
File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v0.11.0/buildtest/schemas/examples/settings.schema.json/valid/combined_executor.yml
________________________________________________________________________________
system:
generic:
hostnames: ['.*']
moduletool: N/A
load_default_buildspecs: True
executors:
local:
bash:
description: submit jobs on local machine
shell: bash -v
slurm:
haswell:
launcher: sbatch
options:
- "-p haswell"
- "-t 00:10"
lsf:
batch:
launcher: bsub
queue: batch
options:
- "-q batch"
- "-t 00:10"
cobalt:
normal:
launcher: qsub
queue: normal
options:
- "-n 1"
- "-t 10"
compilers:
compiler:
gcc:
default:
cc: /usr/bin/gcc
cxx: /usr/bin/g++
fc: /usr/bin/gfortran
If you want to retrieve full json schema file for buildtest configuration you can
run buildtest schema -n settings.schema.json --json
or short option -j
.