Introspection Operation¶
Config Options (buildtest config --help)¶
$ buildtest config --help
usage: buildtest [options] [COMMANDS] config [-h] {view,edit,validate,summary} ...
optional arguments:
-h, --help show this help message and exit
subcommands:
buildtest configuration
{view,edit,validate,summary}
view View Buildtest Configuration File
edit Edit Buildtest Configuration File
validate Validate buildtest settings file with schema.
summary Provide summary of buildtest settings.
The buildtest config command allows user to see view or edit your buildtest
settings file (settings.yml). To see content of your buildtest settings run:
buildtest config view
Shown below is an example output.
$ buildtest config view
executors:
local:
bash:
description: submit jobs on local machine using bash shell
shell: bash
sh:
description: submit jobs on local machine using sh shell
shell: sh
python:
description: submit jobs on local machine using python shell
shell: python
config:
editor: vi
paths:
buildspec_roots:
- /Users/siddiq90/Documents/buildtest-cori
Likewise, you can edit the file by running:
buildtest config edit
To check if your buildtest settings is valid, run buildtest config validate.
This will validate your settings.yml with the schema settings.schema.json.
The output will be the following.
$ buildtest config validate
/Users/siddiq90/.buildtest/config.yml is valid
If there is an error during validation, the output from jsonschema.exceptions.ValidationError
will be displayed in terminal. For example the error below indicates there was an error
on editor key in config object which expects the editor to be one of the
enum types [vi, vim, nano, emacs]:
$ buildtest config validate
Traceback (most recent call last):
File "/Users/siddiq90/.local/share/virtualenvs/buildtest-1gHVG2Pd/bin/buildtest", line 11, in <module>
load_entry_point('buildtest', 'console_scripts', 'buildtest')()
File "/Users/siddiq90/Documents/buildtest/buildtest/main.py", line 32, in main
check_settings()
File "/Users/siddiq90/Documents/buildtest/buildtest/config.py", line 71, in check_settings
validate(instance=user_schema, schema=config_schema)
File "/Users/siddiq90/.local/share/virtualenvs/buildtest-1gHVG2Pd/lib/python3.7/site-packages/jsonschema/validators.py", line 899, in validate
raise error
jsonschema.exceptions.ValidationError: 'gedit' is not one of ['vi', 'vim', 'nano', 'emacs']
Failed validating 'enum' in schema['properties']['config']['properties']['editor']:
{'default': 'vim',
'enum': ['vi', 'vim', 'nano', 'emacs'],
'type': 'string'}
On instance['config']['editor']:
'gedit'
You can get a summary of buildtest using buildtest config summary, this will
display information from several sources into one single command along.
$ buildtest config summary
buildtest version: 0.8.0
buildtest Path: /Users/siddiq90/.local/share/virtualenvs/buildtest-1gHVG2Pd/bin/buildtest
Machine Details
______________________________
Operating System: Darwin 19.5.0
Hostname: DOE-7086392.local
Machine: x86_64
Processor: i386
Python Path /Users/siddiq90/.local/share/virtualenvs/buildtest-1gHVG2Pd/bin/python
Python Version: 3.7.3
User: siddiq90
Buildtest Settings
________________________________________________________________________________
Buildtest Settings: /Users/siddiq90/.buildtest/config.yml
Buildtest Settings is VALID
Executors: ['local.bash', 'local.sh', 'local.python']
Buildtest Repositories:
________________________________________________________________________________
Repo File: /Users/siddiq90/.buildtest/repo.yaml
Active Repos: ['buildtesters/tutorials']
Repo Paths: ['/private/tmp/github.com/buildtesters/tutorials']
Buildspec Cache File: /Users/siddiq90/Documents/buildtest/var/buildspec.cache
Number of buildspecs: 13
Number of Tests: 27
Tests: ['systemd_default_target', '_bin_sh_shell', '_bin_bash_shell', 'bash_shell', 'sh_shell', 'shell_options', 'environment_variables', 'variables', 'selinux_disable', 'exit1_fail', 'exit1_pass', 'returncode_mismatch', 'wrongexecutor', 'circle_area', 'skip', 'unskipped', 'vecadd_gnu', 'hello_f', 'hello_c', 'hello_cplusplus', 'hello_f', 'hello_c', 'hello_cplusplus', 'cc_example', 'fc_example', 'cxx_example', 'executable_arguments']
Buildtest Schemas
________________________________________________________________________________
Available Schemas: ['script-v1.0.schema.json', 'compiler-v1.0.schema.json', 'global.schema.json', 'settings.schema.json']
Supported Sub-Schemas
________________________________________________________________________________
script-v1.0.schema.json : /Users/siddiq90/Documents/buildtest/buildtest/schemas/script/script-v1.0.schema.json
Examples Directory for schema: /Users/siddiq90/Documents/buildtest/buildtest/schemas/script/examples
compiler-v1.0.schema.json : /Users/siddiq90/Documents/buildtest/buildtest/schemas/compiler/compiler-v1.0.schema.json
Examples Directory for schema: /Users/siddiq90/Documents/buildtest/buildtest/schemas/compiler/examples
Access to buildtest schemas¶
The buildtest schema command can show you list of available schemas just run
the command with no options and it will show all the json schemas buildtest supports.
$ buildtest schema
script-v1.0.schema.json
compiler-v1.0.schema.json
global.schema.json
settings.schema.json
Shown below is the command usage of buildtest schema
$ buildtest schema --help
usage: buildtest [options] [COMMANDS] schema [-h] [-n Schema Name] [-e] [-j]
optional arguments:
-h, --help show this help message and exit
-n Schema Name, --name Schema Name
show schema by name (e.g., script)
-e, --example Show schema examples that are validated by corresponding schemafile
-j, --json Display json schema file
The json schemas are hosted on the web at https://buildtesters.github.io/schemas/. buildtest provides a means to display the json schema from the buildtest interface. Note that buildtest will show the schemas provided in buildtest repo and not ones provided by schemas repo. This is because, we let development of schema run independent of the framework.
To select a JSON schema use the --name option to select a schema, for example
to view a JSON Schema for script-v1.0.schema.json run the following:
$ buildtest schema --name script-v1.0.schema.json --json
Similarly, if you want to view example buildspecs for a schema use the --example
option with a schema. For example to view all example schemas for
compiler-v1.0.schema.json run the following:
$ buildtest schema --name compiler-v1.0.schema.json --example
Buildspec Features¶
$ buildtest buildspec --help
usage: buildtest [options] [COMMANDS] buildspec [-h] {find,view,edit} ...
optional arguments:
-h, --help show this help message and exit
subcommands:
Commands options for Buildspecs
{find,view,edit}
find find all buildspecs
view view a buildspec
edit edit a buildspec
The buildtest buildspec find loads all buildspecs specified in buildspec roots
from your configuration file. To build your cache just run:
$ buildtest buildspec find
To rebuild your cache, which you may need to do if you add more directories to buildspec_roots
in your configuration or edit some buildspec run:
$ buildtest buildspec find --clear
This will rebuild your cache and validate all buildspecs with updated files. Currently, we don’t support automatic rebuild of cache.
Shown below is a list of options for buildtest buildspec find command.
$ buildtest buildspec find --help
usage: buildtest [options] [COMMANDS] buildspec find [-h] [-c] [-t] [-bf]
optional arguments:
-h, --help show this help message and exit
-c, --clear Clear buildspec cache and find all buildspecs again
-t, --tags List all available tags
-bf, --buildspec-files
Get all buildspec files from cache
If you want to retrieve all unique tags from all buildspecs you can run
buildtest buildspec find --tags
$ buildtest buildspec find --tags
Searching buildspec in following directories: /Users/siddiq90/Documents/buildtest/tutorials
+-----------+
| Tags |
+===========+
| tutorials |
+-----------+
If you want to find all buildspec files in cache run buildtest buildspec find --buildspec-files
$ buildtest buildspec find --buildspec-files
Searching buildspec in following directories: /Users/siddiq90/Documents/buildtest/tutorials
+--------------------------------------------------------------------------------+
| buildspecs |
+================================================================================+
| /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml |
+--------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml |
+--------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/environment.yml |
+--------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/vars.yml |
+--------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/selinux.yml |
+--------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml |
+--------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml |
+--------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/skip_tests.yml |
+--------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml |
+--------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+--------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml |
+--------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml |
+--------------------------------------------------------------------------------+
Test Reports (buildtest report)¶
buildtest keeps track of all test results which can be retrieved via buildtest report. Shown below is command usage.
$ buildtest report --help
usage: buildtest [options] [COMMANDS] report [-h] [--helpformat] [--format FORMAT]
optional arguments:
-h, --help show this help message and exit
--helpformat List of available format fields
--format FORMAT format field for printing purposes. For more details see --helpformat for list of available fields.
Fields must be separated by comma (--format <field1>,<field2>,...)
You may run buildtest report and buildtest will display report
with default format fields.
$ buildtest report
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| id | state | returncode | starttime | endtime | runtime | tags | buildspec |
+=============================================+=========+==============+=====================+=====================+============+===========+==================================================================================+
| systemd_default_target_2020-09-02-21-41 | FAIL | 1 | 2020/09/02 21:41:56 | 2020/09/02 21:41:56 | 0.0213593 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| systemd_default_target_2020-09-02-21-42 | FAIL | 1 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.0125463 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| systemd_default_target_2020-09-02-21-42 | FAIL | 1 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00771781 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| slurm_down_nodes_reason_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.0110754 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| slurm_down_nodes_reason_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.00830546 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| slurm_not_responding_nodes_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.00802073 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| slurm_not_responding_nodes_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.00736713 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| _bin_sh_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.0125264 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| _bin_sh_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.014642 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| _bin_bash_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.00878803 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| _bin_bash_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.00996381 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| bash_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.00561796 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| bash_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.00678369 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| sh_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.0102359 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| sh_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.0110021 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| shell_options_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.0104085 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| shell_options_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.0115846 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| circle_area_2020-09-02-21-42 | FAIL | 2 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.0096071 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/python-shell.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| circle_area_2020-09-02-21-42 | FAIL | 2 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.0117158 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/python-shell.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| hello_dinosaur_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.00698703 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/environment.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| hello_dinosaur_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.0071777 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/environment.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| variables_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:20 | 2020/09/02 21:42:20 | 0.00946478 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/vars.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| pre_post_build_run_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:20 | 2020/09/02 21:42:20 | 0.661319 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| environment_variables_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:20 | 2020/09/02 21:42:20 | 0.0120327 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/environment.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| hello_f_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:20 | 2020/09/02 21:42:20 | 0.00786752 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| hello_f_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:43 | 2020/09/02 21:42:43 | 0.013958 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| hello_c_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:20 | 2020/09/02 21:42:21 | 0.229417 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| hello_c_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:43 | 2020/09/02 21:42:43 | 0.235872 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| hello_cplusplus_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:21 | 2020/09/02 21:42:21 | 0.461781 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| hello_cplusplus_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:43 | 2020/09/02 21:42:44 | 0.498086 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| cc_example_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:21 | 2020/09/02 21:42:21 | 0.230038 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| cc_example_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:44 | 2020/09/02 21:42:44 | 0.239393 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| fc_example_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:21 | 2020/09/02 21:42:21 | 0.00840816 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| fc_example_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:44 | 2020/09/02 21:42:44 | 0.0103057 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| cxx_example_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:21 | 2020/09/02 21:42:22 | 0.45725 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| cxx_example_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:44 | 2020/09/02 21:42:44 | 0.505495 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| selinux_disable_2020-09-02-21-42 | FAIL | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00954966 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/selinux.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| _bin_sh_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.0126632 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| _bin_bash_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.0085432 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| bash_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00603703 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| sh_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00911067 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| shell_options_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.0104522 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| circle_area_2020-09-02-21-42 | FAIL | 2 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00808543 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| unskipped_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00792155 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/skip_tests.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| unskipped_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:49 | 2020/09/02 21:42:49 | 0.042425 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/skip_tests.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| executable_arguments_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.232554 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| exit1_fail_2020-09-02-21-42 | FAIL | 1 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00782586 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| exit1_fail_2020-09-02-21-42 | FAIL | 1 | 2020/09/02 21:42:48 | 2020/09/02 21:42:48 | 0.0132099 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| exit1_pass_2020-09-02-21-42 | PASS | 1 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00772625 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| exit1_pass_2020-09-02-21-42 | PASS | 1 | 2020/09/02 21:42:48 | 2020/09/02 21:42:48 | 0.0110667 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| returncode_mismatch_2020-09-02-21-42 | FAIL | 2 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00836054 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| returncode_mismatch_2020-09-02-21-42 | FAIL | 2 | 2020/09/02 21:42:48 | 2020/09/02 21:42:48 | 0.00940911 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| vecadd_gnu_2020-09-02-21-42 | FAIL | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.0267169 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| vecadd_gnu_2020-09-02-21-42 | FAIL | 0 | 2020/09/02 21:42:46 | 2020/09/02 21:42:46 | 0.0421426 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
There are more fields captured in the report, so if you want to see a
list of available format fields run buildtest report --helpformat.
$ buildtest report --helpformat
Fields Description
---------- --------------------------------------------
buildspec Buildspec file
name Name of test defined in buildspec
id Unique Build Identifier
testroot Root of test directory
testpath Path to test
command Command executed
outfile Output file
errfile Error File
schemafile Schema file used for validation
executor Executor name
tags Tag name
starttime Start Time of test in date format
endtime End Time for Test in date format
runtime Total runtime in seconds
state Test State reported by buildtest (PASS/FAIL)
returncode Return Code from Test Execution
You can filter report using --format field which expects field
name separated by comma (i.e –format <field1>,<field2>). In this example
we format by fields --format name,type,executor,state,returncode
$ buildtest report --format name,schemafile,executor,state,returncode
+----------------------------+-------------------------+--------------+---------+--------------+
| name | schemafile | executor | state | returncode |
+============================+=========================+==============+=========+==============+
| systemd_default_target | script-v1.0.schema.json | local.bash | FAIL | 1 |
+----------------------------+-------------------------+--------------+---------+--------------+
| systemd_default_target | script-v1.0.schema.json | local.bash | FAIL | 1 |
+----------------------------+-------------------------+--------------+---------+--------------+
| systemd_default_target | script-v1.0.schema.json | local.bash | FAIL | 1 |
+----------------------------+-------------------------+--------------+---------+--------------+
| hello_dinosaur | script-v1.0.schema.json | local.bash | FAIL | 127 |
+----------------------------+-------------------------+--------------+---------+--------------+
| hello_dinosaur | script-v1.0.schema.json | local.bash | PASS | 0 |
+----------------------------+-------------------------+--------------+---------+--------------+
| circle_area | script-v1.0.schema.json | local.python | PASS | 0 |
+----------------------------+-------------------------+--------------+---------+--------------+
| circle_area | script-v1.0.schema.json | local.python | PASS | 0 |
+----------------------------+-------------------------+--------------+---------+--------------+
| _bin_sh_shell | script-v1.0.schema.json | local.sh | PASS | 0 |
+----------------------------+-------------------------+--------------+---------+--------------+
| _bin_sh_shell | script-v1.0.schema.json | local.sh | PASS | 0 |
+----------------------------+-------------------------+--------------+---------+--------------+
| _bin_bash_shell | script-v1.0.schema.json | local.bash | FAIL | 127 |
+----------------------------+-------------------------+--------------+---------+--------------+
| _bin_bash_shell | script-v1.0.schema.json | local.bash | PASS | 0 |
+----------------------------+-------------------------+--------------+---------+--------------+
| bash_shell | script-v1.0.schema.json | local.bash | FAIL | 127 |
+----------------------------+-------------------------+--------------+---------+--------------+
| bash_shell | script-v1.0.schema.json | local.bash | PASS | 0 |
+----------------------------+-------------------------+--------------+---------+--------------+
| sh_shell | script-v1.0.schema.json | local.sh | PASS | 0 |
+----------------------------+-------------------------+--------------+---------+--------------+
| sh_shell | script-v1.0.schema.json | local.sh | PASS | 0 |
+----------------------------+-------------------------+--------------+---------+--------------+
| shell_options | script-v1.0.schema.json | local.sh | PASS | 0 |
+----------------------------+-------------------------+--------------+---------+--------------+
| shell_options | script-v1.0.schema.json | local.sh | PASS | 0 |
+----------------------------+-------------------------+--------------+---------+--------------+
| slurm_down_nodes_reason | script-v1.0.schema.json | local.bash | FAIL | 127 |
+----------------------------+-------------------------+--------------+---------+--------------+
| slurm_down_nodes_reason | script-v1.0.schema.json | local.bash | PASS | 0 |
+----------------------------+-------------------------+--------------+---------+--------------+
| slurm_not_responding_nodes | script-v1.0.schema.json | local.bash | FAIL | 127 |
+----------------------------+-------------------------+--------------+---------+--------------+
| slurm_not_responding_nodes | script-v1.0.schema.json | local.bash | PASS | 0 |
+----------------------------+-------------------------+--------------+---------+--------------+
| selinux_disable | script-v1.0.schema.json | local.bash | PASS | 0 |
+----------------------------+-------------------------+--------------+---------+--------------+