Getting Started with buildtest¶
Interacting with the client¶
Once you install buildtest, you should find the client on your $PATH, you can run the following to see path to buildtest:
$ which buildtest
If you don’t see buildtest go back and review section Installing buildtest.
Build Usage¶
The buildtest build command is used for building and running tests. Buildtest will read one or more Buildspecs (YAML)
file that adheres to one of the buildtest schemas. For a complete list of build options, run buildtest build --help
$ buildtest build --help
usage: buildtest [options] [COMMANDS] build [-h] [-b BUILDSPEC] [-x EXCLUDE] [--tags TAGS] [-e EXECUTOR]
[-s {parse,build}] [-t TESTDIR] [--rebuild REBUILD]
optional arguments:
-h, --help show this help message and exit
-b BUILDSPEC, --buildspec BUILDSPEC
Specify a Buildspec (YAML) file to build and run the test.
-x EXCLUDE, --exclude EXCLUDE
Exclude one or more configs from processing. Configs can be files or directories.
--tags TAGS Specify buildspecs by tags found in buildspec cache
-e EXECUTOR, --executor EXECUTOR
Specify buildspecs by executor name found in buildspec cache
-s {parse,build}, --stage {parse,build}
control behavior of buildtest build
-t TESTDIR, --testdir TESTDIR
specify a custom test directory. By default, use .buildtest in $PWD.
--rebuild REBUILD Rebuild test X number of times. Must be a positive number between [1-50]
Discover Buildspecs¶
The buildspec search resolution is described as follows:
If file doesn’t exist, check for file in buildspec roots and break after first match
If buildspec path is a directory, traverse directory recursively to find all
.ymlextensionsIf buildspec path is a file, check if file extension is not
.yml, exit immediately
Shown below is a diagram on how buildtest discovers buildspecs. The user
inputs a buildspec via --buildspec or tags (--tags) Building By Tags
which will discover the buildspecs. User can Excluding Buildspecs
using --exclude option which is processed after discovering buildspecs. The
excluded buildspecs are removed from list if found and final list of buildspecs
is processed.
Building a Test¶
To build a test, we use the --buildspec or short option -b to specify the
path to Buildspec file.
Let’s see some examples, first we specify a full path to buildspec file
$ buildtest build -b /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+-----------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
+----------------------+
| Stage: Building Test |
+----------------------+
name | id | type | executor | tags | testpath
------------------------+----------+--------+------------+---------------+-------------------------------------------------------------------------------------------------------------
systemd_default_target | 1770533b | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/0/stage/generate.sh
+----------------------+
| Stage: Running Test |
+----------------------+
name | id | executor | status | returncode | testpath
------------------------+----------+------------+----------+--------------+-------------------------------------------------------------------------------------------------------------
systemd_default_target | 1770533b | local.bash | FAIL | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/0/stage/generate.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 1 tests
Passed Tests: 0/1 Percentage: 0.000%
Failed Tests: 1/1 Percentage: 100.000%
buildtest won’t accept .yaml file extension for file, this can be demonstrated as
follows:
$ buildtest build -b invalid_ext.yaml
invalid_ext.yaml does not end in file extension .yml
There are no config files to process.
buildtest can perform a directory build for instance let’s build
for directory tests/examples/buildspecs where buildtest will recursively
search for all .yml files
$ buildtest build -b tests/examples/buildspecs/
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/python-shell.yml
/Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/environment.yml
/Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
/Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+----------------------------------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/python-shell.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/environment.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml
+----------------------+
| Stage: Building Test |
+----------------------+
name | id | type | executor | tags | testpath
----------------------------+----------+--------+--------------+--------+---------------------------------------------------------------------------------------------------------------
circle_area | 0bdaef77 | script | local.python | | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/0/stage/generate.sh
hello_dinosaur | dff7a691 | script | local.bash | | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/environment/hello_dinosaur/0/stage/generate.sh
_bin_sh_shell | f31c3498 | script | local.sh | | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/0/stage/generate.sh
_bin_bash_shell | 9700d000 | script | local.bash | | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/_bin_bash_shell/0/stage/generate.sh
bash_shell | c110d07a | script | local.bash | | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/bash_shell/0/stage/generate.sh
sh_shell | 9cbe76d3 | script | local.sh | | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/0/stage/generate.sh
shell_options | 47330a4a | script | local.sh | | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/0/stage/generate.sh
slurm_down_nodes_reason | 43858c19 | script | local.bash | | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/slurm/slurm_down_nodes_reason/0/stage/generate.sh
slurm_not_responding_nodes | 49a854e8 | script | local.bash | | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/slurm/slurm_not_responding_nodes/0/stage/generate.sh
+----------------------+
| Stage: Running Test |
+----------------------+
name | id | executor | status | returncode | testpath
----------------------------+----------+--------------+----------+--------------+---------------------------------------------------------------------------------------------------------------
circle_area | 0bdaef77 | local.python | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/0/stage/generate.sh
hello_dinosaur | dff7a691 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/environment/hello_dinosaur/0/stage/generate.sh
_bin_sh_shell | f31c3498 | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/0/stage/generate.sh
_bin_bash_shell | 9700d000 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/_bin_bash_shell/0/stage/generate.sh
bash_shell | c110d07a | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/bash_shell/0/stage/generate.sh
sh_shell | 9cbe76d3 | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/0/stage/generate.sh
shell_options | 47330a4a | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/0/stage/generate.sh
slurm_down_nodes_reason | 43858c19 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/slurm/slurm_down_nodes_reason/0/stage/generate.sh
slurm_not_responding_nodes | 49a854e8 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/slurm/slurm_not_responding_nodes/0/stage/generate.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 9 tests
Passed Tests: 9/9 Percentage: 100.000%
Failed Tests: 0/9 Percentage: 0.000%
In next section, you will see, we can build multiple buildspecs and interchange
file and directory with -b option.
Building Multiple Buildspecs¶
Buildtest supports building multiple buildspecs, just specify the -b option
for every Buildspec you want to build. In this example, we specify a file and
directory path. The search resolution is performed for every argument (-b)
independently, and accumulated into list.
$ buildtest build -b tests/examples/buildspecs/ -b tutorials/systemd.yml
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/python-shell.yml
/Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/environment.yml
/Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
/Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml
/Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+----------------------------------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/python-shell.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/environment.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
+----------------------+
| Stage: Building Test |
+----------------------+
name | id | type | executor | tags | testpath
----------------------------+----------+--------+--------------+---------------+---------------------------------------------------------------------------------------------------------------
circle_area | b06f76c2 | script | local.python | | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/1/stage/generate.sh
hello_dinosaur | cebde392 | script | local.bash | | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/environment/hello_dinosaur/1/stage/generate.sh
_bin_sh_shell | d554763a | script | local.sh | | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/1/stage/generate.sh
_bin_bash_shell | e412e216 | script | local.bash | | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/_bin_bash_shell/1/stage/generate.sh
bash_shell | 5c94695b | script | local.bash | | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/bash_shell/1/stage/generate.sh
sh_shell | c61e8164 | script | local.sh | | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/1/stage/generate.sh
shell_options | d5d62f37 | script | local.sh | | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/1/stage/generate.sh
slurm_down_nodes_reason | 60b25553 | script | local.bash | | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/slurm/slurm_down_nodes_reason/1/stage/generate.sh
slurm_not_responding_nodes | 97bf2c4a | script | local.bash | | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/slurm/slurm_not_responding_nodes/1/stage/generate.sh
systemd_default_target | 15a66b55 | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/1/stage/generate.sh
+----------------------+
| Stage: Running Test |
+----------------------+
name | id | executor | status | returncode | testpath
----------------------------+----------+--------------+----------+--------------+---------------------------------------------------------------------------------------------------------------
circle_area | b06f76c2 | local.python | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/1/stage/generate.sh
hello_dinosaur | cebde392 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/environment/hello_dinosaur/1/stage/generate.sh
_bin_sh_shell | d554763a | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/1/stage/generate.sh
_bin_bash_shell | e412e216 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/_bin_bash_shell/1/stage/generate.sh
bash_shell | 5c94695b | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/bash_shell/1/stage/generate.sh
sh_shell | c61e8164 | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/1/stage/generate.sh
shell_options | d5d62f37 | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/1/stage/generate.sh
slurm_down_nodes_reason | 60b25553 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/slurm/slurm_down_nodes_reason/1/stage/generate.sh
slurm_not_responding_nodes | 97bf2c4a | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/slurm/slurm_not_responding_nodes/1/stage/generate.sh
systemd_default_target | 15a66b55 | local.bash | FAIL | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/1/stage/generate.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 10 tests
Passed Tests: 9/10 Percentage: 90.000%
Failed Tests: 1/10 Percentage: 10.000%
Excluding Buildspecs¶
Buildtest provides --exclude option or short option -x to exclude
buildspecs which can be useful when you want to build all buildspecs in a directory
but exclude a few buildspecs or exclude a sub-directory.
For example we can build all buildspecs in examples but exclude file examples/systemd.yml
by running:
$ buildtest build -b examples -x examples/systemd.yml
buildtest will discover all Buildspecs and then exclude any buildspecs specified
by -x option. You can specify -x multiple times just like -b option.
For example, we can undo discovery by passing same option to -b and -x as follows:
$ buildtest build -b examples/ -x examples/
There are no Buildspec files to process.
Buildtest will stop immediately if there are no Buildspecs to process, this is true if you were to specify files instead of directory.
Building By Tags¶
buildtest can perform builds by tags by using --tags option. In order to use this
feature, buildspecs must be in cache so you must run buildtest buildspec find
or see Finding Buildspecs.
To build all tutorials tests you can perform buildtest build --tags tutorials.
In the buildspec there is a field tags: [tutorials] to classify tests.
buildtest will read the cache file var/buildspec-cache.json and see which
buildspecs have a matching tag. You should run buildtest buildspec find
atleast once, in order to detect cache file.
$ buildtest build --tags tutorials
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/environment.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml
/Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
/Users/siddiq90/Documents/buildtest/tutorials/vars.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml
/Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
/Users/siddiq90/Documents/buildtest/tutorials/run_only_platform.yml
/Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml
/Users/siddiq90/Documents/buildtest/tutorials/root_user.yml
/Users/siddiq90/Documents/buildtest/tutorials/skip_tests.yml
/Users/siddiq90/Documents/buildtest/tutorials/sleep.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
/Users/siddiq90/Documents/buildtest/tutorials/shebang.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
/Users/siddiq90/Documents/buildtest/tutorials/hello_world.yml
/Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
/Users/siddiq90/Documents/buildtest/tutorials/selinux.yml
[run_only_platform_linux] test is skipped because ['run_only']['platform'] got value: Linux but detected platform: Darwin.
[run_only_as_root] test is skipped because ['run_only']['user'] got value: root but detected user: siddiq90.
[skip] test is skipped.
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
---------------------------+--------------+--------------------------------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/environment.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/vars.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/run_only_platform.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/root_user.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/skip_tests.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/sleep.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/shebang.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/hello_world.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/selinux.yml
+----------------------+
| Stage: Building Test |
+----------------------+
name | id | type | executor | tags | testpath
--------------------------+----------+----------+--------------+--------------------------+---------------------------------------------------------------------------------------------------------------------------
environment_variables | d0e51818 | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/environment/environment_variables/0/stage/generate.sh
pre_post_build_run | 1e81254f | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/pre_post_build_run/pre_post_build_run/0/stage/generate.sh
_bin_sh_shell | 85c5e433 | script | local.sh | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/2/stage/generate.sh
_bin_bash_shell | 06ef100e | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/_bin_bash_shell/2/stage/generate.sh
bash_shell | 13e306ff | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/bash_shell/2/stage/generate.sh
sh_shell | a018fabc | script | local.sh | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/2/stage/generate.sh
shell_options | a7a23ec8 | script | local.sh | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/2/stage/generate.sh
variables | 3adfeb8b | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vars/variables/0/stage/generate.sh
executable_arguments | 5d670438 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/passing_args/executable_arguments/0/stage/generate.sh
exit1_fail | d405aea9 | script | local.sh | ['tutorials', 'fail'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_fail/0/stage/generate.sh
exit1_pass | 992a08e0 | script | local.sh | ['tutorials', 'pass'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_pass/0/stage/generate.sh
returncode_list_mismatch | 269abcb9 | script | local.sh | ['tutorials', 'fail'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_list_mismatch/0/stage/generate.sh
returncode_int_match | 146a0269 | script | local.sh | ['tutorials', 'pass'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_int_match/0/stage/generate.sh
run_only_platform_darwin | 1d86e162 | script | local.python | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.python/run_only_platform/run_only_platform_darwin/0/stage/generate.sh
circle_area | 59d92815 | script | local.python | ['tutorials', 'python'] | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/2/stage/generate.sh
unskipped | 17706881 | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/skip_tests/unskipped/0/stage/generate.sh
sleep | eadfa0df | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/sleep/sleep/0/stage/generate.sh
vecadd_gnu | 62df938a | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vecadd/vecadd_gnu/0/stage/generate.sh
bash_login_shebang | a4196349 | script | local.bash | tutorials | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shebang/bash_login_shebang/0/stage/generate.sh
bash_nonlogin_shebang | 72c038f0 | script | local.bash | tutorials | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shebang/bash_nonlogin_shebang/0/stage/generate.sh
hello_f | 34d5c6d8 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_f/0/stage/generate.sh
hello_c | 05ed531f | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_c/0/stage/generate.sh
hello_cplusplus | 2543a4e8 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_cplusplus/0/stage/generate.sh
cc_example | 3143c39a | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cc_example/0/stage/generate.sh
fc_example | b84feab0 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/fc_example/0/stage/generate.sh
cxx_example | 855fae37 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cxx_example/0/stage/generate.sh
hello_world | fc9f1058 | script | local.bash | tutorials | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/hello_world/hello_world/0/stage/generate.sh
systemd_default_target | 8864cbc1 | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/2/stage/generate.sh
selinux_disable | 52b88227 | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/selinux/selinux_disable/0/stage/generate.sh
+----------------------+
| Stage: Running Test |
+----------------------+
name | id | executor | status | returncode | testpath
--------------------------+----------+--------------+----------+--------------+---------------------------------------------------------------------------------------------------------------------------
environment_variables | d0e51818 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/environment/environment_variables/0/stage/generate.sh
pre_post_build_run | 1e81254f | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/pre_post_build_run/pre_post_build_run/0/stage/generate.sh
_bin_sh_shell | 85c5e433 | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/2/stage/generate.sh
_bin_bash_shell | 06ef100e | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/_bin_bash_shell/2/stage/generate.sh
bash_shell | 13e306ff | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/bash_shell/2/stage/generate.sh
sh_shell | a018fabc | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/2/stage/generate.sh
shell_options | a7a23ec8 | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/2/stage/generate.sh
variables | 3adfeb8b | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vars/variables/0/stage/generate.sh
executable_arguments | 5d670438 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/passing_args/executable_arguments/0/stage/generate.sh
exit1_fail | d405aea9 | local.sh | FAIL | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_fail/0/stage/generate.sh
exit1_pass | 992a08e0 | local.sh | PASS | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_pass/0/stage/generate.sh
returncode_list_mismatch | 269abcb9 | local.sh | FAIL | 2 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_list_mismatch/0/stage/generate.sh
returncode_int_match | 146a0269 | local.sh | PASS | 128 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_int_match/0/stage/generate.sh
run_only_platform_darwin | 1d86e162 | local.python | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.python/run_only_platform/run_only_platform_darwin/0/stage/generate.sh
circle_area | 59d92815 | local.python | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/2/stage/generate.sh
unskipped | 17706881 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/skip_tests/unskipped/0/stage/generate.sh
sleep | eadfa0df | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/sleep/sleep/0/stage/generate.sh
vecadd_gnu | 62df938a | local.bash | FAIL | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vecadd/vecadd_gnu/0/stage/generate.sh
bash_login_shebang | a4196349 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shebang/bash_login_shebang/0/stage/generate.sh
bash_nonlogin_shebang | 72c038f0 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shebang/bash_nonlogin_shebang/0/stage/generate.sh
hello_f | 34d5c6d8 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_f/0/stage/generate.sh
hello_c | 05ed531f | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_c/0/stage/generate.sh
hello_cplusplus | 2543a4e8 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_cplusplus/0/stage/generate.sh
cc_example | 3143c39a | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cc_example/0/stage/generate.sh
fc_example | b84feab0 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/fc_example/0/stage/generate.sh
cxx_example | 855fae37 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cxx_example/0/stage/generate.sh
hello_world | fc9f1058 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/hello_world/hello_world/0/stage/generate.sh
systemd_default_target | 8864cbc1 | local.bash | FAIL | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/2/stage/generate.sh
selinux_disable | 52b88227 | local.bash | FAIL | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/selinux/selinux_disable/0/stage/generate.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 29 tests
Passed Tests: 24/29 Percentage: 82.759%
Failed Tests: 5/29 Percentage: 17.241%
You can build by multiple tags by specifying --tags multiple times. In next
example we build all tests with tag name compiler and python.
$ buildtest build --tags compile --tags python
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/python-hello.yml
/Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
---------------------------+--------------+--------------------------------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/python-hello.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
+----------------------+
| Stage: Building Test |
+----------------------+
name | id | type | executor | tags | testpath
----------------------+----------+----------+--------------+--------------------------+--------------------------------------------------------------------------------------------------------------------
python_hello | 48caf02e | script | local.bash | python | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/python-hello/python_hello/0/stage/generate.sh
circle_area | 8a235c10 | script | local.python | ['tutorials', 'python'] | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/3/stage/generate.sh
executable_arguments | 77c21de7 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/passing_args/executable_arguments/1/stage/generate.sh
pre_post_build_run | d88a5039 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/pre_post_build_run/pre_post_build_run/1/stage/generate.sh
vecadd_gnu | ce7bbe15 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vecadd/vecadd_gnu/1/stage/generate.sh
hello_f | e525904f | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_f/1/stage/generate.sh
hello_c | 83beacb7 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_c/1/stage/generate.sh
hello_cplusplus | 6131d2d7 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_cplusplus/1/stage/generate.sh
cc_example | 46f76cea | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cc_example/1/stage/generate.sh
fc_example | ca8b4485 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/fc_example/1/stage/generate.sh
cxx_example | 3adf259c | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cxx_example/1/stage/generate.sh
+----------------------+
| Stage: Running Test |
+----------------------+
name | id | executor | status | returncode | testpath
----------------------+----------+--------------+----------+--------------+--------------------------------------------------------------------------------------------------------------------
python_hello | 48caf02e | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/python-hello/python_hello/0/stage/generate.sh
circle_area | 8a235c10 | local.python | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/3/stage/generate.sh
executable_arguments | 77c21de7 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/passing_args/executable_arguments/1/stage/generate.sh
pre_post_build_run | d88a5039 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/pre_post_build_run/pre_post_build_run/1/stage/generate.sh
vecadd_gnu | ce7bbe15 | local.bash | FAIL | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vecadd/vecadd_gnu/1/stage/generate.sh
hello_f | e525904f | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_f/1/stage/generate.sh
hello_c | 83beacb7 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_c/1/stage/generate.sh
hello_cplusplus | 6131d2d7 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_cplusplus/1/stage/generate.sh
cc_example | 46f76cea | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cc_example/1/stage/generate.sh
fc_example | ca8b4485 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/fc_example/1/stage/generate.sh
cxx_example | 3adf259c | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cxx_example/1/stage/generate.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 11 tests
Passed Tests: 10/11 Percentage: 90.909%
Failed Tests: 1/11 Percentage: 9.091%
When multiple tags are specified, we search each tag independently and if it is found in the buildspec cache we retrieve the test. To see a list of available tags in your buildspec cache see Querying buildspec tags.
Note
The --tags is used for discovering buildspecs and filtering tests during build phase.
For example a buildspec file (system.yml) that contain three tests hostname_check, timeout, and ping_test
are generally all run by default if you run as buildtest build -b system.yml, but if you
specify --tags buildtest will exclude tests that don’t have a matching tagname. It is possible
buildtest build --tags system can discover buildspec file system.yml but only
tests timeout and ping_test are built because they have a system tag while
hostname_check is skipped because it’s test doesn’t have a system tag.
You can combine --tags with --buildspec and --exclude in a single command.
buildtest will query tags and buildspecs independently and combine all discovered
buildspecs, any duplicates are ignored and finally we apply the exclusion list to
remove buildspecs.
In next example we combine all of these features together. This example builds all test with python tag, and build all buildspecs in directory - tutorials/compilers but we exclude tutorials/compilers/vecadd.yml.
$ buildtest build --tags python -b tutorials/compilers -x tutorials/compilers/vecadd.yml
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml
/Users/siddiq90/Documents/buildtest/tutorials/python-hello.yml
Excluded Buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
[hello_f] test is skipped because it is not in tag filter list: ['python']
[hello_c] test is skipped because it is not in tag filter list: ['python']
[hello_cplusplus] test is skipped because it is not in tag filter list: ['python']
[cc_example] test is skipped because it is not in tag filter list: ['python']
[fc_example] test is skipped because it is not in tag filter list: ['python']
[cxx_example] test is skipped because it is not in tag filter list: ['python']
[executable_arguments] test is skipped because it is not in tag filter list: ['python']
[pre_post_build_run] test is skipped because it is not in tag filter list: ['python']
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
---------------------------+--------------+--------------------------------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/python-hello.yml
+----------------------+
| Stage: Building Test |
+----------------------+
name | id | type | executor | tags | testpath
--------------+----------+--------+--------------+-------------------------+---------------------------------------------------------------------------------------------------------
circle_area | 888d6562 | script | local.python | ['tutorials', 'python'] | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/4/stage/generate.sh
python_hello | bbc55590 | script | local.bash | python | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/python-hello/python_hello/1/stage/generate.sh
+----------------------+
| Stage: Running Test |
+----------------------+
name | id | executor | status | returncode | testpath
--------------+----------+--------------+----------+--------------+---------------------------------------------------------------------------------------------------------
circle_area | 888d6562 | local.python | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/4/stage/generate.sh
python_hello | bbc55590 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/python-hello/python_hello/1/stage/generate.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 2 tests
Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%
Building by Executors¶
buildtest can build tests by executor name using the --executor option. If you
to build all test associated to an executor such as local.sh you can run:
$ buildtest build --executor local.sh
buildtest will query buildspec cache for the executor name and retrieve a list of buildspecs with matching executor name. Later we process every buildspec and filter tests with executor name. In the first stage we retrieve the buildspec file which may contain one or more test and in second stage we process each test.
To see a list of available executors in buildspec cache see Querying buildspec executor.
Note
By default all tests are run in buildspec file, the –executor is filtering by tests. This option behaves similar to tags, the –executor is used for discovering buildspecs and filtering tests with corresponding executor name.
In this example we run all tests that are associated to local.sh executor. Notice how buildtest skips tests that don’t match executor local.sh even though they were discovered in buildspec file.
$ buildtest build --executor local.sh
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
/Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
[_bin_bash_shell] test is skipped because it is not in executor filter list: ['local.sh']
[bash_shell] test is skipped because it is not in executor filter list: ['local.sh']
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+-------------------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
+----------------------+
| Stage: Building Test |
+----------------------+
name | id | type | executor | tags | testpath
--------------------------+----------+--------+------------+-----------------------+---------------------------------------------------------------------------------------------------------------------
exit1_fail | 0a7c4951 | script | local.sh | ['tutorials', 'fail'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_fail/1/stage/generate.sh
exit1_pass | 943c3d32 | script | local.sh | ['tutorials', 'pass'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_pass/1/stage/generate.sh
returncode_list_mismatch | e5905c73 | script | local.sh | ['tutorials', 'fail'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_list_mismatch/1/stage/generate.sh
returncode_int_match | ac11ac19 | script | local.sh | ['tutorials', 'pass'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_int_match/1/stage/generate.sh
_bin_sh_shell | 6ad1af21 | script | local.sh | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/3/stage/generate.sh
sh_shell | 26ec2ae6 | script | local.sh | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/3/stage/generate.sh
shell_options | 9f7b7cd8 | script | local.sh | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/3/stage/generate.sh
+----------------------+
| Stage: Running Test |
+----------------------+
name | id | executor | status | returncode | testpath
--------------------------+----------+------------+----------+--------------+---------------------------------------------------------------------------------------------------------------------
exit1_fail | 0a7c4951 | local.sh | FAIL | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_fail/1/stage/generate.sh
exit1_pass | 943c3d32 | local.sh | PASS | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_pass/1/stage/generate.sh
returncode_list_mismatch | e5905c73 | local.sh | FAIL | 2 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_list_mismatch/1/stage/generate.sh
returncode_int_match | ac11ac19 | local.sh | PASS | 128 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_int_match/1/stage/generate.sh
_bin_sh_shell | 6ad1af21 | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/3/stage/generate.sh
sh_shell | 26ec2ae6 | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/3/stage/generate.sh
shell_options | 9f7b7cd8 | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/3/stage/generate.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 7 tests
Passed Tests: 5/7 Percentage: 71.429%
Failed Tests: 2/7 Percentage: 28.571%
We can append arguments to --executor to search for multiple executors by
specifying --executor <name1> --executor <name2>. In next example we search
all tests associated with local.sh and local.bash executor.
Note
If you specify multiple executors, buildtest will combine the executors
into list, for example --executor local.bash --executor local.sh is converted
into a list (executor filter) - [local.bash, local.sh], and buildtest will
skip any test whose executor field in testname doesn’t belong to executor
filter list are skipped.
$ buildtest build --executor local.sh --executor local.bash
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/selinux.yml
/Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
/Users/siddiq90/Documents/buildtest/general_tests/sched/slurm/sacctmgr.yml
/Users/siddiq90/Documents/buildtest/tutorials/run_only_distro.yml
/Users/siddiq90/Documents/buildtest/general_tests/sched/lsf/bmgroups.yml
/Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml
/Users/siddiq90/Documents/buildtest/general_tests/sched/lsf/bugroup.yml
/Users/siddiq90/Documents/buildtest/tutorials/hello_world.yml
/Users/siddiq90/Documents/buildtest/general_tests/sched/lsf/bhosts.yml
/Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
/Users/siddiq90/Documents/buildtest/general_tests/sched/lsf/bqueues.yml
/Users/siddiq90/Documents/buildtest/tutorials/environment.yml
/Users/siddiq90/Documents/buildtest/tutorials/sleep.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml
/Users/siddiq90/Documents/buildtest/general_tests/sched/slurm/scontrol.yml
/Users/siddiq90/Documents/buildtest/general_tests/configuration/ssh_localhost.yml
/Users/siddiq90/Documents/buildtest/general_tests/sched/slurm/sinfo.yml
/Users/siddiq90/Documents/buildtest/general_tests/sched/lsf/lsinfo.yml
/Users/siddiq90/Documents/buildtest/general_tests/configuration/systemd-default-target.yml
/Users/siddiq90/Documents/buildtest/tutorials/tags_example.yml
/Users/siddiq90/Documents/buildtest/tutorials/root_user.yml
/Users/siddiq90/Documents/buildtest/tutorials/vars.yml
/Users/siddiq90/Documents/buildtest/tutorials/shebang.yml
/Users/siddiq90/Documents/buildtest/general_tests/configuration/disk_usage.yml
/Users/siddiq90/Documents/buildtest/general_tests/sched/slurm/squeue.yml
/Users/siddiq90/Documents/buildtest/tutorials/python-hello.yml
/Users/siddiq90/Documents/buildtest/tutorials/skip_tests.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
/Users/siddiq90/Documents/buildtest/general_tests/configuration/ulimits.yml
[show_accounts] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[show_users] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[show_qos] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[show_tres] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[show_host_groups] test is skipped because ['run_only']['scheduler'] got value: lsf but detected scheduler: [].
[show_lsf_user_groups] test is skipped because ['run_only']['scheduler'] got value: lsf but detected scheduler: [].
[display_lsf_hosts] test is skipped because ['run_only']['scheduler'] got value: lsf but detected scheduler: [].
[display_hosts_format] test is skipped because ['run_only']['scheduler'] got value: lsf but detected scheduler: [].
[bhosts_version] test is skipped because ['run_only']['scheduler'] got value: lsf but detected scheduler: [].
[show_lsf_queues] test is skipped because ['run_only']['scheduler'] got value: lsf but detected scheduler: [].
[show_lsf_queues_formatted] test is skipped because ['run_only']['scheduler'] got value: lsf but detected scheduler: [].
[show_lsf_queues_current_user] test is skipped because ['run_only']['scheduler'] got value: lsf but detected scheduler: [].
[slurm_config] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[show_partition] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[ssh_localhost_remotecommand] test is skipped because ['run_only']['platform'] got value: Linux but detected platform: Darwin.
[nodes_state_down] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[nodes_state_reboot] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[nodes_state_allocated] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[nodes_state_completing] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[nodes_state_idle] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[node_down_fail_list_reason] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[dead_nodes] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[get_partitions] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[sinfo_version] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[show_lsf_configuration] test is skipped because ['run_only']['scheduler'] got value: lsf but detected scheduler: [].
[show_lsf_models] test is skipped because ['run_only']['scheduler'] got value: lsf but detected scheduler: [].
[show_lsf_resources] test is skipped because ['run_only']['scheduler'] got value: lsf but detected scheduler: [].
[lsf_version] test is skipped because ['run_only']['scheduler'] got value: lsf but detected scheduler: [].
[run_only_as_root] test is skipped because ['run_only']['user'] got value: root but detected user: siddiq90.
[current_user_queue] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[show_all_jobs] test is skipped because ['run_only']['scheduler'] got value: slurm but detected scheduler: [].
[skip] test is skipped.
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
---------------------------+--------------+--------------------------------------------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/selinux.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/general_tests/sched/slurm/sacctmgr.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/run_only_distro.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/general_tests/sched/lsf/bmgroups.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/general_tests/sched/lsf/bugroup.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/hello_world.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/general_tests/sched/lsf/bhosts.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/general_tests/sched/lsf/bqueues.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/environment.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/sleep.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/general_tests/sched/slurm/scontrol.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/general_tests/configuration/ssh_localhost.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/general_tests/sched/slurm/sinfo.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/general_tests/sched/lsf/lsinfo.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/general_tests/configuration/systemd-default-target.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/tags_example.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/root_user.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/vars.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/shebang.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/general_tests/configuration/disk_usage.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/general_tests/sched/slurm/squeue.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/python-hello.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/skip_tests.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/general_tests/configuration/ulimits.yml
+----------------------+
| Stage: Building Test |
+----------------------+
name | id | type | executor | tags | testpath
--------------------------+----------+----------+------------+---------------------------+----------------------------------------------------------------------------------------------------------------------------
selinux_disable | 88fb1b1c | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/selinux/selinux_disable/1/stage/generate.sh
exit1_fail | b1b25a16 | script | local.sh | ['tutorials', 'fail'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_fail/2/stage/generate.sh
exit1_pass | 365fdd6e | script | local.sh | ['tutorials', 'pass'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_pass/2/stage/generate.sh
returncode_list_mismatch | aeb6f626 | script | local.sh | ['tutorials', 'fail'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_list_mismatch/2/stage/generate.sh
returncode_int_match | d817c4fd | script | local.sh | ['tutorials', 'pass'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_int_match/2/stage/generate.sh
run_only_macos_distro | 87c4ddbf | script | local.bash | | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/run_only_distro/run_only_macos_distro/0/stage/generate.sh
systemd_default_target | 6fc3c7b4 | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/3/stage/generate.sh
pre_post_build_run | 878b19fc | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/pre_post_build_run/pre_post_build_run/2/stage/generate.sh
hello_world | 87211773 | script | local.bash | tutorials | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/hello_world/hello_world/1/stage/generate.sh
_bin_sh_shell | f16e1275 | script | local.sh | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/4/stage/generate.sh
_bin_bash_shell | 5786ac8b | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/_bin_bash_shell/3/stage/generate.sh
bash_shell | ad7e4f41 | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/bash_shell/3/stage/generate.sh
sh_shell | b5e23cb1 | script | local.sh | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/4/stage/generate.sh
shell_options | 265177ba | script | local.sh | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/4/stage/generate.sh
environment_variables | 72827962 | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/environment/environment_variables/1/stage/generate.sh
sleep | 1a04b18d | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/sleep/sleep/1/stage/generate.sh
vecadd_gnu | 2f7420a3 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vecadd/vecadd_gnu/2/stage/generate.sh
executable_arguments | 03f1c6af | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/passing_args/executable_arguments/2/stage/generate.sh
systemd_default_target | cc2fdeab | script | local.bash | ['system'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd-default-target/systemd_default_target/0/stage/generate.sh
string_tag | 66f45e29 | script | local.bash | network | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/tags_example/string_tag/0/stage/generate.sh
list_of_strings_tags | 371520d5 | script | local.bash | ['network', 'ping'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/tags_example/list_of_strings_tags/0/stage/generate.sh
variables | b3c8fedf | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vars/variables/1/stage/generate.sh
bash_login_shebang | d54ed2f7 | script | local.bash | tutorials | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shebang/bash_login_shebang/1/stage/generate.sh
bash_nonlogin_shebang | ae73cee9 | script | local.bash | tutorials | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shebang/bash_nonlogin_shebang/1/stage/generate.sh
root_disk_usage | 9771523d | script | local.bash | ['filesystem', 'storage'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/disk_usage/root_disk_usage/0/stage/generate.sh
python_hello | adc8633f | script | local.bash | python | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/python-hello/python_hello/2/stage/generate.sh
unskipped | 33ea31ab | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/skip_tests/unskipped/1/stage/generate.sh
hello_f | 590ad365 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_f/2/stage/generate.sh
hello_c | cc592e31 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_c/2/stage/generate.sh
hello_cplusplus | 8655c367 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_cplusplus/2/stage/generate.sh
cc_example | 1edbc832 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cc_example/2/stage/generate.sh
fc_example | 3e112e84 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/fc_example/2/stage/generate.sh
cxx_example | a8eac662 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cxx_example/2/stage/generate.sh
ulimit_filelock | b86d4a0e | script | local.bash | ['system'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/ulimits/ulimit_filelock/0/stage/generate.sh
ulimit_cputime | b021f45d | script | local.bash | ['system'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/ulimits/ulimit_cputime/0/stage/generate.sh
ulimit_stacksize | 178fd98a | script | local.bash | ['system'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/ulimits/ulimit_stacksize/0/stage/generate.sh
ulimit_vmsize | aaea044c | script | local.bash | ['system'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/ulimits/ulimit_vmsize/0/stage/generate.sh
ulimit_filedescriptor | f9968865 | script | local.bash | ['system'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/ulimits/ulimit_filedescriptor/0/stage/generate.sh
ulimit_max_user_process | f6787b9f | script | local.bash | ['system'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/ulimits/ulimit_max_user_process/0/stage/generate.sh
+----------------------+
| Stage: Running Test |
+----------------------+
name | id | executor | status | returncode | testpath
--------------------------+----------+------------+----------+--------------+----------------------------------------------------------------------------------------------------------------------------
selinux_disable | 88fb1b1c | local.bash | FAIL | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/selinux/selinux_disable/1/stage/generate.sh
exit1_fail | b1b25a16 | local.sh | FAIL | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_fail/2/stage/generate.sh
exit1_pass | 365fdd6e | local.sh | PASS | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_pass/2/stage/generate.sh
returncode_list_mismatch | aeb6f626 | local.sh | FAIL | 2 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_list_mismatch/2/stage/generate.sh
returncode_int_match | d817c4fd | local.sh | PASS | 128 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_int_match/2/stage/generate.sh
run_only_macos_distro | 87c4ddbf | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/run_only_distro/run_only_macos_distro/0/stage/generate.sh
systemd_default_target | 6fc3c7b4 | local.bash | FAIL | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/3/stage/generate.sh
pre_post_build_run | 878b19fc | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/pre_post_build_run/pre_post_build_run/2/stage/generate.sh
hello_world | 87211773 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/hello_world/hello_world/1/stage/generate.sh
_bin_sh_shell | f16e1275 | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/4/stage/generate.sh
_bin_bash_shell | 5786ac8b | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/_bin_bash_shell/3/stage/generate.sh
bash_shell | ad7e4f41 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/bash_shell/3/stage/generate.sh
sh_shell | b5e23cb1 | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/4/stage/generate.sh
shell_options | 265177ba | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/4/stage/generate.sh
environment_variables | 72827962 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/environment/environment_variables/1/stage/generate.sh
sleep | 1a04b18d | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/sleep/sleep/1/stage/generate.sh
vecadd_gnu | 2f7420a3 | local.bash | FAIL | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vecadd/vecadd_gnu/2/stage/generate.sh
executable_arguments | 03f1c6af | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/passing_args/executable_arguments/2/stage/generate.sh
systemd_default_target | cc2fdeab | local.bash | FAIL | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd-default-target/systemd_default_target/0/stage/generate.sh
string_tag | 66f45e29 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/tags_example/string_tag/0/stage/generate.sh
list_of_strings_tags | 371520d5 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/tags_example/list_of_strings_tags/0/stage/generate.sh
variables | b3c8fedf | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vars/variables/1/stage/generate.sh
bash_login_shebang | d54ed2f7 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shebang/bash_login_shebang/1/stage/generate.sh
bash_nonlogin_shebang | ae73cee9 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shebang/bash_nonlogin_shebang/1/stage/generate.sh
root_disk_usage | 9771523d | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/disk_usage/root_disk_usage/0/stage/generate.sh
python_hello | adc8633f | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/python-hello/python_hello/2/stage/generate.sh
unskipped | 33ea31ab | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/skip_tests/unskipped/1/stage/generate.sh
hello_f | 590ad365 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_f/2/stage/generate.sh
hello_c | cc592e31 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_c/2/stage/generate.sh
hello_cplusplus | 8655c367 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_cplusplus/2/stage/generate.sh
cc_example | 1edbc832 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cc_example/2/stage/generate.sh
fc_example | 3e112e84 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/fc_example/2/stage/generate.sh
cxx_example | a8eac662 | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cxx_example/2/stage/generate.sh
ulimit_filelock | b86d4a0e | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/ulimits/ulimit_filelock/0/stage/generate.sh
ulimit_cputime | b021f45d | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/ulimits/ulimit_cputime/0/stage/generate.sh
ulimit_stacksize | 178fd98a | local.bash | FAIL | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/ulimits/ulimit_stacksize/0/stage/generate.sh
ulimit_vmsize | aaea044c | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/ulimits/ulimit_vmsize/0/stage/generate.sh
ulimit_filedescriptor | f9968865 | local.bash | FAIL | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/ulimits/ulimit_filedescriptor/0/stage/generate.sh
ulimit_max_user_process | f6787b9f | local.bash | FAIL | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/ulimits/ulimit_max_user_process/0/stage/generate.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 39 tests
Passed Tests: 30/39 Percentage: 76.923%
Failed Tests: 9/39 Percentage: 23.077%
Control builds by Stages¶
You can control behavior of buildtest build command to stop at certain point
using --stage option. This takes two values parse or build, which will
stop buildtest after parsing buildspecs or building the test content.
If you want to know your buildspecs are valid you can use --stage=parse to stop
after parsing the buildspec. Shown below is an example build where we stop
after parse stage.
$ buildtest build -b tutorials/systemd.yml --stage=parse
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+-----------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
Likewise, if you want to troubleshoot your test script without running them you can
use --stage=build which will stop after building your test script. This can
be extremely useful when writing your buildspecs and not having to run your tests.
In this next example, we stop our after the build stage using --stage=build.
$ buildtest build -b tutorials/systemd.yml --stage=build
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+-----------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
+----------------------+
| Stage: Building Test |
+----------------------+
name | id | type | executor | tags | testpath
------------------------+----------+--------+------------+---------------+-------------------------------------------------------------------------------------------------------------
systemd_default_target | f1c076a7 | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/4/stage/generate.sh
Invalid Buildspecs¶
buildtest will skip any buildspecs that fail to validate, in that case the test script will not be generated. Here is an example where we have an invalid buildspec.
$ buildtest build -b tutorials/invalid_buildspec_section.yml
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/invalid_buildspec_section.yml
Error Messages from Stage: Parse
________________________________________________________________________________
Skipping /Users/siddiq90/Documents/buildtest/tutorials/invalid_buildspec_section.yml since it failed to validate
No buildspecs to process because there are no valid buildspecs
buildtest may skip tests from running if buildspec specifies an invalid executor name since buildtest needs to know this in order to delegate test to Executor class responsible for running the test. Here is an example where test failed to run since we provided invalid executor.
$ buildtest build -b tutorials/invalid_executor.yml
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml
Error Messages from Stage: Parse
________________________________________________________________________________
Skipping /Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml since it failed to validate
No buildspecs to process because there are no valid buildspecs
Rebuild Tests¶
buildtest can rebuild tests using the --rebuild option which can be useful if
you want to test a particular test multiple times. The rebuild option works across
all discovered buildspecs and create a new test instance (unique id) and test directory
path. To demonstrate we will build tutorials/python-shell.yml three times using
--rebuild=3.
$ buildtest build -b tutorials/python-shell.yml --rebuild=3
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+----------------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml
+----------------------+
| Stage: Building Test |
+----------------------+
name | id | type | executor | tags | testpath
-------------+----------+--------+--------------+-------------------------+---------------------------------------------------------------------------------------------------------
circle_area | d20fd0ea | script | local.python | ['tutorials', 'python'] | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/5/stage/generate.sh
circle_area | 6de553e7 | script | local.python | ['tutorials', 'python'] | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/6/stage/generate.sh
circle_area | 0fd67587 | script | local.python | ['tutorials', 'python'] | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/7/stage/generate.sh
+----------------------+
| Stage: Running Test |
+----------------------+
name | id | executor | status | returncode | testpath
-------------+----------+--------------+----------+--------------+---------------------------------------------------------------------------------------------------------
circle_area | d20fd0ea | local.python | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/5/stage/generate.sh
circle_area | 6de553e7 | local.python | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/6/stage/generate.sh
circle_area | 0fd67587 | local.python | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/7/stage/generate.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 3 tests
Passed Tests: 3/3 Percentage: 100.000%
Failed Tests: 0/3 Percentage: 0.000%
The rebuild works with all options including: --buildspec, --exclude, --tags
and --executors.
In the next example we rebuild tests by discovering all tags that contain fail.
$ buildtest build --tags fail --rebuild=2
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
[exit1_pass] test is skipped because it is not in tag filter list: ['fail']
[returncode_int_match] test is skipped because it is not in tag filter list: ['fail']
[exit1_pass] test is skipped because it is not in tag filter list: ['fail']
[returncode_int_match] test is skipped because it is not in tag filter list: ['fail']
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+-------------------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
+----------------------+
| Stage: Building Test |
+----------------------+
name | id | type | executor | tags | testpath
--------------------------+----------+--------+------------+-----------------------+---------------------------------------------------------------------------------------------------------------------
exit1_fail | 46a14403 | script | local.sh | ['tutorials', 'fail'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_fail/3/stage/generate.sh
returncode_list_mismatch | 78981e2b | script | local.sh | ['tutorials', 'fail'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_list_mismatch/3/stage/generate.sh
exit1_fail | f3a827e6 | script | local.sh | ['tutorials', 'fail'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_fail/4/stage/generate.sh
returncode_list_mismatch | 1887648a | script | local.sh | ['tutorials', 'fail'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_list_mismatch/4/stage/generate.sh
+----------------------+
| Stage: Running Test |
+----------------------+
name | id | executor | status | returncode | testpath
--------------------------+----------+------------+----------+--------------+---------------------------------------------------------------------------------------------------------------------
exit1_fail | 46a14403 | local.sh | FAIL | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_fail/3/stage/generate.sh
returncode_list_mismatch | 78981e2b | local.sh | FAIL | 2 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_list_mismatch/3/stage/generate.sh
exit1_fail | f3a827e6 | local.sh | FAIL | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_fail/4/stage/generate.sh
returncode_list_mismatch | 1887648a | local.sh | FAIL | 2 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_list_mismatch/4/stage/generate.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 4 tests
Passed Tests: 0/4 Percentage: 0.000%
Failed Tests: 4/4 Percentage: 100.000%
The rebuild option expects a range between 1-50, the --rebuild=1 is equivalent
to running without --rebuild option. We set a max limit for rebuild option to
avoid system degredation due to high workload.
If you try to exceed this bound you will get an error such as:
$ buildtest build -b tutorials/pass_returncode.yml --rebuild 51
usage: buildtest [options] [COMMANDS] build [-h] [-b BUILDSPEC] [-x EXCLUDE] [--tags TAGS] [-e EXECUTOR]
[-s {parse,build}] [-t TESTDIR] [--rebuild REBUILD] [--settings SETTINGS]
buildtest [options] [COMMANDS] build: error: argument --rebuild: 51 must be a positive number between [1-50]
Buildspecs Interface¶
buildtest is able to find and validate all buildspecs in your repos. The
command buildtest buildspec comes with the following options.
$ buildtest buildspec --help
usage: buildtest [options] [COMMANDS] buildspec [-h] {find,view,edit} ...
optional arguments:
-h, --help show this help message and exit
subcommands:
Commands options for Buildspecs
{find,view,edit}
find find all buildspecs
view view a buildspec
edit edit a buildspec
Finding Buildspecs¶
To find all buildspecs run buildtest buildspec find which will discover
all buildspecs in all repos by recursively finding all .yml extensions.
$ buildtest buildspec find
Found 36 buildspecs
Validated 5/36 buildspecs
Validated 10/36 buildspecs
Validated 15/36 buildspecs
Validated 20/36 buildspecs
Validated 25/36 buildspecs
Validated 30/36 buildspecs
Validated 35/36 buildspecs
Validated 36/36 buildspecs
Detected 3 invalid buildspecs
Writing invalid buildspecs to file: /Users/siddiq90/Documents/buildtest/var/buildspec.error
+------------------------------+----------+--------------+---------------------------+------------------------------------------------------------------+
| Name | Type | Executor | Tags | Description |
+==============================+==========+==============+===========================+==================================================================+
| systemd_default_target | script | local.bash | ['tutorials'] | check if default target is multi-user.target |
+------------------------------+----------+--------------+---------------------------+------------------------------------------------------------------+
| run_only_macos_distro | script | local.bash | [] | Run test only if linux distro is darwin. |
+------------------------------+----------+--------------+---------------------------+------------------------------------------------------------------+
| _bin_sh_shell | script | local.sh | ['tutorials'] | /bin/sh shell example |
+------------------------------+----------+--------------+---------------------------+------------------------------------------------------------------+
| _bin_bash_shell | script | local.bash | ['tutorials'] | /bin/bash shell example |
+------------------------------+----------+--------------+---------------------------+------------------------------------------------------------------+
| bash_shell | script | local.bash | ['tutorials'] | bash shell example |
+------------------------------+----------+--------------+---------------------------+------------------------------------------------------------------+
| sh_shell | script | local.sh | ['tutorials'] | sh shell example |
+------------------------------+----------+--------------+---------------------------+------------------------------------------------------------------+
| shell_options | script | local.sh | ['tutorials'] | shell options |
+------------------------------+----------+--------------+---------------------------+------------------------------------------------------------------+
| environment_variables | script | local.bash | ['tutorials'] | Declare environment variables |
+------------------------------+----------+--------------+---------------------------+------------------------------------------------------------------+
| python_hello | script | local.bash | python | Hello World python |
+------------------------------+----------+--------------+---------------------------+------------------------------------------------------------------+
| variables | script | local.bash | ['tutorials'] | Declare shell variables |
+------------------------------+----------+--------------+---------------------------+------------------------------------------------------------------+
| selinux_disable | script | local.bash | ['tutorials'] | Check if SELinux is Disabled |
+------------------------------+----------+--------------+---------------------------+------------------------------------------------------------------+
| bash_login_shebang | script | local.bash | tutorials | customize shebang line with bash login shell |
+------------------------------+----------+--------------+---------------------------+------------------------------------------------------------------+
| bash_nonlogin_shebang | script | local.bash | tutorials | customize shebang line with default bash (nonlogin) shell |
+------------------------------+----------+--------------+---------------------------+------------------------------------------------------------------+
| exit1_fail | script | local.sh | ['tutorials', 'fail'] | exit 1 by default is FAIL |
+------------------------------+----------+--------------+---------------------------+------------------------------------------------------------------+
| exit1_pass | script | local.sh | ['tutorials', 'pass'] | report exit 1 as PASS |
+------------------------------+----------+--------------+---------------------------+------------------------------------------------------------------+
...
buildtest will validate each buildspec file with the appropriate schema type. buildspecs that pass validation will be displayed on screen. buildtest will report all invalid buildspecs in a text file for you to review.
buildtest will cache the results in var/buildspec-cache.json so subsequent
runs to buildtest buildspec find will be much faster because it is read from cache.
If you make changes to buildspec you may want to rebuild the buildspec cache then
run:
$ buildtest buildspec find --clear
Shown below is a list of options for buildtest buildspec find command.
$ buildtest buildspec find --help
usage: buildtest [options] [COMMANDS] buildspec find [-h] [-c] [-t] [-bf] [-le] [--filter FILTER] [--helpfilter]
optional arguments:
-h, --help show this help message and exit
-c, --clear Clear buildspec cache and find all buildspecs again
-t, --tags List all available tags
-bf, --buildspec-files
Get all buildspec files from cache
-le, --list-executors
get all unique executors from buildspecs
--filter FILTER Filter buildspec cache with filter fields in format --filter key1=val1,key2=val2
--helpfilter Show Filter fields for --filter option for querying buildspec cache
If you want to find all buildspec files in cache run buildtest buildspec find --buildspec-files
$ buildtest buildspec find --buildspec-files
+--------------------------------------------------------------------------------------------+
| buildspecs |
+============================================================================================+
| /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml |
+--------------------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/run_only_distro.yml |
+--------------------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml |
+--------------------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/environment.yml |
+--------------------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/python-hello.yml |
+--------------------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/vars.yml |
+--------------------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/selinux.yml |
+--------------------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/shebang.yml |
+--------------------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/hello_world.yml |
+--------------------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/root_user.yml |
+--------------------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/tags_example.yml |
+--------------------------------------------------------------------------------------------+
| /Users/siddiq90/Documents/buildtest/tutorials/run_only_platform.yml |
+--------------------------------------------------------------------------------------------+
...
Filtering buildspec¶
You can filter buildspec cache using the the --filter option. Let’s take a look
at the available filter fields that are acceptable with filter option.
$ buildtest buildspec find --helpfilter
Field Description Type
-------- ----------------------- ------
executor Filter by executor name STRING
tags Filter by tag name STRING
type Filter by schema type STRING
The --filter option accepts arguments in key/value format as follows:
buildtest buildspec find --filter key1=value1,key2=value2,key3=value3
We can filter buildspec cache by tags=fail which will query all tests with
associated tag field in test.
$ buildtest buildspec find --filter tags=fail
+--------------------------+--------+------------+-----------------------+-----------------------------------------------------+
| Name | Type | Executor | Tags | Description |
+==========================+========+============+=======================+=====================================================+
| exit1_fail | script | local.sh | ['tutorials', 'fail'] | exit 1 by default is FAIL |
+--------------------------+--------+------------+-----------------------+-----------------------------------------------------+
| returncode_list_mismatch | script | local.sh | ['tutorials', 'fail'] | exit 2 failed since it failed to match returncode 1 |
+--------------------------+--------+------------+-----------------------+-----------------------------------------------------+
In addition, we can query buildspecs by schema type, in next example we query all tests using the script schema
$ buildtest buildspec find --filter type=script
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| Name | Type | Executor | Tags | Description |
+==============================+========+==============+===========================+==================================================================+
| systemd_default_target | script | local.bash | ['tutorials'] | check if default target is multi-user.target |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| run_only_macos_distro | script | local.bash | [] | Run test only if linux distro is darwin. |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| _bin_sh_shell | script | local.sh | ['tutorials'] | /bin/sh shell example |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| _bin_bash_shell | script | local.bash | ['tutorials'] | /bin/bash shell example |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| bash_shell | script | local.bash | ['tutorials'] | bash shell example |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| sh_shell | script | local.sh | ['tutorials'] | sh shell example |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| shell_options | script | local.sh | ['tutorials'] | shell options |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| environment_variables | script | local.bash | ['tutorials'] | Declare environment variables |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| python_hello | script | local.bash | python | Hello World python |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| variables | script | local.bash | ['tutorials'] | Declare shell variables |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| selinux_disable | script | local.bash | ['tutorials'] | Check if SELinux is Disabled |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| bash_login_shebang | script | local.bash | tutorials | customize shebang line with bash login shell |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| bash_nonlogin_shebang | script | local.bash | tutorials | customize shebang line with default bash (nonlogin) shell |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| exit1_fail | script | local.sh | ['tutorials', 'fail'] | exit 1 by default is FAIL |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| exit1_pass | script | local.sh | ['tutorials', 'pass'] | report exit 1 as PASS |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| returncode_list_mismatch | script | local.sh | ['tutorials', 'fail'] | exit 2 failed since it failed to match returncode 1 |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| returncode_int_match | script | local.sh | ['tutorials', 'pass'] | exit 128 matches returncode 128 |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| hello_world | script | local.bash | tutorials | hello world example |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| run_only_as_root | script | local.bash | ['tutorials'] | This test will only run if current user is root |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| string_tag | script | local.bash | network | tags can be a string |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| list_of_strings_tags | script | local.bash | ['network', 'ping'] | tags can be a list of strings |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| run_only_platform_darwin | script | local.python | ['tutorials'] | This test will only run if target platform is Darwin |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| run_only_platform_linux | script | local.python | ['tutorials'] | This test will only run if target platform is Linux |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| circle_area | script | local.python | ['tutorials', 'python'] | Calculate circle of area given a radius |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| skip | script | local.bash | ['tutorials'] | |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| unskipped | script | local.bash | ['tutorials'] | |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| sleep | script | local.bash | ['tutorials'] | sleep 2 seconds |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| root_disk_usage | script | local.bash | ['filesystem', 'storage'] | Check root disk usage and report if it exceeds threshold |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| ssh_localhost_remotecommand | script | local.bash | ['ssh'] | Test if ssh on localhost works and if we can run remote command. |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| systemd_default_target | script | local.bash | ['system'] | check if default target is multi-user.target |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| ulimit_filelock | script | local.bash | ['system'] | Check if file lock is set to unlimited in ulimits |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| ulimit_cputime | script | local.bash | ['system'] | Check if cputime is set to unlimited in ulimits |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| ulimit_stacksize | script | local.bash | ['system'] | Check if stack size is set to unlimited in ulimits |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| ulimit_vmsize | script | local.bash | ['system'] | Check virtual memory size and check if its set to unlimited |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| ulimit_filedescriptor | script | local.bash | ['system'] | Check if open file descriptors limit is set to 4096 |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| ulimit_max_user_process | script | local.bash | ['system'] | Check max number of user process limit is set to 2048 |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| show_accounts | script | local.bash | ['slurm'] | run sacctmgr list accounts |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| show_users | script | local.bash | ['slurm'] | run sacctmgr list users |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| show_qos | script | local.bash | ['slurm'] | run sacctmgr list qos |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| show_tres | script | local.bash | ['slurm'] | run sacctmgr list tres |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| slurm_config | script | local.bash | ['slurm'] | run scontrol show config |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| show_partition | script | local.bash | ['slurm'] | run scontrol show partition |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| current_user_queue | script | local.bash | ['slurm'] | show all current pending jobs for current user (squeue -u $USER) |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| show_all_jobs | script | local.bash | ['slurm'] | show all pending + running jobs (squeue -a) |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| nodes_state_down | script | local.bash | ['slurm'] | Show nodes in DOWN state |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| nodes_state_reboot | script | local.bash | ['slurm'] | Show nodes in REBOOT state |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| nodes_state_allocated | script | local.bash | ['slurm'] | Show nodes in ALLOCATED state |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| nodes_state_completing | script | local.bash | ['slurm'] | Show nodes in COMPLETING state |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| nodes_state_idle | script | local.bash | ['slurm'] | Show nodes in IDLE state |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| node_down_fail_list_reason | script | local.bash | ['slurm'] | Show nodes DOWN, DRAINED, FAIL or FAILING and list reason |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| dead_nodes | script | local.bash | ['slurm'] | Show non-responding nodes |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| get_partitions | script | local.bash | ['slurm'] | Get all slurm partitions |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| sinfo_version | script | local.bash | ['slurm'] | get slurm version |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| show_host_groups | script | local.bash | lsf | Show information about host groups using bmgroup |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| show_lsf_configuration | script | local.bash | lsf | Show LSF configuration using lsinfo |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| show_lsf_models | script | local.bash | lsf | Show information about host models in LSF cluster |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| show_lsf_resources | script | local.bash | lsf | Show information about LSF resources |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| lsf_version | script | local.bash | lsf | Display lsf version using lsinfo |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| show_lsf_user_groups | script | local.bash | lsf | Show information about all LSF user groups |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| show_lsf_queues | script | local.bash | lsf | Show LSF queues |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| show_lsf_queues_formatted | script | local.bash | lsf | Show LSF queues with formatted columns |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| show_lsf_queues_current_user | script | local.bash | lsf | Show LSF queues available for current user |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| display_lsf_hosts | script | local.bash | lsf | Show all hosts in LSF cluster |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| display_hosts_format | script | local.bash | lsf | Show all hosts with column hostname and status |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
| bhosts_version | script | local.bash | lsf | Display bhosts version |
+------------------------------+--------+--------------+---------------------------+------------------------------------------------------------------+
Finally, we can combine multiple filter fields separated by comma, in next example
we query all buildspecs with tags=tutorials, executor=local.sh, and type=script
$ buildtest buildspec find --filter tags=tutorials,executor=local.sh,type=script
+--------------------------+--------+------------+-----------------------+-----------------------------------------------------+
| Name | Type | Executor | Tags | Description |
+==========================+========+============+=======================+=====================================================+
| _bin_sh_shell | script | local.sh | ['tutorials'] | /bin/sh shell example |
+--------------------------+--------+------------+-----------------------+-----------------------------------------------------+
| sh_shell | script | local.sh | ['tutorials'] | sh shell example |
+--------------------------+--------+------------+-----------------------+-----------------------------------------------------+
| shell_options | script | local.sh | ['tutorials'] | shell options |
+--------------------------+--------+------------+-----------------------+-----------------------------------------------------+
| exit1_fail | script | local.sh | ['tutorials', 'fail'] | exit 1 by default is FAIL |
+--------------------------+--------+------------+-----------------------+-----------------------------------------------------+
| exit1_pass | script | local.sh | ['tutorials', 'pass'] | report exit 1 as PASS |
+--------------------------+--------+------------+-----------------------+-----------------------------------------------------+
| returncode_list_mismatch | script | local.sh | ['tutorials', 'fail'] | exit 2 failed since it failed to match returncode 1 |
+--------------------------+--------+------------+-----------------------+-----------------------------------------------------+
| returncode_int_match | script | local.sh | ['tutorials', 'pass'] | exit 128 matches returncode 128 |
+--------------------------+--------+------------+-----------------------+-----------------------------------------------------+
Querying buildspec tags¶
If you want to retrieve all unique tags from all buildspecs you can run
buildtest buildspec find --tags
$ buildtest buildspec find --tags
+------------+
| Tags |
+============+
| tutorials |
+------------+
| compile |
+------------+
| ssh |
+------------+
| filesystem |
+------------+
| slurm |
+------------+
| storage |
+------------+
| pass |
+------------+
| ping |
+------------+
| lsf |
+------------+
| python |
+------------+
| system |
+------------+
| network |
+------------+
| fail |
+------------+
Querying buildspec executor¶
To find all executors from cache you can run buildtest buildspec find --list-executors.
This will retrieve the ‘executor’ field from all buildspec and any duplicates will
be ignored.
$ buildtest buildspec find --list-executors
+--------------+
| executors |
+==============+
| local.bash |
+--------------+
| local.python |
+--------------+
| local.sh |
+--------------+
Viewing Buildspecs¶
If you want to view or edit a buildspec you can type the name of test. Since we can have more than one test in a buildspec, opening any of the name entry that map to same file will result in same operation.
For example, we can view systemd_default_target as follows
$ buildtest buildspec view systemd_default_target
version: "1.0"
buildspecs:
systemd_default_target:
executor: local.bash
type: script
description: check if default target is multi-user.target
tags: [tutorials]
run: |
if [ "multi-user.target" == `systemctl get-default` ]; then
echo "multi-user is the default target";
exit 0
fi
echo "multi-user is not the default target";
exit 1
status:
returncode: 0
Editing Buildspecs¶
To edit a buildspec you can run buildtest buildspec edit <name> which
will open file in editor. Once you make change, buildtest will validate the
buildspec upon closure, if there is an issue buildtest will report an error
during validation and you will be prompted to fix issue until it is resolved.
For example we can see an output message after editing file, user will be prompted to press a key which will open the file in editor:
$ buildtest buildspec edit systemd_default_target
version 1.1 is not known for type {'1.0': 'script-v1.0.schema.json', 'latest': 'script-v1.0.schema.json'}. Try using latest.
Press any key to continue
Test Reports (buildtest report)¶
buildtest keeps track of all test results which can be retrieved via buildtest report. Shown below is command usage.
$ buildtest report --help
usage: buildtest [options] [COMMANDS] report [-h] [--helpformat] [--format FORMAT] [--filter FILTER] [--helpfilter]
optional arguments:
-h, --help show this help message and exit
--helpformat List of available format fields
--format FORMAT format field for printing purposes. For more details see --helpformat for list of available fields.
Fields must be separated by comma (--format <field1>,<field2>,...)
--filter FILTER Filter report by filter fields. The filter fields must be set in format: --filter
key1=val1,key2=val2,...
--helpfilter Report a list of filter fields to be used with --filter option
You may run buildtest report and buildtest will display report
with default format fields.
$ buildtest report
+----------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+--------------------------------------------------------------------------------------------+
| name | id | state | returncode | starttime | endtime | runtime | tags | buildspec |
+============================+==========+=========+==============+=====================+=====================+===========+====================+============================================================================================+
| systemd_default_target | 1770533b | FAIL | 1 | 2020/10/19 15:54:23 | 2020/10/19 15:54:23 | 0.182725 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml |
+----------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+--------------------------------------------------------------------------------------------+
| systemd_default_target | 15a66b55 | FAIL | 1 | 2020/10/19 15:54:25 | 2020/10/19 15:54:25 | 0.0458719 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml |
+----------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+--------------------------------------------------------------------------------------------+
| systemd_default_target | 8864cbc1 | FAIL | 1 | 2020/10/19 15:54:31 | 2020/10/19 15:54:31 | 0.0545546 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml |
+----------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+--------------------------------------------------------------------------------------------+
| systemd_default_target | 6fc3c7b4 | FAIL | 1 | 2020/10/19 15:54:36 | 2020/10/19 15:54:36 | 0.0453 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml |
+----------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+--------------------------------------------------------------------------------------------+
| circle_area | 0bdaef77 | PASS | 0 | 2020/10/19 15:54:24 | 2020/10/19 15:54:24 | 0.147359 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/python-shell.yml |
+----------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+--------------------------------------------------------------------------------------------+
| circle_area | b06f76c2 | PASS | 0 | 2020/10/19 15:54:25 | 2020/10/19 15:54:25 | 0.149374 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/python-shell.yml |
+----------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+--------------------------------------------------------------------------------------------+
| hello_dinosaur | dff7a691 | PASS | 0 | 2020/10/19 15:54:24 | 2020/10/19 15:54:24 | 0.0483172 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/environment.yml |
+----------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+--------------------------------------------------------------------------------------------+
| hello_dinosaur | cebde392 | PASS | 0 | 2020/10/19 15:54:25 | 2020/10/19 15:54:25 | 0.0721789 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/environment.yml |
+----------------------------+----------+---------+--------------+---------------------+---------------------+-----------+--------------------+--------------------------------------------------------------------------------------------+
...
Format Reports¶
There are more fields captured in the report, so if you want to see a
list of available format fields run buildtest report --helpformat.
$ buildtest report --helpformat
Fields Description
---------- --------------------------------------------
buildspec Buildspec file
name Name of test defined in buildspec
id Unique Build Identifier (abbreviated)
full_id Full qualified unique build identifier
testroot Root of test directory
testpath Path to test
command Command executed
outfile Output file
errfile Error File
schemafile Schema file used for validation
executor Executor name
tags Tag name
starttime Start Time of test in date format
endtime End Time for Test in date format
runtime Total runtime in seconds
state Test State reported by buildtest (PASS/FAIL)
returncode Return Code from Test Execution
You can format report using --format field which expects field
name separated by comma (i.e –format <field1>,<field2>). In this example
we format by fields --format id,executor,state,returncode
$ buildtest report --format name,id,executor,state,returncode
+----------------------------+----------+--------------+---------+--------------+
| name | id | executor | state | returncode |
+============================+==========+==============+=========+==============+
| systemd_default_target | 1770533b | local.bash | FAIL | 1 |
+----------------------------+----------+--------------+---------+--------------+
| systemd_default_target | 15a66b55 | local.bash | FAIL | 1 |
+----------------------------+----------+--------------+---------+--------------+
| systemd_default_target | 8864cbc1 | local.bash | FAIL | 1 |
+----------------------------+----------+--------------+---------+--------------+
| systemd_default_target | 6fc3c7b4 | local.bash | FAIL | 1 |
+----------------------------+----------+--------------+---------+--------------+
| circle_area | 0bdaef77 | local.python | PASS | 0 |
+----------------------------+----------+--------------+---------+--------------+
| circle_area | b06f76c2 | local.python | PASS | 0 |
+----------------------------+----------+--------------+---------+--------------+
| hello_dinosaur | dff7a691 | local.bash | PASS | 0 |
+----------------------------+----------+--------------+---------+--------------+
| hello_dinosaur | cebde392 | local.bash | PASS | 0 |
+----------------------------+----------+--------------+---------+--------------+
...
Filter Reports¶
You can also filter reports using the --filter option, but first let’s
check the available filter fields. In order to see available filter fields
run buildtest report --helpfilter.
$ buildtest report --helpfilter
Filter Fields Description Expected Value
--------------- -------------------------- ----------------
buildspec Filter by buildspec file FILE
name Filter by test name STRING
executor Filter by executor name STRING
state Filter by test state PASS/FAIL
tags Filter tests by tag name STRING
returncode Filter tests by returncode INT
The --filter expects arguments in key=value format, you can
specify multiple filter fields by a comma. buildtest will treat multiple
filters as logical AND operation. The filter option can be used with
--format field. Let’s see some examples to illustrate the point.
To see all tests with returncode of 2 we set --filter returncode=2.
$ buildtest report --filter returncode=2 --format=name,id,returncode
+--------------------------+----------+--------------+
| name | id | returncode |
+==========================+==========+==============+
| returncode_list_mismatch | 269abcb9 | 2 |
+--------------------------+----------+--------------+
| returncode_list_mismatch | e5905c73 | 2 |
+--------------------------+----------+--------------+
| returncode_list_mismatch | aeb6f626 | 2 |
+--------------------------+----------+--------------+
| returncode_list_mismatch | 78981e2b | 2 |
+--------------------------+----------+--------------+
| returncode_list_mismatch | 1887648a | 2 |
+--------------------------+----------+--------------+
| returncode_list_mismatch | 49d45f72 | 2 |
+--------------------------+----------+--------------+
Note
buildtest automatically converts returncode to integer when matching returncode, so --filter returncode="2" will work too
If you want to filter by test name exit1_pass you can use the
name=exit1_pass field as shown below
$ buildtest report --filter name=exit1_pass --format=name,id,returncode,state
+------------+----------+--------------+---------+
| name | id | returncode | state |
+============+==========+==============+=========+
| exit1_pass | 992a08e0 | 1 | PASS |
+------------+----------+--------------+---------+
| exit1_pass | 943c3d32 | 1 | PASS |
+------------+----------+--------------+---------+
| exit1_pass | 365fdd6e | 1 | PASS |
+------------+----------+--------------+---------+
| exit1_pass | 59997545 | 1 | PASS |
+------------+----------+--------------+---------+
Likewise, we can filter tests by buildspec file using the --filter buildspec=<file>.
In example below we set buildspec=tutorials/pass_returncode.yml. In this example,
buildtest will resolve path and find the buildspec. If file doesn’t exist or is
not found in cache it will raise an error
$ buildtest report --filter buildspec=tutorials/pass_returncode.yml --format=name,id,state,buildspec
+--------------------------+----------+---------+-------------------------------------------------------------------+
| name | id | state | buildspec |
+==========================+==========+=========+===================================================================+
| exit1_fail | d405aea9 | FAIL | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| exit1_fail | 0a7c4951 | FAIL | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| exit1_fail | b1b25a16 | FAIL | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| exit1_fail | 46a14403 | FAIL | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| exit1_fail | f3a827e6 | FAIL | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| exit1_fail | b046ccd8 | FAIL | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| exit1_pass | 992a08e0 | PASS | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| exit1_pass | 943c3d32 | PASS | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| exit1_pass | 365fdd6e | PASS | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| exit1_pass | 59997545 | PASS | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| returncode_list_mismatch | 269abcb9 | FAIL | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| returncode_list_mismatch | e5905c73 | FAIL | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| returncode_list_mismatch | aeb6f626 | FAIL | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| returncode_list_mismatch | 78981e2b | FAIL | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| returncode_list_mismatch | 1887648a | FAIL | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| returncode_list_mismatch | 49d45f72 | FAIL | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| returncode_int_match | 146a0269 | PASS | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| returncode_int_match | ac11ac19 | PASS | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| returncode_int_match | d817c4fd | PASS | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
| returncode_int_match | 502be830 | PASS | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+--------------------------+----------+---------+-------------------------------------------------------------------+
We can also pass multiple filter fields for instance if we want to find all FAIL tests for executor local.sh we can do the following
$ buildtest report --filter state=FAIL,executor=local.sh --format=name,id,state,executor
+--------------------------+----------+---------+------------+
| name | id | state | executor |
+==========================+==========+=========+============+
| exit1_fail | d405aea9 | FAIL | local.sh |
+--------------------------+----------+---------+------------+
| exit1_fail | 0a7c4951 | FAIL | local.sh |
+--------------------------+----------+---------+------------+
| exit1_fail | b1b25a16 | FAIL | local.sh |
+--------------------------+----------+---------+------------+
| exit1_fail | 46a14403 | FAIL | local.sh |
+--------------------------+----------+---------+------------+
| exit1_fail | f3a827e6 | FAIL | local.sh |
+--------------------------+----------+---------+------------+
| exit1_fail | b046ccd8 | FAIL | local.sh |
+--------------------------+----------+---------+------------+
| returncode_list_mismatch | 269abcb9 | FAIL | local.sh |
+--------------------------+----------+---------+------------+
| returncode_list_mismatch | e5905c73 | FAIL | local.sh |
+--------------------------+----------+---------+------------+
| returncode_list_mismatch | aeb6f626 | FAIL | local.sh |
+--------------------------+----------+---------+------------+
| returncode_list_mismatch | 78981e2b | FAIL | local.sh |
+--------------------------+----------+---------+------------+
| returncode_list_mismatch | 1887648a | FAIL | local.sh |
+--------------------------+----------+---------+------------+
| returncode_list_mismatch | 49d45f72 | FAIL | local.sh |
+--------------------------+----------+---------+------------+
Filter Exception Cases¶
The returncode filter field expects an integer value, so if you try a non-integer
returncode you will get the following message:
$ buildtest report --filter returncode=1.5
Traceback (most recent call last):
File "/Users/siddiq90/Documents/buildtest/bin/buildtest", line 17, in <module>
buildtest.main.main()
File "/Users/siddiq90/Documents/buildtest/buildtest/main.py", line 45, in main
args.func(args)
File "/Users/siddiq90/Documents/buildtest/buildtest/menu/report.py", line 128, in func_report
raise BuildTestError(f"Invalid returncode:{filter_args[key]} must be an integer")
buildtest.exceptions.BuildTestError: 'Invalid returncode:1.5 must be an integer'
The state filter field expects value of PASS or FAIL so if you specify an
invalid state you will get an error as follows:
$ buildtest report --filter state=UNKNOWN
filter argument 'state' must be 'PASS' or 'FAIL' got value UNKNOWN
The buildspec field expects a valid file path, it can be an absolute or relative
path, buildtest will resolve absolute path and check if file exist and is in the report
file. If it’s an invalid file we get an error such as:
$ buildtest report --filter buildspec=/path/to/invalid.yml
Invalid File Path for filter field 'buildspec': /path/to/invalid.yml
You may have a valid filepath for buildspec filter field such as
tutorials/invalid_executor.yml, but there is no record in the report cache
because this test can’t be run. In this case you will get the following message:
$ buildtest report --filter buildspec=tutorials/invalid_executor.yml
buildspec file: /Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml not found in cache
Test Inspection¶
buildtest provides an interface via buildtest inspect to query test details once
test is recorded in var/report.json. The command usage is the following.
$ buildtest inspect --help
usage: buildtest [options] [COMMANDS] inspect [-h] test
positional arguments:
test select unique test
optional arguments:
-h, --help show this help message and exit
The buildtest inspect expects a unique test id this can be
retrieve using the full_id format field if you are not sure:
$ buildtest report --format name, full_id
For example, let’s assume we have the following tests in our report:
$ buildtest report --format name,full_id
+-------------------------+--------------------------------------+
| name | full_id |
+=========================+======================================+
| bash_login_shebang | eb6e26b2-938b-4913-8b98-e21528c82778 |
+-------------------------+--------------------------------------+
| bash_login_shebang | d7937a9a-d3fb-4d3f-95e1-465488757820 |
+-------------------------+--------------------------------------+
| bash_login_shebang | dea6c6fd-b9a6-4b07-a3fc-b483d02d7ff9 |
+-------------------------+--------------------------------------+
| bash_nonlogin_shebang | bbf94b94-949d-4f97-987a-9a93309f1dc2 |
+-------------------------+--------------------------------------+
| bash_nonlogin_shebang | 7ca9db2f-1e2b-4739-b9a2-71c8cc00249e |
+-------------------------+--------------------------------------+
| bash_nonlogin_shebang | 4c5caf85-6ba0-4ca0-90b0-c769a2fcf501 |
+-------------------------+--------------------------------------+
| root_disk_usage | e78071ef-6444-4228-b7f9-b4eb39071fdd |
+-------------------------+--------------------------------------+
| ulimit_filelock | c6294cfa-c559-493b-b44f-b17b54ec276d |
+-------------------------+--------------------------------------+
| ulimit_cputime | aa5530e2-be09-4d49-b8c0-0e818f855a40 |
+-------------------------+--------------------------------------+
| ulimit_stacksize | 3591925d-7dfa-4bc7-a3b1-fb9dfadf956e |
+-------------------------+--------------------------------------+
| ulimit_vmsize | 4a01f26b-9c8a-4870-8e33-51923c8c46ad |
+-------------------------+--------------------------------------+
| ulimit_filedescriptor | 565b85ac-e51f-46f9-8c6f-c2899a370609 |
+-------------------------+--------------------------------------+
| ulimit_max_user_process | 0486c11c-5733-4d8e-822e-c0adddbb2af7 |
+-------------------------+--------------------------------------+
| systemd_default_target | 7cfc9057-6338-403c-a7af-b1301d04d817 |
+-------------------------+--------------------------------------+
Let’s assume we are interested in viewing test bash_login_shebang, since we
have multiple instance for same test we must specify a unique id. In example below
we query the the test id eb6e26b2-938b-4913-8b98-e21528c82778:
$ buildtest inspect eb6e26b2-938b-4913-8b98-e21528c82778
{
"id": "eb6e26b2",
"full_id": "eb6e26b2-938b-4913-8b98-e21528c82778",
"testroot": "/Users/siddiq90/Documents/buildtest/var/tests/local.bash/shebang/bash_login_shebang/0",
"testpath": "/Users/siddiq90/Documents/buildtest/var/tests/local.bash/shebang/bash_login_shebang/0/stage/generate.sh",
"command": "/Users/siddiq90/Documents/buildtest/var/tests/local.bash/shebang/bash_login_shebang/0/stage/generate.sh",
"outfile": "/Users/siddiq90/Documents/buildtest/var/tests/local.bash/shebang/bash_login_shebang/0/run/bash_login_shebang.out",
"errfile": "/Users/siddiq90/Documents/buildtest/var/tests/local.bash/shebang/bash_login_shebang/0/run/bash_login_shebang.err",
"schemafile": "script-v1.0.schema.json",
"executor": "local.bash",
"tags": "tutorials",
"starttime": "2020/10/21 16:27:18",
"endtime": "2020/10/21 16:27:18",
"runtime": 0.26172968399999996,
"state": "PASS",
"returncode": 0
}
Output File
______________________________
Login Shell
Error File
______________________________
Test Content
______________________________
#!/bin/bash -l
source /Users/siddiq90/Documents/buildtest/var/executors/local.bash/before_script.sh
shopt -q login_shell && echo 'Login Shell' || echo 'Not Login Shell'
source /Users/siddiq90/Documents/buildtest/var/executors/local.bash/after_script.sh
buildspec: /Users/siddiq90/Documents/buildtest/tutorials/shebang.yml
______________________________
version: "1.0"
buildspecs:
bash_login_shebang:
type: script
executor: local.bash
shebang: "#!/bin/bash -l"
description: customize shebang line with bash login shell
tags: tutorials
run: shopt -q login_shell && echo 'Login Shell' || echo 'Not Login Shell'
status:
regex:
exp: "^Login Shell$"
stream: stdout
bash_nonlogin_shebang:
type: script
executor: local.bash
shebang: "#!/bin/bash"
description: customize shebang line with default bash (nonlogin) shell
tags: tutorials
run: shopt -q login_shell && echo 'Login Shell' || echo 'Not Login Shell'
status:
regex:
exp: "^Not Login Shell$"
stream: stdout
buildtest will present the test record from JSON record including contents of output file, error file, testscript and buildspec file.
User can can specify first few characters of the id and buildtest will detect if its a unique test id. If buildtest discovers more than one test id, then buildtest will report all the ids where there is a conflict. In example below we find two tests with id 7c:
$ buildtest inspect 7c
Detected 2 test records, please specify a unique test id
7ca9db2f-1e2b-4739-b9a2-71c8cc00249e
7cfc9057-6338-403c-a7af-b1301d04d817
Note
This feature is in development and may change in future
buildtest schemas¶
The buildtest schema command can show you list of available schemas just run
the command with no options and it will show all the json schemas supported by buildtest.
$ buildtest schema
global.schema.json
definitions.schema.json
settings.schema.json
compiler-v1.0.schema.json
script-v1.0.schema.json
Shown below is the command usage of buildtest schema
$ buildtest schema --help
usage: buildtest [options] [COMMANDS] schema [-h] [-n Schema Name] [-e] [-j] [-v]
optional arguments:
-h, --help show this help message and exit
-n Schema Name, --name Schema Name
show schema by name (e.g., script)
-e, --example Show schema examples
-j, --json Display json schema file
-v, --validate Validate all schema examples with corresponding schemafile
The json schemas are hosted on the web at https://buildtesters.github.io/schemas/. buildtest provides a means to display the json schema from the buildtest interface. Note that buildtest will show the schemas provided in buildtest repo and not ones provided by schemas repo. This is because, we let development of schema run independent of the framework.
To select a JSON schema use the --name option to select a schema, for example
to view a JSON Schema for script-v1.0.schema.json run the following:
$ buildtest schema --name script-v1.0.schema.json --json
Similarly, if you want to view example buildspecs for a schema use the --example
option with a schema. For example to view all example schemas for
compiler-v1.0.schema.json run the following:
$ buildtest schema --name compiler-v1.0.schema.json --example
Debug Mode¶
buildtest can stream logs to stdout stream for debugging. You can use buildtest -d <DEBUGLEVEL>
or long option --debug with any buildtest commands. The DEBUGLEVEL are the following:
DEBUG
INFO
WARNING
ERROR
CRITICAL
buildtest is using logging.setLevel
to control log level. The content is logged in file buildtest.log in your current
directory with default log level of DEBUG. If you want to get all logs use
-d DEBUG with your buildtest command:
buildtest -d DEBUG <command>
The debug mode can be useful when troubleshooting builds, in this example we
set debug level to DEBUG for an invalid buildspec.
$ buildtest -d DEBUG build -b tutorials/invalid_executor.yml
2020-10-19 15:54:47,065 [config.py:42 - check_settings() ] - [DEBUG] Loading default settings schema: /Users/siddiq90/Documents/buildtest/buildtest/schemas/settings.schema.json
2020-10-19 15:54:47,065 [utils.py:34 - load_schema() ] - [DEBUG] Successfully loaded schema file: /Users/siddiq90/Documents/buildtest/buildtest/schemas/settings.schema.json
2020-10-19 15:54:47,065 [config.py:45 - check_settings() ] - [DEBUG] Validating user schema with schema: /Users/siddiq90/Documents/buildtest/buildtest/schemas/settings.schema.json
2020-10-19 15:54:47,067 [config.py:47 - check_settings() ] - [DEBUG] Validation was successful
2020-10-19 15:54:47,067 [build.py:139 - discover_by_buildspecs() ] - [DEBUG] BuildSpec: tutorials/invalid_executor.yml is a file
2020-10-19 15:54:47,067 [build.py:151 - discover_by_buildspecs() ] - [INFO] Found the following config files: ['/Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml']
2020-10-19 15:54:47,068 [build.py:212 - discover_buildspecs() ] - [DEBUG] Based on input argument: --buildspec ['tutorials/invalid_executor.yml'] buildtest discovered the following Buildspecs: ['/Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml']
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml
2020-10-19 15:54:47,068 [setup.py:40 - __init__() ] - [DEBUG] Getting Executors from buildtest settings
2020-10-19 15:54:47,070 [parser.py:94 - _validate_global() ] - [INFO] Validating /Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml with schema: /Users/siddiq90/Documents/buildtest/buildtest/schemas/global.schema.json
2020-10-19 15:54:47,070 [parser.py:115 - _validate() ] - [INFO] Validating test - 'wrongexecutor' in recipe: /Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml
2020-10-19 15:54:47,070 [parser.py:123 - _validate() ] - [INFO] wrongexecutor is a dictionary
2020-10-19 15:54:47,070 [parser.py:133 - _validate() ] - [INFO] Detected field 'type: script'
2020-10-19 15:54:47,070 [parser.py:140 - _validate() ] - [INFO] Checking script in supported type schemas: ['script', 'compiler']
2020-10-19 15:54:47,071 [buildspec.py:406 - parse_buildspecs() ] - [ERROR] executor: badexecutor not found in executor list: ['local.bash', 'local.sh', 'local.python']
Error Messages from Stage: Parse
________________________________________________________________________________
Skipping /Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml since it failed to validate
No buildspecs to process because there are no valid buildspecs
Accessing buildtest documentation¶
We provide two command line options to access buildtest and schema docs. To access buildtest docs you can run:
$ buildtest docs
To access schema docs run:
$ buildtest schemadocs
Logfile¶
Currently, buildtest will write the log file for any buildtest build command
in buildtest.log of the current directory. The logfile will be overwritten
if you run repeative commands from same directory. A permanent log file location
will be implemented (TBD).