Getting Started with buildtest¶
Interacting with the client¶
After you install buildtest, you should find the client on your path:
$ which buildtest
~/.local/bin/buildtest
If you don’t see buildtest go back and review section Installing buildtest.
Buildspecs¶
Finding Buildspecs¶
buildtest is able to find and validate all buildspecs in your repos. The
command buildtest buildspec comes with the following options.
$ buildtest buildspec --help
usage: buildtest [options] [COMMANDS] buildspec [-h] {find,view,edit} ...
optional arguments:
-h, --help show this help message and exit
subcommands:
Commands options for Buildspecs
{find,view,edit}
find find all buildspecs
view view a buildspec
edit edit a buildspec
To find all buildspecs run buildtest buildspec find which will discover
all buildspecs in all repos. buildtest will recursively traverse all repos
and seek out all .yml extensions so make sure your buildspecs conform to
the file extension.
$ buildtest buildspec find
Searching buildspec in following directories: /Users/siddiq90/Documents/buildtest/tutorials
Found 14 buildspecs
Validated 5/14 buildspecs
Validated 10/14 buildspecs
Validated 14/14 buildspecs
Detected 1 invalid buildspecs
Writing invalid buildspecs to file: /Users/siddiq90/Documents/buildtest/var/buildspec.error
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| Name | Type | Executor | Tags | Description |
+========================+==========+==============+===============+================================================================+
| systemd_default_target | script | local.bash | ['tutorials'] | check if default target is multi-user.target |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| _bin_sh_shell | script | local.sh | ['tutorials'] | /bin/sh shell example |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| _bin_bash_shell | script | local.bash | ['tutorials'] | /bin/bash shell example |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| bash_shell | script | local.bash | ['tutorials'] | bash shell example |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| sh_shell | script | local.sh | ['tutorials'] | sh shell example |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| shell_options | script | local.sh | ['tutorials'] | shell options |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| environment_variables | script | local.bash | ['tutorials'] | Declare environment variables |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| variables | script | local.bash | ['tutorials'] | Declare shell variables |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| selinux_disable | script | local.bash | ['tutorials'] | Check if SELinux is Disabled |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| exit1_fail | script | local.sh | ['tutorials'] | exit 1 by default is FAIL |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| exit1_pass | script | local.sh | ['tutorials'] | report exit 1 as PASS |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| returncode_mismatch | script | local.sh | ['tutorials'] | exit 2 failed since it failed to match returncode 1 |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| wrongexecutor | script | badexecutor | ['tutorials'] | valid test but invalid executor name so test won't run |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| circle_area | script | local.python | ['tutorials'] | Calculate circle of area given a radius |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| skip | script | local.bash | ['tutorials'] | |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| unskipped | script | local.bash | ['tutorials'] | |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| vecadd_gnu | compiler | local.bash | ['tutorials'] | Vector Addition example with GNU compiler |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| hello_f | compiler | local.bash | ['tutorials'] | Hello World Fortran Compilation |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| hello_c | compiler | local.bash | ['tutorials'] | Hello World C Compilation |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| hello_cplusplus | compiler | local.bash | ['tutorials'] | Hello World C++ Compilation |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| cc_example | compiler | local.bash | ['tutorials'] | Example by using cc to set C compiler |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| fc_example | compiler | local.bash | ['tutorials'] | Example by using fc to set Fortran compiler |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| cxx_example | compiler | local.bash | ['tutorials'] | Example by using cxx to set C++ compiler |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| pre_post_build_run | compiler | local.bash | ['tutorials'] | example using pre_build, post_build, pre_run, post_run example |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
| executable_arguments | compiler | local.bash | ['tutorials'] | Passing arguments example |
+------------------------+----------+--------------+---------------+----------------------------------------------------------------+
buildtest will find all buildspecs and validate each file with the appropriate schema type. buildspecs that pass validation will be displayed on screen. buildtest will report all invalid buildspecs in a text file for you to review.
buildtest will cache the results in var/buildspec-cache.json so subsequent
runs to buildtest buildspec find will be much faster since we read from cache.
If you make changes to buildspec you may want to rebuild cache using buildtest buildspec find --clear.
Viewing Buildspecs¶
If you want to view or edit a buildspec you can type the name of test. Since we can have more than one test in a buildspec, opening any of the name entry that map to same file will result in same operation.
For example, we can view systemd_default_target as follows
$ buildtest buildspec view systemd_default_target
version: "1.0"
buildspecs:
systemd_default_target:
executor: local.bash
type: script
description: check if default target is multi-user.target
tags: [tutorials]
run: |
if [ "multi-user.target" == `systemctl get-default` ]; then
echo "multi-user is the default target";
exit 0
fi
echo "multi-user is not the default target";
exit 1
status:
returncode: 0
Editing Buildspecs¶
To edit a buildspec you can run buildtest buildspec edit <name> which
will open file in editor. Once you make change, buildtest will validate the
buildspec upon closure, if there is an issue buildtest will report an error
during validation and you will be prompted to fix issue until it is resolved.
For example we can see an output message after editing file, user will be prompted to press a key which will open the file in editor:
$ buildtest buildspec edit systemd_default_target
version 1.1 is not known for type {'1.0': 'script-v1.0.schema.json', 'latest': 'script-v1.0.schema.json'}. Try using latest.
Press any key to continue
Build Usage¶
The buildtest build command is used for building and running tests. Buildtest will read one or more Buildspecs (YAML)
file that adheres to one of the buildtest schemas. For a complete list of build options, run buildtest build --help
$ buildtest build --help
usage: buildtest [options] [COMMANDS] build [-h] [-b BUILDSPEC] [-t TESTDIR] [--settings SETTINGS] [-x EXCLUDE]
[--tags TAGS] [-s {parse,build}]
optional arguments:
-h, --help show this help message and exit
-b BUILDSPEC, --buildspec BUILDSPEC
Specify a Buildspec (YAML) file to build and run the test.
-t TESTDIR, --testdir TESTDIR
specify a custom test directory. By default, use .buildtest in $PWD.
--settings SETTINGS Specify an alternate buildtest settings file to use
-x EXCLUDE, --exclude EXCLUDE
Exclude one or more configs from processing. Configs can be files or directories.
--tags TAGS Specify buildspecs by tags
-s {parse,build}, --stage {parse,build}
control behavior of buildtest build
Discover Buildspecs¶
The buildspec search resolution is described as follows:
If file doesn’t exist, check for file in buildspec roots and break after first match
If buildspec path is a directory, traverse directory recursively to find all
.ymlextensionsIf buildspec path is a file, check if file extension is not
.yml, exit immediately
Shown below is a diagram on how buildtest discovers buildspecs. The user
inputs a buildspec via --buildspec or tags (--tags) Building By Tags
which will discover the buildspecs. User can Excluding Buildspecs
using --exclude option which is processed after discovering buildspecs. The
excluded buildspecs are removed from list if found and final list of buildspecs
is processed.
Building a Test¶
To build a test, we use the --buildspec or short option -b to specify the
path to Buildspec file.
Let’s see some examples, first we specify a full path to buildspec file
$ buildtest build -b /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
Paths:
__________
Test Directory: /Users/siddiq90/Documents/buildtest/var/tests
+-------------------------------+
| Stage: Discovered Buildspecs |
+-------------------------------+
/Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+-----------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
+----------------------+
| Stage: Building Test |
+----------------------+
Name | Schema File | Test Path | Buildspec
------------------------+-------------------------+-----------------------------------------------------------------------------------------------------+-----------------------------------------------------------
systemd_default_target | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
+----------------------+
| Stage: Running Test |
+----------------------+
name | executor | status | returncode | testpath
------------------------+------------+----------+--------------+-----------------------------------------------------------------------------------------------------
systemd_default_target | local.bash | FAIL | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/generate.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 1 tests
Passed Tests: 0/1 Percentage: 0.000%
Failed Tests: 1/1 Percentage: 100.000%
buildtest won’t accept .yaml file extension for file, this can be demonstrated as
follows:
$ buildtest build -b tests/examples/buildspecs/os.yaml
Paths:
__________
Prefix: /private/tmp
Buildspec Search Path: ['/Users/siddiq90/.buildtest/site']
Test Directory: /private/tmp/tests
tests/examples/buildspecs/os.yaml does not end in file extension .yml
buildtest can perform a directory build for instance let’s build
for directory tests/examples/buildspecs where buildtest will recursively
search for all .yml files
$ buildtest build -b tests/examples/buildspecs/
Paths:
__________
Test Directory: /Users/siddiq90/Documents/buildtest/var/tests
+-------------------------------+
| Stage: Discovered Buildspecs |
+-------------------------------+
/Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml
/Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
/Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/python-shell.yml
/Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/environment.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+----------------------------------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/python-shell.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/environment.yml
+----------------------+
| Stage: Building Test |
+----------------------+
Name | Schema File | Test Path | Buildspec
----------------------------+-------------------------+-------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------
slurm_down_nodes_reason | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/slurm/slurm_down_nodes_reason/generate.sh | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml
slurm_not_responding_nodes | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/slurm/slurm_not_responding_nodes/generate.sh | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml
_bin_sh_shell | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/generate.sh | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
_bin_bash_shell | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/_bin_bash_shell/generate.sh | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
bash_shell | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/bash_shell/generate.sh | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
sh_shell | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/generate.sh | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
shell_options | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/generate.sh | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
circle_area | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/generate.py | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/python-shell.yml
hello_dinosaur | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/environment/hello_dinosaur/generate.sh | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/environment.yml
+----------------------+
| Stage: Running Test |
+----------------------+
name | executor | status | returncode | testpath
----------------------------+--------------+----------+--------------+-------------------------------------------------------------------------------------------------------
slurm_down_nodes_reason | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/slurm/slurm_down_nodes_reason/generate.sh
slurm_not_responding_nodes | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/slurm/slurm_not_responding_nodes/generate.sh
_bin_sh_shell | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/generate.sh
_bin_bash_shell | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/_bin_bash_shell/generate.sh
bash_shell | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/bash_shell/generate.sh
sh_shell | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/generate.sh
shell_options | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/generate.sh
circle_area | local.python | FAIL | 2 | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/generate.py
hello_dinosaur | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/environment/hello_dinosaur/generate.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 9 tests
Passed Tests: 8/9 Percentage: 88.889%
Failed Tests: 1/9 Percentage: 11.111%
In next section, you will see, we can build multiple buildspecs and interchange
file and directory with -b option.
Building Multiple Buildspecs¶
Buildtest supports building multiple buildspecs, just specify the -b option
for every Buildspec you want to build. In this example, we specify a file and
directory path. The search resolution is performed for every argument (-b)
independently, and accumulated into list.
$ buildtest build -b tests/examples/buildspecs/ -b tutorials/systemd.yml
Paths:
__________
Test Directory: /Users/siddiq90/Documents/buildtest/var/tests
+-------------------------------+
| Stage: Discovered Buildspecs |
+-------------------------------+
/Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/python-shell.yml
/Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
/Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/environment.yml
/Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
/Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+----------------------------------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/python-shell.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/environment.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml
+----------------------+
| Stage: Building Test |
+----------------------+
Name | Schema File | Test Path | Buildspec
----------------------------+-------------------------+-------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------
circle_area | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/generate.py | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/python-shell.yml
systemd_default_target | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
hello_dinosaur | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/environment/hello_dinosaur/generate.sh | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/environment.yml
_bin_sh_shell | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/generate.sh | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
_bin_bash_shell | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/_bin_bash_shell/generate.sh | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
bash_shell | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/bash_shell/generate.sh | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
sh_shell | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/generate.sh | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
shell_options | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/generate.sh | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml
slurm_down_nodes_reason | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/slurm/slurm_down_nodes_reason/generate.sh | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml
slurm_not_responding_nodes | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/slurm/slurm_not_responding_nodes/generate.sh | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml
+----------------------+
| Stage: Running Test |
+----------------------+
name | executor | status | returncode | testpath
----------------------------+--------------+----------+--------------+-------------------------------------------------------------------------------------------------------
circle_area | local.python | FAIL | 2 | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/generate.py
systemd_default_target | local.bash | FAIL | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/generate.sh
hello_dinosaur | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/environment/hello_dinosaur/generate.sh
_bin_sh_shell | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/generate.sh
_bin_bash_shell | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/_bin_bash_shell/generate.sh
bash_shell | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/bash_shell/generate.sh
sh_shell | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/generate.sh
shell_options | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/generate.sh
slurm_down_nodes_reason | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/slurm/slurm_down_nodes_reason/generate.sh
slurm_not_responding_nodes | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/slurm/slurm_not_responding_nodes/generate.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 10 tests
Passed Tests: 8/10 Percentage: 80.000%
Failed Tests: 2/10 Percentage: 20.000%
Excluding Buildspecs¶
Buildtest provides --exclude option or short option -x to exclude
buildspecs which can be useful when you want to build all buildspecs in a directory
but exclude a few buildspecs or exclude a sub-directory.
For example we can build all buildspecs in examples but exclude file examples/systemd.yml
by running:
$ buildtest build -b examples -x examples/systemd.yml
buildtest will discover all Buildspecs and then exclude any buildspecs specified
by -x option. You can specify -x multiple times just like -b option.
For example, we can undo discovery by passing same option to -b and -x as follows:
$ buildtest build -b examples/ -x examples/
There are no Buildspec files to process.
Buildtest will stop immediately if there are no Buildspecs to process, this is true if you were to specify files instead of directory.
Building By Tags¶
buildtest can perform builds by tags by using --tags option. In order to use this
feature, buildspecs must be in cache so you must run buildtest buildspec find
or see Finding Buildspecs.
To build all tutorials tests you can perform buildtest build --tags tutorials.
In the buildspec there is a field tags: [tutorials] to classify tests.
buildtest will read the cache file var/buildspec-cache.json and see which
buildspecs have a matching tag. You should run buildtest buildspec find
atleast once, in order to detect cache file.
$ buildtest build --tags tutorials
Paths:
__________
Test Directory: /Users/siddiq90/Documents/buildtest/var/tests
+-------------------------------+
| Stage: Discovered Buildspecs |
+-------------------------------+
/Users/siddiq90/Documents/buildtest/tutorials/vars.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml
/Users/siddiq90/Documents/buildtest/tutorials/environment.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
/Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
/Users/siddiq90/Documents/buildtest/tutorials/selinux.yml
/Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
/Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml
/Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml
/Users/siddiq90/Documents/buildtest/tutorials/skip_tests.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml
/Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
[skip] test is skipped.
schemafile | validstate | buildspec
---------------------------+--------------+--------------------------------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/vars.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/environment.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/selinux.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/skip_tests.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
+----------------------+
| Stage: Building Test |
+----------------------+
Name | Schema File | Test Path | Buildspec
------------------------+---------------------------+------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------
variables | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vars/variables/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/vars.yml
pre_post_build_run | compiler-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/pre_post_build_run/pre_post_build_run/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml
environment_variables | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/environment/environment_variables/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/environment.yml
hello_f | compiler-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_f/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
hello_c | compiler-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_c/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
hello_cplusplus | compiler-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_cplusplus/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
cc_example | compiler-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cc_example/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
fc_example | compiler-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/fc_example/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
cxx_example | compiler-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cxx_example/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
systemd_default_target | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
selinux_disable | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/selinux/selinux_disable/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/selinux.yml
_bin_sh_shell | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
_bin_bash_shell | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/_bin_bash_shell/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
bash_shell | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/bash_shell/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
sh_shell | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
shell_options | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
circle_area | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/generate.py | /Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml
wrongexecutor | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/badexecutor/invalid_executor/wrongexecutor/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml
unskipped | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/skip_tests/unskipped/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/skip_tests.yml
executable_arguments | compiler-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/passing_args/executable_arguments/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml
exit1_fail | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_fail/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
exit1_pass | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_pass/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
returncode_mismatch | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_mismatch/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
vecadd_gnu | compiler-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vecadd/vecadd_gnu/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
+----------------------+
| Stage: Running Test |
+----------------------+
[wrongexecutor]: Failed to Run Test
name | executor | status | returncode | testpath
------------------------+--------------+----------+--------------+------------------------------------------------------------------------------------------------------------
variables | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vars/variables/generate.sh
pre_post_build_run | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/pre_post_build_run/pre_post_build_run/generate.sh
environment_variables | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/environment/environment_variables/generate.sh
hello_f | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_f/generate.sh
hello_c | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_c/generate.sh
hello_cplusplus | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_cplusplus/generate.sh
cc_example | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cc_example/generate.sh
fc_example | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/fc_example/generate.sh
cxx_example | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cxx_example/generate.sh
systemd_default_target | local.bash | FAIL | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/generate.sh
selinux_disable | local.bash | FAIL | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/selinux/selinux_disable/generate.sh
_bin_sh_shell | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/generate.sh
_bin_bash_shell | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/_bin_bash_shell/generate.sh
bash_shell | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/bash_shell/generate.sh
sh_shell | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/generate.sh
shell_options | local.sh | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/generate.sh
circle_area | local.python | FAIL | 2 | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/generate.py
unskipped | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/skip_tests/unskipped/generate.sh
executable_arguments | local.bash | PASS | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/passing_args/executable_arguments/generate.sh
exit1_fail | local.sh | FAIL | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_fail/generate.sh
exit1_pass | local.sh | PASS | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_pass/generate.sh
returncode_mismatch | local.sh | FAIL | 2 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_mismatch/generate.sh
vecadd_gnu | local.bash | FAIL | 0 | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vecadd/vecadd_gnu/generate.sh
Error Messages from Stage: Run
________________________________________________________________________________
[wrongexecutor]: executor badexecutor is not defined in /Users/siddiq90/.buildtest/config.yml
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 23 tests
Passed Tests: 17/23 Percentage: 73.913%
Failed Tests: 6/23 Percentage: 26.087%
Control builds by Stages¶
You can control behavior of buildtest build command to stop at certain point
using --stage option. This takes two values parse or build, which will
stop buildtest after parsing buildspecs or building the test content.
If you want to know your buildspecs are valid you can use --stage=parse to stop
after parsing the buildspec. Shown below is an example build where we stop
after parse stage.
$ buildtest build -b tutorials/systemd.yml --stage=parse
Paths:
__________
Test Directory: /Users/siddiq90/Documents/buildtest/var/tests
+-------------------------------+
| Stage: Discovered Buildspecs |
+-------------------------------+
/Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+-----------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
Likewise, if you want to troubleshoot your test script without running them you can
use --stage=build which will stop after building your test script. This can
be extremely useful when writing your buildspecs and not having to run your tests.
In this next example, we stop our after the build stage using --stage=build.
$ buildtest build -b tutorials/systemd.yml --stage=build
Paths:
__________
Test Directory: /Users/siddiq90/Documents/buildtest/var/tests
+-------------------------------+
| Stage: Discovered Buildspecs |
+-------------------------------+
/Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+-----------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
+----------------------+
| Stage: Building Test |
+----------------------+
Name | Schema File | Test Path | Buildspec
------------------------+-------------------------+-----------------------------------------------------------------------------------------------------+-----------------------------------------------------------
systemd_default_target | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
Invalid Buildspecs¶
buildtest will skip any buildspecs that fail to validate, in that case the test script will not be generated. Here is an example where we have an invalid buildspec.
$ buildtest build -b tutorials/invalid_buildspec_section.yml
Paths:
__________
Test Directory: /Users/siddiq90/Documents/buildtest/var/tests
+-------------------------------+
| Stage: Discovered Buildspecs |
+-------------------------------+
/Users/siddiq90/Documents/buildtest/tutorials/invalid_buildspec_section.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
--------------+--------------+-------------
Error Messages from Stage: Parse
________________________________________________________________________________
Skipping /Users/siddiq90/Documents/buildtest/tutorials/invalid_buildspec_section.yml since it failed to validate
+----------------------+
| Stage: Building Test |
+----------------------+
Name | Schema File | Test Path | Buildspec
--------+---------------+-------------+-------------
+----------------------+
| Stage: Running Test |
+----------------------+
name | executor | status | returncode | testpath
--------+------------+----------+--------------+------------
No tests were executed
buildtest may skip tests from running if buildspec specifies an invalid executor name since buildtest needs to know this in order to delegate test to Executor class responsible for running the test. Here is an example where test failed to run since we provided invalid executor.
$ buildtest build -b tutorials/invalid_executor.yml
Paths:
__________
Test Directory: /Users/siddiq90/Documents/buildtest/var/tests
+-------------------------------+
| Stage: Discovered Buildspecs |
+-------------------------------+
/Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+--------------------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml
+----------------------+
| Stage: Building Test |
+----------------------+
Name | Schema File | Test Path | Buildspec
---------------+-------------------------+------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------
wrongexecutor | script-v1.0.schema.json | /Users/siddiq90/Documents/buildtest/var/tests/badexecutor/invalid_executor/wrongexecutor/generate.sh | /Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml
+----------------------+
| Stage: Running Test |
+----------------------+
[wrongexecutor]: Failed to Run Test
name | executor | status | returncode | testpath
--------+------------+----------+--------------+------------
Error Messages from Stage: Run
________________________________________________________________________________
[wrongexecutor]: executor badexecutor is not defined in /Users/siddiq90/.buildtest/config.yml
No tests were executed
Buildtest Report¶
The buildtest report command will show result of all tests in a tabular
form. Shown below is an example
$ buildtest report
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| id | state | returncode | starttime | endtime | runtime | tags | buildspec |
+=============================================+=========+==============+=====================+=====================+============+===========+==================================================================================+
| systemd_default_target_2020-09-02-21-41 | FAIL | 1 | 2020/09/02 21:41:56 | 2020/09/02 21:41:56 | 0.0213593 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| systemd_default_target_2020-09-02-21-42 | FAIL | 1 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.0125463 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| systemd_default_target_2020-09-02-21-42 | FAIL | 1 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00771781 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| slurm_down_nodes_reason_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.0110754 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| slurm_down_nodes_reason_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.00830546 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| slurm_not_responding_nodes_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.00802073 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| slurm_not_responding_nodes_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.00736713 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/slurm.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| _bin_sh_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.0125264 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| _bin_sh_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.014642 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| _bin_bash_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.00878803 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| _bin_bash_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.00996381 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| bash_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.00561796 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| bash_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.00678369 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| sh_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.0102359 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| sh_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.0110021 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| shell_options_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.0104085 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| shell_options_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.0115846 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| circle_area_2020-09-02-21-42 | FAIL | 2 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.0096071 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/python-shell.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| circle_area_2020-09-02-21-42 | FAIL | 2 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.0117158 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/python-shell.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| hello_dinosaur_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:01 | 2020/09/02 21:42:01 | 0.00698703 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/environment.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| hello_dinosaur_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:06 | 2020/09/02 21:42:06 | 0.0071777 | | /Users/siddiq90/Documents/buildtest/tests/examples/buildspecs/environment.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| variables_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:20 | 2020/09/02 21:42:20 | 0.00946478 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/vars.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| pre_post_build_run_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:20 | 2020/09/02 21:42:20 | 0.661319 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| environment_variables_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:20 | 2020/09/02 21:42:20 | 0.0120327 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/environment.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| hello_f_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:20 | 2020/09/02 21:42:20 | 0.00786752 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| hello_f_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:43 | 2020/09/02 21:42:43 | 0.013958 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| hello_c_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:20 | 2020/09/02 21:42:21 | 0.229417 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| hello_c_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:43 | 2020/09/02 21:42:43 | 0.235872 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| hello_cplusplus_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:21 | 2020/09/02 21:42:21 | 0.461781 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| hello_cplusplus_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:43 | 2020/09/02 21:42:44 | 0.498086 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| cc_example_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:21 | 2020/09/02 21:42:21 | 0.230038 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| cc_example_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:44 | 2020/09/02 21:42:44 | 0.239393 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| fc_example_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:21 | 2020/09/02 21:42:21 | 0.00840816 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| fc_example_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:44 | 2020/09/02 21:42:44 | 0.0103057 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| cxx_example_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:21 | 2020/09/02 21:42:22 | 0.45725 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| cxx_example_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:44 | 2020/09/02 21:42:44 | 0.505495 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| selinux_disable_2020-09-02-21-42 | FAIL | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00954966 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/selinux.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| _bin_sh_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.0126632 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| _bin_bash_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.0085432 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| bash_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00603703 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| sh_shell_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00911067 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| shell_options_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.0104522 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| circle_area_2020-09-02-21-42 | FAIL | 2 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00808543 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| unskipped_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00792155 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/skip_tests.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| unskipped_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:49 | 2020/09/02 21:42:49 | 0.042425 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/skip_tests.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| executable_arguments_2020-09-02-21-42 | PASS | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.232554 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| exit1_fail_2020-09-02-21-42 | FAIL | 1 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00782586 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| exit1_fail_2020-09-02-21-42 | FAIL | 1 | 2020/09/02 21:42:48 | 2020/09/02 21:42:48 | 0.0132099 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| exit1_pass_2020-09-02-21-42 | PASS | 1 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00772625 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| exit1_pass_2020-09-02-21-42 | PASS | 1 | 2020/09/02 21:42:48 | 2020/09/02 21:42:48 | 0.0110667 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| returncode_mismatch_2020-09-02-21-42 | FAIL | 2 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.00836054 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| returncode_mismatch_2020-09-02-21-42 | FAIL | 2 | 2020/09/02 21:42:48 | 2020/09/02 21:42:48 | 0.00940911 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| vecadd_gnu_2020-09-02-21-42 | FAIL | 0 | 2020/09/02 21:42:22 | 2020/09/02 21:42:22 | 0.0267169 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
| vecadd_gnu_2020-09-02-21-42 | FAIL | 0 | 2020/09/02 21:42:46 | 2020/09/02 21:42:46 | 0.0421426 | tutorials | /Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml |
+---------------------------------------------+---------+--------------+---------------------+---------------------+------------+-----------+----------------------------------------------------------------------------------+
buildtest will store result metadata of each test in a file var/report.json which
is found in root of buildtest. This file is updated upon every buildtest build command.
For more information see Test Reports (buildtest report).
Debug Mode¶
buildtest can stream logs to stdout stream for debugging. You can use buildtest -d <DEBUGLEVEL>
or long option --debug with any buildtest commands. The DEBUGLEVEL are:
DEBUG, INFO, WARNING, ERROR, CRITICAL which controls
log level to be displayed in console. buildtest is using
logging.setLevel
to control log level.
The same content is logged in buildtest.log with default log level of DEBUG.
If you want to get all logs use -d DEBUG with your buildtest command:
buildtest -d DEBUG <command>
Logfile¶
Currently, buildtest will write the log file for any buildtest build command
in buildtest.log of the current directory. The logfile will be overwritten
if you run repeative commands from same directory. A permanent log file location
will be implemented (TBD).