Scripting in buildtest¶
This guide will walk you through on how to script with buildtest.
Discovering Buildspecs¶
Let’s take this first example where we discover all buildspecs found in top-level
tutorials directory.
import os
from buildtest.defaults import BUILDTEST_ROOT
from buildtest.menu.build import discover_buildspecs
included_bp, excluded_bp = discover_buildspecs(
buildspec=[os.path.join(BUILDTEST_ROOT, "tutorials")]
)
print("\n Discovered buildspecs: \n")
[print(f) for f in included_bp]
print("\n Excluded buildspecs: \n")
[print(f) for f in excluded_bp]
The variable BUILDTEST_ROOT is the root of buildtest and typically setup once
you install buildtest. The discover_buildspecs method can be invoked to retrieve
a list of buildspecs discovered. The method will return two list, one for discovered
and excluded buildspecs.
Now let’s run this example and note we see all buildspecs in tutorials directory
were retrieved. This is equivalent to running buildtest build --buildspec tutorials.
Discovered buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/python-hello.yml
/Users/siddiq90/Documents/buildtest/tutorials/run_only_platform.yml
/Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml
/Users/siddiq90/Documents/buildtest/tutorials/skip_tests.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml
/Users/siddiq90/Documents/buildtest/tutorials/invalid_executor.yml
/Users/siddiq90/Documents/buildtest/tutorials/shebang.yml
/Users/siddiq90/Documents/buildtest/tutorials/environment.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
/Users/siddiq90/Documents/buildtest/tutorials/root_user.yml
/Users/siddiq90/Documents/buildtest/tutorials/run_only_distro.yml
/Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml
/Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
/Users/siddiq90/Documents/buildtest/tutorials/selinux.yml
/Users/siddiq90/Documents/buildtest/tutorials/invalid_tags.yml
/Users/siddiq90/Documents/buildtest/tutorials/hello_world.yml
/Users/siddiq90/Documents/buildtest/tutorials/sleep.yml
/Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
/Users/siddiq90/Documents/buildtest/tutorials/tags_example.yml
/Users/siddiq90/Documents/buildtest/tutorials/invalid_buildspec_section.yml
/Users/siddiq90/Documents/buildtest/tutorials/vars.yml
Excluded buildspecs:
We can also discover buildspecs by tags, in next example we discover all buildspecs by tutorials tag. This can be done by passing a tagname for argument tags in discover_buildspecs method.
This is equivalent to running buildtest build --tags tutorials.
from buildtest.menu.build import discover_buildspecs
tagname = ["tutorials"]
print(f"Searching by tagname: {tagname}")
included_bp, excluded_bp = discover_buildspecs(tags=tagname)
print("\n Discovered buildspecs: \n")
[print(f) for f in included_bp]
Note
You must have a buildspec cache in order to discover tags (buildtest buildspec find)
Now let’s run this test
Searching by tagname: ['tutorials']
Discovered buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/hello_world.yml
/Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
/Users/siddiq90/Documents/buildtest/tutorials/run_only_platform.yml
/Users/siddiq90/Documents/buildtest/tutorials/vars.yml
/Users/siddiq90/Documents/buildtest/tutorials/root_user.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml
/Users/siddiq90/Documents/buildtest/tutorials/skip_tests.yml
/Users/siddiq90/Documents/buildtest/tutorials/shebang.yml
/Users/siddiq90/Documents/buildtest/tutorials/selinux.yml
/Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
/Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml
/Users/siddiq90/Documents/buildtest/tutorials/environment.yml
/Users/siddiq90/Documents/buildtest/tutorials/sleep.yml
/Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
You can combine tags and buildspecs with discover_buildspecs method and buildtest will combine the results.
Build Phase¶
Now that we can find buildspecs, let’s try to parse and build the tests. In next example we will discover, parse, and build all tests with tag name tutorials.
from buildtest.config import load_settings
from buildtest.menu.build import discover_buildspecs, resolve_testdirectory, build_phase
from buildtest.menu.buildspec import parse_buildspecs
tagname = ["tutorials"]
print(f"Searching by tagname: {tagname}")
included_bp, excluded_bp = discover_buildspecs(tags=tagname, debug=True)
configuration = load_settings()
testdir = resolve_testdirectory(configuration)
builders = parse_buildspecs(included_bp, testdir, rebuild=1, printTable=True)
build_phase(builders, printTable=True)
We retrieve all buildspecs by tag tutorials as mentioned in previous example. Next we load buildtest configuration using load_settings which returns a dictionary containing buildtest configuration. During this process, we validate the buildtest configuration.
Next, we need to figure out our test directory in order to write tests. This
can be achieved by passing the loaded configuration to method resolve_testdirectory.
The return will be path to test directory. The test directory can be specified
on command line buildtest build --testdir or path in configuration. If its not
set in configuration we default to $BUILDTEST_ROOT/var/tests
Next we invoke parse_buildspecs which expects a list of buildspecs, test directory
and executor. The printTable=True will print parse table of buildspecs that are validated. The
parse_buildspecs will validate each buildspec, and skip any buildspecs that fail validation.
The parser is implemented in class BuildspecParser. For all valid buildspecs
we return a list of builders that is a list of tests for each buildspec that
is an instance of BuilderBase class that is responsible for building the test.
Next we pass all builders to build_phase method which will generate testscript
for each builder. The printTable=True will print table for builder phase.
Note
Each builder corresponds to a single test name.
Now let’s run this script and notice the output resembles similar to running
buildtest build --tags tutorials but we stop right after build. In other words
this is equivalent to buildtest build --tags tutorials --stage=build.
Searching by tagname: ['tutorials']
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
/Users/siddiq90/Documents/buildtest/tutorials/root_user.yml
/Users/siddiq90/Documents/buildtest/tutorials/run_only_platform.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml
/Users/siddiq90/Documents/buildtest/tutorials/environment.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
/Users/siddiq90/Documents/buildtest/tutorials/shebang.yml
/Users/siddiq90/Documents/buildtest/tutorials/skip_tests.yml
/Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml
/Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
/Users/siddiq90/Documents/buildtest/tutorials/selinux.yml
/Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml
/Users/siddiq90/Documents/buildtest/tutorials/vars.yml
/Users/siddiq90/Documents/buildtest/tutorials/sleep.yml
/Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
/Users/siddiq90/Documents/buildtest/tutorials/hello_world.yml
[run_only_as_root] test is skipped because ['run_only']['user'] got value: root but detected user: siddiq90.
[run_only_platform_linux] test is skipped because ['run_only']['platform'] got value: Linux but detected platform: Darwin.
[skip] test is skipped.
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
---------------------------+--------------+--------------------------------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/root_user.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/run_only_platform.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/pre_post_build_run.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/environment.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/vecadd.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/shebang.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/skip_tests.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/shell_examples.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/passing_args.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/systemd.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/selinux.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/python-shell.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/vars.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/sleep.yml
compiler-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/compilers/gnu_hello.yml
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/hello_world.yml
+----------------------+
| Stage: Building Test |
+----------------------+
name | id | type | executor | tags | testpath
--------------------------+----------+----------+--------------+--------------------------+---------------------------------------------------------------------------------------------------------------------------
exit1_fail | caeb1cd5 | script | local.sh | ['tutorials', 'fail'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_fail/8/stage/generate.sh
exit1_pass | 0b2fac5b | script | local.sh | ['tutorials', 'pass'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_pass/6/stage/generate.sh
returncode_list_mismatch | 653f6fae | script | local.sh | ['tutorials', 'fail'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_list_mismatch/8/stage/generate.sh
returncode_int_match | 13d7cc98 | script | local.sh | ['tutorials', 'pass'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_int_match/6/stage/generate.sh
run_only_platform_darwin | b27688fd | script | local.python | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.python/run_only_platform/run_only_platform_darwin/3/stage/generate.sh
pre_post_build_run | 143a9bb4 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/pre_post_build_run/pre_post_build_run/4/stage/generate.sh
environment_variables | 27625a4e | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/environment/environment_variables/3/stage/generate.sh
vecadd_gnu | b048e564 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vecadd/vecadd_gnu/5/stage/generate.sh
bash_login_shebang | 7d4303d1 | script | local.bash | tutorials | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shebang/bash_login_shebang/4/stage/generate.sh
bash_nonlogin_shebang | d87c79c1 | script | local.bash | tutorials | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shebang/bash_nonlogin_shebang/4/stage/generate.sh
unskipped | 013f04a2 | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/skip_tests/unskipped/4/stage/generate.sh
_bin_sh_shell | 764fc41a | script | local.sh | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/_bin_sh_shell/6/stage/generate.sh
_bin_bash_shell | be2673cd | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/_bin_bash_shell/5/stage/generate.sh
bash_shell | cb2805d5 | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/shell_examples/bash_shell/5/stage/generate.sh
sh_shell | 5966a1c2 | script | local.sh | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/sh_shell/6/stage/generate.sh
shell_options | 05ffa6cb | script | local.sh | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/shell_examples/shell_options/6/stage/generate.sh
executable_arguments | c6619fcc | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/passing_args/executable_arguments/4/stage/generate.sh
systemd_default_target | 5d717ba8 | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/systemd/systemd_default_target/6/stage/generate.sh
selinux_disable | 1df1ac5f | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/selinux/selinux_disable/3/stage/generate.sh
circle_area | f0d12e1e | script | local.python | ['tutorials', 'python'] | /Users/siddiq90/Documents/buildtest/var/tests/local.python/python-shell/circle_area/9/stage/generate.sh
variables | 3bd1a67a | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/vars/variables/3/stage/generate.sh
sleep | c39f3421 | script | local.bash | ['tutorials'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/sleep/sleep/3/stage/generate.sh
hello_f | 16de9e5d | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_f/5/stage/generate.sh
hello_c | c53af412 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_c/5/stage/generate.sh
hello_cplusplus | 5a599d47 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/hello_cplusplus/5/stage/generate.sh
cc_example | 0818a978 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cc_example/5/stage/generate.sh
fc_example | 9b502366 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/fc_example/5/stage/generate.sh
cxx_example | 79590f19 | compiler | local.bash | ['tutorials', 'compile'] | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/gnu_hello/cxx_example/5/stage/generate.sh
hello_world | 342ea38e | script | local.bash | tutorials | /Users/siddiq90/Documents/buildtest/var/tests/local.bash/hello_world/hello_world/3/stage/generate.sh
Now you may have guessed it, --stage=parse will stop right after the parse stage, in this
case we won’t invoke build_phase method.
Run Phase¶
In the Build Phase example, we discovered and validated buildspecs and built all the tests, but
tests were not run. In this example, we will build off this example to run the test.
In this example, we demonstrate a script that is emulating the command buildtest build --buildspec tutorials/pass_returncode.yml
import os
from buildtest.config import load_settings
from buildtest.defaults import BUILDTEST_ROOT
from buildtest.executors.setup import BuildExecutor
from buildtest.menu.build import (
discover_buildspecs,
resolve_testdirectory,
build_phase,
run_phase,
)
from buildtest.menu.buildspec import parse_buildspecs
input_buildspecs = [os.path.join(BUILDTEST_ROOT, "tutorials", "pass_returncode.yml")]
included_bp, excluded_bp = discover_buildspecs(buildspec=input_buildspecs, debug=True)
configuration = load_settings()
testdir = resolve_testdirectory(configuration)
executor = BuildExecutor(configuration)
print("List of executors: ", executor.executors)
builders = parse_buildspecs(included_bp, testdir, rebuild=1, printTable=True)
build_phase(builders, printTable=True)
run_phase(builders, executor, configuration, printTable=True)
In-order to run the tests, we need to initialize the executors defined in buildtest settings see What is an executor?. This action is performed in line:
executor = BuildExecutor(configuration)
The BuildExecutor takes an input buildtest settings, and builds a list of executors
objects that is responsible for running tests. Next, we parse and build buildspecs
by invoking parse_buildspecs and build_phase as discussed previously. Finally,
we invoke run_phase which runs the test.
+-------------------------------+
| Stage: Discovering Buildspecs |
+-------------------------------+
Discovered Buildspecs:
/Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
List of executors: {'local.bash': local.bash, 'local.sh': local.sh, 'local.python': local.python}
+---------------------------+
| Stage: Parsing Buildspecs |
+---------------------------+
schemafile | validstate | buildspec
-------------------------+--------------+-------------------------------------------------------------------
script-v1.0.schema.json | True | /Users/siddiq90/Documents/buildtest/tutorials/pass_returncode.yml
+----------------------+
| Stage: Building Test |
+----------------------+
name | id | type | executor | tags | testpath
--------------------------+----------+--------+------------+-----------------------+---------------------------------------------------------------------------------------------------------------------
exit1_fail | 98a2e55c | script | local.sh | ['tutorials', 'fail'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_fail/9/stage/generate.sh
exit1_pass | 73b4fd50 | script | local.sh | ['tutorials', 'pass'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_pass/7/stage/generate.sh
returncode_list_mismatch | 87285388 | script | local.sh | ['tutorials', 'fail'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_list_mismatch/9/stage/generate.sh
returncode_int_match | 88197672 | script | local.sh | ['tutorials', 'pass'] | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_int_match/7/stage/generate.sh
+----------------------+
| Stage: Running Test |
+----------------------+
name | id | executor | status | returncode | testpath
--------------------------+----------+------------+----------+--------------+---------------------------------------------------------------------------------------------------------------------
exit1_fail | 98a2e55c | local.sh | FAIL | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_fail/9/stage/generate.sh
exit1_pass | 73b4fd50 | local.sh | PASS | 1 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/exit1_pass/7/stage/generate.sh
returncode_list_mismatch | 87285388 | local.sh | FAIL | 2 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_list_mismatch/9/stage/generate.sh
returncode_int_match | 88197672 | local.sh | PASS | 128 | /Users/siddiq90/Documents/buildtest/var/tests/local.sh/pass_returncode/returncode_int_match/7/stage/generate.sh
+----------------------+
| Stage: Test Summary |
+----------------------+
Executed 4 tests
Passed Tests: 2/4 Percentage: 50.000%
Failed Tests: 2/4 Percentage: 50.000%