Buildspec Overview¶
What is a buildspec?¶
A buildspec is a YAML file that defines your test in buildtest which is validated by schema followed by building a shell script and running the generated test. Buildtest will parse the buildspec with the global schema file which defines the top-level structure of buildspec file.
Example¶
Let’s start off with a simple example that declares two variables X and Y and prints the sum of X+Y.
buildspecs:
add_numbers:
type: script
executor: generic.local.bash
description: Add X+Y
tags: [tutorials]
vars:
X: 1
Y: 2
run: echo "$X+$Y=" $(($X+$Y))
buildtest will validate the entire file with global.schema.json, the schema
requires version and buildspec in order to validate file. The buildspec
is where you define each test. The name of the test is add_numbers.
The test requires a type field which is the sub-schema used to validate the
test section. In this example type: script informs buildtest to use the Script Schema
when validating test section.
Each subschema has a list of field attributes that are supported, for example the fields: type, executor, vars and run are all valid fields supported by the script schema.
Let’s look at a more interesting example, shown below is a multi line run example using the script schema with test name called systemd_default_target, shown below is the content of test:
buildspecs:
systemd_default_target:
executor: generic.local.bash
type: script
tags: [system]
description: check if default target is multi-user.target
run: |
if [ "multi-user.target" == `systemctl get-default` ]; then
echo "multi-user is the default target";
exit 0
fi
echo "multi-user is not the default target";
exit 1
The test name systemd_default_target defined in buildspec section is
validated with the following pattern "^[A-Za-z_][A-Za-z0-9_]*$". This test
will use the executor generic.local.bash which means it will use the Local Executor
with an executor name bash defined in the buildtest settings. The default
buildtest settings will provide a bash executor as follows:
system:
generic:
hostnames: ["localhost"]
executors:
local:
bash:
description: submit jobs on local machine using bash shell
shell: bash
The shell: bash indicates this executor will use bash to run the test scripts.
To reference this executor use the format <system>.<type>.<name> in this case generic.local.bash
refers to bash executor.
The description field is an optional key that can be used to provide a brief
summary of the test. The description field is limited to 80 characters.
In this example we can specify multiple commands in run section, this
can be done in YAML using run: | followed by content of run section tab indented
2 spaces.
In this next example, we introduce the summary field, which can be used as an extended description of test. It has no
impact on the test. Unlike the description field, the summary field has no limit on character count and one can define multi-line
string using the pipe symbol |.
buildspecs:
summary_example:
type: script
executor: generic.local.bash
description: The summary field can be a multi-line string and exceed 80 char
tags: [tutorials]
summary: |
This is a long description of test that
can exceed 80 characters and be multiline
run: hostname
Script Schema¶
The script schema is used for writing simple scripts (bash, sh, python) in Buildspec.
To use this schema you must set type: script. The run field is responsible
for writing the content of test.
Shown below is schema header for script.schema.json.
{
"$id": "script.schema.json",
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "script schema version",
"description": "The script schema is of ``type: script`` in sub-schema which is used for running shell scripts",
"type": "object",
"required": [
"type",
The "type": "object" means sub-schema is a JSON object
where we define a list of key/value pair. The "required" field specifies a list of
fields that must be specified in order to validate the Buildspec. In this example, type, run, and executor
are required fields. The additionalProperties: false informs schema to reject
any extra properties not defined in the schema.
The executor key is required for all sub-schemas which instructs buildtest
which executor to use when running the test. The executors are defined in
How to configure buildtest. In our first example we define variables using the
vars property which is a Key/Value pair for variable assignment.
The run section is required for script schema which defines the content of
the test script.
For more details on script schema see schema docs at https://buildtesters.github.io/buildtest/
Declaring Environment Variables¶
You can define environment variables using the env property, this is compatible
with shells: bash, sh, zsh, csh and tcsh. It does not work with
shell: python. In example below we declare three tests using environment
variable with default shell (bash), csh, and tcsh
buildspecs:
bash_env_variables:
executor: generic.local.bash
description: Declare environment variables in default shell (bash)
type: script
env:
FIRST_NAME: avocado
LAST_NAME: dinosaur
tags: [tutorials]
run: |
hostname
whoami
echo $USER
printf "${FIRST_NAME} ${LAST_NAME}\n"
csh_env_declaration:
executor: generic.local.csh
type: script
description: "csh shell example to declare environment variables"
shell: /bin/csh
tags: [tutorials]
env:
SHELL_NAME: "csh"
run: echo "This is running $SHELL_NAME"
tcsh_env_declaration:
executor: generic.local.csh
type: script
description: "tcsh shell example to declare environment variables"
shell: /bin/tcsh
tags: [tutorials]
env:
path: "/usr/local/bin:$PATH"
run: echo $path
This test can be run by issuing the following command: buildtest build -b tutorials/environment.yml.
If we inspect one of the test script we will see that buildtest generates a build script that invokes the test using the
shell wrapper /bin/csh for the csh test and gets the returncode.
#!/bin/bash
############# START VARIABLE DECLARATION ########################
export BUILDTEST_TEST_NAME=csh_env_declaration
export BUILDTEST_TEST_ROOT=/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.csh/environment/csh_env_declaration/0
export BUILDTEST_BUILDSPEC_DIR=/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials
export BUILDTEST_STAGE_DIR=/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.csh/environment/csh_env_declaration/0/stage
export BUILDTEST_TEST_ID=501ec5d3-e614-4ae8-9c1e-4849ce340c76
############# END VARIABLE DECLARATION ########################
# source executor startup script
source /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/executor/generic.local.csh/before_script.sh
# Run generated script
/bin/csh /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.csh/environment/csh_env_declaration/0/stage/csh_env_declaration.csh
# Get return code
returncode=$?
# Exit with return code
exit $returncode
This generated test looks something like this
#!/bin/csh
# Declare environment variables
setenv SHELL_NAME csh
# Content of run section
echo "This is running $SHELL_NAME"
Environment variables are defined using export in bash, sh, zsh while csh and
tcsh use setenv.
Declaring Variables¶
Variables can be defined using vars property, this is compatible with all shells
except for python. The variables are defined slightly different in csh, tcsh as pose
to bash, sh, and zsh. In example below we define tests with bash and csh.
In YAML strings can be specified with or without quotes however in bash, variables
need to be enclosed in quotes " if you are defining a multi word string (name="First Last").
If you need define a literal string it is recommended
to use the literal block | that is a special character in YAML.
If you want to specify " or ' in string you can use the escape character
\ followed by any of the special character. In example below we define
several variables such as X, Y that contain numbers, variable literalstring
is a literal string processed by YAML. The variable singlequote and doublequote
defines a variable with the special character ' and ". The variables
current_user and num_files store result of a shell command. This can
be done using var=$(<command>) or var=`<command>` where <command> is
a Linux command.
buildspecs:
variables_bash:
type: script
executor: generic.local.bash
description: Declare shell variables in bash
tags: [tutorials]
vars:
X: 1
Y: 2
literalstring: this is a literal string
singlequote: \'singlequote\'
doublequote: \"doublequote\"
current_user: "$(whoami)"
num_files: "`find $HOME -type f -maxdepth 1 | wc -l`"
multiline_string: |
Hello my name is Bob \n
I am 30 years old
run: |
echo "$X+$Y="$(($X+$Y))
echo $literalstring
echo $singlequote
echo $doublequote
echo "current user:" $current_user
echo "number of files:" $num_files
echo -e $multiline_string
Next we build this test by running buildtest build -b $BUILDTEST_ROOT/tutorials/vars.yml.
buildtest build -b $BUILDTEST_ROOT/tutorials/vars.yml
$ buildtest build -b $BUILDTEST_ROOT/tutorials/vars.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19398495-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/02/06 21:49:46 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/tutorials/vars.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ variabl… │ script │ generic… │ None │ None │ None │ Declare │ /home/… │
│ │ │ │ │ │ │ shell │ │
│ │ │ │ │ │ │ variabl… │ │
│ │ │ │ │ │ │ in bash │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
variables_bash/84dd6959: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/vars/variables_bash/84dd6959
variables_bash/84dd6959: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/vars/variables_bash/84dd6959/stage
variables_bash/84dd6959: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/vars/variables_bash/84dd6959/variables_bash_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
variables_bash/84dd6959 does not have any dependencies adding test to queue
variables_bash/84dd6959: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/vars/variables_bash/84dd6959/stage
variables_bash/84dd6959: Running Test via command: bash --norc --noprofile -eo pipefail variables_bash_build.sh
variables_bash/84dd6959: Test completed in 0.077881 seconds
variables_bash/84dd6959: Test completed with returncode: 0
variables_bash/84dd6959: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/vars/variables_bash/84dd6959/variables_bash.out
variables_bash/84dd6959: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/vars/variables_bash/84dd6959/variables_bash.err
In this iteration we are going to run the following tests: [variables_bash/84dd6959]
Test Summary
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ variables_ba │ generic.loc… │ PASS │ N/A N/A N/A │ 0 │ 0.077881 │
│ sh/84dd6959 │ │ │ │ │ │
└──────────────┴──────────────┴────────┴───────────────┴────────────┴──────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/logs/buildtest__whknrp3.log
Let’s check the generated script from the previous build, you can run buildtest inspect query -o variables_bash where
-o refers to output file for testname variables_bash. Take note of the output file we
buildtest inspect query -o variables_bash
$ buildtest inspect query -o variables_bash
───────────── variables_bash/84dd6959-df55-4bd4-8bf1-8b1133d94648 ──────────────
Executor: generic.local.bash
Description: Declare shell variables in bash
State: PASS
Returncode: 0
Runtime: 0.077881 sec
Starttime: 2023/02/06 21:49:46
Endtime: 2023/02/06 21:49:46
Command: bash --norc --noprofile -eo pipefail variables_bash_build.sh
Test Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/vars/variables_bash/84dd6959/variables_bash.sh
Build Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/vars/variables_bash/84dd6959/variables_bash_build.sh
Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/vars/variables_bash/84dd6959/variables_bash.out
Error File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/vars/variables_bash/84dd6959/variables_bash.err
Log File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/logs/buildtest__whknrp3.log
─ Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/che… ─
1+2=3
this is a literal string
\'singlequote\'
"doublequote"
current user: docs
number of files: 4
Hello my name is Bob
I am 30 years old
Defining Tags¶
The tags field can be used to classify tests which can be used to organize tests
or if you want to Building By Tags (buildtest build --tags <TAGNAME>).
Tags can be defined as a string or list of strings. In this example, the test
string_tag defines a tag name network while test list_of_strings_tags
define a list of tags named network and ping.
buildspecs:
string_tag:
type: script
executor: generic.local.bash
description: tags can be a string
tags: network
run: hostname
list_of_strings_tags:
type: script
executor: generic.local.bash
description: tags can be a list of strings
tags: [network, ping]
run: ping -c 4 www.google.com
Each item in tags must be a string and no duplicates are allowed, for example in this test, we define a duplicate tag network which is not allowed.
buildspecs:
duplicate_string_tags:
type: script
executor: generic.local.bash
description: duplicate strings in tags list is not allowed
tags: [network, network]
run: hostname
If tags is a list, it must contain atleast one item.
Test Status¶
buildtest will record state of each test which can be PASS or FAIL. By default a 0 exit code is
PASS and everything else is a FAIL. The status property can be used to determine how test will report its state.
Currently, we can match state based on the following:
Return Code Matching¶
buildtest can report PASS/FAIL based on returncode, by default a 0 exit code is PASS
and everything else is FAIL. The returncode can be a list of exit codes to match.
In this example we have four tests called exit1_fail, exit1_pass,
returncode_list_mismatch and returncode_int_match. We expect exit1_fail and
returncode_mismatch to FAIL while exit1_pass and returncode_int_match
will PASS.
buildspecs:
exit1_fail:
executor: generic.local.bash
type: script
description: exit 1 by default is FAIL
tags: [tutorials, fail]
run: exit 1
exit1_pass:
executor: generic.local.bash
type: script
description: report exit 1 as PASS
run: exit 1
tags: [tutorials, pass]
status:
returncode: [1]
returncode_list_mismatch:
executor: generic.local.bash
type: script
description: exit 2 failed since it failed to match returncode 1
run: exit 2
tags: [tutorials, fail]
status:
returncode: [1, 3]
returncode_int_match:
executor: generic.local.bash
type: script
description: exit 128 matches returncode 128
run: exit 128
tags: [tutorials, pass]
status:
returncode: 128
Let’s build this test and pay close attention to the status column in output.
buildtest build -b tutorials/test_status/pass_returncode.yml
$ buildtest build -b tutorials/test_status/pass_returncode.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19398495-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/02/06 21:49:47 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/tutorials/test_status/pass_returncode.yml: VALID
Total builder objects created: 4
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ exit1_f… │ script │ generic… │ None │ None │ None │ exit 1 │ /home/… │
│ │ │ │ │ │ │ by │ │
│ │ │ │ │ │ │ default │ │
│ │ │ │ │ │ │ is FAIL │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ exit1_p… │ script │ generic… │ None │ None │ None │ report │ /home/… │
│ │ │ │ │ │ │ exit 1 │ │
│ │ │ │ │ │ │ as PASS │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ returnc… │ script │ generic… │ None │ None │ None │ exit 2 │ /home/… │
│ │ │ │ │ │ │ failed │ │
│ │ │ │ │ │ │ since it │ │
│ │ │ │ │ │ │ failed │ │
│ │ │ │ │ │ │ to match │ │
│ │ │ │ │ │ │ returnc… │ │
│ │ │ │ │ │ │ 1 │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ returnc… │ script │ generic… │ None │ None │ None │ exit 128 │ /home/… │
│ │ │ │ │ │ │ matches │ │
│ │ │ │ │ │ │ returnc… │ │
│ │ │ │ │ │ │ 128 │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
exit1_fail/096c9745: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/exit1_fail/096c9745
exit1_fail/096c9745: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/exit1_fail/096c9745/stage
exit1_fail/096c9745: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/exit1_fail/096c9745/exit1_fail_build.sh
exit1_pass/2e02e91a: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/exit1_pass/2e02e91a
exit1_pass/2e02e91a: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/exit1_pass/2e02e91a/stage
exit1_pass/2e02e91a: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/exit1_pass/2e02e91a/exit1_pass_build.sh
returncode_list_mismatch/71c3c207: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/returncode_list_mismatch/71c3c207
returncode_list_mismatch/71c3c207: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/returncode_list_mismatch/71c3c207/stage
returncode_list_mismatch/71c3c207: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/returncode_list_mismatch/71c3c207/returncode_list_mismatch_build.sh
returncode_int_match/fe249f0d: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/returncode_int_match/fe249f0d
returncode_int_match/fe249f0d: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/returncode_int_match/fe249f0d/stage
returncode_int_match/fe249f0d: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/returncode_int_match/fe249f0d/returncode_int_match_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
exit1_fail/096c9745 does not have any dependencies adding test to queue
exit1_pass/2e02e91a does not have any dependencies adding test to queue
returncode_list_mismatch/71c3c207 does not have any dependencies adding test to queue
returncode_int_match/fe249f0d does not have any dependencies adding test to queue
exit1_fail/096c9745: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/exit1_fail/096c9745/stage
exit1_fail/096c9745: Running Test via command: bash --norc --noprofile -eo pipefail exit1_fail_build.sh
exit1_fail/096c9745: Test completed in 0.005385 seconds
exit1_fail/096c9745: Test completed with returncode: 1
exit1_fail/096c9745: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/exit1_fail/096c9745/exit1_fail.out
exit1_fail/096c9745: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/exit1_fail/096c9745/exit1_fail.err
exit1_pass/2e02e91a: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/exit1_pass/2e02e91a/stage
exit1_pass/2e02e91a: Running Test via command: bash --norc --noprofile -eo pipefail exit1_pass_build.sh
exit1_pass/2e02e91a: Test completed in 0.005113 seconds
exit1_pass/2e02e91a: Test completed with returncode: 1
exit1_pass/2e02e91a: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/exit1_pass/2e02e91a/exit1_pass.out
exit1_pass/2e02e91a: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/exit1_pass/2e02e91a/exit1_pass.err
exit1_pass/2e02e91a: Checking returncode - 1 is matched in list [1]
returncode_list_mismatch/71c3c207: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/returncode_list_mismatch/71c3c207/stage
returncode_list_mismatch/71c3c207: Running Test via command: bash --norc --noprofile -eo pipefail returncode_list_mismatch_build.sh
returncode_list_mismatch/71c3c207: Test completed in 0.005011 seconds
returncode_list_mismatch/71c3c207: Test completed with returncode: 2
returncode_list_mismatch/71c3c207: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/returncode_list_mismatch/71c3c207/returncode_list_mismatch.out
returncode_list_mismatch/71c3c207: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/returncode_list_mismatch/71c3c207/returncode_list_mismatch.err
returncode_list_mismatch/71c3c207: Checking returncode - 2 is matched in list [1, 3]
returncode_int_match/fe249f0d: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/returncode_int_match/fe249f0d/stage
returncode_int_match/fe249f0d: Running Test via command: bash --norc --noprofile -eo pipefail returncode_int_match_build.sh
returncode_int_match/fe249f0d: Test completed in 0.00502 seconds
returncode_int_match/fe249f0d: Test completed with returncode: 128
returncode_int_match/fe249f0d: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/returncode_int_match/fe249f0d/returncode_int_match.out
returncode_int_match/fe249f0d: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/pass_returncode/returncode_int_match/fe249f0d/returncode_int_match.err
returncode_int_match/fe249f0d: Checking returncode - 128 is matched in list [128]
In this iteration we are going to run the following tests: [exit1_fail/096c9745, exit1_pass/2e02e91a, returncode_list_mismatch/71c3c207, returncode_int_match/fe249f0d]
Test Summary
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ exit1_fail/0 │ generic.loc… │ FAIL │ N/A N/A N/A │ 1 │ 0.005385 │
│ 96c9745 │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ exit1_pass/2 │ generic.loc… │ PASS │ True False │ 1 │ 0.005113 │
│ e02e91a │ │ │ False │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ returncode_i │ generic.loc… │ PASS │ True False │ 128 │ 0.00502 │
│ nt_match/fe2 │ │ │ False │ │ │
│ 49f0d │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ returncode_l │ generic.loc… │ FAIL │ False False │ 2 │ 0.005011 │
│ ist_mismatch │ │ │ False │ │ │
│ /71c3c207 │ │ │ │ │ │
└──────────────┴──────────────┴────────┴───────────────┴────────────┴──────────┘
Passed Tests: 2/4 Percentage: 50.000%
Failed Tests: 2/4 Percentage: 50.000%
Adding 4 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/logs/buildtest_wapx6xng.log
The returncode field can be an integer or list of integers but it may not accept
duplicate values. If you specify a list of exit codes, buildtest will check actual returncode
with list of expected returncodes specified by returncode field.
Shown below are examples of invalid returncodes:
# empty list is not allowed
returncode: []
# floating point is not accepted in list
returncode: [1, 1.5]
# floating point not accepted
returncode: 1.5
# duplicates are not allowed
returncode: [1, 2, 5, 5]
Passing Test based on regular expression¶
buildtest can configure PASS/FAIL of test based on regular expression on output or error file. This can be useful if you are expecting a certain output from the test as pose to returncode check.
In this example we introduce, the regex field which is part of status that expects
a regular expression via exp. The stream property must be stdout or stderr which indicates
buildtest will read output or error file and apply regular expression. If there is a match, buildtest will record the
test state as PASS otherwise it will be a FAIL. In this example, we have two tests that will apply regular expression
on output file.
buildspecs:
status_regex_pass:
executor: generic.local.bash
type: script
tags: [system]
description: Pass test based on regular expression
run: echo "PASS"
status:
regex:
stream: stdout
exp: "^(PASS)$"
status_regex_fail:
executor: generic.local.bash
type: script
tags: [system]
description: Pass test based on regular expression
run: echo "FAIL"
status:
regex:
stream: stdout
exp: "^(123FAIL)$"
Now if we run this test, we will see first test will pass while second one will fail even though the returncode is a 0. Take a close look at the status property
buildtest build -b tutorials/test_status/status_regex.yml
$ buildtest build -b tutorials/test_status/status_regex.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19398495-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/02/06 21:49:47 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/tutorials/test_status/status_regex.yml: VALID
Total builder objects created: 2
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ status_… │ script │ generic… │ None │ None │ None │ Pass │ /home/… │
│ │ │ │ │ │ │ test │ │
│ │ │ │ │ │ │ based on │ │
│ │ │ │ │ │ │ regular │ │
│ │ │ │ │ │ │ express… │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ status_… │ script │ generic… │ None │ None │ None │ Pass │ /home/… │
│ │ │ │ │ │ │ test │ │
│ │ │ │ │ │ │ based on │ │
│ │ │ │ │ │ │ regular │ │
│ │ │ │ │ │ │ express… │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
status_regex_pass/f7e1ee4f: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/status_regex/status_regex_pass/f7e1ee4f
status_regex_pass/f7e1ee4f: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/status_regex/status_regex_pass/f7e1ee4f/stage
status_regex_pass/f7e1ee4f: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/status_regex/status_regex_pass/f7e1ee4f/status_regex_pass_build.sh
status_regex_fail/9ee3389f: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/status_regex/status_regex_fail/9ee3389f
status_regex_fail/9ee3389f: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/status_regex/status_regex_fail/9ee3389f/stage
status_regex_fail/9ee3389f: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/status_regex/status_regex_fail/9ee3389f/status_regex_fail_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
status_regex_fail/9ee3389f does not have any dependencies adding test to queue
status_regex_pass/f7e1ee4f does not have any dependencies adding test to queue
status_regex_fail/9ee3389f: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/status_regex/status_regex_fail/9ee3389f/stage
status_regex_fail/9ee3389f: Running Test via command: bash --norc --noprofile -eo pipefail status_regex_fail_build.sh
status_regex_fail/9ee3389f: Test completed in 0.005593 seconds
status_regex_fail/9ee3389f: Test completed with returncode: 0
status_regex_fail/9ee3389f: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/status_regex/status_regex_fail/9ee3389f/status_regex_fail.out
status_regex_fail/9ee3389f: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/status_regex/status_regex_fail/9ee3389f/status_regex_fail.err
status_regex_fail/9ee3389f: performing regular expression - '^(123FAIL)$' on file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/status_regex/status_regex_fail/9ee3389f/status_regex_fail.out
status_regex_fail/9ee3389f: Regular Expression Match - Failed!
status_regex_pass/f7e1ee4f: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/status_regex/status_regex_pass/f7e1ee4f/stage
status_regex_pass/f7e1ee4f: Running Test via command: bash --norc --noprofile -eo pipefail status_regex_pass_build.sh
status_regex_pass/f7e1ee4f: Test completed in 0.005146 seconds
status_regex_pass/f7e1ee4f: Test completed with returncode: 0
status_regex_pass/f7e1ee4f: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/status_regex/status_regex_pass/f7e1ee4f/status_regex_pass.out
status_regex_pass/f7e1ee4f: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/status_regex/status_regex_pass/f7e1ee4f/status_regex_pass.err
status_regex_pass/f7e1ee4f: performing regular expression - '^(PASS)$' on file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/status_regex/status_regex_pass/f7e1ee4f/status_regex_pass.out
status_regex_pass/f7e1ee4f: Regular Expression Match - Success!
In this iteration we are going to run the following tests: [status_regex_fail/9ee3389f, status_regex_pass/f7e1ee4f]
Test Summary
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ status_regex │ generic.loc… │ FAIL │ False False │ 0 │ 0.005593 │
│ _fail/9ee338 │ │ │ False │ │ │
│ 9f │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ status_regex │ generic.loc… │ PASS │ False True │ 0 │ 0.005146 │
│ _pass/f7e1ee │ │ │ False │ │ │
│ 4f │ │ │ │ │ │
└──────────────┴──────────────┴────────┴───────────────┴────────────┴──────────┘
Passed Tests: 1/2 Percentage: 50.000%
Failed Tests: 1/2 Percentage: 50.000%
Adding 2 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/logs/buildtest_yszcezkn.log
Passing Test based on runtime¶
buildtest can determine state of test based on runtime property which is part of
status object. This can be used if you want to control how test PASS or FAIL based on
execution time of test. In example below we have five tests that make use of runtime property
for passing a test. The runtime property support min and max property that can mark test
pass based on minimum and maximum runtime. A test will pass if it’s execution time is greater than min
time and less than max time. If min is specified without max property the upperbound is not set, likewise
max without min will pass if test is less than max time. The lower bound is not set, but test runtime
will be greater than 0 sec.
In test timelimit_min, we sleep for 2 seconds and it will pass because minimum runtime is 1.0 seconds. Similarly, timelimit_max will pass because we sleep for 2 seconds with a max time of 5.0.
buildspecs:
timelimit_min_max:
type: script
executor: generic.local.sh
description: "Run a sleep job for 2 seconds and test pass if its within 1.0-3.0sec"
tags: ["tutorials"]
run: sleep 2
status:
runtime:
min: 1.0
max: 3.0
timelimit_min:
type: script
executor: generic.local.sh
description: "Run a sleep job for 2 seconds and test pass if its exceeds min time of 1.0 sec"
tags: ["tutorials"]
run: sleep 2
status:
runtime:
min: 1.0
timelimit_max:
type: script
executor: generic.local.sh
description: "Run a sleep job for 2 seconds and test pass if it's within max time: 5.0 sec"
tags: ["tutorials"]
run: sleep 2
status:
runtime:
max: 5.0
timelimit_min_fail:
type: script
executor: generic.local.sh
description: "This test fails because it runs less than mintime of 10 second"
tags: ["tutorials"]
run: sleep 2
status:
runtime:
min: 10.0
timelimit_max_fail:
type: script
executor: generic.local.sh
description: "This test fails because it exceeds maxtime of 1.0 second"
tags: ["tutorials"]
run: sleep 3
status:
runtime:
max: 1.0
buildtest build -b tutorials/test_status/runtime_status_test.yml
$ buildtest build -b tutorials/test_status/runtime_status_test.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19398495-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/02/06 21:49:48 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/tutorials/test_status/runtime_status_test.yml: VALID
Total builder objects created: 5
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ timelim… │ script │ generic… │ None │ None │ None │ Run a │ /home/… │
│ │ │ │ │ │ │ sleep │ │
│ │ │ │ │ │ │ job for │ │
│ │ │ │ │ │ │ 2 │ │
│ │ │ │ │ │ │ seconds │ │
│ │ │ │ │ │ │ and test │ │
│ │ │ │ │ │ │ pass if │ │
│ │ │ │ │ │ │ its │ │
│ │ │ │ │ │ │ within │ │
│ │ │ │ │ │ │ 1.0-3.0… │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ timelim… │ script │ generic… │ None │ None │ None │ Run a │ /home/… │
│ │ │ │ │ │ │ sleep │ │
│ │ │ │ │ │ │ job for │ │
│ │ │ │ │ │ │ 2 │ │
│ │ │ │ │ │ │ seconds │ │
│ │ │ │ │ │ │ and test │ │
│ │ │ │ │ │ │ pass if │ │
│ │ │ │ │ │ │ its │ │
│ │ │ │ │ │ │ exceeds │ │
│ │ │ │ │ │ │ min time │ │
│ │ │ │ │ │ │ of 1.0 │ │
│ │ │ │ │ │ │ sec │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ timelim… │ script │ generic… │ None │ None │ None │ Run a │ /home/… │
│ │ │ │ │ │ │ sleep │ │
│ │ │ │ │ │ │ job for │ │
│ │ │ │ │ │ │ 2 │ │
│ │ │ │ │ │ │ seconds │ │
│ │ │ │ │ │ │ and test │ │
│ │ │ │ │ │ │ pass if │ │
│ │ │ │ │ │ │ it's │ │
│ │ │ │ │ │ │ within │ │
│ │ │ │ │ │ │ max │ │
│ │ │ │ │ │ │ time: │ │
│ │ │ │ │ │ │ 5.0 sec │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ timelim… │ script │ generic… │ None │ None │ None │ This │ /home/… │
│ │ │ │ │ │ │ test │ │
│ │ │ │ │ │ │ fails │ │
│ │ │ │ │ │ │ because │ │
│ │ │ │ │ │ │ it runs │ │
│ │ │ │ │ │ │ less │ │
│ │ │ │ │ │ │ than │ │
│ │ │ │ │ │ │ mintime │ │
│ │ │ │ │ │ │ of 10 │ │
│ │ │ │ │ │ │ second │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ timelim… │ script │ generic… │ None │ None │ None │ This │ /home/… │
│ │ │ │ │ │ │ test │ │
│ │ │ │ │ │ │ fails │ │
│ │ │ │ │ │ │ because │ │
│ │ │ │ │ │ │ it │ │
│ │ │ │ │ │ │ exceeds │ │
│ │ │ │ │ │ │ maxtime │ │
│ │ │ │ │ │ │ of 1.0 │ │
│ │ │ │ │ │ │ second │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
timelimit_min_max/d057efce: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min_max/d057efce
timelimit_min_max/d057efce: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min_max/d057efce/stage
timelimit_min_max/d057efce: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min_max/d057efce/timelimit_min_max_build.sh
timelimit_min/18b1e84d: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min/18b1e84d
timelimit_min/18b1e84d: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min/18b1e84d/stage
timelimit_min/18b1e84d: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min/18b1e84d/timelimit_min_build.sh
timelimit_max/87b52b36: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_max/87b52b36
timelimit_max/87b52b36: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_max/87b52b36/stage
timelimit_max/87b52b36: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_max/87b52b36/timelimit_max_build.sh
timelimit_min_fail/36275664: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min_fail/36275664
timelimit_min_fail/36275664: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min_fail/36275664/stage
timelimit_min_fail/36275664: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min_fail/36275664/timelimit_min_fail_build.sh
timelimit_max_fail/31133453: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_max_fail/31133453
timelimit_max_fail/31133453: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_max_fail/31133453/stage
timelimit_max_fail/31133453: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_max_fail/31133453/timelimit_max_fail_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
timelimit_max/87b52b36 does not have any dependencies adding test to queue
timelimit_min_max/d057efce does not have any dependencies adding test to queue
timelimit_min_fail/36275664 does not have any dependencies adding test to queue
timelimit_max_fail/31133453 does not have any dependencies adding test to queue
timelimit_min/18b1e84d does not have any dependencies adding test to queue
timelimit_max/87b52b36: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_max/87b52b36/stage
timelimit_max/87b52b36: Running Test via command: sh --norc --noprofile -eo pipefail timelimit_max_build.sh
timelimit_max/87b52b36: Test completed in 0.003149 seconds
timelimit_max/87b52b36: Test completed with returncode: 2
timelimit_max/87b52b36: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_max/87b52b36/timelimit_max.out
timelimit_max/87b52b36: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_max/87b52b36/timelimit_max.err
timelimit_max/87b52b36: Checking runtime < maxtime: 0.003149 < 5.0
timelimit_min_fail/36275664: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min_fail/36275664/stage
timelimit_min_fail/36275664: Running Test via command: sh --norc --noprofile -eo pipefail timelimit_min_fail_build.sh
timelimit_min_fail/36275664: Test completed in 0.003015 seconds
timelimit_min_fail/36275664: Test completed with returncode: 2
timelimit_min_fail/36275664: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min_fail/36275664/timelimit_min_fail.out
timelimit_min_fail/36275664: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min_fail/36275664/timelimit_min_fail.err
timelimit_min_fail/36275664: Checking mintime < runtime: 10.0 < 0.003015
timelimit_max_fail/31133453: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_max_fail/31133453/stage
timelimit_max_fail/31133453: Running Test via command: sh --norc --noprofile -eo pipefail timelimit_max_fail_build.sh
timelimit_max_fail/31133453: Test completed in 0.002991 seconds
timelimit_max_fail/31133453: Test completed with returncode: 2
timelimit_max_fail/31133453: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_max_fail/31133453/timelimit_max_fail.out
timelimit_max_fail/31133453: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_max_fail/31133453/timelimit_max_fail.err
timelimit_max_fail/31133453: Checking runtime < maxtime: 0.002991 < 1.0
timelimit_min_max/d057efce: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min_max/d057efce/stage
timelimit_min_max/d057efce: Running Test via command: sh --norc --noprofile -eo pipefail timelimit_min_max_build.sh
timelimit_min_max/d057efce: Test completed in 0.00299 seconds
timelimit_min_max/d057efce: Test completed with returncode: 2
timelimit_min_max/d057efce: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min_max/d057efce/timelimit_min_max.out
timelimit_min_max/d057efce: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min_max/d057efce/timelimit_min_max.err
timelimit_min_max/d057efce: Checking mintime < runtime < maxtime: 1.0 < 0.00299 < 3.0
timelimit_min/18b1e84d: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min/18b1e84d/stage
timelimit_min/18b1e84d: Running Test via command: sh --norc --noprofile -eo pipefail timelimit_min_build.sh
timelimit_min/18b1e84d: Test completed in 0.002956 seconds
timelimit_min/18b1e84d: Test completed with returncode: 2
timelimit_min/18b1e84d: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min/18b1e84d/timelimit_min.out
timelimit_min/18b1e84d: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/runtime_status_test/timelimit_min/18b1e84d/timelimit_min.err
timelimit_min/18b1e84d: Checking mintime < runtime: 1.0 < 0.002956
In this iteration we are going to run the following tests: [timelimit_max/87b52b36, timelimit_min_fail/36275664, timelimit_max_fail/31133453, timelimit_min_max/d057efce, timelimit_min/18b1e84d]
Test Summary
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ timelimit_mi │ generic.loc… │ FAIL │ False False │ 2 │ 0.003015 │
│ n_fail/36275 │ │ │ False │ │ │
│ 664 │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ timelimit_ma │ generic.loc… │ PASS │ False False │ 2 │ 0.002991 │
│ x_fail/31133 │ │ │ True │ │ │
│ 453 │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ timelimit_mi │ generic.loc… │ FAIL │ False False │ 2 │ 0.00299 │
│ n_max/d057ef │ │ │ False │ │ │
│ ce │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ timelimit_mi │ generic.loc… │ FAIL │ False False │ 2 │ 0.002956 │
│ n/18b1e84d │ │ │ False │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ timelimit_ma │ generic.loc… │ PASS │ False False │ 2 │ 0.003149 │
│ x/87b52b36 │ │ │ True │ │ │
└──────────────┴──────────────┴────────┴───────────────┴────────────┴──────────┘
Passed Tests: 2/5 Percentage: 40.000%
Failed Tests: 3/5 Percentage: 60.000%
Adding 5 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/logs/buildtest__ght8y4v.log
If we look at the test results, we expect the first three tests timelimit_min, timelimit_max, timelimit_min_max will pass while the last two tests fail because it fails to comply with runtime property.
buildtest report --filter buildspec=tutorials/test_status/runtime_status_test.yml --format name,id,state,runtime --latest
$ buildtest report --filter buildspec=tutorials/test_status/runtime_status_test.yml --format name,id,state,runtime --latest
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ name ┃ id ┃ state ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ timelimit_min_fail │ 36275664 │ FAIL │ 0.003015 │
├────────────────────────────────┼────────────────┼────────────┼───────────────┤
│ timelimit_max_fail │ 31133453 │ PASS │ 0.002991 │
├────────────────────────────────┼────────────────┼────────────┼───────────────┤
│ timelimit_min_max │ d057efce │ FAIL │ 0.00299 │
├────────────────────────────────┼────────────────┼────────────┼───────────────┤
│ timelimit_min │ 18b1e84d │ FAIL │ 0.002956 │
├────────────────────────────────┼────────────────┼────────────┼───────────────┤
│ timelimit_max │ 87b52b36 │ PASS │ 0.003149 │
└────────────────────────────────┴────────────────┴────────────┴───────────────┘
Explicitly Declaring Status of Test¶
You can explicitly define status of test regardless of what buildtest does for checking status of test. This
can be useful if you want to explicitly mark a test as PASS or FAIL regardless of how test behaves. This can be done via
state property which expects one of two types PASS or FAIL. If state property is specified, buildtest will ignore any checks
including returncode, regex, or runtime match.
In this next example we will demonstrate how one can use state property for marking test state. In this
example we have four tests. The first test always_pass will PASS even though we have a non-zero returncode. The
second test always_fail will FAIL even though it has a 0 returncode. The last two test demonstrate how one
can define state regardless of what is specified for returncode match. buildtest will honor the state property even if
their is a match on the returncode.
buildspecs:
always_pass:
type: script
executor: 'generic.local.sh'
description: This test will always 'PASS'
run: exit 1
status:
state: PASS
always_fail:
type: script
executor: 'generic.local.sh'
description: This test will always 'FAIL'
run: exit 0
status:
state: FAIL
test_fail_returncode_match:
type: script
executor: 'generic.local.sh'
description: This test will 'FAIL' even if we have returncode match
run: exit 1
status:
state: FAIL
returncode: 1
test_pass_returncode_mismatch:
type: script
executor: 'generic.local.sh'
description: This test will 'PASS' even if we have returncode mismatch
run: exit 1
status:
state: PASS
returncode: 2
If we build this test, we expect buildtest to honor the value of state property
buildtest build -b tutorials/test_status/explicit_state.yml
$ buildtest build -b tutorials/test_status/explicit_state.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19398495-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/02/06 21:49:49 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/tutorials/test_status/explicit_state.yml: VALID
Total builder objects created: 4
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ always_… │ script │ generic… │ None │ None │ None │ This │ /home/… │
│ │ │ │ │ │ │ test │ │
│ │ │ │ │ │ │ will │ │
│ │ │ │ │ │ │ always │ │
│ │ │ │ │ │ │ 'PASS' │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ always_… │ script │ generic… │ None │ None │ None │ This │ /home/… │
│ │ │ │ │ │ │ test │ │
│ │ │ │ │ │ │ will │ │
│ │ │ │ │ │ │ always │ │
│ │ │ │ │ │ │ 'FAIL' │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ test_fa… │ script │ generic… │ None │ None │ None │ This │ /home/… │
│ │ │ │ │ │ │ test │ │
│ │ │ │ │ │ │ will │ │
│ │ │ │ │ │ │ 'FAIL' │ │
│ │ │ │ │ │ │ even if │ │
│ │ │ │ │ │ │ we have │ │
│ │ │ │ │ │ │ returnc… │ │
│ │ │ │ │ │ │ match │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ test_pa… │ script │ generic… │ None │ None │ None │ This │ /home/… │
│ │ │ │ │ │ │ test │ │
│ │ │ │ │ │ │ will │ │
│ │ │ │ │ │ │ 'PASS' │ │
│ │ │ │ │ │ │ even if │ │
│ │ │ │ │ │ │ we have │ │
│ │ │ │ │ │ │ returnc… │ │
│ │ │ │ │ │ │ mismatch │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
always_pass/11472dfa: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/always_pass/11472dfa
always_pass/11472dfa: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/always_pass/11472dfa/stage
always_pass/11472dfa: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/always_pass/11472dfa/always_pass_build.sh
always_fail/2635a62b: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/always_fail/2635a62b
always_fail/2635a62b: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/always_fail/2635a62b/stage
always_fail/2635a62b: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/always_fail/2635a62b/always_fail_build.sh
test_fail_returncode_match/88ae2a4e: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/test_fail_returncode_match/88ae2a4e
test_fail_returncode_match/88ae2a4e: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/test_fail_returncode_match/88ae2a4e/stage
test_fail_returncode_match/88ae2a4e: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/test_fail_returncode_match/88ae2a4e/test_fail_returncode_match_build.sh
test_pass_returncode_mismatch/5e3117cc: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/test_pass_returncode_mismatch/5e3117cc
test_pass_returncode_mismatch/5e3117cc: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/test_pass_returncode_mismatch/5e3117cc/stage
test_pass_returncode_mismatch/5e3117cc: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/test_pass_returncode_mismatch/5e3117cc/test_pass_returncode_mismatch_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
always_pass/11472dfa does not have any dependencies adding test to queue
test_fail_returncode_match/88ae2a4e does not have any dependencies adding test to queue
test_pass_returncode_mismatch/5e3117cc does not have any dependencies adding test to queue
always_fail/2635a62b does not have any dependencies adding test to queue
always_pass/11472dfa: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/always_pass/11472dfa/stage
always_pass/11472dfa: Running Test via command: sh --norc --noprofile -eo pipefail always_pass_build.sh
always_pass/11472dfa: Test completed in 0.003157 seconds
always_pass/11472dfa: Test completed with returncode: 2
always_pass/11472dfa: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/always_pass/11472dfa/always_pass.out
always_pass/11472dfa: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/always_pass/11472dfa/always_pass.err
always_fail/2635a62b: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/always_fail/2635a62b/stage
always_fail/2635a62b: Running Test via command: sh --norc --noprofile -eo pipefail always_fail_build.sh
always_fail/2635a62b: Test completed in 0.003267 seconds
always_fail/2635a62b: Test completed with returncode: 2
always_fail/2635a62b: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/always_fail/2635a62b/always_fail.out
always_fail/2635a62b: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/always_fail/2635a62b/always_fail.err
test_pass_returncode_mismatch/5e3117cc: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/test_pass_returncode_mismatch/5e3117cc/stage
test_pass_returncode_mismatch/5e3117cc: Running Test via command: sh --norc --noprofile -eo pipefail test_pass_returncode_mismatch_build.sh
test_pass_returncode_mismatch/5e3117cc: Test completed in 0.002911 seconds
test_pass_returncode_mismatch/5e3117cc: Test completed with returncode: 2
test_pass_returncode_mismatch/5e3117cc: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/test_pass_returncode_mismatch/5e3117cc/test_pass_returncode_mismatch.out
test_pass_returncode_mismatch/5e3117cc: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/test_pass_returncode_mismatch/5e3117cc/test_pass_returncode_mismatch.err
test_pass_returncode_mismatch/5e3117cc: Checking returncode - 2 is matched in list [2]
test_fail_returncode_match/88ae2a4e: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/test_fail_returncode_match/88ae2a4e/stage
test_fail_returncode_match/88ae2a4e: Running Test via command: sh --norc --noprofile -eo pipefail test_fail_returncode_match_build.sh
test_fail_returncode_match/88ae2a4e: Test completed in 0.00292 seconds
test_fail_returncode_match/88ae2a4e: Test completed with returncode: 2
test_fail_returncode_match/88ae2a4e: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/test_fail_returncode_match/88ae2a4e/test_fail_returncode_match.out
test_fail_returncode_match/88ae2a4e: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.sh/explicit_state/test_fail_returncode_match/88ae2a4e/test_fail_returncode_match.err
test_fail_returncode_match/88ae2a4e: Checking returncode - 2 is matched in list [1]
In this iteration we are going to run the following tests: [always_pass/11472dfa, always_fail/2635a62b, test_pass_returncode_mismatch/5e3117cc, test_fail_returncode_match/88ae2a4e]
Test Summary
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ always_fail/ │ generic.loc… │ FAIL │ False False │ 2 │ 0.003267 │
│ 2635a62b │ │ │ False │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ test_fail_re │ generic.loc… │ FAIL │ False False │ 2 │ 0.00292 │
│ turncode_mat │ │ │ False │ │ │
│ ch/88ae2a4e │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ always_pass/ │ generic.loc… │ PASS │ False False │ 2 │ 0.003157 │
│ 11472dfa │ │ │ False │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ test_pass_re │ generic.loc… │ PASS │ True False │ 2 │ 0.002911 │
│ turncode_mis │ │ │ False │ │ │
│ match/5e3117 │ │ │ │ │ │
│ cc │ │ │ │ │ │
└──────────────┴──────────────┴────────┴───────────────┴────────────┴──────────┘
Passed Tests: 2/4 Percentage: 50.000%
Failed Tests: 2/4 Percentage: 50.000%
Adding 4 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/logs/buildtest_z1spzwvs.log
File Checks¶
buildtest supports various file checks that can be used as means for passing test.
For instance, if you want to check for file existence, you can use exists property
which expects a list of file or directory names to check. This can be useful if your test
will write some output file or directory and test will pass based on existence of file/directory.
In the example below we have two tests, first test will pass, where all files exist. We check for files and directory path, note variable and shell expansion is supported.
In the second example, we expect this test to fail because filename bar does not exist.
buildspecs:
status_exists:
type: script
executor: generic.local.bash
description: status check based for file and directory
run: |
mkdir -p $HOME/dirA
mkdir -p /tmp/ABC
touch file1
status:
exists:
- $HOME/dirA
- ~/.bashrc
- /tmp/ABC
- file1
status_exists_failure:
type: script
executor: generic.local.bash
description: status check failure for existence
run: touch foo
status:
exists:
- bar
We can run this test by running the following, take note of the output.
buildtest build -b tutorials/test_status/exists.yml
$ buildtest build -b tutorials/test_status/exists.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19398495-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/02/06 21:49:50 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/tutorials/test_status/exists.yml: VALID
Total builder objects created: 2
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ status_… │ script │ generic… │ None │ None │ None │ status │ /home/… │
│ │ │ │ │ │ │ check │ │
│ │ │ │ │ │ │ based │ │
│ │ │ │ │ │ │ for file │ │
│ │ │ │ │ │ │ and │ │
│ │ │ │ │ │ │ directo… │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ status_… │ script │ generic… │ None │ None │ None │ status │ /home/… │
│ │ │ │ │ │ │ check │ │
│ │ │ │ │ │ │ failure │ │
│ │ │ │ │ │ │ for │ │
│ │ │ │ │ │ │ existen… │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
status_exists/cda002fb: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/exists/status_exists/cda002fb
status_exists/cda002fb: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/exists/status_exists/cda002fb/stage
status_exists/cda002fb: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/exists/status_exists/cda002fb/status_exists_build.sh
status_exists_failure/725564f2: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/exists/status_exists_failure/725564f2
status_exists_failure/725564f2: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/exists/status_exists_failure/725564f2/stage
status_exists_failure/725564f2: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/exists/status_exists_failure/725564f2/status_exists_failure_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
status_exists_failure/725564f2 does not have any dependencies adding test to queue
status_exists/cda002fb does not have any dependencies adding test to queue
status_exists/cda002fb: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/exists/status_exists/cda002fb/stage
status_exists/cda002fb: Running Test via command: bash --norc --noprofile -eo pipefail status_exists_build.sh
status_exists/cda002fb: Test completed in 0.01232 seconds
status_exists/cda002fb: Test completed with returncode: 0
status_exists/cda002fb: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/exists/status_exists/cda002fb/status_exists.out
status_exists/cda002fb: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/exists/status_exists/cda002fb/status_exists.err
status_exists/cda002fb: Test all files: ['$HOME/dirA', '~/.bashrc', '/tmp/ABC', 'file1'] existences
status_exists/cda002fb: file: /home/docs/dirA exists
status_exists/cda002fb: file: /home/docs/.bashrc exists
status_exists/cda002fb: file: /tmp/ABC exists
status_exists/cda002fb: file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/exists/status_exists/cda002fb/stage/file1 exists
status_exists/cda002fb: Exist Check: True
status_exists_failure/725564f2: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/exists/status_exists_failure/725564f2/stage
status_exists_failure/725564f2: Running Test via command: bash --norc --noprofile -eo pipefail status_exists_failure_build.sh
status_exists_failure/725564f2: Test completed in 0.007664 seconds
status_exists_failure/725564f2: Test completed with returncode: 0
status_exists_failure/725564f2: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/exists/status_exists_failure/725564f2/status_exists_failure.out
status_exists_failure/725564f2: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/exists/status_exists_failure/725564f2/status_exists_failure.err
status_exists_failure/725564f2: Test all files: ['bar'] existences
status_exists_failure/725564f2: file: bar does not exist
status_exists_failure/725564f2: Exist Check: False
In this iteration we are going to run the following tests: [status_exists/cda002fb, status_exists_failure/725564f2]
Test Summary
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ status_exist │ generic.loc… │ FAIL │ False False │ 0 │ 0.007664 │
│ s_failure/72 │ │ │ False │ │ │
│ 5564f2 │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ status_exist │ generic.loc… │ PASS │ False False │ 0 │ 0.01232 │
│ s/cda002fb │ │ │ False │ │ │
└──────────────┴──────────────┴────────┴───────────────┴────────────┴──────────┘
Passed Tests: 1/2 Percentage: 50.000%
Failed Tests: 1/2 Percentage: 50.000%
Adding 2 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/logs/buildtest_wydi689m.log
Each item in the exists field must be a string, which can lead to issue in example
below let’s assume we want a test to pass based on a directory name 1, if we specify
as follows, this test will fail validation.
buildspecs:
file_exists_failure:
type: script
executor: generic.local.bash
description: this test will fail validation, because item must be a string
run: mkdir -p 1
status:
exists: [1]
We can validate this buildspec by running the following
buildtest bc validate -b tutorials/test_status/file_exists_exception.yml
$ buildtest bc validate -b tutorials/test_status/file_exists_exception.yml ─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/t… ─ 1 is not of type 'string' Failed validating 'type' in schema['properties']['status']['properties']['exists']['items']: {'type': 'string'} On instance['status']['exists'][0]: 1 1 buildspecs failed to validate
In order to run this test, we need to enclose each item in quotes. Shown below is the same test with quotations.
buildspecs:
file_exists_pass:
type: script
executor: generic.local.bash
description: this test will pass
run: mkdir -p 1
status:
exists: [ '1' ]
Let’s validate and build this test.
buildtest bc validate -b tutorials/test_status/file_exists_with_number.yml
$ buildtest bc validate -b tutorials/test_status/file_exists_with_number.yml
buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/tutorials/test_status/file_exists_with_number.yml is valid
All buildspecs passed validation!!!
buildtest build -b tutorials/test_status/file_exists_with_number.yml
$ buildtest build -b tutorials/test_status/file_exists_with_number.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19398495-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/02/06 21:49:51 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/tutorials/test_status/file_exists_with_number.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ file_ex… │ script │ generic… │ None │ None │ None │ this │ /home/… │
│ │ │ │ │ │ │ test │ │
│ │ │ │ │ │ │ will │ │
│ │ │ │ │ │ │ pass │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
file_exists_pass/d30d29ea: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/d30d29ea
file_exists_pass/d30d29ea: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/d30d29ea/stage
file_exists_pass/d30d29ea: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/d30d29ea/file_exists_pass_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
file_exists_pass/d30d29ea does not have any dependencies adding test to queue
file_exists_pass/d30d29ea: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/d30d29ea/stage
file_exists_pass/d30d29ea: Running Test via command: bash --norc --noprofile -eo pipefail file_exists_pass_build.sh
file_exists_pass/d30d29ea: Test completed in 0.006606 seconds
file_exists_pass/d30d29ea: Test completed with returncode: 0
file_exists_pass/d30d29ea: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/d30d29ea/file_exists_pass.out
file_exists_pass/d30d29ea: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/d30d29ea/file_exists_pass.err
file_exists_pass/d30d29ea: Test all files: ['1'] existences
file_exists_pass/d30d29ea: file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/d30d29ea/stage/1 exists
file_exists_pass/d30d29ea: Exist Check: True
In this iteration we are going to run the following tests: [file_exists_pass/d30d29ea]
Test Summary
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ file_exists_ │ generic.loc… │ PASS │ False False │ 0 │ 0.006606 │
│ pass/d30d29e │ │ │ False │ │ │
│ a │ │ │ │ │ │
└──────────────┴──────────────┴────────┴───────────────┴────────────┴──────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/logs/buildtest_bsp0zcff.log
In the next example, we introduce checks for files and directory via is_file and
is_dir property, which behaves similar to exists except they will check if each item
is a file or directory. We expect the first test to fail, because $HOME/.bashrc is
not a directory but a file. The second test will incorporate the same test and
use is_file for status check.
buildspecs:
file_and_dir_checks:
type: script
executor: generic.local.bash
description: status check for files and directories
run: hostname
status:
is_dir:
- $HOME
- $HOME/.bashrc
- /tmp
combined_file_and_dir_checks:
type: script
executor: generic.local.bash
description: status check for files and directories
run: hostname
status:
is_dir:
- $HOME
- /tmp
is_file:
- $HOME/.bashrc
Let’s build the test and see the output.
buildtest build -b tutorials/test_status/file_and_dir_check.yml
$ buildtest build -b tutorials/test_status/file_and_dir_check.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19398495-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/02/06 21:49:52 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/tutorials/test_status/file_and_dir_check.yml: VALID
Total builder objects created: 2
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ file_an… │ script │ generic… │ None │ None │ None │ status │ /home/… │
│ │ │ │ │ │ │ check │ │
│ │ │ │ │ │ │ for │ │
│ │ │ │ │ │ │ files │ │
│ │ │ │ │ │ │ and │ │
│ │ │ │ │ │ │ directo… │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ combine… │ script │ generic… │ None │ None │ None │ status │ /home/… │
│ │ │ │ │ │ │ check │ │
│ │ │ │ │ │ │ for │ │
│ │ │ │ │ │ │ files │ │
│ │ │ │ │ │ │ and │ │
│ │ │ │ │ │ │ directo… │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
file_and_dir_checks/43b106e8: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_and_dir_check/file_and_dir_checks/43b106e8
file_and_dir_checks/43b106e8: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_and_dir_check/file_and_dir_checks/43b106e8/stage
file_and_dir_checks/43b106e8: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_and_dir_check/file_and_dir_checks/43b106e8/file_and_dir_checks_build.sh
combined_file_and_dir_checks/046a0ff1: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_and_dir_check/combined_file_and_dir_checks/046a0ff1
combined_file_and_dir_checks/046a0ff1: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_and_dir_check/combined_file_and_dir_checks/046a0ff1/stage
combined_file_and_dir_checks/046a0ff1: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_and_dir_check/combined_file_and_dir_checks/046a0ff1/combined_file_and_dir_checks_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
file_and_dir_checks/43b106e8 does not have any dependencies adding test to queue
combined_file_and_dir_checks/046a0ff1 does not have any dependencies adding test to queue
file_and_dir_checks/43b106e8: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_and_dir_check/file_and_dir_checks/43b106e8/stage
file_and_dir_checks/43b106e8: Running Test via command: bash --norc --noprofile -eo pipefail file_and_dir_checks_build.sh
file_and_dir_checks/43b106e8: Test completed in 0.006266 seconds
file_and_dir_checks/43b106e8: Test completed with returncode: 0
file_and_dir_checks/43b106e8: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_and_dir_check/file_and_dir_checks/43b106e8/file_and_dir_checks.out
file_and_dir_checks/43b106e8: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_and_dir_check/file_and_dir_checks/43b106e8/file_and_dir_checks.err
file_and_dir_checks/43b106e8: Test all files: ['$HOME', '$HOME/.bashrc', '/tmp'] existences
file_and_dir_checks/43b106e8: file: /home/docs is a directory
file_and_dir_checks/43b106e8: file: $HOME/.bashrc is not a directory
file_and_dir_checks/43b106e8: file: /tmp is a directory
file_and_dir_checks/43b106e8: Directory Existence Check: False
combined_file_and_dir_checks/046a0ff1: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_and_dir_check/combined_file_and_dir_checks/046a0ff1/stage
combined_file_and_dir_checks/046a0ff1: Running Test via command: bash --norc --noprofile -eo pipefail combined_file_and_dir_checks_build.sh
combined_file_and_dir_checks/046a0ff1: Test completed in 0.005858 seconds
combined_file_and_dir_checks/046a0ff1: Test completed with returncode: 0
combined_file_and_dir_checks/046a0ff1: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_and_dir_check/combined_file_and_dir_checks/046a0ff1/combined_file_and_dir_checks.out
combined_file_and_dir_checks/046a0ff1: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/file_and_dir_check/combined_file_and_dir_checks/046a0ff1/combined_file_and_dir_checks.err
combined_file_and_dir_checks/046a0ff1: Test all files: ['$HOME', '/tmp'] existences
combined_file_and_dir_checks/046a0ff1: file: /home/docs is a directory
combined_file_and_dir_checks/046a0ff1: file: /tmp is a directory
combined_file_and_dir_checks/046a0ff1: Directory Existence Check: True
combined_file_and_dir_checks/046a0ff1: Test all files: ['$HOME/.bashrc'] existences
combined_file_and_dir_checks/046a0ff1: file: /home/docs/.bashrc is a file
combined_file_and_dir_checks/046a0ff1: File Existence Check: True
In this iteration we are going to run the following tests: [file_and_dir_checks/43b106e8, combined_file_and_dir_checks/046a0ff1]
Test Summary
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ file_and_dir │ generic.loc… │ FAIL │ False False │ 0 │ 0.006266 │
│ _checks/43b1 │ │ │ False │ │ │
│ 06e8 │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ combined_fil │ generic.loc… │ PASS │ False False │ 0 │ 0.005858 │
│ e_and_dir_ch │ │ │ False │ │ │
│ ecks/046a0ff │ │ │ │ │ │
│ 1 │ │ │ │ │ │
└──────────────┴──────────────┴────────┴───────────────┴────────────┴──────────┘
Passed Tests: 1/2 Percentage: 50.000%
Failed Tests: 1/2 Percentage: 50.000%
Adding 2 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/logs/buildtest_e57uzz_o.log
Skipping test¶
By default, buildtest will run all tests defined in buildspecs section, if you
want to skip a test use the skip field which expects a boolean value. Shown
below is an example test.
buildspecs:
skip:
type: script
executor: generic.local.bash
description: This test is skipped
skip: Yes
tags: [tutorials]
run: hostname
unskipped:
type: script
executor: generic.local.bash
description: This test is not skipped
skip: No
tags: [tutorials]
run: hostname
The first test skip will be ignored by buildtest because skip: true is defined
while unskipped will be processed as usual.
Note
YAML and JSON have different representation for boolean. For json schema
valid values are true and false see https://json-schema.org/understanding-json-schema/reference/boolean.html
however YAML has many more representation for boolean see https://yaml.org/type/bool.html. You
may use any of the YAML boolean, however it’s best to stick with json schema values
true and false.
Here is an example build, notice message [skip] test is skipped during the build
stage
buildtest build -b tutorials/skip_tests.yml
$ buildtest build -b tutorials/skip_tests.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19398495-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/02/06 21:49:53 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
skip: skipping test due to 'skip' property.
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/tutorials/skip_tests.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ unskipp… │ script │ generic… │ None │ None │ None │ This │ /home/… │
│ │ │ │ │ │ │ test is │ │
│ │ │ │ │ │ │ not │ │
│ │ │ │ │ │ │ skipped │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
unskipped/8df01f55: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/skip_tests/unskipped/8df01f55
unskipped/8df01f55: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/skip_tests/unskipped/8df01f55/stage
unskipped/8df01f55: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/skip_tests/unskipped/8df01f55/unskipped_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
unskipped/8df01f55 does not have any dependencies adding test to queue
unskipped/8df01f55: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/skip_tests/unskipped/8df01f55/stage
unskipped/8df01f55: Running Test via command: bash --norc --noprofile -eo pipefail unskipped_build.sh
unskipped/8df01f55: Test completed in 0.006114 seconds
unskipped/8df01f55: Test completed with returncode: 0
unskipped/8df01f55: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/skip_tests/unskipped/8df01f55/unskipped.out
unskipped/8df01f55: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/tests/generic.local.bash/skip_tests/unskipped/8df01f55/unskipped.err
In this iteration we are going to run the following tests: [unskipped/8df01f55]
Test Summary
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ unskipped/8d │ generic.loc… │ PASS │ N/A N/A N/A │ 0 │ 0.006114 │
│ f01f55 │ │ │ │ │ │
└──────────────┴──────────────┴────────┴───────────────┴────────────┴──────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/logs/buildtest_5f2oggt7.log
Skipping a buildspec¶
Sometimes you may want to skip all test in a buildspec instead of updating every test with skip property, this can be done by setting
skip at the top-level. This can be useful if you are running several test in a directory such as buildtest build -b dir1/ and you don’t
want to explicitly exclude file via -x option every time, instead you can hardcode this into the buildspec. A typical use-case of skipping
test is when a test is broken and you don’t want to run it then its good idea to set skip: yes on the buildspec and fix it later.
In this next example we set skip: yes, buildtest will skip the buildspec and no test will be processed even if skip is set in each test.
skip: yes
buildspecs:
skip_all_tests:
type: script
executor: generic.local.bash
description: "All test in this buildspec are skipped"
tags: [tutorials]
run: hostname
this_test_is_also_skipped:
type: script
skip: no
executor: generic.local.bash
description: "This test is also skipped even if skip is defined in test"
tags: [ tutorials ]
run: hostname
If you try building this buildspec, you will see buildtest will skip the buildspec and terminate.
buildtest build -b tutorials/skip_buildspec.yml
$ buildtest build -b tutorials/skip_buildspec.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19398495-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/02/06 21:49:53 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/tutorials/skip_buildspec.yml: skipping all test since 'skip' is defined
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/tutorials/skip_buildspec.yml: VALID
Buildspecs Filtered out
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ buildspecs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/t… │
└──────────────────────────────────────────────────────────────────────────────┘
buildtest is unable to create any tests because there are no valid buildspecs.
Please see logfile: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.2/var/buildtest.log