Status Checks
buildtest will record state of each test which can be PASS or FAIL. By default a 0 exit code is
PASS and anything else is a FAIL. The status property can be used to determine how test will report its state.
Currently, we can match state based on the following:
returncode: Return Code Matching
buildtest can report PASS/FAIL based on returncode, by default a 0 exit code is PASS
and everything else is FAIL. The returncode can be a list of exit codes to match.
In this example we have four tests called exit1_fail, exit1_pass,
returncode_list_mismatch and returncode_int_match. We expect exit1_fail and
returncode_mismatch to FAIL while exit1_pass and returncode_int_match
will PASS.
buildspecs:
exit1_fail:
executor: generic.local.bash
type: script
description: exit 1 by default is FAIL
tags: [tutorials, fail]
run: exit 1
exit1_pass:
executor: generic.local.bash
type: script
description: report exit 1 as PASS
run: exit 1
tags: [tutorials, pass]
status:
returncode: [1]
returncode_list_mismatch:
executor: generic.local.bash
type: script
description: exit 2 failed since it failed to match returncode 1
run: exit 2
tags: [tutorials, fail]
status:
returncode: [1, 3]
returncode_int_match:
executor: generic.local.bash
type: script
description: exit 128 matches returncode 128
run: exit 128
tags: [tutorials, pass]
status:
returncode: 128
Let’s build this test and pay close attention to the status column in output.
buildtest build -b tutorials/test_status/pass_returncode.yml
$ buildtest build -b tutorials/test_status/pass_returncode.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:44:56 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/pass_returncode.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/pass_returncode.yml: VALID
Total builder objects created: 4
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ exit1_fa │ script │ generic. │ None │ None │ None │ exit 1 │ /home/d │
│ il/4f0f9 │ │ local.ba │ │ │ │ by │ ocs/che │
│ 56d │ │ sh │ │ │ │ default │ ckouts/ │
│ │ │ │ │ │ │ is FAIL │ readthe │
│ │ │ │ │ │ │ │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/p │
│ │ │ │ │ │ │ │ ass_ret │
│ │ │ │ │ │ │ │ urncode │
│ │ │ │ │ │ │ │ .yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ exit1_pa │ script │ generic. │ None │ None │ None │ report │ /home/d │
│ ss/62337 │ │ local.ba │ │ │ │ exit 1 │ ocs/che │
│ 672 │ │ sh │ │ │ │ as PASS │ ckouts/ │
│ │ │ │ │ │ │ │ readthe │
│ │ │ │ │ │ │ │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/p │
│ │ │ │ │ │ │ │ ass_ret │
│ │ │ │ │ │ │ │ urncode │
│ │ │ │ │ │ │ │ .yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ returnco │ script │ generic. │ None │ None │ None │ exit 2 │ /home/d │
│ de_list_ │ │ local.ba │ │ │ │ failed │ ocs/che │
│ mismatch │ │ sh │ │ │ │ since it │ ckouts/ │
│ /c2b2859 │ │ │ │ │ │ failed │ readthe │
│ d │ │ │ │ │ │ to match │ docs.or │
│ │ │ │ │ │ │ returnco │ g/user_ │
│ │ │ │ │ │ │ de 1 │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/p │
│ │ │ │ │ │ │ │ ass_ret │
│ │ │ │ │ │ │ │ urncode │
│ │ │ │ │ │ │ │ .yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ returnco │ script │ generic. │ None │ None │ None │ exit 128 │ /home/d │
│ de_int_m │ │ local.ba │ │ │ │ matches │ ocs/che │
│ atch/60e │ │ sh │ │ │ │ returnco │ ckouts/ │
│ e1b1a │ │ │ │ │ │ de 128 │ readthe │
│ │ │ │ │ │ │ │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/p │
│ │ │ │ │ │ │ │ ass_ret │
│ │ │ │ │ │ │ │ urncode │
│ │ │ │ │ │ │ │ .yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
exit1_fail/4f0f956d: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/pass_returncode/exit1_fail/4f0f956d
exit1_pass/62337672: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/pass_returncode/exit1_pass/62337672
returncode_list_mismatch/c2b2859d: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/pass_returncode/returncode_list_mismatch/c2b2859d
returncode_int_match/60ee1b1a: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/pass_returncode/returncode_int_match/60ee1b1a
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
returncode_int_match/60ee1b1a does not have any dependencies adding test to queue
returncode_list_mismatch/c2b2859d does not have any dependencies adding test to queue
exit1_fail/4f0f956d does not have any dependencies adding test to queue
exit1_pass/62337672 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ returncode_int_match/60ee1b1a │
│ returncode_list_mismatch/c2b2859d │
│ exit1_fail/4f0f956d │
│ exit1_pass/62337672 │
└───────────────────────────────────┘
returncode_int_match/60ee1b1a: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/pass_returncode/returncode_int_match/60ee1b1a/stage
returncode_int_match/60ee1b1a: Running Test via command: bash returncode_int_match_build.sh
returncode_int_match/60ee1b1a: failed to submit job with returncode: 128
─────────────── Error Message for returncode_int_match/60ee1b1a ────────────────
returncode_int_match/60ee1b1a: Detected failure in running test, will attempt to retry test: 1 times
returncode_int_match/60ee1b1a: Run - 1/1
returncode_int_match/60ee1b1a: Running Test via command: bash returncode_int_match_build.sh
returncode_int_match/60ee1b1a: failed to submit job with returncode: 128
returncode_int_match/60ee1b1a: Test completed in 0.014886 seconds with returncode: 128
returncode_int_match/60ee1b1a: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/pass_returncode/returncode_int_match/60ee1b1a/returncode_int_match.out
returncode_int_match/60ee1b1a: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/pass_returncode/returncode_int_match/60ee1b1a/returncode_int_match.err
returncode_int_match/60ee1b1a: Checking returncode - 128 is matched in list [128]
returncode_list_mismatch/c2b2859d: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/pass_returncode/returncode_list_mismatch/c2b2859d/stage
returncode_list_mismatch/c2b2859d: Running Test via command: bash returncode_list_mismatch_build.sh
returncode_list_mismatch/c2b2859d: failed to submit job with returncode: 2
───────────── Error Message for returncode_list_mismatch/c2b2859d ──────────────
returncode_list_mismatch/c2b2859d: Detected failure in running test, will attempt to retry test: 1 times
returncode_list_mismatch/c2b2859d: Run - 1/1
returncode_list_mismatch/c2b2859d: Running Test via command: bash returncode_list_mismatch_build.sh
returncode_list_mismatch/c2b2859d: failed to submit job with returncode: 2
returncode_list_mismatch/c2b2859d: Test completed in 0.014623 seconds with returncode: 2
returncode_list_mismatch/c2b2859d: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/pass_returncode/returncode_list_mismatch/c2b2859d/returncode_list_mismatch.out
returncode_list_mismatch/c2b2859d: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/pass_returncode/returncode_list_mismatch/c2b2859d/returncode_list_mismatch.err
returncode_list_mismatch/c2b2859d: Checking returncode - 2 is matched in list [1, 3]
exit1_fail/4f0f956d: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/pass_returncode/exit1_fail/4f0f956d/stage
exit1_fail/4f0f956d: Running Test via command: bash exit1_fail_build.sh
exit1_fail/4f0f956d: failed to submit job with returncode: 1
──────────────────── Error Message for exit1_fail/4f0f956d ─────────────────────
exit1_fail/4f0f956d: Detected failure in running test, will attempt to retry test: 1 times
exit1_fail/4f0f956d: Run - 1/1
exit1_fail/4f0f956d: Running Test via command: bash exit1_fail_build.sh
exit1_fail/4f0f956d: failed to submit job with returncode: 1
exit1_fail/4f0f956d: Test completed in 0.015435 seconds with returncode: 1
exit1_fail/4f0f956d: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/pass_returncode/exit1_fail/4f0f956d/exit1_fail.out
exit1_fail/4f0f956d: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/pass_returncode/exit1_fail/4f0f956d/exit1_fail.err
exit1_pass/62337672: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/pass_returncode/exit1_pass/62337672/stage
exit1_pass/62337672: Running Test via command: bash exit1_pass_build.sh
exit1_pass/62337672: failed to submit job with returncode: 1
──────────────────── Error Message for exit1_pass/62337672 ─────────────────────
exit1_pass/62337672: Detected failure in running test, will attempt to retry test: 1 times
exit1_pass/62337672: Run - 1/1
exit1_pass/62337672: Running Test via command: bash exit1_pass_build.sh
exit1_pass/62337672: failed to submit job with returncode: 1
exit1_pass/62337672: Test completed in 0.014779 seconds with returncode: 1
exit1_pass/62337672: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/pass_returncode/exit1_pass/62337672/exit1_pass.out
exit1_pass/62337672: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/pass_returncode/exit1_pass/62337672/exit1_pass.err
exit1_pass/62337672: Checking returncode - 1 is matched in list [1]
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ exit1_pass/62337672 │ generic.local.bash │ PASS │ 1 │ 0.015 │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ returncode_int_match/60 │ generic.local.bash │ PASS │ 128 │ 0.015 │
│ ee1b1a │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ returncode_list_mismatc │ generic.local.bash │ FAIL │ 2 │ 0.015 │
│ h/c2b2859d │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ exit1_fail/4f0f956d │ generic.local.bash │ FAIL │ 1 │ 0.015 │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 2/4 Percentage: 50.000%
Failed Tests: 2/4 Percentage: 50.000%
Adding 4 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_n5bdogvn.log
The returncode field can be an integer or list of integers but it may not accept
duplicate values. If you specify a list of exit codes, buildtest will check actual returncode
with list of expected returncodes specified by returncode field.
Shown below are examples of invalid returncodes:
# empty list is not allowed
returncode: []
# floating point is not accepted in list
returncode: [1, 1.5]
# floating point not accepted
returncode: 1.5
# duplicates are not allowed
returncode: [1, 2, 5, 5]
regex: Regular Expression on stdout/stderr
buildtest can configure PASS/FAIL of test based on regular expression on output or error file. This can be useful if you are expecting a certain output from the test as pose to returncode check.
In this example we introduce, the regex field which is part of status that expects
a regular expression via exp. The stream property must be stdout or stderr which indicates
buildtest will read output or error file and apply regular expression. If there is a match, buildtest will record the
test state as PASS otherwise it will be a FAIL. In this example, we have four tests that will apply two regular expression
on output file and two regular expression on error file.
buildspecs:
status_regex_stdout_pass:
executor: generic.local.bash
type: script
tags: [system]
description: Pass test based on regular expression
run: echo "PASS"
status:
regex:
stream: stdout
exp: "^(PASS)$"
status_regex_stdout_fail:
executor: generic.local.bash
type: script
tags: [system]
description: Pass test based on regular expression
run: echo "FAIL"
status:
regex:
stream: stdout
exp: "^(123FAIL)$"
status_regex_stderr_pass:
executor: generic.local.bash
type: script
tags: [system]
description: Pass test based on regular expression
run: echo "PASS" 1>&2
status:
regex:
stream: stderr
exp: '^(PASS)$'
status_regex_stderr_fail:
executor: generic.local.bash
type: script
tags: [system]
description: Pass test based on regular expression
run: echo "FAIL" 1>&2
status:
regex:
stream: stderr
exp: "^(123FAIL)$"
Now if we run this test, we will see first test will pass while second one will fail even though the returncode is a 0. Take a close look at the status property
buildtest build -b tutorials/test_status/status_regex.yml
$ buildtest build -b tutorials/test_status/status_regex.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:44:57 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/status_regex.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/status_regex.yml: VALID
Total builder objects created: 4
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ status_r │ script │ generic. │ None │ None │ None │ Pass │ /home/d │
│ egex_std │ │ local.ba │ │ │ │ test │ ocs/che │
│ out_pass │ │ sh │ │ │ │ based on │ ckouts/ │
│ /e87e20d │ │ │ │ │ │ regular │ readthe │
│ a │ │ │ │ │ │ expressi │ docs.or │
│ │ │ │ │ │ │ on │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/s │
│ │ │ │ │ │ │ │ tatus_r │
│ │ │ │ │ │ │ │ egex.ym │
│ │ │ │ │ │ │ │ l │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ status_r │ script │ generic. │ None │ None │ None │ Pass │ /home/d │
│ egex_std │ │ local.ba │ │ │ │ test │ ocs/che │
│ out_fail │ │ sh │ │ │ │ based on │ ckouts/ │
│ /e9f147b │ │ │ │ │ │ regular │ readthe │
│ 6 │ │ │ │ │ │ expressi │ docs.or │
│ │ │ │ │ │ │ on │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/s │
│ │ │ │ │ │ │ │ tatus_r │
│ │ │ │ │ │ │ │ egex.ym │
│ │ │ │ │ │ │ │ l │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ status_r │ script │ generic. │ None │ None │ None │ Pass │ /home/d │
│ egex_std │ │ local.ba │ │ │ │ test │ ocs/che │
│ err_pass │ │ sh │ │ │ │ based on │ ckouts/ │
│ /9547d1f │ │ │ │ │ │ regular │ readthe │
│ d │ │ │ │ │ │ expressi │ docs.or │
│ │ │ │ │ │ │ on │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/s │
│ │ │ │ │ │ │ │ tatus_r │
│ │ │ │ │ │ │ │ egex.ym │
│ │ │ │ │ │ │ │ l │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ status_r │ script │ generic. │ None │ None │ None │ Pass │ /home/d │
│ egex_std │ │ local.ba │ │ │ │ test │ ocs/che │
│ err_fail │ │ sh │ │ │ │ based on │ ckouts/ │
│ /dfae701 │ │ │ │ │ │ regular │ readthe │
│ 7 │ │ │ │ │ │ expressi │ docs.or │
│ │ │ │ │ │ │ on │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/s │
│ │ │ │ │ │ │ │ tatus_r │
│ │ │ │ │ │ │ │ egex.ym │
│ │ │ │ │ │ │ │ l │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
status_regex_stdout_pass/e87e20da: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stdout_pass/e87e20da
status_regex_stdout_fail/e9f147b6: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stdout_fail/e9f147b6
status_regex_stderr_pass/9547d1fd: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stderr_pass/9547d1fd
status_regex_stderr_fail/dfae7017: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stderr_fail/dfae7017
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
status_regex_stderr_fail/dfae7017 does not have any dependencies adding test to queue
status_regex_stdout_fail/e9f147b6 does not have any dependencies adding test to queue
status_regex_stderr_pass/9547d1fd does not have any dependencies adding test to queue
status_regex_stdout_pass/e87e20da does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ status_regex_stdout_pass/e87e20da │
│ status_regex_stdout_fail/e9f147b6 │
│ status_regex_stderr_fail/dfae7017 │
│ status_regex_stderr_pass/9547d1fd │
└───────────────────────────────────┘
status_regex_stdout_pass/e87e20da: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stdout_pass/e87e20da/stage
status_regex_stdout_pass/e87e20da: Running Test via command: bash status_regex_stdout_pass_build.sh
status_regex_stdout_pass/e87e20da: Test completed in 0.005897 seconds with returncode: 0
status_regex_stdout_pass/e87e20da: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stdout_pass/e87e20da/status_regex_stdout_pass.out
status_regex_stdout_pass/e87e20da: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stdout_pass/e87e20da/status_regex_stdout_pass.err
status_regex_stdout_pass/e87e20da: performing regular expression - '^(PASS)$' on file: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stdout_pass/e87e20da/status_regex_stdout_pass.out
status_regex_stdout_pass/e87e20da: Regular Expression Match - Success!
status_regex_stdout_fail/e9f147b6: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stdout_fail/e9f147b6/stage
status_regex_stdout_fail/e9f147b6: Running Test via command: bash status_regex_stdout_fail_build.sh
status_regex_stdout_fail/e9f147b6: Test completed in 0.005738 seconds with returncode: 0
status_regex_stdout_fail/e9f147b6: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stdout_fail/e9f147b6/status_regex_stdout_fail.out
status_regex_stdout_fail/e9f147b6: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stdout_fail/e9f147b6/status_regex_stdout_fail.err
status_regex_stdout_fail/e9f147b6: performing regular expression - '^(123FAIL)$' on file: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stdout_fail/e9f147b6/status_regex_stdout_fail.out
status_regex_stdout_fail/e9f147b6: Regular Expression Match - Failed!
status_regex_stderr_fail/dfae7017: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stderr_fail/dfae7017/stage
status_regex_stderr_fail/dfae7017: Running Test via command: bash status_regex_stderr_fail_build.sh
status_regex_stderr_fail/dfae7017: Test completed in 0.00559 seconds with returncode: 0
status_regex_stderr_fail/dfae7017: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stderr_fail/dfae7017/status_regex_stderr_fail.out
status_regex_stderr_fail/dfae7017: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stderr_fail/dfae7017/status_regex_stderr_fail.err
status_regex_stderr_fail/dfae7017: performing regular expression - '^(123FAIL)$' on file: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stderr_fail/dfae7017/status_regex_stderr_fail.err
status_regex_stderr_fail/dfae7017: Regular Expression Match - Failed!
status_regex_stderr_pass/9547d1fd: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stderr_pass/9547d1fd/stage
status_regex_stderr_pass/9547d1fd: Running Test via command: bash status_regex_stderr_pass_build.sh
status_regex_stderr_pass/9547d1fd: Test completed in 0.005757 seconds with returncode: 0
status_regex_stderr_pass/9547d1fd: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stderr_pass/9547d1fd/status_regex_stderr_pass.out
status_regex_stderr_pass/9547d1fd: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stderr_pass/9547d1fd/status_regex_stderr_pass.err
status_regex_stderr_pass/9547d1fd: performing regular expression - '^(PASS)$' on file: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/status_regex/status_regex_stderr_pass/9547d1fd/status_regex_stderr_pass.err
status_regex_stderr_pass/9547d1fd: Regular Expression Match - Success!
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ status_regex_stderr_pas │ generic.local.bash │ PASS │ 0 │ 0.006 │
│ s/9547d1fd │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ status_regex_stderr_fai │ generic.local.bash │ FAIL │ 0 │ 0.006 │
│ l/dfae7017 │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ status_regex_stdout_pas │ generic.local.bash │ PASS │ 0 │ 0.006 │
│ s/e87e20da │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ status_regex_stdout_fai │ generic.local.bash │ FAIL │ 0 │ 0.006 │
│ l/e9f147b6 │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 2/4 Percentage: 50.000%
Failed Tests: 2/4 Percentage: 50.000%
Adding 4 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_mt0n9zsm.log
file_regex: Regular Expression on files
In the previous example, we applied regular expression on output or error file; however, you may want to apply
regular expression on arbitrary files. This can be done by specifying file_regex property. The file_regex property is
an array of assertions, where each assertion must have filename and exp property. The filename property is the path
to filename and exp is the regular expression you want to apply.
In this example, we have three tests that make use of file_regex property. The first test will perform a regular expression
on multiple file names. The second test will attempt to check on a directory name which is not supported since regular expression must
be applied to file name. The third test will show that variable expansion and environment variable expansion is supported.
buildspecs:
regex_on_multiple_files:
type: script
executor: generic.local.bash
description: Test regex on multiple files
run: |
echo "Hello" > hello.txt
buildtest --help > buildtest_help.txt
status:
file_regex:
- file: hello.txt
exp: '^(Hello)$'
- file: buildtest_help.txt
exp: '^(usage: buildtest)'
regex_on_directory_not_supported:
type: script
executor: generic.local.bash
description: Test regex on directory is not supported
run: |
mkdir -p hello
echo "Hello" > hello/hello.txt
status:
file_regex:
- file: hello
exp: '^(Hello)$'
file_expansion_supported:
type: script
executor: generic.local.bash
description: Test regex with variable and shell expansion
run: |
echo "Hello" > $BUILDTEST_ROOT/hello.txt
echo "Hello" > $HOME/hello.txt
status:
file_regex:
- file: $BUILDTEST_ROOT/hello.txt
exp: '^(Hello)$'
- file: ~/hello.txt
exp: '^(Hello)$'
We can build this test by running the following command:
buildtest build -b tutorials/test_status/regex_on_filename.yml
$ buildtest build -b tutorials/test_status/regex_on_filename.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:44:57 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/regex_on_filename.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/regex_on_filename.yml: VALID
Total builder objects created: 3
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ regex_on │ script │ generic. │ None │ None │ None │ Test │ /home/d │
│ _multipl │ │ local.ba │ │ │ │ regex on │ ocs/che │
│ e_files/ │ │ sh │ │ │ │ multiple │ ckouts/ │
│ 0c9b08c8 │ │ │ │ │ │ files │ readthe │
│ │ │ │ │ │ │ │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/r │
│ │ │ │ │ │ │ │ egex_on │
│ │ │ │ │ │ │ │ _filena │
│ │ │ │ │ │ │ │ me.yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ regex_on │ script │ generic. │ None │ None │ None │ Test │ /home/d │
│ _directo │ │ local.ba │ │ │ │ regex on │ ocs/che │
│ ry_not_s │ │ sh │ │ │ │ director │ ckouts/ │
│ upported │ │ │ │ │ │ y is not │ readthe │
│ /1b2918c │ │ │ │ │ │ supporte │ docs.or │
│ a │ │ │ │ │ │ d │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/r │
│ │ │ │ │ │ │ │ egex_on │
│ │ │ │ │ │ │ │ _filena │
│ │ │ │ │ │ │ │ me.yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ file_exp │ script │ generic. │ None │ None │ None │ Test │ /home/d │
│ ansion_s │ │ local.ba │ │ │ │ regex │ ocs/che │
│ upported │ │ sh │ │ │ │ with │ ckouts/ │
│ /c90f65b │ │ │ │ │ │ variable │ readthe │
│ a │ │ │ │ │ │ and │ docs.or │
│ │ │ │ │ │ │ shell │ g/user_ │
│ │ │ │ │ │ │ expansio │ builds/ │
│ │ │ │ │ │ │ n │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/r │
│ │ │ │ │ │ │ │ egex_on │
│ │ │ │ │ │ │ │ _filena │
│ │ │ │ │ │ │ │ me.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
regex_on_multiple_files/0c9b08c8: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/regex_on_multiple_files/0c9b08c8
regex_on_directory_not_supported/1b2918ca: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/regex_on_directory_not_supported/1b2918ca
file_expansion_supported/c90f65ba: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/file_expansion_supported/c90f65ba
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
regex_on_directory_not_supported/1b2918ca does not have any dependencies adding test to queue
file_expansion_supported/c90f65ba does not have any dependencies adding test to queue
regex_on_multiple_files/0c9b08c8 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ regex_on_directory_not_supported/1b2918ca │
│ file_expansion_supported/c90f65ba │
│ regex_on_multiple_files/0c9b08c8 │
└───────────────────────────────────────────┘
regex_on_directory_not_supported/1b2918ca: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/regex_on_directory_not_supported/1b2918ca/stage
regex_on_directory_not_supported/1b2918ca: Running Test via command: bash regex_on_directory_not_supported_build.sh
regex_on_directory_not_supported/1b2918ca: Test completed in 0.007204 seconds with returncode: 0
regex_on_directory_not_supported/1b2918ca: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/regex_on_directory_not_supported/1b2918ca/regex_on_directory_not_supported.out
regex_on_directory_not_supported/1b2918ca: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/regex_on_directory_not_supported/1b2918ca/regex_on_directory_not_supported.err
regex_on_directory_not_supported/1b2918ca: File: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/regex_on_directory_not_supported/1b2918ca/stage/hello is not a file
file_expansion_supported/c90f65ba: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/file_expansion_supported/c90f65ba/stage
file_expansion_supported/c90f65ba: Running Test via command: bash file_expansion_supported_build.sh
file_expansion_supported/c90f65ba: Test completed in 0.005663 seconds with returncode: 0
file_expansion_supported/c90f65ba: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/file_expansion_supported/c90f65ba/file_expansion_supported.out
file_expansion_supported/c90f65ba: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/file_expansion_supported/c90f65ba/file_expansion_supported.err
file_expansion_supported/c90f65ba: Performing regex expression '^(Hello)$' on file /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/hello.txt
file_expansion_supported/c90f65ba: Regular expression on file /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/hello.txt is a MATCH!
file_expansion_supported/c90f65ba: Performing regex expression '^(Hello)$' on file /home/docs/hello.txt
file_expansion_supported/c90f65ba: Regular expression on file /home/docs/hello.txt is a MATCH!
regex_on_multiple_files/0c9b08c8: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/regex_on_multiple_files/0c9b08c8/stage
regex_on_multiple_files/0c9b08c8: Running Test via command: bash regex_on_multiple_files_build.sh
regex_on_multiple_files/0c9b08c8: Test completed in 0.499809 seconds with returncode: 0
regex_on_multiple_files/0c9b08c8: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/regex_on_multiple_files/0c9b08c8/regex_on_multiple_files.out
regex_on_multiple_files/0c9b08c8: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/regex_on_multiple_files/0c9b08c8/regex_on_multiple_files.err
regex_on_multiple_files/0c9b08c8: Performing regex expression '^(Hello)$' on file /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/regex_on_multiple_files/0c9b08c8/stage/hello.txt
regex_on_multiple_files/0c9b08c8: Regular expression on file /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/regex_on_multiple_files/0c9b08c8/stage/hello.txt is a MATCH!
regex_on_multiple_files/0c9b08c8: Performing regex expression '^(usage: buildtest)' on file /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/regex_on_multiple_files/0c9b08c8/stage/buildtest_help.txt
regex_on_multiple_files/0c9b08c8: Regular expression on file /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/regex_on_filename/regex_on_multiple_files/0c9b08c8/stage/buildtest_help.txt is a MATCH!
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ regex_on_directory_not_ │ generic.local.bash │ FAIL │ 0 │ 0.007 │
│ supported/1b2918ca │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ file_expansion_supporte │ generic.local.bash │ PASS │ 0 │ 0.006 │
│ d/c90f65ba │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ regex_on_multiple_files │ generic.local.bash │ PASS │ 0 │ 0.500 │
│ /0c9b08c8 │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 2/3 Percentage: 66.667%
Failed Tests: 1/3 Percentage: 33.333%
Adding 3 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_q_cybq8n.log
runtime: Passing Test based on runtime
buildtest can determine state of test based on runtime property which is part of
status object. This can be used if you want to control how test PASS or FAIL based on
execution time of test. In example below we have five tests that make use of runtime property
for passing a test. The runtime property support min and max property that can mark test
pass based on minimum and maximum runtime. A test will pass if it’s execution time is greater than min
time and less than max time. If min is specified without max property the upperbound is not set, likewise
max without min will pass if test is less than max time. The lower bound is not set, but test runtime
will be greater than 0 sec.
In test timelimit_min, we sleep for 2 seconds and it will pass because minimum runtime is 1.0 seconds. Similarly, timelimit_max will pass because we sleep for 2 seconds with a max time of 5.0.
buildspecs:
timelimit_min_max:
type: script
executor: generic.local.sh
description: "Run a sleep job for 2 seconds and test pass if its within 1.0-3.0sec"
tags: ["tutorials"]
run: sleep 2
status:
runtime:
min: 1.0
max: 3.0
timelimit_min:
type: script
executor: generic.local.sh
description: "Run a sleep job for 2 seconds and test pass if its exceeds min time of 1.0 sec"
tags: ["tutorials"]
run: sleep 2
status:
runtime:
min: 1.0
timelimit_max:
type: script
executor: generic.local.sh
description: "Run a sleep job for 2 seconds and test pass if it's within max time: 5.0 sec"
tags: ["tutorials"]
run: sleep 2
status:
runtime:
max: 5.0
timelimit_min_fail:
type: script
executor: generic.local.sh
description: "This test fails because it runs less than mintime of 10 second"
tags: ["tutorials"]
run: sleep 2
status:
runtime:
min: 10.0
timelimit_max_fail:
type: script
executor: generic.local.sh
description: "This test fails because it exceeds maxtime of 1.0 second"
tags: ["tutorials"]
run: sleep 3
status:
runtime:
max: 1.0
buildtest build -b tutorials/test_status/runtime_status_test.yml
$ buildtest build -b tutorials/test_status/runtime_status_test.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:44:59 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/runtime_status_test.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/runtime_status_test.yml: VALID
Total builder objects created: 5
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ timelimi │ script │ generic. │ None │ None │ None │ Run a │ /home/d │
│ t_min_ma │ │ local.sh │ │ │ │ sleep │ ocs/che │
│ x/3a3027 │ │ │ │ │ │ job for │ ckouts/ │
│ 59 │ │ │ │ │ │ 2 │ readthe │
│ │ │ │ │ │ │ seconds │ docs.or │
│ │ │ │ │ │ │ and test │ g/user_ │
│ │ │ │ │ │ │ pass if │ builds/ │
│ │ │ │ │ │ │ its │ buildte │
│ │ │ │ │ │ │ within │ st/chec │
│ │ │ │ │ │ │ 1.0-3.0s │ kouts/d │
│ │ │ │ │ │ │ ec │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/r │
│ │ │ │ │ │ │ │ untime_ │
│ │ │ │ │ │ │ │ status_ │
│ │ │ │ │ │ │ │ test.ym │
│ │ │ │ │ │ │ │ l │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ timelimi │ script │ generic. │ None │ None │ None │ Run a │ /home/d │
│ t_min/52 │ │ local.sh │ │ │ │ sleep │ ocs/che │
│ 9eb1ec │ │ │ │ │ │ job for │ ckouts/ │
│ │ │ │ │ │ │ 2 │ readthe │
│ │ │ │ │ │ │ seconds │ docs.or │
│ │ │ │ │ │ │ and test │ g/user_ │
│ │ │ │ │ │ │ pass if │ builds/ │
│ │ │ │ │ │ │ its │ buildte │
│ │ │ │ │ │ │ exceeds │ st/chec │
│ │ │ │ │ │ │ min time │ kouts/d │
│ │ │ │ │ │ │ of 1.0 │ evel/tu │
│ │ │ │ │ │ │ sec │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/r │
│ │ │ │ │ │ │ │ untime_ │
│ │ │ │ │ │ │ │ status_ │
│ │ │ │ │ │ │ │ test.ym │
│ │ │ │ │ │ │ │ l │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ timelimi │ script │ generic. │ None │ None │ None │ Run a │ /home/d │
│ t_max/5a │ │ local.sh │ │ │ │ sleep │ ocs/che │
│ 4b5940 │ │ │ │ │ │ job for │ ckouts/ │
│ │ │ │ │ │ │ 2 │ readthe │
│ │ │ │ │ │ │ seconds │ docs.or │
│ │ │ │ │ │ │ and test │ g/user_ │
│ │ │ │ │ │ │ pass if │ builds/ │
│ │ │ │ │ │ │ it's │ buildte │
│ │ │ │ │ │ │ within │ st/chec │
│ │ │ │ │ │ │ max │ kouts/d │
│ │ │ │ │ │ │ time: │ evel/tu │
│ │ │ │ │ │ │ 5.0 sec │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/r │
│ │ │ │ │ │ │ │ untime_ │
│ │ │ │ │ │ │ │ status_ │
│ │ │ │ │ │ │ │ test.ym │
│ │ │ │ │ │ │ │ l │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ timelimi │ script │ generic. │ None │ None │ None │ This │ /home/d │
│ t_min_fa │ │ local.sh │ │ │ │ test │ ocs/che │
│ il/4a3da │ │ │ │ │ │ fails │ ckouts/ │
│ 6a4 │ │ │ │ │ │ because │ readthe │
│ │ │ │ │ │ │ it runs │ docs.or │
│ │ │ │ │ │ │ less │ g/user_ │
│ │ │ │ │ │ │ than │ builds/ │
│ │ │ │ │ │ │ mintime │ buildte │
│ │ │ │ │ │ │ of 10 │ st/chec │
│ │ │ │ │ │ │ second │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/r │
│ │ │ │ │ │ │ │ untime_ │
│ │ │ │ │ │ │ │ status_ │
│ │ │ │ │ │ │ │ test.ym │
│ │ │ │ │ │ │ │ l │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ timelimi │ script │ generic. │ None │ None │ None │ This │ /home/d │
│ t_max_fa │ │ local.sh │ │ │ │ test │ ocs/che │
│ il/e12d6 │ │ │ │ │ │ fails │ ckouts/ │
│ ce0 │ │ │ │ │ │ because │ readthe │
│ │ │ │ │ │ │ it │ docs.or │
│ │ │ │ │ │ │ exceeds │ g/user_ │
│ │ │ │ │ │ │ maxtime │ builds/ │
│ │ │ │ │ │ │ of 1.0 │ buildte │
│ │ │ │ │ │ │ second │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/r │
│ │ │ │ │ │ │ │ untime_ │
│ │ │ │ │ │ │ │ status_ │
│ │ │ │ │ │ │ │ test.ym │
│ │ │ │ │ │ │ │ l │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
timelimit_min_max/3a302759: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_min_max/3a302759
timelimit_min/529eb1ec: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_min/529eb1ec
timelimit_max/5a4b5940: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_max/5a4b5940
timelimit_min_fail/4a3da6a4: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_min_fail/4a3da6a4
timelimit_max_fail/e12d6ce0: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_max_fail/e12d6ce0
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
timelimit_min_fail/4a3da6a4 does not have any dependencies adding test to queue
timelimit_min/529eb1ec does not have any dependencies adding test to queue
timelimit_max_fail/e12d6ce0 does not have any dependencies adding test to queue
timelimit_min_max/3a302759 does not have any dependencies adding test to queue
timelimit_max/5a4b5940 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ timelimit_min_fail/4a3da6a4 │
│ timelimit_min/529eb1ec │
│ timelimit_max_fail/e12d6ce0 │
│ timelimit_min_max/3a302759 │
│ timelimit_max/5a4b5940 │
└─────────────────────────────┘
timelimit_min_fail/4a3da6a4: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_min_fail/4a3da6a4/stage
timelimit_min_fail/4a3da6a4: Running Test via command: bash timelimit_min_fail_build.sh
timelimit_min_fail/4a3da6a4: Test completed in 2.006247 seconds with returncode: 0
timelimit_min_fail/4a3da6a4: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_min_fail/4a3da6a4/timelimit_min_fail.out
timelimit_min_fail/4a3da6a4: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_min_fail/4a3da6a4/timelimit_min_fail.err
timelimit_min_fail/4a3da6a4: Checking mintime < runtime: 10.0 < 2.006247
timelimit_min/529eb1ec: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_min/529eb1ec/stage
timelimit_min/529eb1ec: Running Test via command: bash timelimit_min_build.sh
timelimit_min/529eb1ec: Test completed in 2.006259 seconds with returncode: 0
timelimit_min/529eb1ec: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_min/529eb1ec/timelimit_min.out
timelimit_min/529eb1ec: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_min/529eb1ec/timelimit_min.err
timelimit_min/529eb1ec: Checking mintime < runtime: 1.0 < 2.006259
timelimit_max_fail/e12d6ce0: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_max_fail/e12d6ce0/stage
timelimit_max_fail/e12d6ce0: Running Test via command: bash timelimit_max_fail_build.sh
timelimit_max_fail/e12d6ce0: Test completed in 3.006265 seconds with returncode: 0
timelimit_max_fail/e12d6ce0: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_max_fail/e12d6ce0/timelimit_max_fail.out
timelimit_max_fail/e12d6ce0: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_max_fail/e12d6ce0/timelimit_max_fail.err
timelimit_max_fail/e12d6ce0: Checking runtime < maxtime: 3.006265 < 1.0
timelimit_min_max/3a302759: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_min_max/3a302759/stage
timelimit_min_max/3a302759: Running Test via command: bash timelimit_min_max_build.sh
timelimit_min_max/3a302759: Test completed in 2.006235 seconds with returncode: 0
timelimit_min_max/3a302759: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_min_max/3a302759/timelimit_min_max.out
timelimit_min_max/3a302759: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_min_max/3a302759/timelimit_min_max.err
timelimit_min_max/3a302759: Checking mintime < runtime < maxtime: 1.0 < 2.006235 < 3.0
timelimit_max/5a4b5940: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_max/5a4b5940/stage
timelimit_max/5a4b5940: Running Test via command: bash timelimit_max_build.sh
timelimit_max/5a4b5940: Test completed in 2.006189 seconds with returncode: 0
timelimit_max/5a4b5940: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_max/5a4b5940/timelimit_max.out
timelimit_max/5a4b5940: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/runtime_status_test/timelimit_max/5a4b5940/timelimit_max.err
timelimit_max/5a4b5940: Checking runtime < maxtime: 2.006189 < 5.0
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ timelimit_min/529eb1ec │ generic.local.sh │ PASS │ 0 │ 2.006 │
├───────────────────────────┼──────────────────┼────────┼────────────┼─────────┤
│ timelimit_max/5a4b5940 │ generic.local.sh │ PASS │ 0 │ 2.006 │
├───────────────────────────┼──────────────────┼────────┼────────────┼─────────┤
│ timelimit_max_fail/e12d6c │ generic.local.sh │ FAIL │ 0 │ 3.006 │
│ e0 │ │ │ │ │
├───────────────────────────┼──────────────────┼────────┼────────────┼─────────┤
│ timelimit_min_max/3a30275 │ generic.local.sh │ PASS │ 0 │ 2.006 │
│ 9 │ │ │ │ │
├───────────────────────────┼──────────────────┼────────┼────────────┼─────────┤
│ timelimit_min_fail/4a3da6 │ generic.local.sh │ FAIL │ 0 │ 2.006 │
│ a4 │ │ │ │ │
└───────────────────────────┴──────────────────┴────────┴────────────┴─────────┘
Passed Tests: 3/5 Percentage: 60.000%
Failed Tests: 2/5 Percentage: 40.000%
Adding 5 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_lrk21qn5.log
If we look at the test results, we expect the first three tests timelimit_min, timelimit_max, timelimit_min_max will pass while the last two tests fail because it fails to comply with runtime property.
buildtest report --filter buildspec=tutorials/test_status/runtime_status_test.yml --format name,id,state,runtime --latest
$ buildtest report --filter buildspec=tutorials/test_status/runtime_status_test.yml --format name,id,state,runtime --latest
Report File: /tmp/tmp0ns2hqkt/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ name ┃ id ┃ state ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ timelimit_min │ 529eb1ec │ PASS │ 2.006259 │
├────────────────────────────────┼────────────────┼────────────┼───────────────┤
│ timelimit_max │ 5a4b5940 │ PASS │ 2.006189 │
├────────────────────────────────┼────────────────┼────────────┼───────────────┤
│ timelimit_max_fail │ e12d6ce0 │ FAIL │ 3.006265 │
├────────────────────────────────┼────────────────┼────────────┼───────────────┤
│ timelimit_min_max │ 3a302759 │ PASS │ 2.006235 │
├────────────────────────────────┼────────────────┼────────────┼───────────────┤
│ timelimit_min_fail │ 4a3da6a4 │ FAIL │ 2.006247 │
└────────────────────────────────┴────────────────┴────────────┴───────────────┘
state: Explicitly Declare Status of Test
You can explicitly define status of test regardless of what buildtest does for checking status of test. This
can be useful if you want to explicitly mark a test as PASS or FAIL regardless of how test behaves. This can be done via
state property which expects one of two types PASS or FAIL. If state property is specified, buildtest will ignore any checks
including returncode, regex, or runtime match.
In this next example we will demonstrate how one can use state property for marking test state. In this
example we have four tests. The first test always_pass will PASS even though we have a non-zero returncode. The
second test always_fail will FAIL even though it has a 0 returncode. The last two test demonstrate how one
can define state regardless of what is specified for returncode match. buildtest will honor the state property even if
their is a match on the returncode.
buildspecs:
always_pass:
type: script
executor: 'generic.local.sh'
description: This test will always 'PASS'
run: exit 1
status:
state: PASS
always_fail:
type: script
executor: 'generic.local.sh'
description: This test will always 'FAIL'
run: exit 0
status:
state: FAIL
test_fail_returncode_match:
type: script
executor: 'generic.local.sh'
description: This test will 'FAIL' even if we have returncode match
run: exit 1
status:
state: FAIL
returncode: 1
test_pass_returncode_mismatch:
type: script
executor: 'generic.local.sh'
description: This test will 'PASS' even if we have returncode mismatch
run: exit 1
status:
state: PASS
returncode: 2
If we build this test, we expect buildtest to honor the value of state property
buildtest build -b tutorials/test_status/explicit_state.yml
$ buildtest build -b tutorials/test_status/explicit_state.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:45:11 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/explicit_state.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/explicit_state.yml: VALID
Total builder objects created: 4
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ always_p │ script │ generic. │ None │ None │ None │ This │ /home/d │
│ ass/ab25 │ │ local.sh │ │ │ │ test │ ocs/che │
│ 7a5f │ │ │ │ │ │ will │ ckouts/ │
│ │ │ │ │ │ │ always │ readthe │
│ │ │ │ │ │ │ 'PASS' │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/e │
│ │ │ │ │ │ │ │ xplicit │
│ │ │ │ │ │ │ │ _state. │
│ │ │ │ │ │ │ │ yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ always_f │ script │ generic. │ None │ None │ None │ This │ /home/d │
│ ail/44ff │ │ local.sh │ │ │ │ test │ ocs/che │
│ 55e2 │ │ │ │ │ │ will │ ckouts/ │
│ │ │ │ │ │ │ always │ readthe │
│ │ │ │ │ │ │ 'FAIL' │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/e │
│ │ │ │ │ │ │ │ xplicit │
│ │ │ │ │ │ │ │ _state. │
│ │ │ │ │ │ │ │ yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ test_fai │ script │ generic. │ None │ None │ None │ This │ /home/d │
│ l_return │ │ local.sh │ │ │ │ test │ ocs/che │
│ code_mat │ │ │ │ │ │ will │ ckouts/ │
│ ch/2d197 │ │ │ │ │ │ 'FAIL' │ readthe │
│ f97 │ │ │ │ │ │ even if │ docs.or │
│ │ │ │ │ │ │ we have │ g/user_ │
│ │ │ │ │ │ │ returnco │ builds/ │
│ │ │ │ │ │ │ de match │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/e │
│ │ │ │ │ │ │ │ xplicit │
│ │ │ │ │ │ │ │ _state. │
│ │ │ │ │ │ │ │ yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ test_pas │ script │ generic. │ None │ None │ None │ This │ /home/d │
│ s_return │ │ local.sh │ │ │ │ test │ ocs/che │
│ code_mis │ │ │ │ │ │ will │ ckouts/ │
│ match/38 │ │ │ │ │ │ 'PASS' │ readthe │
│ a9b8cb │ │ │ │ │ │ even if │ docs.or │
│ │ │ │ │ │ │ we have │ g/user_ │
│ │ │ │ │ │ │ returnco │ builds/ │
│ │ │ │ │ │ │ de │ buildte │
│ │ │ │ │ │ │ mismatch │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/e │
│ │ │ │ │ │ │ │ xplicit │
│ │ │ │ │ │ │ │ _state. │
│ │ │ │ │ │ │ │ yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
always_pass/ab257a5f: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/explicit_state/always_pass/ab257a5f
always_fail/44ff55e2: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/explicit_state/always_fail/44ff55e2
test_fail_returncode_match/2d197f97: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/explicit_state/test_fail_returncode_match/2d197f97
test_pass_returncode_mismatch/38a9b8cb: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/explicit_state/test_pass_returncode_mismatch/38a9b8cb
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
always_pass/ab257a5f does not have any dependencies adding test to queue
always_fail/44ff55e2 does not have any dependencies adding test to queue
test_fail_returncode_match/2d197f97 does not have any dependencies adding test to queue
test_pass_returncode_mismatch/38a9b8cb does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ always_pass/ab257a5f │
│ always_fail/44ff55e2 │
│ test_fail_returncode_match/2d197f97 │
│ test_pass_returncode_mismatch/38a9b8cb │
└────────────────────────────────────────┘
always_pass/ab257a5f: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/explicit_state/always_pass/ab257a5f/stage
always_pass/ab257a5f: Running Test via command: bash always_pass_build.sh
always_pass/ab257a5f: failed to submit job with returncode: 1
──────────────────── Error Message for always_pass/ab257a5f ────────────────────
always_pass/ab257a5f: Detected failure in running test, will attempt to retry test: 1 times
always_pass/ab257a5f: Run - 1/1
always_pass/ab257a5f: Running Test via command: bash always_pass_build.sh
always_pass/ab257a5f: failed to submit job with returncode: 1
always_pass/ab257a5f: Test completed in 0.013108 seconds with returncode: 1
always_pass/ab257a5f: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/explicit_state/always_pass/ab257a5f/always_pass.out
always_pass/ab257a5f: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/explicit_state/always_pass/ab257a5f/always_pass.err
always_fail/44ff55e2: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/explicit_state/always_fail/44ff55e2/stage
always_fail/44ff55e2: Running Test via command: bash always_fail_build.sh
always_fail/44ff55e2: Test completed in 0.004791 seconds with returncode: 0
always_fail/44ff55e2: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/explicit_state/always_fail/44ff55e2/always_fail.out
always_fail/44ff55e2: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/explicit_state/always_fail/44ff55e2/always_fail.err
test_fail_returncode_match/2d197f97: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/explicit_state/test_fail_returncode_match/2d197f97/stage
test_fail_returncode_match/2d197f97: Running Test via command: bash test_fail_returncode_match_build.sh
test_fail_returncode_match/2d197f97: failed to submit job with returncode: 1
──────────── Error Message for test_fail_returncode_match/2d197f97 ─────────────
test_fail_returncode_match/2d197f97: Detected failure in running test, will attempt to retry test: 1 times
test_fail_returncode_match/2d197f97: Run - 1/1
test_fail_returncode_match/2d197f97: Running Test via command: bash test_fail_returncode_match_build.sh
test_fail_returncode_match/2d197f97: failed to submit job with returncode: 1
test_fail_returncode_match/2d197f97: Test completed in 0.013004 seconds with returncode: 1
test_fail_returncode_match/2d197f97: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/explicit_state/test_fail_returncode_match/2d197f97/test_fail_returncode_match.out
test_fail_returncode_match/2d197f97: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/explicit_state/test_fail_returncode_match/2d197f97/test_fail_returncode_match.err
test_pass_returncode_mismatch/38a9b8cb: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/explicit_state/test_pass_returncode_mismatch/38a9b8cb/stage
test_pass_returncode_mismatch/38a9b8cb: Running Test via command: bash test_pass_returncode_mismatch_build.sh
test_pass_returncode_mismatch/38a9b8cb: failed to submit job with returncode: 1
─────────── Error Message for test_pass_returncode_mismatch/38a9b8cb ───────────
test_pass_returncode_mismatch/38a9b8cb: Detected failure in running test, will attempt to retry test: 1 times
test_pass_returncode_mismatch/38a9b8cb: Run - 1/1
test_pass_returncode_mismatch/38a9b8cb: Running Test via command: bash test_pass_returncode_mismatch_build.sh
test_pass_returncode_mismatch/38a9b8cb: failed to submit job with returncode: 1
test_pass_returncode_mismatch/38a9b8cb: Test completed in 0.013138 seconds with returncode: 1
test_pass_returncode_mismatch/38a9b8cb: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/explicit_state/test_pass_returncode_mismatch/38a9b8cb/test_pass_returncode_mismatch.out
test_pass_returncode_mismatch/38a9b8cb: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.sh/explicit_state/test_pass_returncode_mismatch/38a9b8cb/test_pass_returncode_mismatch.err
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ test_pass_returncode_mism │ generic.local.sh │ PASS │ 1 │ 0.013 │
│ atch/38a9b8cb │ │ │ │ │
├───────────────────────────┼──────────────────┼────────┼────────────┼─────────┤
│ test_fail_returncode_matc │ generic.local.sh │ FAIL │ 1 │ 0.013 │
│ h/2d197f97 │ │ │ │ │
├───────────────────────────┼──────────────────┼────────┼────────────┼─────────┤
│ always_pass/ab257a5f │ generic.local.sh │ PASS │ 1 │ 0.013 │
├───────────────────────────┼──────────────────┼────────┼────────────┼─────────┤
│ always_fail/44ff55e2 │ generic.local.sh │ FAIL │ 0 │ 0.005 │
└───────────────────────────┴──────────────────┴────────┴────────────┴─────────┘
Passed Tests: 2/4 Percentage: 50.000%
Failed Tests: 2/4 Percentage: 50.000%
Adding 4 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_ee3y1o61.log
File Checks
buildtest supports various file checks that can be used as means for passing test. This can include checking for exists: File Existence, is_file, is_dir: File and Directory Checks, is_symlink: Symbolic Link Check, and file_count: File Count.
exists: File Existence
For instance, if you want to check for file existence, you can use exists property
which expects a list of file or directory names to check. This can be useful if your test
will write some output file or directory and test will pass based on existence of file/directory.
In the example below we have two tests, first test will pass, where all files exist. We check for files and directory path, note variable and shell expansion is supported.
In the second example, we expect this test to fail because filename bar does not exist.
buildspecs:
status_exists:
type: script
executor: generic.local.bash
description: status check based for file and directory
run: |
mkdir -p $HOME/dirA
mkdir -p /tmp/ABC
touch file1
status:
exists:
- $HOME/dirA
- ~/.bashrc
- /tmp/ABC
- file1
status_exists_failure:
type: script
executor: generic.local.bash
description: status check failure for existence
run: touch foo
status:
exists:
- bar
We can run this test by running the following, take note of the output.
buildtest build -b tutorials/test_status/exists.yml
$ buildtest build -b tutorials/test_status/exists.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:45:12 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/exists.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/exists.yml: VALID
Total builder objects created: 2
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ status_e │ script │ generic. │ None │ None │ None │ status │ /home/d │
│ xists/4e │ │ local.ba │ │ │ │ check │ ocs/che │
│ bbfcc9 │ │ sh │ │ │ │ based │ ckouts/ │
│ │ │ │ │ │ │ for file │ readthe │
│ │ │ │ │ │ │ and │ docs.or │
│ │ │ │ │ │ │ director │ g/user_ │
│ │ │ │ │ │ │ y │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/e │
│ │ │ │ │ │ │ │ xists.y │
│ │ │ │ │ │ │ │ ml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ status_e │ script │ generic. │ None │ None │ None │ status │ /home/d │
│ xists_fa │ │ local.ba │ │ │ │ check │ ocs/che │
│ ilure/30 │ │ sh │ │ │ │ failure │ ckouts/ │
│ 5f1cf6 │ │ │ │ │ │ for │ readthe │
│ │ │ │ │ │ │ existenc │ docs.or │
│ │ │ │ │ │ │ e │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/e │
│ │ │ │ │ │ │ │ xists.y │
│ │ │ │ │ │ │ │ ml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
status_exists/4ebbfcc9: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/exists/status_exists/4ebbfcc9
status_exists_failure/305f1cf6: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/exists/status_exists_failure/305f1cf6
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
status_exists_failure/305f1cf6 does not have any dependencies adding test to queue
status_exists/4ebbfcc9 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ status_exists/4ebbfcc9 │
│ status_exists_failure/305f1cf6 │
└────────────────────────────────┘
status_exists/4ebbfcc9: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/exists/status_exists/4ebbfcc9/stage
status_exists/4ebbfcc9: Running Test via command: bash status_exists_build.sh
status_exists/4ebbfcc9: Test completed in 0.009231 seconds with returncode: 0
status_exists/4ebbfcc9: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/exists/status_exists/4ebbfcc9/status_exists.out
status_exists/4ebbfcc9: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/exists/status_exists/4ebbfcc9/status_exists.err
status_exists/4ebbfcc9: Test all files: ['$HOME/dirA', '~/.bashrc', '/tmp/ABC', 'file1'] existences
status_exists/4ebbfcc9: file: /home/docs/dirA exists
status_exists/4ebbfcc9: file: /home/docs/.bashrc exists
status_exists/4ebbfcc9: file: /tmp/ABC exists
status_exists/4ebbfcc9: file: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/exists/status_exists/4ebbfcc9/stage/file1 exists
status_exists/4ebbfcc9: Exist Check: True
status_exists_failure/305f1cf6: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/exists/status_exists_failure/305f1cf6/stage
status_exists_failure/305f1cf6: Running Test via command: bash status_exists_failure_build.sh
status_exists_failure/305f1cf6: Test completed in 0.006795 seconds with returncode: 0
status_exists_failure/305f1cf6: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/exists/status_exists_failure/305f1cf6/status_exists_failure.out
status_exists_failure/305f1cf6: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/exists/status_exists_failure/305f1cf6/status_exists_failure.err
status_exists_failure/305f1cf6: Test all files: ['bar'] existences
status_exists_failure/305f1cf6: file: bar does not exist
status_exists_failure/305f1cf6: Exist Check: False
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ status_exists_failure/3 │ generic.local.bash │ FAIL │ 0 │ 0.007 │
│ 05f1cf6 │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ status_exists/4ebbfcc9 │ generic.local.bash │ PASS │ 0 │ 0.009 │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 1/2 Percentage: 50.000%
Failed Tests: 1/2 Percentage: 50.000%
Adding 2 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_9m0b9__m.log
Each item in the exists field must be a string, which can lead to issue in example
below let’s assume we want a test to pass based on a directory name 1, if we specify
as follows, this test will fail validation.
buildspecs:
file_exists_failure:
type: script
executor: generic.local.bash
description: this test will fail validation, because item must be a string
run: mkdir -p 1
status:
exists: [1]
We can validate this buildspec by running the following
buildtest bc validate -b tutorials/test_status/file_exists_exception.yml
$ buildtest bc validate -b tutorials/test_status/file_exists_exception.yml ─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… ─ 1 is not of type 'string' Failed validating 'type' in schema['properties']['status']['properties']['exists']['items']: {'type': 'string'} On instance['status']['exists'][0]: 1 1 buildspecs failed to validate
In order to run this test, we need to enclose each item in quotes. Shown below is the same test with quotations.
buildspecs:
file_exists_pass:
type: script
executor: generic.local.bash
description: this test will pass
run: mkdir -p 1
status:
exists: [ '1' ]
Let’s validate and build this test.
buildtest bc validate -b tutorials/test_status/file_exists_with_number.yml
$ buildtest bc validate -b tutorials/test_status/file_exists_with_number.yml
buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/file_exists_with_number.yml is valid
All buildspecs passed validation!!!
buildtest build -b tutorials/test_status/file_exists_with_number.yml
$ buildtest build -b tutorials/test_status/file_exists_with_number.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:45:13 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/file_exists_with_number.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/file_exists_with_number.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ file_exi │ script │ generic. │ None │ None │ None │ this │ /home/d │
│ sts_pass │ │ local.ba │ │ │ │ test │ ocs/che │
│ /2795dfa │ │ sh │ │ │ │ will │ ckouts/ │
│ 4 │ │ │ │ │ │ pass │ readthe │
│ │ │ │ │ │ │ │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/f │
│ │ │ │ │ │ │ │ ile_exi │
│ │ │ │ │ │ │ │ sts_wit │
│ │ │ │ │ │ │ │ h_numbe │
│ │ │ │ │ │ │ │ r.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
file_exists_pass/2795dfa4: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/2795dfa4
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
file_exists_pass/2795dfa4 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ file_exists_pass/2795dfa4 │
└───────────────────────────┘
file_exists_pass/2795dfa4: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/2795dfa4/stage
file_exists_pass/2795dfa4: Running Test via command: bash file_exists_pass_build.sh
file_exists_pass/2795dfa4: Test completed in 0.007276 seconds with returncode: 0
file_exists_pass/2795dfa4: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/2795dfa4/file_exists_pass.out
file_exists_pass/2795dfa4: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/2795dfa4/file_exists_pass.err
file_exists_pass/2795dfa4: Test all files: ['1'] existences
file_exists_pass/2795dfa4: file: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/2795dfa4/stage/1 exists
file_exists_pass/2795dfa4: Exist Check: True
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ file_exists_pass/2795df │ generic.local.bash │ PASS │ 0 │ 0.007 │
│ a4 │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_69dsgy85.log
is_file, is_dir: File and Directory Checks
In the next example, we introduce checks for files and directory via is_file and
is_dir property, which behaves similar to exists except they will check if each item
is a file or directory. We expect the first test to fail, because $HOME/.bashrc is
not a directory but a file. The second test will incorporate the same test and
use is_file for status check.
buildspecs:
file_and_dir_checks:
type: script
executor: generic.local.bash
description: status check for files and directories
run: hostname
status:
is_dir:
- $HOME
- $HOME/.bashrc
- /tmp
combined_file_and_dir_checks:
type: script
executor: generic.local.bash
description: status check for files and directories
run: hostname
status:
is_dir:
- $HOME
- /tmp
is_file:
- $HOME/.bashrc
Let’s build the test and see the output.
buildtest build -b tutorials/test_status/file_and_dir_check.yml
$ buildtest build -b tutorials/test_status/file_and_dir_check.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:45:14 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/file_and_dir_check.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/file_and_dir_check.yml: VALID
Total builder objects created: 2
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ file_and │ script │ generic. │ None │ None │ None │ status │ /home/d │
│ _dir_che │ │ local.ba │ │ │ │ check │ ocs/che │
│ cks/bc80 │ │ sh │ │ │ │ for │ ckouts/ │
│ 32df │ │ │ │ │ │ files │ readthe │
│ │ │ │ │ │ │ and │ docs.or │
│ │ │ │ │ │ │ director │ g/user_ │
│ │ │ │ │ │ │ ies │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/f │
│ │ │ │ │ │ │ │ ile_and │
│ │ │ │ │ │ │ │ _dir_ch │
│ │ │ │ │ │ │ │ eck.yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ combined │ script │ generic. │ None │ None │ None │ status │ /home/d │
│ _file_an │ │ local.ba │ │ │ │ check │ ocs/che │
│ d_dir_ch │ │ sh │ │ │ │ for │ ckouts/ │
│ ecks/06b │ │ │ │ │ │ files │ readthe │
│ 4c097 │ │ │ │ │ │ and │ docs.or │
│ │ │ │ │ │ │ director │ g/user_ │
│ │ │ │ │ │ │ ies │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/f │
│ │ │ │ │ │ │ │ ile_and │
│ │ │ │ │ │ │ │ _dir_ch │
│ │ │ │ │ │ │ │ eck.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
file_and_dir_checks/bc8032df: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_and_dir_check/file_and_dir_checks/bc8032df
combined_file_and_dir_checks/06b4c097: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_and_dir_check/combined_file_and_dir_checks/06b4c097
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
combined_file_and_dir_checks/06b4c097 does not have any dependencies adding test to queue
file_and_dir_checks/bc8032df does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ combined_file_and_dir_checks/06b4c097 │
│ file_and_dir_checks/bc8032df │
└───────────────────────────────────────┘
combined_file_and_dir_checks/06b4c097: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_and_dir_check/combined_file_and_dir_checks/06b4c097/stage
combined_file_and_dir_checks/06b4c097: Running Test via command: bash combined_file_and_dir_checks_build.sh
combined_file_and_dir_checks/06b4c097: Test completed in 0.007129 seconds with returncode: 0
combined_file_and_dir_checks/06b4c097: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_and_dir_check/combined_file_and_dir_checks/06b4c097/combined_file_and_dir_checks.out
combined_file_and_dir_checks/06b4c097: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_and_dir_check/combined_file_and_dir_checks/06b4c097/combined_file_and_dir_checks.err
combined_file_and_dir_checks/06b4c097: Test all files: ['$HOME', '/tmp'] existences
combined_file_and_dir_checks/06b4c097: file: /home/docs is a directory
combined_file_and_dir_checks/06b4c097: file: /tmp is a directory
combined_file_and_dir_checks/06b4c097: Directory Existence Check: True
combined_file_and_dir_checks/06b4c097: Test all files: ['$HOME/.bashrc'] existences
combined_file_and_dir_checks/06b4c097: file: /home/docs/.bashrc is a file
combined_file_and_dir_checks/06b4c097: File Existence Check: True
file_and_dir_checks/bc8032df: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_and_dir_check/file_and_dir_checks/bc8032df/stage
file_and_dir_checks/bc8032df: Running Test via command: bash file_and_dir_checks_build.sh
file_and_dir_checks/bc8032df: Test completed in 0.00716 seconds with returncode: 0
file_and_dir_checks/bc8032df: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_and_dir_check/file_and_dir_checks/bc8032df/file_and_dir_checks.out
file_and_dir_checks/bc8032df: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_and_dir_check/file_and_dir_checks/bc8032df/file_and_dir_checks.err
file_and_dir_checks/bc8032df: Test all files: ['$HOME', '$HOME/.bashrc', '/tmp'] existences
file_and_dir_checks/bc8032df: file: /home/docs is a directory
file_and_dir_checks/bc8032df: file: $HOME/.bashrc is not a directory
file_and_dir_checks/bc8032df: file: /tmp is a directory
file_and_dir_checks/bc8032df: Directory Existence Check: False
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ combined_file_and_dir_c │ generic.local.bash │ PASS │ 0 │ 0.007 │
│ hecks/06b4c097 │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ file_and_dir_checks/bc8 │ generic.local.bash │ FAIL │ 0 │ 0.007 │
│ 032df │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 1/2 Percentage: 50.000%
Failed Tests: 1/2 Percentage: 50.000%
Adding 2 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_kk8tn11d.log
is_symlink: Symbolic Link Check
buildtest can configure PASS/FAIL of test based on the status of symbolic link. This can be useful if your test will create a symbolic link to a file or directory and test will pass if the symbolic link is present.
You can use the is_symlink property which expects a list of values that are checked for symbolic links. In the example below, the test will pass as all the values are valid symbolic links and are not broken. Note that variable and shell expansion
is supported.
buildspecs:
symlink_test:
type: script
executor: generic.local.bash
description: status check based on symbolic link
run: |
ln -s /tmp scratch
ln -s $HOME/.bashrc $HOME/.bashrc_link
status:
is_symlink:
- scratch
- $HOME/.bashrc_link
- ~/.bashrc_link
We can run this test by running the following.
buildtest build -b tutorials/test_status/is_symlink.yml
$ buildtest build -b tutorials/test_status/is_symlink.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:45:15 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/is_symlink.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/is_symlink.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ symlink_ │ script │ generic. │ None │ None │ None │ status │ /home/d │
│ test/c23 │ │ local.ba │ │ │ │ check │ ocs/che │
│ 770ff │ │ sh │ │ │ │ based on │ ckouts/ │
│ │ │ │ │ │ │ symbolic │ readthe │
│ │ │ │ │ │ │ link │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/i │
│ │ │ │ │ │ │ │ s_symli │
│ │ │ │ │ │ │ │ nk.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
symlink_test/c23770ff: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/is_symlink/symlink_test/c23770ff
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
symlink_test/c23770ff does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━┩
│ symlink_test/c23770ff │
└───────────────────────┘
symlink_test/c23770ff: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/is_symlink/symlink_test/c23770ff/stage
symlink_test/c23770ff: Running Test via command: bash symlink_test_build.sh
symlink_test/c23770ff: Test completed in 0.047699 seconds with returncode: 0
symlink_test/c23770ff: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/is_symlink/symlink_test/c23770ff/symlink_test.out
symlink_test/c23770ff: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/is_symlink/symlink_test/c23770ff/symlink_test.err
symlink_test/c23770ff: Check all items: ['scratch', '$HOME/.bashrc_link', '~/.bashrc_link'] for symbolic links
symlink_test/c23770ff: scratch is a symbolic link to /tmp
symlink_test/c23770ff: $HOME/.bashrc_link is a symbolic link to /home/docs/.bashrc
symlink_test/c23770ff: ~/.bashrc_link is a symbolic link to /home/docs/.bashrc
symlink_test/c23770ff: Symlink Check: True
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ symlink_test/c23770ff │ generic.local.bash │ PASS │ 0 │ 0.048 │
└───────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_lptnfbmn.log
file_count: File Count
buildtest can check for number of files in a directory. This can be useful if your test writes number of files and you
want to check if the number of files is as expected. You can use the file_count property to perform file count. This
property is a list of assertion, where each item is an object. The dir and count are required keys.
The dir is the path to directory to perform directory traversal, and count key is the number of expected files that will be
used for comparison. In the first test, we perform a directory walk and expect 5 files in the directory. We can perform directory
search based on file extension by using ext key. The ext property can be a string or a list. The second test will perform
directory walk on directory named foo and search for file extensions .sh, .py, .txt. The depth property controls
the maximum depth for directory traversal, this must be of an integer type. The depth property is optional and if not specified, we will
perform full directory traversal.
buildspecs:
file_count_on_directory:
type: script
executor: generic.local.bash
description: file count check in directory
run: |
mkdir -p foo
touch foo/{1..5}
status:
file_count:
- dir: foo
count: 5
file_count_by_extension:
type: script
executor: generic.local.bash
description: file count by extension
run: |
mkdir -p foo/bar
touch foo/{1..5}.sh
touch foo/bar/{1..3}.py foo/bar/{1..3}.txt
status:
file_count:
- dir: foo
ext: '.sh'
depth: 1
count: 5
- dir: foo/bar
ext: ['.py', '.txt']
count: 6
We can run this test by running the following.
buildtest build -b tutorials/test_status/file_count.yml
$ buildtest build -b tutorials/test_status/file_count.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:45:15 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/file_count.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/file_count.yml: VALID
Total builder objects created: 2
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ file_cou │ script │ generic. │ None │ None │ None │ file │ /home/d │
│ nt_on_di │ │ local.ba │ │ │ │ count │ ocs/che │
│ rectory/ │ │ sh │ │ │ │ check in │ ckouts/ │
│ 71e367e1 │ │ │ │ │ │ director │ readthe │
│ │ │ │ │ │ │ y │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/f │
│ │ │ │ │ │ │ │ ile_cou │
│ │ │ │ │ │ │ │ nt.yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ file_cou │ script │ generic. │ None │ None │ None │ file │ /home/d │
│ nt_by_ex │ │ local.ba │ │ │ │ count by │ ocs/che │
│ tension/ │ │ sh │ │ │ │ extensio │ ckouts/ │
│ 1f1fdb6f │ │ │ │ │ │ n │ readthe │
│ │ │ │ │ │ │ │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/f │
│ │ │ │ │ │ │ │ ile_cou │
│ │ │ │ │ │ │ │ nt.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
file_count_on_directory/71e367e1: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count/file_count_on_directory/71e367e1
file_count_by_extension/1f1fdb6f: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count/file_count_by_extension/1f1fdb6f
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
file_count_on_directory/71e367e1 does not have any dependencies adding test to queue
file_count_by_extension/1f1fdb6f does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ file_count_on_directory/71e367e1 │
│ file_count_by_extension/1f1fdb6f │
└──────────────────────────────────┘
file_count_on_directory/71e367e1: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count/file_count_on_directory/71e367e1/stage
file_count_on_directory/71e367e1: Running Test via command: bash file_count_on_directory_build.sh
file_count_on_directory/71e367e1: Test completed in 0.008067 seconds with returncode: 0
file_count_on_directory/71e367e1: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count/file_count_on_directory/71e367e1/file_count_on_directory.out
file_count_on_directory/71e367e1: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count/file_count_on_directory/71e367e1/file_count_on_directory.err
file_count_on_directory/71e367e1: Found 5 file in directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count/file_count_on_directory/71e367e1/stage/foo. Comparing with reference count: 5. Comparison check is 5 == 5 which evaluates to True
file_count_on_directory/71e367e1: File Count Check: True
file_count_by_extension/1f1fdb6f: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count/file_count_by_extension/1f1fdb6f/stage
file_count_by_extension/1f1fdb6f: Running Test via command: bash file_count_by_extension_build.sh
file_count_by_extension/1f1fdb6f: Test completed in 0.008786 seconds with returncode: 0
file_count_by_extension/1f1fdb6f: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count/file_count_by_extension/1f1fdb6f/file_count_by_extension.out
file_count_by_extension/1f1fdb6f: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count/file_count_by_extension/1f1fdb6f/file_count_by_extension.err
file_count_by_extension/1f1fdb6f: Found 5 file in directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count/file_count_by_extension/1f1fdb6f/stage/foo. Comparing with reference count: 5. Comparison check is 5 == 5 which evaluates to True
file_count_by_extension/1f1fdb6f: Found 6 file in directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count/file_count_by_extension/1f1fdb6f/stage/foo/bar. Comparing with reference count: 6. Comparison check is 6 == 6 which evaluates to True
file_count_by_extension/1f1fdb6f: File Count Check: True
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ file_count_on_directory │ generic.local.bash │ PASS │ 0 │ 0.008 │
│ /71e367e1 │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ file_count_by_extension │ generic.local.bash │ PASS │ 0 │ 0.009 │
│ /1f1fdb6f │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%
Adding 2 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_62mzqyz3.log
In the next example, we introduce filepattern property which allows you to check for files based on a pattern. The filepattern property
is a regular expression which is compiled via re.compile and applied to a
directory path. Please note the regular expression must be valid, otherwise buildtest will not return any files during directory traversal.
You can use filepattern and ext property together to search for files. If both are specified, then we will search for files
with both methods and join the two list prior to performing comparison.
buildspecs:
file_count_by_expression:
type: script
executor: generic.local.bash
description: file count by expression
run: |
mkdir -p /tmp
touch /tmp/foo{1..5}.txt
ls -l /tmp/foo*.txt
filenames=$(find $BUILDTEST_ROOT/buildtest -type f \( -name "defaults.py" -o -name "main.py" \) -maxdepth 1)
totalfiles=$(find $BUILDTEST_ROOT/buildtest -type f \( -name "defaults.py" -o -name "main.py" \) -maxdepth 1 | wc -l)
echo "Filenames: $filenames"
echo "Total files: $totalfiles"
status:
file_count:
- dir: /tmp
filepattern: 'foo[1-5].txt$'
count: 5
- dir: '$BUILDTEST_ROOT/buildtest'
filepattern: '(defaults|main).py$'
count: 2
depth: 1
file_extension_and_filepattern:
type: script
executor: generic.local.bash
description: file count by file extension and file pattern
run: |
touch foo{1..5}.txt
touch {conf,main}.py
ls -l
status:
file_count:
- dir: .
ext: '.txt'
filepattern: '(conf|main).py$'
count: 7
Let’s build this test by running the following:
buildtest build -b tutorials/test_status/file_count_pattern.yml
$ buildtest build -b tutorials/test_status/file_count_pattern.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:45:16 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/file_count_pattern.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/file_count_pattern.yml: VALID
Total builder objects created: 2
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ file_cou │ script │ generic. │ None │ None │ None │ file │ /home/d │
│ nt_by_ex │ │ local.ba │ │ │ │ count by │ ocs/che │
│ pression │ │ sh │ │ │ │ expressi │ ckouts/ │
│ /949f95c │ │ │ │ │ │ on │ readthe │
│ 2 │ │ │ │ │ │ │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/f │
│ │ │ │ │ │ │ │ ile_cou │
│ │ │ │ │ │ │ │ nt_patt │
│ │ │ │ │ │ │ │ ern.yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ file_ext │ script │ generic. │ None │ None │ None │ file │ /home/d │
│ ension_a │ │ local.ba │ │ │ │ count by │ ocs/che │
│ nd_filep │ │ sh │ │ │ │ file │ ckouts/ │
│ attern/f │ │ │ │ │ │ extensio │ readthe │
│ 068d9a2 │ │ │ │ │ │ n and │ docs.or │
│ │ │ │ │ │ │ file │ g/user_ │
│ │ │ │ │ │ │ pattern │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/f │
│ │ │ │ │ │ │ │ ile_cou │
│ │ │ │ │ │ │ │ nt_patt │
│ │ │ │ │ │ │ │ ern.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
file_count_by_expression/949f95c2: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_pattern/file_count_by_expression/949f95c2
file_extension_and_filepattern/f068d9a2: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_pattern/file_extension_and_filepattern/f068d9a2
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
file_count_by_expression/949f95c2 does not have any dependencies adding test to queue
file_extension_and_filepattern/f068d9a2 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ file_count_by_expression/949f95c2 │
│ file_extension_and_filepattern/f068d9a2 │
└─────────────────────────────────────────┘
file_count_by_expression/949f95c2: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_pattern/file_count_by_expression/949f95c2/stage
file_count_by_expression/949f95c2: Running Test via command: bash file_count_by_expression_build.sh
file_count_by_expression/949f95c2: Test completed in 0.014478 seconds with returncode: 0
file_count_by_expression/949f95c2: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_pattern/file_count_by_expression/949f95c2/file_count_by_expression.out
file_count_by_expression/949f95c2: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_pattern/file_count_by_expression/949f95c2/file_count_by_expression.err
file_count_by_expression/949f95c2: Found 5 file in directory: /tmp. Comparing with reference count: 5. Comparison check is 5 == 5 which evaluates to True
file_count_by_expression/949f95c2: Found 2 file in directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/buildtest. Comparing with reference count: 2. Comparison check is 2 == 2 which evaluates to True
file_count_by_expression/949f95c2: File Count Check: True
file_extension_and_filepattern/f068d9a2: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_pattern/file_extension_and_filepattern/f068d9a2/stage
file_extension_and_filepattern/f068d9a2: Running Test via command: bash file_extension_and_filepattern_build.sh
file_extension_and_filepattern/f068d9a2: Test completed in 0.009638 seconds with returncode: 0
file_extension_and_filepattern/f068d9a2: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_pattern/file_extension_and_filepattern/f068d9a2/file_extension_and_filepattern.out
file_extension_and_filepattern/f068d9a2: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_pattern/file_extension_and_filepattern/f068d9a2/file_extension_and_filepattern.err
file_extension_and_filepattern/f068d9a2: Found 7 file in directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_pattern/file_extension_and_filepattern/f068d9a2/stage. Comparing with reference count: 7. Comparison check is 7 == 7 which evaluates to True
file_extension_and_filepattern/f068d9a2: File Count Check: True
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ file_extension_and_file │ generic.local.bash │ PASS │ 0 │ 0.010 │
│ pattern/f068d9a2 │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ file_count_by_expressio │ generic.local.bash │ PASS │ 0 │ 0.014 │
│ n/949f95c2 │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%
Adding 2 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_nswv5cf6.log
In the next example, we will introduce filetype property that can be used to filter directory search based on file type.
The filetype property can one of the following values file, dir, symlink. Once set, the directory traversal will
seek out files based on the file type. Note that when filetype is set to dir we will return the parent directory and
all sub-directories. This test will create a few subdirectories and create symbolic link, next we will perform directory search
by directory and symbolic link. We expect this test to pass as we will find 3 directories and 2 symbolic links.
buildspecs:
file_count_by_filetype:
type: script
executor: generic.local.bash
description: Count the number of directories and symbolic links
run: |
mkdir -p foo/{bar,baz}
find foo -type dir
ln -s foo/bar foo/bar.link
ln -s foo/baz foo/baz.link
status:
file_count:
- dir: foo
count: 3
filetype: 'dir'
- dir: foo
count: 2
filetype: 'symlink'
Let’s build this test by running the following:
buildtest build -b tutorials/test_status/file_count_filetype.yml
$ buildtest build -b tutorials/test_status/file_count_filetype.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:45:17 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/file_count_filetype.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/file_count_filetype.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ file_cou │ script │ generic. │ None │ None │ None │ Count │ /home/d │
│ nt_by_fi │ │ local.ba │ │ │ │ the │ ocs/che │
│ letype/e │ │ sh │ │ │ │ number │ ckouts/ │
│ 60136cd │ │ │ │ │ │ of │ readthe │
│ │ │ │ │ │ │ director │ docs.or │
│ │ │ │ │ │ │ ies and │ g/user_ │
│ │ │ │ │ │ │ symbolic │ builds/ │
│ │ │ │ │ │ │ links │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/f │
│ │ │ │ │ │ │ │ ile_cou │
│ │ │ │ │ │ │ │ nt_file │
│ │ │ │ │ │ │ │ type.ym │
│ │ │ │ │ │ │ │ l │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
file_count_by_filetype/e60136cd: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_filetype/file_count_by_filetype/e60136cd
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
file_count_by_filetype/e60136cd does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ file_count_by_filetype/e60136cd │
└─────────────────────────────────┘
file_count_by_filetype/e60136cd: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_filetype/file_count_by_filetype/e60136cd/stage
file_count_by_filetype/e60136cd: Running Test via command: bash file_count_by_filetype_build.sh
file_count_by_filetype/e60136cd: failed to submit job with returncode: 1
────────────── Error Message for file_count_by_filetype/e60136cd ───────────────
find: Must separate multiple arguments to -type using: ','
file_count_by_filetype/e60136cd: Detected failure in running test, will attempt to retry test: 1 times
file_count_by_filetype/e60136cd: Run - 1/1
file_count_by_filetype/e60136cd: Running Test via command: bash file_count_by_filetype_build.sh
file_count_by_filetype/e60136cd: failed to submit job with returncode: 1
file_count_by_filetype/e60136cd: Test completed in 0.020145 seconds with returncode: 1
file_count_by_filetype/e60136cd: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_filetype/file_count_by_filetype/e60136cd/file_count_by_filetype.out
file_count_by_filetype/e60136cd: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_filetype/file_count_by_filetype/e60136cd/file_count_by_filetype.err
file_count_by_filetype/e60136cd: Found 3 file in directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_filetype/file_count_by_filetype/e60136cd/stage/foo. Comparing with reference count: 3. Comparison check is 3 == 3 which evaluates to True
file_count_by_filetype/e60136cd: Found 0 file in directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_filetype/file_count_by_filetype/e60136cd/stage/foo. Comparing with reference count: 2. Comparison check is 0 == 2 which evaluates to False
file_count_by_filetype/e60136cd: File Count Check: False
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ file_count_by_filetype/ │ generic.local.bash │ FAIL │ 1 │ 0.020 │
│ e60136cd │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 0/1 Percentage: 0.000%
Failed Tests: 1/1 Percentage: 100.000%
Adding 1 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_ly3g6zkz.log
Buildtest will perform a directory walk when using file_count, which can run into performance issues if you have a large directory.
We have added a safety check during directory traversal to a maximum of 999999 files. You have the option to configure the directory
traversal limit using file_traversal_limit which is an integer, the default value is 10000 if not specified. The minimum value and
maximum value can be 1 and 999999 respectively.
In this next example, we will illustrate how this feature works. We will create 99 .txt files in directory foo. We will
perform two assertions with different values for file_traversal_limit. In the first example, we will set this to 50 and
expect 50 files returned. We expect this check to be True. In the next example, we will set file_traversal_limit to 20 and
set count: 10 where we should expect a failure. In principle, we should retrieve 20 files but this will mismatch the comparison check.
buildspecs:
file_traverse_limit:
type: script
executor: generic.local.bash
description: Use of file_traverse_limit to limit number of files searched in a directory
run: |
mkdir foo
touch foo/{1..99}.txt
status:
file_count:
- dir: foo
count: 50
file_traverse_limit: 50
- dir: foo
count: 10
file_traverse_limit: 20
We can try building this test by running the following:
buildtest build -b tutorials/test_status/file_count_file_traverse_limit.yml
$ buildtest build -b tutorials/test_status/file_count_file_traverse_limit.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:45:18 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/file_count_file_traverse_limit.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/file_count_file_traverse_limit.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ file_tra │ script │ generic. │ None │ None │ None │ Use of │ /home/d │
│ verse_li │ │ local.ba │ │ │ │ file_tra │ ocs/che │
│ mit/1d5c │ │ sh │ │ │ │ verse_li │ ckouts/ │
│ c175 │ │ │ │ │ │ mit to │ readthe │
│ │ │ │ │ │ │ limit │ docs.or │
│ │ │ │ │ │ │ number │ g/user_ │
│ │ │ │ │ │ │ of files │ builds/ │
│ │ │ │ │ │ │ searched │ buildte │
│ │ │ │ │ │ │ in a │ st/chec │
│ │ │ │ │ │ │ director │ kouts/d │
│ │ │ │ │ │ │ y │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/f │
│ │ │ │ │ │ │ │ ile_cou │
│ │ │ │ │ │ │ │ nt_file │
│ │ │ │ │ │ │ │ _traver │
│ │ │ │ │ │ │ │ se_limi │
│ │ │ │ │ │ │ │ t.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
file_traverse_limit/1d5cc175: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_file_traverse_limit/file_traverse_limit/1d5cc175
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
file_traverse_limit/1d5cc175 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ file_traverse_limit/1d5cc175 │
└──────────────────────────────┘
file_traverse_limit/1d5cc175: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_file_traverse_limit/file_traverse_limit/1d5cc175/stage
file_traverse_limit/1d5cc175: Running Test via command: bash file_traverse_limit_build.sh
file_traverse_limit/1d5cc175: Test completed in 0.009816 seconds with returncode: 0
file_traverse_limit/1d5cc175: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_file_traverse_limit/file_traverse_limit/1d5cc175/file_traverse_limit.out
file_traverse_limit/1d5cc175: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_file_traverse_limit/file_traverse_limit/1d5cc175/file_traverse_limit.err
file_traverse_limit/1d5cc175: Found 50 file in directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_file_traverse_limit/file_traverse_limit/1d5cc175/stage/foo. Comparing with reference count: 50. Comparison check is 50 == 50 which evaluates to True
file_traverse_limit/1d5cc175: Found 20 file in directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_count_file_traverse_limit/file_traverse_limit/1d5cc175/stage/foo. Comparing with reference count: 10. Comparison check is 20 == 10 which evaluates to False
file_traverse_limit/1d5cc175: File Count Check: False
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ file_traverse_limit/1d5 │ generic.local.bash │ FAIL │ 0 │ 0.010 │
│ cc175 │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 0/1 Percentage: 0.000%
Failed Tests: 1/1 Percentage: 100.000%
Adding 1 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_wk_xdpt4.log
mode: Change Status Check Behavior to Logical AND/OR
By default, the status check performed by buildtest is a logical OR, where if any of the status check is True, then the test will
PASS. However, if you want to change this behavior to logical AND, you can use the mode property. The valid values can be
[AND, and, OR, or]. In the example below, we have two tests that illustrate the use of mode.
buildspecs:
status_logical_and:
type: script
executor: 'generic.local.bash'
description: 'Using logical AND to check status'
run: |
echo "This is a test"
exit 1
status:
mode: and
returncode: 1
regex:
stream: stdout
exp: 'This is a test'
status_logical_or:
type: script
executor: 'generic.local.bash'
description: 'Using logical OR to check status'
run: |
echo "This is a test"
exit 1
status:
mode: or
returncode: 0
regex:
stream: stdout
exp: 'This is a test'
The first test uses mode: and which implies all status check are evaluated as logical AND, we expect this test to PASS.
In the second test, we use mode: or where status check are evaluated as logical OR which is the default behavior. Note if mode
is not specified, it is equivalent to mode: or. In second test, we expect this to pass because regex check will PASS however,
the returncode check will fail due to mismatch in returncode. If we changed this to mode: and then we would expect this test
to fail.
Shown below is the output of running this test.
buildtest build -b tutorials/test_status/mode.yml
$ buildtest build -b tutorials/test_status/mode.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:45:18 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/mode.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/mode.yml: VALID
Total builder objects created: 2
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ status_l │ script │ generic. │ None │ None │ None │ Using │ /home/d │
│ ogical_a │ │ local.ba │ │ │ │ logical │ ocs/che │
│ nd/4f533 │ │ sh │ │ │ │ AND to │ ckouts/ │
│ bce │ │ │ │ │ │ check │ readthe │
│ │ │ │ │ │ │ status │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/m │
│ │ │ │ │ │ │ │ ode.yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ status_l │ script │ generic. │ None │ None │ None │ Using │ /home/d │
│ ogical_o │ │ local.ba │ │ │ │ logical │ ocs/che │
│ r/032027 │ │ sh │ │ │ │ OR to │ ckouts/ │
│ a8 │ │ │ │ │ │ check │ readthe │
│ │ │ │ │ │ │ status │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/m │
│ │ │ │ │ │ │ │ ode.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
status_logical_and/4f533bce: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/mode/status_logical_and/4f533bce
status_logical_or/032027a8: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/mode/status_logical_or/032027a8
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
status_logical_and/4f533bce does not have any dependencies adding test to queue
status_logical_or/032027a8 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ status_logical_and/4f533bce │
│ status_logical_or/032027a8 │
└─────────────────────────────┘
status_logical_and/4f533bce: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/mode/status_logical_and/4f533bce/stage
status_logical_and/4f533bce: Running Test via command: bash status_logical_and_build.sh
status_logical_and/4f533bce: failed to submit job with returncode: 1
──────────────── Error Message for status_logical_and/4f533bce ─────────────────
status_logical_and/4f533bce: Detected failure in running test, will attempt to retry test: 1 times
status_logical_and/4f533bce: Run - 1/1
status_logical_and/4f533bce: Running Test via command: bash status_logical_and_build.sh
status_logical_and/4f533bce: failed to submit job with returncode: 1
status_logical_and/4f533bce: Test completed in 0.014938 seconds with returncode: 1
status_logical_and/4f533bce: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/mode/status_logical_and/4f533bce/status_logical_and.out
status_logical_and/4f533bce: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/mode/status_logical_and/4f533bce/status_logical_and.err
status_logical_and/4f533bce: Checking returncode - 1 is matched in list [1]
status_logical_and/4f533bce: performing regular expression - 'This is a test' on file: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/mode/status_logical_and/4f533bce/status_logical_and.out
status_logical_and/4f533bce: Regular Expression Match - Success!
status_logical_or/032027a8: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/mode/status_logical_or/032027a8/stage
status_logical_or/032027a8: Running Test via command: bash status_logical_or_build.sh
status_logical_or/032027a8: failed to submit job with returncode: 1
───────────────── Error Message for status_logical_or/032027a8 ─────────────────
status_logical_or/032027a8: Detected failure in running test, will attempt to retry test: 1 times
status_logical_or/032027a8: Run - 1/1
status_logical_or/032027a8: Running Test via command: bash status_logical_or_build.sh
status_logical_or/032027a8: failed to submit job with returncode: 1
status_logical_or/032027a8: Test completed in 0.014677 seconds with returncode: 1
status_logical_or/032027a8: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/mode/status_logical_or/032027a8/status_logical_or.out
status_logical_or/032027a8: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/mode/status_logical_or/032027a8/status_logical_or.err
status_logical_or/032027a8: Checking returncode - 1 is matched in list [0]
status_logical_or/032027a8: performing regular expression - 'This is a test' on file: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/mode/status_logical_or/032027a8/status_logical_or.out
status_logical_or/032027a8: Regular Expression Match - Success!
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ status_logical_or/03202 │ generic.local.bash │ PASS │ 1 │ 0.015 │
│ 7a8 │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ status_logical_and/4f53 │ generic.local.bash │ PASS │ 1 │ 0.015 │
│ 3bce │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%
Adding 2 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_ms472ju1.log
linecount: Line Count on Stdout/Stderr
This check counts the number of lines in stdout or stderr and compares with reference count. This can be useful if you want to make a test pass based on number of lines. buildtest will split lines by newline character and perform the comparison.
In the example below we will write 10 lines to standard output (stdout) and use the linecount status check to perform a comparison using the
reference count of 10. The linecount is an object which requires the fields: stream and count. The stream can be stdout or stderr
and count is the reference count to compare against. This must be an integer value and greater than 0. In second test we will compare output
with standard error (stream: stderr) and compare with reference count of 5. Since the output is going to write 10 lines to error file we
should expect the second test to fail.
buildspecs:
linecount_stdout:
type: script
executor: generic.local.bash
description: "Write 10 lines to stdout and run linecount check"
run: |
for i in {1..10}; do
echo $i
done
status:
linecount:
stream: stdout
count: 10
linecount_stderr_mismatch:
type: script
executor: generic.local.bash
description: "Write 10 lines to stderr and run linecount check"
run: |
for i in {1..10}; do
echo $i >&2
done
status:
linecount:
stream: stderr
count: 5
Let’s try building this test and you will see the test will pass based on line count comparison.
buildtest build -b tutorials/test_status/linecount.yml
$ buildtest build -b tutorials/test_status/linecount.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:45:19 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/linecount.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/linecount.yml: VALID
Total builder objects created: 2
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ linecoun │ script │ generic. │ None │ None │ None │ Write 10 │ /home/d │
│ t_stdout │ │ local.ba │ │ │ │ lines to │ ocs/che │
│ /4b31f18 │ │ sh │ │ │ │ stdout │ ckouts/ │
│ 9 │ │ │ │ │ │ and run │ readthe │
│ │ │ │ │ │ │ linecoun │ docs.or │
│ │ │ │ │ │ │ t check │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/l │
│ │ │ │ │ │ │ │ inecoun │
│ │ │ │ │ │ │ │ t.yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ linecoun │ script │ generic. │ None │ None │ None │ Write 10 │ /home/d │
│ t_stderr │ │ local.ba │ │ │ │ lines to │ ocs/che │
│ _mismatc │ │ sh │ │ │ │ stderr │ ckouts/ │
│ h/cdb062 │ │ │ │ │ │ and run │ readthe │
│ 14 │ │ │ │ │ │ linecoun │ docs.or │
│ │ │ │ │ │ │ t check │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/l │
│ │ │ │ │ │ │ │ inecoun │
│ │ │ │ │ │ │ │ t.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
linecount_stdout/4b31f189: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stdout/4b31f189
linecount_stderr_mismatch/cdb06214: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stderr_mismatch/cdb06214
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
linecount_stdout/4b31f189 does not have any dependencies adding test to queue
linecount_stderr_mismatch/cdb06214 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ linecount_stdout/4b31f189 │
│ linecount_stderr_mismatch/cdb06214 │
└────────────────────────────────────┘
linecount_stdout/4b31f189: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stdout/4b31f189/stage
linecount_stdout/4b31f189: Running Test via command: bash linecount_stdout_build.sh
linecount_stdout/4b31f189: Test completed in 0.005926 seconds with returncode: 0
linecount_stdout/4b31f189: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stdout/4b31f189/linecount_stdout.out
linecount_stdout/4b31f189: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stdout/4b31f189/linecount_stdout.err
linecount_stdout/4b31f189: Performing line count check on file: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stdout/4b31f189/linecount_stdout.out with 10 (ref count) == 10 (actual count). linecount Check: True
linecount_stderr_mismatch/cdb06214: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stderr_mismatch/cdb06214/stage
linecount_stderr_mismatch/cdb06214: Running Test via command: bash linecount_stderr_mismatch_build.sh
linecount_stderr_mismatch/cdb06214: Test completed in 0.005703 seconds with returncode: 0
linecount_stderr_mismatch/cdb06214: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stderr_mismatch/cdb06214/linecount_stderr_mismatch.out
linecount_stderr_mismatch/cdb06214: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stderr_mismatch/cdb06214/linecount_stderr_mismatch.err
linecount_stderr_mismatch/cdb06214: Performing line count check on file: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stderr_mismatch/cdb06214/linecount_stderr_mismatch.err with 5 (ref count) == 10 (actual count). linecount Check: False
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ linecount_stdout/4b31f1 │ generic.local.bash │ PASS │ 0 │ 0.006 │
│ 89 │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ linecount_stderr_mismat │ generic.local.bash │ FAIL │ 0 │ 0.006 │
│ ch/cdb06214 │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 1/2 Percentage: 50.000%
Failed Tests: 1/2 Percentage: 50.000%
Adding 2 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_wg25xf_p.log
We can inspect the output and error file for each corresponding test and we will notice that both tests will contain 10 lines of output.
$ buildtest inspect query -o linecount_stdout
──────────── linecount_stdout/4b31f189-b8dc-490f-a30e-1a95ba9486df ─────────────
Executor: generic.local.bash
Description: Write 10 lines to stdout and run linecount check
State: PASS
Returncode: 0
Runtime: 0.005926 sec
Starttime: 2024/04/25 15:45:19
Endtime: 2024/04/25 15:45:19
Command: bash linecount_stdout_build.sh
Test Script: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stdout/4b31f189/linecount_stdout.sh
Build Script: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stdout/4b31f189/linecount_stdout_build.sh
Output File: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stdout/4b31f189/linecount_stdout.out
Error File: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stdout/4b31f189/linecount_stdout.err
Log File: /tmp/tmp0ns2hqkt/var/logs/buildtest_wg25xf_p.log
─ Output File: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/lineco… ─
1
2
3
4
5
6
7
8
9
10
$ buildtest inspect query -e linecount_stderr_mismatch
──────── linecount_stderr_mismatch/cdb06214-3385-4a2e-bd77-cb35eb7fa3f8 ────────
Executor: generic.local.bash
Description: Write 10 lines to stderr and run linecount check
State: FAIL
Returncode: 0
Runtime: 0.005703 sec
Starttime: 2024/04/25 15:45:19
Endtime: 2024/04/25 15:45:19
Command: bash linecount_stderr_mismatch_build.sh
Test Script: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stderr_mismatch/cdb06214/linecount_stderr_mismatch.sh
Build Script: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stderr_mismatch/cdb06214/linecount_stderr_mismatch_build.sh
Output File: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stderr_mismatch/cdb06214/linecount_stderr_mismatch.out
Error File: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecount_stderr_mismatch/cdb06214/linecount_stderr_mismatch.err
Log File: /tmp/tmp0ns2hqkt/var/logs/buildtest_wg25xf_p.log
─ Error File: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/linecount/linecou… ─
1
2
3
4
5
6
7
8
9
10
file_linecount: Line Count on File
The file_linecount status check is similar to linecount: Line Count on Stdout/Stderr but instead of comparing output from stdout or stderr,
it compares output from a file. This is useful if you want to compare output from a file with a reference count.
The file_linecount is an array of objects which requires the fields: file and count.
The file is the path to file and count is the reference count to compare against.
To demonstrate this feature we will write 10 lines to a file named count.txt and write output of one echo statement to file hello.txt.
We will use file_linecount field to read both files and compare with reference count. We expect both checks to pass.
buildspecs:
file_linecount:
type: script
executor: generic.local.bash
description: "Perform linecount comparison on files"
run: |
for i in {1..10}; do
echo $i >> count.txt
done
echo "hello world" > hello.txt
status:
file_linecount:
- file: count.txt
count: 10
- file: hello.txt
count: 1
Let’s try building this test and you will see in the build output comparison check is performed on both files. Buildtest will perform a logical AND operation when comparing multiple files for linecount match.
buildtest build -b tutorials/test_status/file_linecount.yml
$ buildtest build -b tutorials/test_status/file_linecount.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:45:21 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/file_linecount.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/file_linecount.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ file_lin │ script │ generic. │ None │ None │ None │ Perform │ /home/d │
│ ecount/f │ │ local.ba │ │ │ │ linecoun │ ocs/che │
│ 143ea70 │ │ sh │ │ │ │ t │ ckouts/ │
│ │ │ │ │ │ │ comparis │ readthe │
│ │ │ │ │ │ │ on on │ docs.or │
│ │ │ │ │ │ │ files │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/f │
│ │ │ │ │ │ │ │ ile_lin │
│ │ │ │ │ │ │ │ ecount. │
│ │ │ │ │ │ │ │ yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
file_linecount/f143ea70: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_linecount/file_linecount/f143ea70
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
file_linecount/f143ea70 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ file_linecount/f143ea70 │
└─────────────────────────┘
file_linecount/f143ea70: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_linecount/file_linecount/f143ea70/stage
file_linecount/f143ea70: Running Test via command: bash file_linecount_build.sh
file_linecount/f143ea70: Test completed in 0.006293 seconds with returncode: 0
file_linecount/f143ea70: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_linecount/file_linecount/f143ea70/file_linecount.out
file_linecount/f143ea70: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_linecount/file_linecount/f143ea70/file_linecount.err
file_linecount/f143ea70: Performing line count check on file: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_linecount/file_linecount/f143ea70/stage/count.txt with 10 (ref count) == 10 (actual count). linecount Check: True
file_linecount/f143ea70: Performing line count check on file: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_linecount/file_linecount/f143ea70/stage/hello.txt with 1 (ref count) == 1 (actual count). linecount Check: True
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ file_linecount/f143ea70 │ generic.local.bash │ PASS │ 0 │ 0.006 │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_ofrgfxj2.log
The file property must be a valid file that is readable, if you specify a directory name or non-existent file then buildtest will
be unable to read the file, hence test will fail to perform comparison. In this next example, we perform a comparison on file: /tmp and
file: badfile.txt both of which are invalid for file comparison.
buildspecs:
file_linecount_exceptions:
type: script
executor: generic.local.bash
description: "Performing file count on directory or invalid files can result in failure"
run: |
for i in {1..10}; do
echo $i >> count.txt
done
echo "hello world" > hello.txt
status:
file_linecount:
- file: /tmp
count: 10
- file: /badfile.txt
count: 1
Buildtest will be able to run the test; however, the test will fail because its unable to read the files to get actual line count for comparison.
buildtest build -b tutorials/test_status/file_linecount_failure.yml
$ buildtest build -b tutorials/test_status/file_linecount_failure.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:45:21 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/file_linecount_failure.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/file_linecount_failure.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ file_lin │ script │ generic. │ None │ None │ None │ Performi │ /home/d │
│ ecount_e │ │ local.ba │ │ │ │ ng file │ ocs/che │
│ xception │ │ sh │ │ │ │ count on │ ckouts/ │
│ s/20e587 │ │ │ │ │ │ director │ readthe │
│ 26 │ │ │ │ │ │ y or │ docs.or │
│ │ │ │ │ │ │ invalid │ g/user_ │
│ │ │ │ │ │ │ files │ builds/ │
│ │ │ │ │ │ │ can │ buildte │
│ │ │ │ │ │ │ result │ st/chec │
│ │ │ │ │ │ │ in │ kouts/d │
│ │ │ │ │ │ │ failure │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/f │
│ │ │ │ │ │ │ │ ile_lin │
│ │ │ │ │ │ │ │ ecount_ │
│ │ │ │ │ │ │ │ failure │
│ │ │ │ │ │ │ │ .yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
file_linecount_exceptions/20e58726: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_linecount_failure/file_linecount_exceptions/20e58726
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
file_linecount_exceptions/20e58726 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ file_linecount_exceptions/20e58726 │
└────────────────────────────────────┘
file_linecount_exceptions/20e58726: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_linecount_failure/file_linecount_exceptions/20e58726/stage
file_linecount_exceptions/20e58726: Running Test via command: bash file_linecount_exceptions_build.sh
file_linecount_exceptions/20e58726: Test completed in 0.006169 seconds with returncode: 0
file_linecount_exceptions/20e58726: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_linecount_failure/file_linecount_exceptions/20e58726/file_linecount_exceptions.out
file_linecount_exceptions/20e58726: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/file_linecount_failure/file_linecount_exceptions/20e58726/file_linecount_exceptions.err
file_linecount_exceptions/20e58726: File: /tmp is not a file
file_linecount_exceptions/20e58726: Unable to resolve file path: /badfile.txt
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ file_linecount_exceptio │ generic.local.bash │ FAIL │ 0 │ 0.006 │
│ ns/20e58726 │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 0/1 Percentage: 0.000%
Failed Tests: 1/1 Percentage: 100.000%
Adding 1 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_sp0nduve.log
Finally, we must note that line count is expected to be 0 or greater, if you specify a negative value for count
then buildtest will raise an error according to the JSON schema.
buildspecs:
file_linecount_negative_value:
type: script
executor: generic.local.bash
description: "Performing file count with a negative count value will result in error"
run: |
> empty.txt
status:
file_linecount:
- file: empty.txt
count: -1
We will simply try validating this buildspec and you will see the error message from the JSON schema
buildtest buildspec validate -b tutorials/test_status/file_linecount_invalid.yml
$ buildtest buildspec validate -b tutorials/test_status/file_linecount_invalid.yml ─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… ─ -1 is less than the minimum of 0 Failed validating 'minimum' in schema['properties']['status']['properties']['file_linecount']['items']['properties']['count']: {'description': 'Specify number of lines as reference value to compare ' 'with actual output', 'minimum': 0, 'type': 'integer'} On instance['status']['file_linecount'][0]['count']: -1 1 buildspecs failed to validate
re: Modify regular expression method
The re property can be used to select the type of regular expression to use with regex or file_regex
which can be re.search, re.match
or re.fullmatch. The re property is a string type which is used to select the
regular expression function to use. In this next example, we will demonstrate the use of this feature with both regex and file_regex.
The re property is optional and if not specified it defaults to re.search.
Since re.search will search for text at any position in the string, the first test re.search.stdout will match the
string is with the output. In the second test re.match.stdout we use re.match which matches from beginning of
string with input pattern is with output. We expect this match to FAIL since the output starts with This is ….
In the third test re.fullmatch.stdout we set re: re.fullmatch which will match the entire string with the pattern.
We expect this match to PASS since the output and pattern are exactly the same. In the fourth test match_on_file_regex we have
have three regular expression, one for each type re.search, re.match and re.fullmatch. All of these expressions will find a match
and this test will PASS.
buildspecs:
re.search.stdout:
executor: generic.local.bash
type: script
description: Test re.search on stdout
run: echo "This is a string"
status:
regex:
stream: stdout
exp: 'is'
re: "re.search"
re.match.stdout:
executor: generic.local.bash
type: script
description: Test re.match on stdout
run: echo "This is a string"
status:
regex:
stream: stdout
exp: 'is'
re: "re.match"
re.fullmatch.stdout:
executor: generic.local.bash
type: script
description: Test re.fullmatch on stdout
run: echo "This is a string"
status:
regex:
stream: stdout
exp: 'This is a string'
re: "re.fullmatch"
re.match_on_file_regex:
executor: generic.local.bash
type: script
description: Test re.match on file regex
run: |
echo "This is a string" > file.txt
echo "Hello World" > hello.txt
status:
file_regex:
- file: file.txt
exp: 'string'
re: "re.search"
- file: hello.txt
exp: 'Hello'
re: "re.match"
- file: hello.txt
exp: 'Hello World'
re: "re.fullmatch"
Let’s try running this test example and see the generated output, all test should pass with exception of re.match.stdout.
buildtest build -b tutorials/test_status/specify_regex_type.yml
$ buildtest build -b tutorials/test_status/specify_regex_type.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:45:23 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/test_status/specify_regex_type.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/test_status/specify_regex_type.yml: VALID
Total builder objects created: 4
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ re.searc │ script │ generic. │ None │ None │ None │ Test │ /home/d │
│ h.stdout │ │ local.ba │ │ │ │ re.searc │ ocs/che │
│ /9e0d38e │ │ sh │ │ │ │ h on │ ckouts/ │
│ 9 │ │ │ │ │ │ stdout │ readthe │
│ │ │ │ │ │ │ │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/s │
│ │ │ │ │ │ │ │ pecify_ │
│ │ │ │ │ │ │ │ regex_t │
│ │ │ │ │ │ │ │ ype.yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ re.match │ script │ generic. │ None │ None │ None │ Test │ /home/d │
│ .stdout/ │ │ local.ba │ │ │ │ re.match │ ocs/che │
│ 6c7deb90 │ │ sh │ │ │ │ on │ ckouts/ │
│ │ │ │ │ │ │ stdout │ readthe │
│ │ │ │ │ │ │ │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/s │
│ │ │ │ │ │ │ │ pecify_ │
│ │ │ │ │ │ │ │ regex_t │
│ │ │ │ │ │ │ │ ype.yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ re.fullm │ script │ generic. │ None │ None │ None │ Test │ /home/d │
│ atch.std │ │ local.ba │ │ │ │ re.fullm │ ocs/che │
│ out/cfc8 │ │ sh │ │ │ │ atch on │ ckouts/ │
│ 7ca8 │ │ │ │ │ │ stdout │ readthe │
│ │ │ │ │ │ │ │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/s │
│ │ │ │ │ │ │ │ pecify_ │
│ │ │ │ │ │ │ │ regex_t │
│ │ │ │ │ │ │ │ ype.yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ re.match │ script │ generic. │ None │ None │ None │ Test │ /home/d │
│ _on_file │ │ local.ba │ │ │ │ re.match │ ocs/che │
│ _regex/4 │ │ sh │ │ │ │ on file │ ckouts/ │
│ 9531359 │ │ │ │ │ │ regex │ readthe │
│ │ │ │ │ │ │ │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /test_s │
│ │ │ │ │ │ │ │ tatus/s │
│ │ │ │ │ │ │ │ pecify_ │
│ │ │ │ │ │ │ │ regex_t │
│ │ │ │ │ │ │ │ ype.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
re.search.stdout/9e0d38e9: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.search.stdout/9e0d38e9
re.match.stdout/6c7deb90: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.match.stdout/6c7deb90
re.fullmatch.stdout/cfc87ca8: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.fullmatch.stdout/cfc87ca8
re.match_on_file_regex/49531359: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.match_on_file_regex/49531359
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
re.match.stdout/6c7deb90 does not have any dependencies adding test to queue
re.search.stdout/9e0d38e9 does not have any dependencies adding test to queue
re.fullmatch.stdout/cfc87ca8 does not have any dependencies adding test to queue
re.match_on_file_regex/49531359 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ re.match.stdout/6c7deb90 │
│ re.search.stdout/9e0d38e9 │
│ re.fullmatch.stdout/cfc87ca8 │
│ re.match_on_file_regex/49531359 │
└─────────────────────────────────┘
re.match.stdout/6c7deb90: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.match.stdout/6c7deb90/stage
re.match.stdout/6c7deb90: Running Test via command: bash re.match.stdout_build.sh
re.match.stdout/6c7deb90: Test completed in 0.005874 seconds with returncode: 0
re.match.stdout/6c7deb90: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.match.stdout/6c7deb90/re.match.stdout.out
re.match.stdout/6c7deb90: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.match.stdout/6c7deb90/re.match.stdout.err
re.match.stdout/6c7deb90: performing regular expression - 'is' on file: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.match.stdout/6c7deb90/re.match.stdout.out
re.match.stdout/6c7deb90: Regular Expression Match - Failed!
re.search.stdout/9e0d38e9: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.search.stdout/9e0d38e9/stage
re.search.stdout/9e0d38e9: Running Test via command: bash re.search.stdout_build.sh
re.search.stdout/9e0d38e9: Test completed in 0.005682 seconds with returncode: 0
re.search.stdout/9e0d38e9: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.search.stdout/9e0d38e9/re.search.stdout.out
re.search.stdout/9e0d38e9: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.search.stdout/9e0d38e9/re.search.stdout.err
re.search.stdout/9e0d38e9: performing regular expression - 'is' on file: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.search.stdout/9e0d38e9/re.search.stdout.out
re.search.stdout/9e0d38e9: Regular Expression Match - Success!
re.fullmatch.stdout/cfc87ca8: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.fullmatch.stdout/cfc87ca8/stage
re.fullmatch.stdout/cfc87ca8: Running Test via command: bash re.fullmatch.stdout_build.sh
re.fullmatch.stdout/cfc87ca8: Test completed in 0.005603 seconds with returncode: 0
re.fullmatch.stdout/cfc87ca8: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.fullmatch.stdout/cfc87ca8/re.fullmatch.stdout.out
re.fullmatch.stdout/cfc87ca8: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.fullmatch.stdout/cfc87ca8/re.fullmatch.stdout.err
re.fullmatch.stdout/cfc87ca8: performing regular expression - 'This is a string' on file: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.fullmatch.stdout/cfc87ca8/re.fullmatch.stdout.out
re.fullmatch.stdout/cfc87ca8: Regular Expression Match - Success!
re.match_on_file_regex/49531359: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.match_on_file_regex/49531359/stage
re.match_on_file_regex/49531359: Running Test via command: bash re.match_on_file_regex_build.sh
re.match_on_file_regex/49531359: Test completed in 0.00562 seconds with returncode: 0
re.match_on_file_regex/49531359: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.match_on_file_regex/49531359/re.match_on_file_regex.out
re.match_on_file_regex/49531359: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.match_on_file_regex/49531359/re.match_on_file_regex.err
re.match_on_file_regex/49531359: Performing regex expression 'string' on file /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.match_on_file_regex/49531359/stage/file.txt
re.match_on_file_regex/49531359: Regular expression on file /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.match_on_file_regex/49531359/stage/file.txt is a MATCH!
re.match_on_file_regex/49531359: Performing regex expression 'Hello' on file /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.match_on_file_regex/49531359/stage/hello.txt
re.match_on_file_regex/49531359: Regular expression on file /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.match_on_file_regex/49531359/stage/hello.txt is a MATCH!
re.match_on_file_regex/49531359: Performing regex expression 'Hello World' on file /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.match_on_file_regex/49531359/stage/hello.txt
re.match_on_file_regex/49531359: Regular expression on file /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/specify_regex_type/re.match_on_file_regex/49531359/stage/hello.txt is a MATCH!
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ re.fullmatch.stdout/cfc │ generic.local.bash │ PASS │ 0 │ 0.006 │
│ 87ca8 │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ re.search.stdout/9e0d38 │ generic.local.bash │ PASS │ 0 │ 0.006 │
│ e9 │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ re.match_on_file_regex/ │ generic.local.bash │ PASS │ 0 │ 0.006 │
│ 49531359 │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ re.match.stdout/6c7deb9 │ generic.local.bash │ FAIL │ 0 │ 0.006 │
│ 0 │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 3/4 Percentage: 75.000%
Failed Tests: 1/4 Percentage: 25.000%
Adding 4 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_t4w51l35.log