Buildtest Command Line Tutorial¶
For this session, we assume you have installed buildtest on your system.
You can check if buildtest command is available by running:
$ buildtest --help
If you receive an error please go back and re-install buildtest.
If you are new to buildtest you can review the quick start guide to learn the basics of buildtest.
Building Test¶
The buildtest build command is used for running a test on your system given a buildspec
file (YAML). The most common way to build a test is specifying a file path via buildtest build -b <path>. To get started,
let’s build our first test by running the following.
buildtest build -b $BUILDTEST_ROOT/tutorials/hello_world.yml
$ buildtest build -b $BUILDTEST_ROOT/tutorials/hello_world.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19701871-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/03/07 15:56:37 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.3 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/tutorials/hello_world.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ hello_w… │ script │ generic… │ None │ None │ None │ hello │ /home/… │
│ │ │ │ │ │ │ world │ │
│ │ │ │ │ │ │ example │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
hello_world/b4920727: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/hello_world/hello_world/b4920727
hello_world/b4920727: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/hello_world/hello_world/b4920727/stage
hello_world/b4920727: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/hello_world/hello_world/b4920727/hello_world_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
hello_world/b4920727 does not have any dependencies adding test to queue
hello_world/b4920727: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/hello_world/hello_world/b4920727/stage
hello_world/b4920727: Running Test via command: bash --norc --noprofile -eo pipefail hello_world_build.sh
hello_world/b4920727: Test completed in 0.005411 seconds
hello_world/b4920727: Test completed with returncode: 0
hello_world/b4920727: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/hello_world/hello_world/b4920727/hello_world.out
hello_world/b4920727: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/hello_world/hello_world/b4920727/hello_world.err
In this iteration we are going to run the following tests: [hello_world/b4920727]
Test Summary
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ hello_world/ │ generic.loc… │ PASS │ N/A N/A N/A │ 0 │ 0.005411 │
│ b4920727 │ │ │ │ │ │
└──────────────┴──────────────┴────────┴───────────────┴────────────┴──────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/logs/buildtest_wt806tpp.log
The -b option can be specified multiple times and it can be used with directory path. buildtest
will recursively search for all .yml extensions and attempt to build all tests.
Let’s try running the following command, where we will build by file and directory.
buildtest build -b $BUILDTEST_ROOT/tutorials/hello_world.yml -b $BUILDTEST_ROOT/general_tests/configuration
$ buildtest build -b $BUILDTEST_ROOT/tutorials/hello_world.yml -b $BUILDTEST_ROOT/general_tests/configuration
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19701871-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/03/07 15:56:38 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.3 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/g… ║
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/g… ║
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/t… ║
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/g… ║
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/g… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 5
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 5
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 5
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/general_tests/configuration/systemd-default-target.yml: VALID
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/general_tests/configuration/ulimits.yml: VALID
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/tutorials/hello_world.yml: VALID
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/general_tests/configuration/kernel_state.yml: VALID
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/general_tests/configuration/disk_usage.yml: VALID
Total builder objects created: 10
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ systemd… │ script │ generic… │ None │ None │ None │ check if │ /home/… │
│ │ │ │ │ │ │ default │ │
│ │ │ │ │ │ │ target │ │
│ │ │ │ │ │ │ is │ │
│ │ │ │ │ │ │ multi-u… │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ ulimit_… │ script │ generic… │ None │ None │ None │ Check if │ /home/… │
│ │ │ │ │ │ │ file │ │
│ │ │ │ │ │ │ lock is │ │
│ │ │ │ │ │ │ set to │ │
│ │ │ │ │ │ │ unlimit… │ │
│ │ │ │ │ │ │ in │ │
│ │ │ │ │ │ │ ulimits │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ ulimit_… │ script │ generic… │ None │ None │ None │ Check if │ /home/… │
│ │ │ │ │ │ │ cputime │ │
│ │ │ │ │ │ │ is set │ │
│ │ │ │ │ │ │ to │ │
│ │ │ │ │ │ │ unlimit… │ │
│ │ │ │ │ │ │ in │ │
│ │ │ │ │ │ │ ulimits │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ ulimit_… │ script │ generic… │ None │ None │ None │ Check if │ /home/… │
│ │ │ │ │ │ │ stack │ │
│ │ │ │ │ │ │ size is │ │
│ │ │ │ │ │ │ set to │ │
│ │ │ │ │ │ │ unlimit… │ │
│ │ │ │ │ │ │ in │ │
│ │ │ │ │ │ │ ulimits │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ ulimit_… │ script │ generic… │ None │ None │ None │ Check │ /home/… │
│ │ │ │ │ │ │ virtual │ │
│ │ │ │ │ │ │ memory │ │
│ │ │ │ │ │ │ size and │ │
│ │ │ │ │ │ │ check if │ │
│ │ │ │ │ │ │ its set │ │
│ │ │ │ │ │ │ to │ │
│ │ │ │ │ │ │ unlimit… │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ ulimit_… │ script │ generic… │ None │ None │ None │ Check if │ /home/… │
│ │ │ │ │ │ │ open │ │
│ │ │ │ │ │ │ file │ │
│ │ │ │ │ │ │ descrip… │ │
│ │ │ │ │ │ │ limit is │ │
│ │ │ │ │ │ │ set to │ │
│ │ │ │ │ │ │ 4096 │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ ulimit_… │ script │ generic… │ None │ None │ None │ Check │ /home/… │
│ │ │ │ │ │ │ max │ │
│ │ │ │ │ │ │ number │ │
│ │ │ │ │ │ │ of user │ │
│ │ │ │ │ │ │ process │ │
│ │ │ │ │ │ │ limit is │ │
│ │ │ │ │ │ │ set to │ │
│ │ │ │ │ │ │ 2048 │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ hello_w… │ script │ generic… │ None │ None │ None │ hello │ /home/… │
│ │ │ │ │ │ │ world │ │
│ │ │ │ │ │ │ example │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ kernel_… │ script │ generic… │ None │ None │ None │ Retrieve │ /home/… │
│ │ │ │ │ │ │ Kernel │ │
│ │ │ │ │ │ │ Swap │ │
│ │ │ │ │ │ │ Usage │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ root_di… │ script │ generic… │ None │ None │ None │ Check │ /home/… │
│ │ │ │ │ │ │ root │ │
│ │ │ │ │ │ │ disk │ │
│ │ │ │ │ │ │ usage │ │
│ │ │ │ │ │ │ and │ │
│ │ │ │ │ │ │ report │ │
│ │ │ │ │ │ │ if it │ │
│ │ │ │ │ │ │ exceeds │ │
│ │ │ │ │ │ │ thresho… │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
systemd_default_target/1993f17f: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/systemd-default-target/systemd_default_target/1993f17f
systemd_default_target/1993f17f: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/systemd-default-target/systemd_default_target/1993f17f/stage
systemd_default_target/1993f17f: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/systemd-default-target/systemd_default_target/1993f17f/systemd_default_target_build.sh
ulimit_filelock_unlimited/0f3245a6: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_filelock_unlimited/0f3245a6
ulimit_filelock_unlimited/0f3245a6: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_filelock_unlimited/0f3245a6/stage
ulimit_filelock_unlimited/0f3245a6: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_filelock_unlimited/0f3245a6/ulimit_filelock_unlimited_build.sh
ulimit_cputime_unlimited/83d9b990: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_cputime_unlimited/83d9b990
ulimit_cputime_unlimited/83d9b990: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_cputime_unlimited/83d9b990/stage
ulimit_cputime_unlimited/83d9b990: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_cputime_unlimited/83d9b990/ulimit_cputime_unlimited_build.sh
ulimit_stacksize_unlimited/265b25f5: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_stacksize_unlimited/265b25f5
ulimit_stacksize_unlimited/265b25f5: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_stacksize_unlimited/265b25f5/stage
ulimit_stacksize_unlimited/265b25f5: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_stacksize_unlimited/265b25f5/ulimit_stacksize_unlimited_build.sh
ulimit_vmsize_unlimited/979d2a29: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_vmsize_unlimited/979d2a29
ulimit_vmsize_unlimited/979d2a29: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_vmsize_unlimited/979d2a29/stage
ulimit_vmsize_unlimited/979d2a29: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_vmsize_unlimited/979d2a29/ulimit_vmsize_unlimited_build.sh
ulimit_filedescriptor_4096/22a57529: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_filedescriptor_4096/22a57529
ulimit_filedescriptor_4096/22a57529: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_filedescriptor_4096/22a57529/stage
ulimit_filedescriptor_4096/22a57529: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_filedescriptor_4096/22a57529/ulimit_filedescriptor_4096_build.sh
ulimit_max_user_process_2048/55b18696: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_max_user_process_2048/55b18696
ulimit_max_user_process_2048/55b18696: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_max_user_process_2048/55b18696/stage
ulimit_max_user_process_2048/55b18696: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_max_user_process_2048/55b18696/ulimit_max_user_process_2048_build.sh
hello_world/e4779f27: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/hello_world/hello_world/e4779f27
hello_world/e4779f27: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/hello_world/hello_world/e4779f27/stage
hello_world/e4779f27: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/hello_world/hello_world/e4779f27/hello_world_build.sh
kernel_swapusage/7c746c30: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/kernel_state/kernel_swapusage/7c746c30
kernel_swapusage/7c746c30: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/kernel_state/kernel_swapusage/7c746c30/stage
kernel_swapusage/7c746c30: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/kernel_state/kernel_swapusage/7c746c30/kernel_swapusage_build.sh
root_disk_usage/02e2bd35: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/disk_usage/root_disk_usage/02e2bd35
root_disk_usage/02e2bd35: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/disk_usage/root_disk_usage/02e2bd35/stage
root_disk_usage/02e2bd35: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/disk_usage/root_disk_usage/02e2bd35/root_disk_usage_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
ulimit_filedescriptor_4096/22a57529 does not have any dependencies adding test to queue
hello_world/e4779f27 does not have any dependencies adding test to queue
ulimit_cputime_unlimited/83d9b990 does not have any dependencies adding test to queue
root_disk_usage/02e2bd35 does not have any dependencies adding test to queue
ulimit_stacksize_unlimited/265b25f5 does not have any dependencies adding test to queue
kernel_swapusage/7c746c30 does not have any dependencies adding test to queue
ulimit_vmsize_unlimited/979d2a29 does not have any dependencies adding test to queue
ulimit_filelock_unlimited/0f3245a6 does not have any dependencies adding test to queue
ulimit_max_user_process_2048/55b18696 does not have any dependencies adding test to queue
systemd_default_target/1993f17f does not have any dependencies adding test to queue
ulimit_filedescriptor_4096/22a57529: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_filedescriptor_4096/22a57529/stage
ulimit_filedescriptor_4096/22a57529: Running Test via command: bash --norc --noprofile -eo pipefail ulimit_filedescriptor_4096_build.sh
ulimit_filedescriptor_4096/22a57529: Test completed in 0.005548 seconds
ulimit_filedescriptor_4096/22a57529: Test completed with returncode: 0
ulimit_filedescriptor_4096/22a57529: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_filedescriptor_4096/22a57529/ulimit_filedescriptor_4096.out
ulimit_filedescriptor_4096/22a57529: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_filedescriptor_4096/22a57529/ulimit_filedescriptor_4096.err
ulimit_filedescriptor_4096/22a57529: performing regular expression - '^4096$' on file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_filedescriptor_4096/22a57529/ulimit_filedescriptor_4096.out
ulimit_filedescriptor_4096/22a57529: Regular Expression Match - Failed!
hello_world/e4779f27: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/hello_world/hello_world/e4779f27/stage
hello_world/e4779f27: Running Test via command: bash --norc --noprofile -eo pipefail hello_world_build.sh
hello_world/e4779f27: Test completed in 0.00515 seconds
hello_world/e4779f27: Test completed with returncode: 0
hello_world/e4779f27: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/hello_world/hello_world/e4779f27/hello_world.out
hello_world/e4779f27: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/hello_world/hello_world/e4779f27/hello_world.err
ulimit_cputime_unlimited/83d9b990: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_cputime_unlimited/83d9b990/stage
ulimit_cputime_unlimited/83d9b990: Running Test via command: bash --norc --noprofile -eo pipefail ulimit_cputime_unlimited_build.sh
ulimit_cputime_unlimited/83d9b990: Test completed in 0.005159 seconds
ulimit_cputime_unlimited/83d9b990: Test completed with returncode: 0
ulimit_cputime_unlimited/83d9b990: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_cputime_unlimited/83d9b990/ulimit_cputime_unlimited.out
ulimit_cputime_unlimited/83d9b990: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_cputime_unlimited/83d9b990/ulimit_cputime_unlimited.err
ulimit_cputime_unlimited/83d9b990: performing regular expression - '^unlimited$' on file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_cputime_unlimited/83d9b990/ulimit_cputime_unlimited.out
ulimit_cputime_unlimited/83d9b990: Regular Expression Match - Success!
root_disk_usage/02e2bd35: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/disk_usage/root_disk_usage/02e2bd35/stage
root_disk_usage/02e2bd35: Running Test via command: bash --norc --noprofile -eo pipefail root_disk_usage_build.sh
root_disk_usage/02e2bd35: Test completed in 0.013178 seconds
root_disk_usage/02e2bd35: Test completed with returncode: 0
root_disk_usage/02e2bd35: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/disk_usage/root_disk_usage/02e2bd35/root_disk_usage.out
root_disk_usage/02e2bd35: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/disk_usage/root_disk_usage/02e2bd35/root_disk_usage.err
ulimit_stacksize_unlimited/265b25f5: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_stacksize_unlimited/265b25f5/stage
ulimit_stacksize_unlimited/265b25f5: Running Test via command: bash --norc --noprofile -eo pipefail ulimit_stacksize_unlimited_build.sh
ulimit_stacksize_unlimited/265b25f5: Test completed in 0.005182 seconds
ulimit_stacksize_unlimited/265b25f5: Test completed with returncode: 0
ulimit_stacksize_unlimited/265b25f5: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_stacksize_unlimited/265b25f5/ulimit_stacksize_unlimited.out
ulimit_stacksize_unlimited/265b25f5: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_stacksize_unlimited/265b25f5/ulimit_stacksize_unlimited.err
ulimit_stacksize_unlimited/265b25f5: performing regular expression - '^unlimited$' on file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_stacksize_unlimited/265b25f5/ulimit_stacksize_unlimited.out
ulimit_stacksize_unlimited/265b25f5: Regular Expression Match - Failed!
kernel_swapusage/7c746c30: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/kernel_state/kernel_swapusage/7c746c30/stage
kernel_swapusage/7c746c30: Running Test via command: bash --norc --noprofile -eo pipefail kernel_swapusage_build.sh
kernel_swapusage/7c746c30: Test completed in 0.069359 seconds
kernel_swapusage/7c746c30: Test completed with returncode: 255
kernel_swapusage/7c746c30: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/kernel_state/kernel_swapusage/7c746c30/kernel_swapusage.out
kernel_swapusage/7c746c30: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/kernel_state/kernel_swapusage/7c746c30/kernel_swapusage.err
ulimit_vmsize_unlimited/979d2a29: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_vmsize_unlimited/979d2a29/stage
ulimit_vmsize_unlimited/979d2a29: Running Test via command: bash --norc --noprofile -eo pipefail ulimit_vmsize_unlimited_build.sh
ulimit_vmsize_unlimited/979d2a29: Test completed in 0.005554 seconds
ulimit_vmsize_unlimited/979d2a29: Test completed with returncode: 0
ulimit_vmsize_unlimited/979d2a29: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_vmsize_unlimited/979d2a29/ulimit_vmsize_unlimited.out
ulimit_vmsize_unlimited/979d2a29: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_vmsize_unlimited/979d2a29/ulimit_vmsize_unlimited.err
ulimit_vmsize_unlimited/979d2a29: performing regular expression - '^unlimited$' on file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_vmsize_unlimited/979d2a29/ulimit_vmsize_unlimited.out
ulimit_vmsize_unlimited/979d2a29: Regular Expression Match - Success!
ulimit_filelock_unlimited/0f3245a6: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_filelock_unlimited/0f3245a6/stage
ulimit_filelock_unlimited/0f3245a6: Running Test via command: bash --norc --noprofile -eo pipefail ulimit_filelock_unlimited_build.sh
ulimit_filelock_unlimited/0f3245a6: Test completed in 0.005591 seconds
ulimit_filelock_unlimited/0f3245a6: Test completed with returncode: 0
ulimit_filelock_unlimited/0f3245a6: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_filelock_unlimited/0f3245a6/ulimit_filelock_unlimited.out
ulimit_filelock_unlimited/0f3245a6: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_filelock_unlimited/0f3245a6/ulimit_filelock_unlimited.err
ulimit_filelock_unlimited/0f3245a6: performing regular expression - '^unlimited$' on file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_filelock_unlimited/0f3245a6/ulimit_filelock_unlimited.out
ulimit_filelock_unlimited/0f3245a6: Regular Expression Match - Success!
ulimit_max_user_process_2048/55b18696: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_max_user_process_2048/55b18696/stage
ulimit_max_user_process_2048/55b18696: Running Test via command: bash --norc --noprofile -eo pipefail ulimit_max_user_process_2048_build.sh
ulimit_max_user_process_2048/55b18696: Test completed in 0.005575 seconds
ulimit_max_user_process_2048/55b18696: Test completed with returncode: 0
ulimit_max_user_process_2048/55b18696: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_max_user_process_2048/55b18696/ulimit_max_user_process_2048.out
ulimit_max_user_process_2048/55b18696: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_max_user_process_2048/55b18696/ulimit_max_user_process_2048.err
ulimit_max_user_process_2048/55b18696: performing regular expression - '^2048$' on file: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/ulimits/ulimit_max_user_process_2048/55b18696/ulimit_max_user_process_2048.out
ulimit_max_user_process_2048/55b18696: Regular Expression Match - Failed!
systemd_default_target/1993f17f: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/systemd-default-target/systemd_default_target/1993f17f/stage
systemd_default_target/1993f17f: Running Test via command: bash --norc --noprofile -eo pipefail systemd_default_target_build.sh
systemd_default_target/1993f17f: Test completed in 0.189919 seconds
systemd_default_target/1993f17f: Test completed with returncode: 1
systemd_default_target/1993f17f: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/systemd-default-target/systemd_default_target/1993f17f/systemd_default_target.out
systemd_default_target/1993f17f: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/systemd-default-target/systemd_default_target/1993f17f/systemd_default_target.err
In this iteration we are going to run the following tests: [ulimit_filedescriptor_4096/22a57529, hello_world/e4779f27, ulimit_cputime_unlimited/83d9b990, root_disk_usage/02e2bd35, ulimit_stacksize_unlimited/265b25f5, kernel_swapusage/7c746c30, ulimit_vmsize_unlimited/979d2a29, ulimit_filelock_unlimited/0f3245a6, ulimit_max_user_process_2048/55b18696, systemd_default_target/1993f17f]
Test Summary
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ ulimit_vmsiz │ generic.loc… │ PASS │ False True │ 0 │ 0.005554 │
│ e_unlimited/ │ │ │ False │ │ │
│ 979d2a29 │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ kernel_swapu │ generic.loc… │ FAIL │ N/A N/A N/A │ 255 │ 0.069359 │
│ sage/7c746c3 │ │ │ │ │ │
│ 0 │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ ulimit_cputi │ generic.loc… │ PASS │ False True │ 0 │ 0.005159 │
│ me_unlimited │ │ │ False │ │ │
│ /83d9b990 │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ ulimit_filel │ generic.loc… │ PASS │ False True │ 0 │ 0.005591 │
│ ock_unlimite │ │ │ False │ │ │
│ d/0f3245a6 │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ ulimit_max_u │ generic.loc… │ FAIL │ False False │ 0 │ 0.005575 │
│ ser_process_ │ │ │ False │ │ │
│ 2048/55b1869 │ │ │ │ │ │
│ 6 │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ systemd_defa │ generic.loc… │ FAIL │ N/A N/A N/A │ 1 │ 0.189919 │
│ ult_target/1 │ │ │ │ │ │
│ 993f17f │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ ulimit_filed │ generic.loc… │ FAIL │ False False │ 0 │ 0.005548 │
│ escriptor_40 │ │ │ False │ │ │
│ 96/22a57529 │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ root_disk_us │ generic.loc… │ PASS │ N/A N/A N/A │ 0 │ 0.013178 │
│ age/02e2bd35 │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ hello_world/ │ generic.loc… │ PASS │ N/A N/A N/A │ 0 │ 0.00515 │
│ e4779f27 │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ ulimit_stack │ generic.loc… │ FAIL │ False False │ 0 │ 0.005182 │
│ size_unlimit │ │ │ False │ │ │
│ ed/265b25f5 │ │ │ │ │ │
└──────────────┴──────────────┴────────┴───────────────┴────────────┴──────────┘
Passed Tests: 5/10 Percentage: 50.000%
Failed Tests: 5/10 Percentage: 50.000%
Adding 10 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/logs/buildtest_5o4kkqpy.log
The -x option can be used to exclude buildspecs which works similar to -b where it can be a file or directory path.
In the next example try running the following commands:
buildtest build -b general_tests/configuration -x general_tests/configuration/ulimits.yml
buildtest build -b general_tests/configuration -x general_tests/configuration
You will notice in the second command when buildtest has no buildspecs to build it will terminate immediately.
Buildtest supports test timeout which can be used if you don’t want to wait indefinitely for test to complete. You
can use --timeout option which will terminate test if it exceeds the timelimit. The timeout is in number of
seconds. Let’s try running the following example and take note of how timeout affects the test status:
buildtest build -b tutorials/sleep.yml
buildtest build -b tutorials/sleep.yml --timeout=1
buildtest build -b tutorials/sleep.yml --timeout=5
Buildtest supports test discovery based on tags via
buildtest build --tags or short option -t. Let’s try running the following test, take note
of the output as buildtest will show discovered buildspecs based on tag names
buildtest build -t python
$ buildtest build -t python
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19701871-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/03/07 15:56:39 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.3 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/t… ║
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Buildspecs By Tag=python
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/t… ║
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 2
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 2
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 2
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/tutorials/python-shell.yml: VALID
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/tutorials/python-hello.yml: VALID
Total builder objects created: 2
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ circle_… │ script │ generic… │ None │ None │ None │ Calcula… │ /home/… │
│ │ │ │ │ │ │ circle │ │
│ │ │ │ │ │ │ of area │ │
│ │ │ │ │ │ │ given a │ │
│ │ │ │ │ │ │ radius │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ python_… │ script │ generic… │ None │ None │ None │ Hello │ /home/… │
│ │ │ │ │ │ │ World │ │
│ │ │ │ │ │ │ python │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
circle_area/ab02800a: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/python-shell/circle_area/ab02800a
circle_area/ab02800a: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/python-shell/circle_area/ab02800a/stage
circle_area/ab02800a: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/python-shell/circle_area/ab02800a/circle_area_build.sh
python_hello/2bc15608: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/python-hello/python_hello/2bc15608
python_hello/2bc15608: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/python-hello/python_hello/2bc15608/stage
python_hello/2bc15608: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/python-hello/python_hello/2bc15608/python_hello_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
circle_area/ab02800a does not have any dependencies adding test to queue
python_hello/2bc15608 does not have any dependencies adding test to queue
circle_area/ab02800a: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/python-shell/circle_area/ab02800a/stage
circle_area/ab02800a: Running Test via command: bash --norc --noprofile -eo pipefail circle_area_build.sh
circle_area/ab02800a: Test completed in 0.030487 seconds
circle_area/ab02800a: Test completed with returncode: 0
circle_area/ab02800a: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/python-shell/circle_area/ab02800a/circle_area.out
circle_area/ab02800a: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/python-shell/circle_area/ab02800a/circle_area.err
python_hello/2bc15608: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/python-hello/python_hello/2bc15608/stage
python_hello/2bc15608: Running Test via command: bash --norc --noprofile -eo pipefail python_hello_build.sh
python_hello/2bc15608: Test completed in 0.029305 seconds
python_hello/2bc15608: Test completed with returncode: 0
python_hello/2bc15608: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/python-hello/python_hello/2bc15608/python_hello.out
python_hello/2bc15608: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/python-hello/python_hello/2bc15608/python_hello.err
In this iteration we are going to run the following tests: [circle_area/ab02800a, python_hello/2bc15608]
Test Summary
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ circle_area/ │ generic.loc… │ PASS │ N/A N/A N/A │ 0 │ 0.030487 │
│ ab02800a │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ python_hello │ generic.loc… │ PASS │ N/A N/A N/A │ 0 │ 0.029305 │
│ /2bc15608 │ │ │ │ │ │
└──────────────┴──────────────┴────────┴───────────────┴────────────┴──────────┘
Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%
Adding 2 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/logs/buildtest_yiz1_3ut.log
To learn more about how to run tests, please refer to Building Test (buildtest build) guide.
Querying Test Report¶
Buildtest will keep track of all tests in a report file (JSON) that can be used to display content of all test runs
and query metadata specific to test. To fetch all test runs you can use buildtest report command which will displays
output in a table format.
You can fetch the path to report file by running:
buildtest report path
Buildtest can write test results to alternate report file if buildtest --report is used, you can list all report files by
running:
buildtest report list
buildtest rt is an alias for buildtest report command for those that hate typing :-)
We can fetch all pass and failed tests via --fail and --pass. Let’s try running the following:
buildtest rt --fail
buildtest rt --pass
Now let’s assume you want to know total failed tests in report file, you can use --row-count option which
displays total row count. Let’s run the following and see total fail count:
buildtest rt --fail --row-count
The buildtest rt summary can be useful if you want to summary of report file.
Buildtest supports paging support with buildtest rt which can be useful when you
have lots of tests. To enable pagination you can run:
buildtest rt --pager
Finally we can filter test records and format table columns via --filter and --format option. Let’s try
running the following command
buildtest rt --filter tags=python --format name,id,tags
$ buildtest rt --filter tags=python --format name,id,tags
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name ┃ id ┃ tags ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ circle_area │ ab02800a │ tutorials python │
├──────────────────────────┼──────────────────┼────────────────────────────────┤
│ python_hello │ 2bc15608 │ python │
└──────────────────────────┴──────────────────┴────────────────────────────────┘
The --format option are comma separated list of format fields while --filter option are key=value pair. To see
list of available format and filter fields you can run:
buildtest rt --helpfilter
buildtest rt --helpformat
Inspecting Test¶
The buildtest inspect command can be used to query test details and display metadata for one or more test. First you will
want to see all available test and their corresponding unique identifiers. Let’s run the following
buildtest it list
$ buildtest it list
Test Summary by name, id, buildspec
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ id ┃ name ┃ buildspec ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 1f342859-0a18-494b-ab6… │ variables_bash │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/vars.yml │
│ 059c0374-d75c-42ba-81a… │ exit1_fail │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/pass_returncode.yml │
│ 335783f5-e15c-4651-ba4… │ returncode_int_match │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/pass_returncode.yml │
│ 3d1c9aa2-fda3-4a8a-ae1… │ exit1_pass │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/pass_returncode.yml │
│ 6fcffd52-9aaa-4324-888… │ returncode_list_mismatch │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/pass_returncode.yml │
│ 206d8b10-aae1-41ab-a94… │ status_regex_pass │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/status_regex.yml │
│ c337d39b-174f-4b46-9a0… │ status_regex_fail │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/status_regex.yml │
│ 3068439d-17f7-477a-a48… │ regex_on_multiple_files │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/regex_on_filename.y │
│ │ │ ml │
│ 1e22514b-08d3-4fa1-888… │ file_expansion_supported │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/regex_on_filename.y │
│ │ │ ml │
│ 1b4e6d7f-47ea-4fbe-986… │ regex_on_directory_not_… │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/regex_on_filename.y │
│ │ │ ml │
│ 18d1f5b3-7d9c-48f4-a41… │ timelimit_max │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/runtime_status_test │
│ │ │ .yml │
│ 0b1bcb74-7075-4b5c-be2… │ timelimit_max_fail │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/runtime_status_test │
│ │ │ .yml │
│ ac538371-7ed6-4dfe-8e5… │ timelimit_min_fail │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/runtime_status_test │
│ │ │ .yml │
│ 3c01e749-a2d2-49df-a2a… │ timelimit_min │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/runtime_status_test │
│ │ │ .yml │
│ 7ee210e8-2b68-46f5-a19… │ timelimit_min_max │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/runtime_status_test │
│ │ │ .yml │
│ 0e6ad89c-f7d4-4e0b-888… │ always_fail │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/explicit_state.yml │
│ 7862ff4d-c898-4599-8d9… │ always_pass │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/explicit_state.yml │
│ fcb7d4a3-d12d-4f9f-baf… │ test_pass_returncode_mi… │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/explicit_state.yml │
│ 8d954e4a-515f-4a68-8a0… │ test_fail_returncode_ma… │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/explicit_state.yml │
│ 02f29e50-cfa2-467e-9c0… │ status_exists_failure │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/exists.yml │
│ acf08bc5-05ab-41cd-84c… │ status_exists │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/exists.yml │
│ 1c124f71-c733-4f07-896… │ file_exists_pass │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/file_exists_with_nu │
│ │ │ mber.yml │
│ e7271701-74ce-406e-879… │ combined_file_and_dir_c… │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/file_and_dir_check. │
│ │ │ yml │
│ bc07f702-2f61-4a32-bf2… │ file_and_dir_checks │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/file_and_dir_check. │
│ │ │ yml │
│ 28311891-1d09-45a8-950… │ symlink_test │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/is_symlink.yml │
│ cbe5f753-9f20-402f-9f4… │ file_count_by_extension │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/file_count.yml │
│ 58184f9d-7656-4ba2-a1e… │ file_count_on_directory │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/file_count.yml │
│ bc16a6d4-c1a6-4902-93c… │ file_count_by_expression │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/file_count_pattern. │
│ │ │ yml │
│ 685c6e6b-e079-4706-b96… │ file_extension_and_file… │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/file_count_pattern. │
│ │ │ yml │
│ e47f2de0-5bae-40af-a7e… │ file_count_by_filetype │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/file_count_filetype │
│ │ │ .yml │
│ 78d91386-d5d0-41c0-9a7… │ file_traverse_limit │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/test_sta │
│ │ │ tus/file_count_file_tra │
│ │ │ verse_limit.yml │
│ 6fe9c354-3172-40f7-8ac… │ unskipped │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/skip_tes │
│ │ │ ts.yml │
│ b4920727-abe1-4437-80f… │ hello_world │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/hello_wo │
│ │ │ rld.yml │
│ e4779f27-9124-41a1-b51… │ hello_world │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/hello_wo │
│ │ │ rld.yml │
│ 979d2a29-5779-4364-acd… │ ulimit_vmsize_unlimited │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/general_tests/conf │
│ │ │ iguration/ulimits.yml │
│ 83d9b990-2c39-4395-814… │ ulimit_cputime_unlimited │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/general_tests/conf │
│ │ │ iguration/ulimits.yml │
│ 0f3245a6-19ab-4179-97c… │ ulimit_filelock_unlimit… │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/general_tests/conf │
│ │ │ iguration/ulimits.yml │
│ 55b18696-f845-4f6c-ae3… │ ulimit_max_user_process… │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/general_tests/conf │
│ │ │ iguration/ulimits.yml │
│ 22a57529-b312-4346-b4e… │ ulimit_filedescriptor_4… │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/general_tests/conf │
│ │ │ iguration/ulimits.yml │
│ 265b25f5-09f1-4c69-990… │ ulimit_stacksize_unlimi… │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/general_tests/conf │
│ │ │ iguration/ulimits.yml │
│ 7c746c30-51bb-4e9c-92f… │ kernel_swapusage │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/general_tests/conf │
│ │ │ iguration/kernel_state. │
│ │ │ yml │
│ 1993f17f-1fba-4079-baf… │ systemd_default_target │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/general_tests/conf │
│ │ │ iguration/systemd-defau │
│ │ │ lt-target.yml │
│ 02e2bd35-7f20-4927-b94… │ root_disk_usage │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/general_tests/conf │
│ │ │ iguration/disk_usage.ym │
│ │ │ l │
│ ab02800a-b8d7-42c3-a16… │ circle_area │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/python-s │
│ │ │ hell.yml │
│ 2bc15608-80fc-4355-aab… │ python_hello │ /home/docs/checkouts/re │
│ │ │ adthedocs.org/user_buil │
│ │ │ ds/buildtest/checkouts/ │
│ │ │ v1.3/tutorials/python-h │
│ │ │ ello.yml │
└─────────────────────────┴──────────────────────────┴─────────────────────────┘
In buildtest, test are referred as builders which is in format <name>/<ID> where each test has a unique identifier separated by backslash / character. To see all builders you can run:
buildtest it list -b
Note, we will be using the builder notation when querying test via buildtest it name and buildtest it query. The
command buildtest it name will display raw JSON record from the report file for a given test. The test names can be positional
arguments so you can query multiple tests simulataneously. Let’s run the following:
buildtest it name hello_world circle_area
The buildtest it query is used to query test records in human readable format. This command is useful once you
run test via buildtest build and you want to inspect test result. buildtest can display test content, output and
error file and support multiple test queries including regular expression!!
Let’s try running the following
buildtest it query -o -e -t hello_world
$ buildtest it query -o -e -t hello_world
─────────────── hello_world/e4779f27-9124-41a1-b510-d83b25758143 ───────────────
Executor: generic.local.bash
Description: hello world example
State: PASS
Returncode: 0
Runtime: 0.00515 sec
Starttime: 2023/03/07 15:56:38
Endtime: 2023/03/07 15:56:38
Command: bash --norc --noprofile -eo pipefail hello_world_build.sh
Test Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/hello_world/hello_world/e4779f27/hello_world.sh
Build Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/hello_world/hello_world/e4779f27/hello_world_build.sh
Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/hello_world/hello_world/e4779f27/hello_world.out
Error File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/tests/generic.local.bash/hello_world/hello_world/e4779f27/hello_world.err
Log File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/var/logs/buildtest_5o4kkqpy.log
─ Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/che… ─
hello world!
─ Error File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/chec… ─
─ Test File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/check… ─
#!/usr/bin/bash
# Content of run section
echo "hello world!"
You can retrieve paths to given test via buildtest path that can be useful if you want to navigate to directory or list
contents. By default buildtest path will retrieve root directory of test. You can retrieve output and error via
buildtest path -o and buildtest path -e. Let’s try running:
buildtest path hello_world
buildtest path -o hello_world
buildtest path -e hello_world
We encourage you review Query Test Report for a detailed guide on how to query test in buildtest.
Interacting with Buildspecs¶
Buildtest supports several ways to interact with buildspecs, such as querying buildspec cache,
validating buildspecs, showing content of buildspecs, and editing buildspecs in editor.
The buildtest buildspec command contains several subcommands that we will discuss in this
session. To learn more we encourage you see Buildspecs Interface for detailed guide.
The buildtest help command can be used to provide a brief help message for each subcommand. Let’s run
the following command since there are lots of commands that can be used to query buildspec.
buildtest help buildspec
$ buildtest help buildspec
Finding Buildspecs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command ┃ Description ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest buildspec find │ Discover and validate all buildspecs │
│ │ and load all validated buildspecs in │
│ │ cache │
│ buildtest buildspec find --rebuild │ Rebuild cache file │
│ buildtest buildspec find --pager │ Paginate output of buildspec cache │
│ buildtest buildspec find --root /tmp │ Discover buildspecs in /tmp and │
│ --rebuild │ rebuild buildspec cache │
│ buildtest buildspec find --quiet │ Rebuild cache file but don't display │
│ --rebuild │ output of cache │
│ buildtest buildspec find --paths │ Print all root directories for │
│ │ buildspecs │
│ buildtest buildspec find --buildspec │ List all available buildspecs from │
│ │ cache │
│ buildtest buildspec find --executors │ List all unique executors from cache │
│ buildtest buildspec find --filter │ Filter buildspec cache based on │
│ type=script,tags=pass │ type=script and tags='pass' │
│ buildtest buildspec find --filter │ Filter cache by buildspec file │
│ buildspec=<path> │ │
│ buildtest buildspec find --format │ Format table columns by field: │
│ name,description │ 'name', and 'description' │
│ buildtest buildspec find │ Group tests by tag name │
│ --group-by-tags │ │
│ buildtest buildspec find │ Group tests by executor name │
│ --group-by-executor │ │
│ buildtest buildspec find --helpfilter │ Show all filter fields │
│ buildtest buildspec find --helpformat │ Show all format fields │
│ buildtest buildspec find --terse │ Display output in terse format │
│ buildtest buildspec find --row-count │ Print total count of records from │
│ │ the table │
│ buildtest buildspec find --count=5 │ Limit output of buildspec cache to 5 │
│ │ elements │
│ buildtest buildspec find invalid │ Show invalid buildspecs │
│ buildtest buildspec find invalid │ Show invalid buildspecs with error │
│ --error │ messages │
└───────────────────────────────────────┴──────────────────────────────────────┘
Validating Buildspecs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command ┃ Description ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest buildspec validate -b │ Validate a buildspec with JSON │
│ <file> │ Schema │
│ buildtest buildspec validate -b /tmp/ │ Validate all buildspecs in directory │
│ -x /tmp/network │ /tmp but exclude /tmp/network │
│ buildtest buildspec validate -t │ Validate all buildspecs for tagname │
│ python -t mac │ 'python' and 'mac' │
│ buildtest buildspec validate -e │ Validate all buildspecs for executor │
│ generic.local.bash │ 'generic.local.bash' │
└───────────────────────────────────────┴──────────────────────────────────────┘
Additional Features of Buildspecs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command ┃ Description ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest buildspec summary │ Show summary of buildspec cache file │
│ buildtest buildspec summary --pager │ Paginate the output of summary for │
│ │ buildspec cache │
│ buildtest buildspec show python_hello │ Show content of buildspec based on │
│ │ test name 'python_hello' │
│ buildtest buildspec show python_hello │ Use color theme 'emacs' for showing │
│ --theme emacs │ content of test │
│ buildtest buildspec show-fail │ Show content of buildspec on all │
│ │ failed tests │
│ buildtest buildspec show-fail │ Show content of test 'exit1_fail' │
│ exit1_fail │ │
│ buildtest buildspec edit-test │ Open test 'python_hello' in editor │
│ python_hello │ and validate file upon closing │
│ buildtest buildspec edit-file │ Open file │
│ $BUILDTEST_ROOT/tutorials/sleep.yml │ $BUILDTEST_ROOT/tutorials/sleep.yml │
│ │ in editor and validate file upon │
│ │ closing │
│ buildtest buildspec maintainers find │ Find buildspec with maintainer name │
│ johndoe │ 'johndoe' │
│ buildtest buildspec maintainers │ List all maintainers from buildspec │
│ --list │ cache │
│ buildtest buildspec maintainers │ List all maintainers in machine │
│ --list --terse --no-header │ readable format without header │
│ buildtest buildspec maintainers │ Show breakdown of maintainers by │
│ --breakdown │ buildspecs │
└───────────────────────────────────────┴──────────────────────────────────────┘
To build the buildspec cache you will need to run the following:
buildtest buildspec find --rebuild -q
The --rebuild option will rebuild the cache and -q will supress output. If you want to see all
valid buildspecs in cache you can run:
buildtest buildspec find
To retrieve all tags you can run:
buildtest buildspec find --tags
We can filter tests via --filter option which expects a key=value pair. Let’s filter by tagname python by running:
buildtest buildspec find --filter tags=python
We can format the columns using --format option where each field is comma separated. Let’s format by fields
name, tags, description
buildtest buildspec find --filter tags=python --format name,tags,description
To see all filter and format fields you can use --helpfilter and --helpformat to list all fields and their description.
If you want to see a summary of the buildspec cache you can run:
buildtest buildspec summary
Buildtest has an alias buildtest bc for buildtest buildspec command so let’s use this going forward.
To validate a buildspec you can use buildtest bc validate command there are several options analogous to buildtest build
for discovering buildspecs such as -b, -x, -t, -e. For instance let’s validate the following buildspecs:
buildtest bc validate -b tutorials/hello_world.yml -b general_tests/configuration
buildtest bc validate -t python
Let’s try validating an invalid buildspec so you can see what happens
buildtest bc validate -b tutorials/invalid_executor.yml
$ buildtest bc validate -b tutorials/invalid_executor.yml
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/t… ─
"[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/tutorials/invalid_executor.yml]: Unable to find executor: badexecutor in ['generic.local.bash', 'generic.local.sh', 'generic.local.csh', 'generic.local.zsh']"
1 buildspecs failed to validate
To see content of buildspec you can use buildtest bc show which expects name of test. Note tab completion
is supported.
Let’s run the following:
buildtest bc show sleep hello_world
Buildtest uses rich python library for coloring which is used extensively throughout the buildtest output.
Rich supports several built-in themes that can be used for your preference. The buildtest bc show -t <THEME> can be used
select a color theme.
Currently, buildtest supports the following themes, feel free to tab complete:
buildtest bc show -t
abap borland emacs gruvbox-dark lovelace native paraiso-light sas stata-dark vs
algol bw friendly gruvbox-light manni nord pastie solarized-dark stata-light xcode
algol_nu colorful friendly_grayscale igor material nord-darker perldoc solarized-light tango zenburn
arduino default fruity inkpot monokai one-dark rainbow_dash staroffice trac
autumn dracula github-dark lilypond murphy paraiso-dark rrt stata vim
Let’s try running the same example with emacs theme:
buildtest bc show -t emacs sleep
If you want to see list of invalid buildspecs you can run:
buildtest bc find invalid
Note, if you fix your invalid buildspec, buildtest will have no way of knowing if buildspec is valid until you
rebuild the buildspec cache buildtest bc find --rebuild.
Buildtest Configuration¶
In order to use buildtest, you need to configure buildtest. We will not discuss buildtest configuration in this tutorial, but show how you can interact with configuration file via command line.
Buildtest provides a default configuration file that is sufficient to get started. To view path to configuration file you can run:
buildtest config path
We have an alias buildtest cg for buildtest config command. If you want to view content of configuration file you can run
buildtest cg view
$ buildtest cg view
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.3/b… ─
1 system:
2 generic:
3 # specify list of hostnames where buildtest can run for given system record
4 hostnames: [".*"]
5
6 # system description
7 description: Generic System
8 # specify module system used at your site (environment-modules, lmod)
9 moduletool: N/A
10
11 # specify test timeout duration in number of seconds
12 # timeout: 60
13
14 # pool size for parallel processing using multiprocessing.Pool. Pool Size must be 1 or higher. The pool size is calculated as follows: poolsize = min(poolsize, os.cpu_count())
15 poolsize: 1
16
17 executors:
18 # define local executors for running jobs locally
19 local:
20 bash:
21 description: submit jobs on local machine using bash shell
22 shell: bash
23 sh:
24 description: submit jobs on local machine using sh shell
25 shell: sh
26 csh:
27 description: submit jobs on local machine using csh shell
28 shell: csh
29 zsh:
30 description: submit jobs on local machine using zsh shell
31 shell: zsh
32 # compiler block
33 compilers:
34 # a list of directories to search for modules
35 # modulepath: ['/tmp', '/var/']
36
37 # regular expression to search for compilers based on module pattern. Used with 'buildtest config compilers find' to generate compiler instance
38 # find:
39 # gcc: "^(gcc)"
40 # intel: "^(intel)"
41 # cray: "^(craype)"
42 # pgi: "^(pgi)"
43 # cuda: ^(cuda)"
44 # clang: "^(clang)"
45
46 # declare compiler instance which can be site-specific. You can let 'buildtest config compilers find' generate compiler section
47 compiler:
48 gcc:
49 builtin_gcc:
50 cc: gcc
51 fc: gfortran
52 cxx: g++
53
54 # location of log directory
55 # logdir: /tmp/
56
57 # specify location where buildtest will write tests
58 # testdir: /tmp
59
60 # specify one or more directory where buildtest should load buildspecs
61 # buildspec_roots: []
62
63 cdash:
64 url: https://my.cdash.org/
65 project: buildtest
66 site: generic
67 buildname: tutorials
68
We also support color themes (buildtest cg view --theme <theme>) when showing content of buildtest configuration.
Buildtest configuration file defines one or more executors that are used when writing test. Every test must be run by an executor. To retrieve all executors in a flat-listing you can run:
buildtest cg executors
Buildtest can show executor details in JSON and YAML format, you can fetch the details by running:
buildtest cg executors --json
buildtest cg executors --yaml