Command Line Tutorial

For this session, we assume you have installed buildtest on your system. You can check if buildtest command is available by running:

$ buildtest --help

If you receive an error please go back and re-install buildtest.

If you are new to buildtest you can review the quick start guide to learn the basics of buildtest.

Building Test

The buildtest build command is used for running a test on your system given a buildspec file (YAML). The most common way to build a test is specifying a file path via buildtest build -b <path>. To get started, let’s build our first test by running the following.

buildtest build -b $BUILDTEST_ROOT/tutorials/hello_world.yml
$ buildtest build -b $BUILDTEST_ROOT/tutorials/hello_world.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-26220538-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/11/08 18:02:53                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  2.1                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.10.14                                                  │
│ Configuration File: /tmp/tmpg3p4j9in/config.yml                              │
│ Test Directory:     /tmp/tmpg3p4j9in/var/tests                               │
│ Report File:        /tmp/tmpg3p4j9in/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/hello_world.yml                                                     ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1                                                                     ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/hello_world.yml: VALID
Total builder objects created: 1
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ hello_wo │ script │ generic. │ None     │ None  │ None  │ hello    │ /home/d │
│ rld/d695 │        │ local.ba │          │       │       │ world    │ ocs/che │
│ 9442     │        │ sh       │          │       │       │ example  │ ckouts/ │
│          │        │          │          │       │       │          │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/tu │
│          │        │          │          │       │       │          │ torials │
│          │        │          │          │       │       │          │ /hello_ │
│          │        │          │          │       │       │          │ world.y │
│          │        │          │          │       │       │          │ ml      │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
hello_world/d6959442: Creating Test Directory: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/hello_world/hello_world/d6959442
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
hello_world/d6959442 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder              ┃
┡━━━━━━━━━━━━━━━━━━━━━━┩
│ hello_world/d6959442 │
└──────────────────────┘
hello_world/d6959442: Current Working Directory : /tmp/tmpg3p4j9in/var/tests/generic.local.bash/hello_world/hello_world/d6959442/stage
hello_world/d6959442: Running Test via command: bash hello_world_build.sh
hello_world/d6959442: Test completed in 0.00479 seconds with returncode: 0
hello_world/d6959442: Writing output file -  /tmp/tmpg3p4j9in/var/tests/generic.local.bash/hello_world/hello_world/d6959442/hello_world.out
hello_world/d6959442: Writing error file - /tmp/tmpg3p4j9in/var/tests/generic.local.bash/hello_world/hello_world/d6959442/hello_world.err
                                Test Summary                                 
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder              ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ hello_world/d6959442 │ generic.local.bash │ PASS   │ 0          │ 0.005   │
└──────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to report file: /tmp/tmpg3p4j9in/var/report.json
Writing Logfile to /tmp/tmpg3p4j9in/var/logs/buildtest_4qvxyl1l.log

The -b option can be specified multiple times and it can be used with directory path. buildtest will recursively search for all .yml extensions and attempt to build all tests.

Let’s try running the following command, where we will build by file and directory.

buildtest build -b $BUILDTEST_ROOT/tutorials/hello_world.yml -b $BUILDTEST_ROOT/general_tests/configuration
$ buildtest build -b $BUILDTEST_ROOT/tutorials/hello_world.yml -b $BUILDTEST_ROOT/general_tests/configuration
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-26220538-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/11/08 18:02:54                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  2.1                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.10.14                                                  │
│ Configuration File: /tmp/tmpg3p4j9in/config.yml                              │
│ Test Directory:     /tmp/tmpg3p4j9in/var/tests                               │
│ Report File:        /tmp/tmpg3p4j9in/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/hello_world.yml                                                     ║
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/g ║
║ eneral_tests/configuration/disk_usage.yml                                    ║
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/g ║
║ eneral_tests/configuration/kernel_state.yml                                  ║
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/g ║
║ eneral_tests/configuration/systemd-default-target.yml                        ║
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/g ║
║ eneral_tests/configuration/ulimits.yml                                       ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 5                                                                     ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  5
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  5
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 5
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/hello_world.yml: VALID
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/general_tests/configuration/disk_usage.yml: VALID
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/general_tests/configuration/kernel_state.yml: VALID
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/general_tests/configuration/systemd-default-target.yml: VALID
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/general_tests/configuration/ulimits.yml: VALID
Total builder objects created: 10
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ hello_wo │ script │ generic. │ None     │ None  │ None  │ hello    │ /home/d │
│ rld/daed │        │ local.ba │          │       │       │ world    │ ocs/che │
│ 79f4     │        │ sh       │          │       │       │ example  │ ckouts/ │
│          │        │          │          │       │       │          │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/tu │
│          │        │          │          │       │       │          │ torials │
│          │        │          │          │       │       │          │ /hello_ │
│          │        │          │          │       │       │          │ world.y │
│          │        │          │          │       │       │          │ ml      │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ root_dis │ script │ generic. │ None     │ None  │ None  │ Check    │ /home/d │
│ k_usage/ │        │ local.ba │          │       │       │ root     │ ocs/che │
│ 0b7b48b8 │        │ sh       │          │       │       │ disk     │ ckouts/ │
│          │        │          │          │       │       │ usage    │ readthe │
│          │        │          │          │       │       │ and      │ docs.or │
│          │        │          │          │       │       │ report   │ g/user_ │
│          │        │          │          │       │       │ if it    │ builds/ │
│          │        │          │          │       │       │ exceeds  │ buildte │
│          │        │          │          │       │       │ threshol │ st/chec │
│          │        │          │          │       │       │ d        │ kouts/d │
│          │        │          │          │       │       │          │ evel/ge │
│          │        │          │          │       │       │          │ neral_t │
│          │        │          │          │       │       │          │ ests/co │
│          │        │          │          │       │       │          │ nfigura │
│          │        │          │          │       │       │          │ tion/di │
│          │        │          │          │       │       │          │ sk_usag │
│          │        │          │          │       │       │          │ e.yml   │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ kernel_s │ script │ generic. │ None     │ None  │ None  │ Retrieve │ /home/d │
│ wapusage │        │ local.ba │          │       │       │ Kernel   │ ocs/che │
│ /fcad0be │        │ sh       │          │       │       │ Swap     │ ckouts/ │
│ 2        │        │          │          │       │       │ Usage    │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/ge │
│          │        │          │          │       │       │          │ neral_t │
│          │        │          │          │       │       │          │ ests/co │
│          │        │          │          │       │       │          │ nfigura │
│          │        │          │          │       │       │          │ tion/ke │
│          │        │          │          │       │       │          │ rnel_st │
│          │        │          │          │       │       │          │ ate.yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ systemd_ │ script │ generic. │ None     │ None  │ None  │ check if │ /home/d │
│ default_ │        │ local.ba │          │       │       │ default  │ ocs/che │
│ target/7 │        │ sh       │          │       │       │ target   │ ckouts/ │
│ a3586a1  │        │          │          │       │       │ is       │ readthe │
│          │        │          │          │       │       │ multi-us │ docs.or │
│          │        │          │          │       │       │ er.targe │ g/user_ │
│          │        │          │          │       │       │ t        │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/ge │
│          │        │          │          │       │       │          │ neral_t │
│          │        │          │          │       │       │          │ ests/co │
│          │        │          │          │       │       │          │ nfigura │
│          │        │          │          │       │       │          │ tion/sy │
│          │        │          │          │       │       │          │ stemd-d │
│          │        │          │          │       │       │          │ efault- │
│          │        │          │          │       │       │          │ target. │
│          │        │          │          │       │       │          │ yml     │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ ulimit_f │ script │ generic. │ None     │ None  │ None  │ Check if │ /home/d │
│ ilelock_ │        │ local.ba │          │       │       │ file     │ ocs/che │
│ unlimite │        │ sh       │          │       │       │ lock is  │ ckouts/ │
│ d/0192b7 │        │          │          │       │       │ set to   │ readthe │
│ 44       │        │          │          │       │       │ unlimite │ docs.or │
│          │        │          │          │       │       │ d in     │ g/user_ │
│          │        │          │          │       │       │ ulimits  │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/ge │
│          │        │          │          │       │       │          │ neral_t │
│          │        │          │          │       │       │          │ ests/co │
│          │        │          │          │       │       │          │ nfigura │
│          │        │          │          │       │       │          │ tion/ul │
│          │        │          │          │       │       │          │ imits.y │
│          │        │          │          │       │       │          │ ml      │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ ulimit_c │ script │ generic. │ None     │ None  │ None  │ Check if │ /home/d │
│ putime_u │        │ local.ba │          │       │       │ cputime  │ ocs/che │
│ nlimited │        │ sh       │          │       │       │ is set   │ ckouts/ │
│ /eca6d25 │        │          │          │       │       │ to       │ readthe │
│ f        │        │          │          │       │       │ unlimite │ docs.or │
│          │        │          │          │       │       │ d in     │ g/user_ │
│          │        │          │          │       │       │ ulimits  │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/ge │
│          │        │          │          │       │       │          │ neral_t │
│          │        │          │          │       │       │          │ ests/co │
│          │        │          │          │       │       │          │ nfigura │
│          │        │          │          │       │       │          │ tion/ul │
│          │        │          │          │       │       │          │ imits.y │
│          │        │          │          │       │       │          │ ml      │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ ulimit_s │ script │ generic. │ None     │ None  │ None  │ Check if │ /home/d │
│ tacksize │        │ local.ba │          │       │       │ stack    │ ocs/che │
│ _unlimit │        │ sh       │          │       │       │ size is  │ ckouts/ │
│ ed/dd2ac │        │          │          │       │       │ set to   │ readthe │
│ f61      │        │          │          │       │       │ unlimite │ docs.or │
│          │        │          │          │       │       │ d in     │ g/user_ │
│          │        │          │          │       │       │ ulimits  │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/ge │
│          │        │          │          │       │       │          │ neral_t │
│          │        │          │          │       │       │          │ ests/co │
│          │        │          │          │       │       │          │ nfigura │
│          │        │          │          │       │       │          │ tion/ul │
│          │        │          │          │       │       │          │ imits.y │
│          │        │          │          │       │       │          │ ml      │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ ulimit_v │ script │ generic. │ None     │ None  │ None  │ Check    │ /home/d │
│ msize_un │        │ local.ba │          │       │       │ virtual  │ ocs/che │
│ limited/ │        │ sh       │          │       │       │ memory   │ ckouts/ │
│ 21cd51e2 │        │          │          │       │       │ size and │ readthe │
│          │        │          │          │       │       │ check if │ docs.or │
│          │        │          │          │       │       │ its set  │ g/user_ │
│          │        │          │          │       │       │ to       │ builds/ │
│          │        │          │          │       │       │ unlimite │ buildte │
│          │        │          │          │       │       │ d        │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/ge │
│          │        │          │          │       │       │          │ neral_t │
│          │        │          │          │       │       │          │ ests/co │
│          │        │          │          │       │       │          │ nfigura │
│          │        │          │          │       │       │          │ tion/ul │
│          │        │          │          │       │       │          │ imits.y │
│          │        │          │          │       │       │          │ ml      │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ ulimit_f │ script │ generic. │ None     │ None  │ None  │ Check if │ /home/d │
│ iledescr │        │ local.ba │          │       │       │ open     │ ocs/che │
│ iptor_40 │        │ sh       │          │       │       │ file     │ ckouts/ │
│ 96/1f0c6 │        │          │          │       │       │ descript │ readthe │
│ 1f6      │        │          │          │       │       │ ors      │ docs.or │
│          │        │          │          │       │       │ limit is │ g/user_ │
│          │        │          │          │       │       │ set to   │ builds/ │
│          │        │          │          │       │       │ 4096     │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/ge │
│          │        │          │          │       │       │          │ neral_t │
│          │        │          │          │       │       │          │ ests/co │
│          │        │          │          │       │       │          │ nfigura │
│          │        │          │          │       │       │          │ tion/ul │
│          │        │          │          │       │       │          │ imits.y │
│          │        │          │          │       │       │          │ ml      │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ ulimit_m │ script │ generic. │ None     │ None  │ None  │ Check    │ /home/d │
│ ax_user_ │        │ local.ba │          │       │       │ max      │ ocs/che │
│ process_ │        │ sh       │          │       │       │ number   │ ckouts/ │
│ 2048/0c7 │        │          │          │       │       │ of user  │ readthe │
│ 5ca60    │        │          │          │       │       │ process  │ docs.or │
│          │        │          │          │       │       │ limit is │ g/user_ │
│          │        │          │          │       │       │ set to   │ builds/ │
│          │        │          │          │       │       │ 2048     │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/ge │
│          │        │          │          │       │       │          │ neral_t │
│          │        │          │          │       │       │          │ ests/co │
│          │        │          │          │       │       │          │ nfigura │
│          │        │          │          │       │       │          │ tion/ul │
│          │        │          │          │       │       │          │ imits.y │
│          │        │          │          │       │       │          │ ml      │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
hello_world/daed79f4: Creating Test Directory: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/hello_world/hello_world/daed79f4
root_disk_usage/0b7b48b8: Creating Test Directory: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/disk_usage/root_disk_usage/0b7b48b8
kernel_swapusage/fcad0be2: Creating Test Directory: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/kernel_state/kernel_swapusage/fcad0be2
systemd_default_target/7a3586a1: Creating Test Directory: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/systemd-default-target/systemd_default_target/7a3586a1
ulimit_filelock_unlimited/0192b744: Creating Test Directory: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_filelock_unlimited/0192b744
ulimit_cputime_unlimited/eca6d25f: Creating Test Directory: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_cputime_unlimited/eca6d25f
ulimit_stacksize_unlimited/dd2acf61: Creating Test Directory: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_stacksize_unlimited/dd2acf61
ulimit_vmsize_unlimited/21cd51e2: Creating Test Directory: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_vmsize_unlimited/21cd51e2
ulimit_filedescriptor_4096/1f0c61f6: Creating Test Directory: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_filedescriptor_4096/1f0c61f6
ulimit_max_user_process_2048/0c75ca60: Creating Test Directory: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_max_user_process_2048/0c75ca60
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
ulimit_filedescriptor_4096/1f0c61f6 does not have any dependencies adding test to queue
kernel_swapusage/fcad0be2 does not have any dependencies adding test to queue
ulimit_vmsize_unlimited/21cd51e2 does not have any dependencies adding test to queue
hello_world/daed79f4 does not have any dependencies adding test to queue
ulimit_max_user_process_2048/0c75ca60 does not have any dependencies adding test to queue
root_disk_usage/0b7b48b8 does not have any dependencies adding test to queue
systemd_default_target/7a3586a1 does not have any dependencies adding test to queue
ulimit_cputime_unlimited/eca6d25f does not have any dependencies adding test to queue
ulimit_stacksize_unlimited/dd2acf61 does not have any dependencies adding test to queue
ulimit_filelock_unlimited/0192b744 does not have any dependencies adding test to queue
        Builders Eligible to Run         
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ulimit_filedescriptor_4096/1f0c61f6   │
│ kernel_swapusage/fcad0be2             │
│ ulimit_vmsize_unlimited/21cd51e2      │
│ hello_world/daed79f4                  │
│ ulimit_max_user_process_2048/0c75ca60 │
│ root_disk_usage/0b7b48b8              │
│ systemd_default_target/7a3586a1       │
│ ulimit_cputime_unlimited/eca6d25f     │
│ ulimit_stacksize_unlimited/dd2acf61   │
│ ulimit_filelock_unlimited/0192b744    │
└───────────────────────────────────────┘
ulimit_filedescriptor_4096/1f0c61f6: Current Working Directory : /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_filedescriptor_4096/1f0c61f6/stage
ulimit_filedescriptor_4096/1f0c61f6: Running Test via command: bash ulimit_filedescriptor_4096_build.sh
ulimit_filedescriptor_4096/1f0c61f6: Test completed in 0.004719 seconds with returncode: 0
ulimit_filedescriptor_4096/1f0c61f6: Writing output file -  /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_filedescriptor_4096/1f0c61f6/ulimit_filedescriptor_4096.out
ulimit_filedescriptor_4096/1f0c61f6: Writing error file - /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_filedescriptor_4096/1f0c61f6/ulimit_filedescriptor_4096.err
ulimit_filedescriptor_4096/1f0c61f6: performing regular expression - '^4096$' on file: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_filedescriptor_4096/1f0c61f6/ulimit_filedescriptor_4096.out
ulimit_filedescriptor_4096/1f0c61f6: Regular Expression Match - Failed!
kernel_swapusage/fcad0be2: Current Working Directory : /tmp/tmpg3p4j9in/var/tests/generic.local.bash/kernel_state/kernel_swapusage/fcad0be2/stage
kernel_swapusage/fcad0be2: Running Test via command: bash kernel_swapusage_build.sh
kernel_swapusage/fcad0be2: failed to submit job with returncode: 255
kernel_swapusage/fcad0be2: Detected failure in running test, will attempt to retry test: 1 times
kernel_swapusage/fcad0be2: Run - 1/1
kernel_swapusage/fcad0be2: Running Test via command: bash kernel_swapusage_build.sh
kernel_swapusage/fcad0be2: failed to submit job with returncode: 255
kernel_swapusage/fcad0be2: Test completed in 0.013474 seconds with returncode: 255
kernel_swapusage/fcad0be2: Writing output file -  /tmp/tmpg3p4j9in/var/tests/generic.local.bash/kernel_state/kernel_swapusage/fcad0be2/kernel_swapusage.out
kernel_swapusage/fcad0be2: Writing error file - /tmp/tmpg3p4j9in/var/tests/generic.local.bash/kernel_state/kernel_swapusage/fcad0be2/kernel_swapusage.err
ulimit_vmsize_unlimited/21cd51e2: Current Working Directory : /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_vmsize_unlimited/21cd51e2/stage
ulimit_vmsize_unlimited/21cd51e2: Running Test via command: bash ulimit_vmsize_unlimited_build.sh
ulimit_vmsize_unlimited/21cd51e2: Test completed in 0.003906 seconds with returncode: 0
ulimit_vmsize_unlimited/21cd51e2: Writing output file -  /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_vmsize_unlimited/21cd51e2/ulimit_vmsize_unlimited.out
ulimit_vmsize_unlimited/21cd51e2: Writing error file - /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_vmsize_unlimited/21cd51e2/ulimit_vmsize_unlimited.err
ulimit_vmsize_unlimited/21cd51e2: performing regular expression - '^unlimited$' on file: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_vmsize_unlimited/21cd51e2/ulimit_vmsize_unlimited.out
ulimit_vmsize_unlimited/21cd51e2: Regular Expression Match - Success!
hello_world/daed79f4: Current Working Directory : /tmp/tmpg3p4j9in/var/tests/generic.local.bash/hello_world/hello_world/daed79f4/stage
hello_world/daed79f4: Running Test via command: bash hello_world_build.sh
hello_world/daed79f4: Test completed in 0.004048 seconds with returncode: 0
hello_world/daed79f4: Writing output file -  /tmp/tmpg3p4j9in/var/tests/generic.local.bash/hello_world/hello_world/daed79f4/hello_world.out
hello_world/daed79f4: Writing error file - /tmp/tmpg3p4j9in/var/tests/generic.local.bash/hello_world/hello_world/daed79f4/hello_world.err
ulimit_max_user_process_2048/0c75ca60: Current Working Directory : /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_max_user_process_2048/0c75ca60/stage
ulimit_max_user_process_2048/0c75ca60: Running Test via command: bash ulimit_max_user_process_2048_build.sh
ulimit_max_user_process_2048/0c75ca60: Test completed in 0.004017 seconds with returncode: 0
ulimit_max_user_process_2048/0c75ca60: Writing output file -  /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_max_user_process_2048/0c75ca60/ulimit_max_user_process_2048.out
ulimit_max_user_process_2048/0c75ca60: Writing error file - /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_max_user_process_2048/0c75ca60/ulimit_max_user_process_2048.err
ulimit_max_user_process_2048/0c75ca60: performing regular expression - '^2048$' on file: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_max_user_process_2048/0c75ca60/ulimit_max_user_process_2048.out
ulimit_max_user_process_2048/0c75ca60: Regular Expression Match - Failed!
root_disk_usage/0b7b48b8: Current Working Directory : /tmp/tmpg3p4j9in/var/tests/generic.local.bash/disk_usage/root_disk_usage/0b7b48b8/stage
root_disk_usage/0b7b48b8: Running Test via command: bash root_disk_usage_build.sh
root_disk_usage/0b7b48b8: Test completed in 0.009364 seconds with returncode: 0
root_disk_usage/0b7b48b8: Writing output file -  /tmp/tmpg3p4j9in/var/tests/generic.local.bash/disk_usage/root_disk_usage/0b7b48b8/root_disk_usage.out
root_disk_usage/0b7b48b8: Writing error file - /tmp/tmpg3p4j9in/var/tests/generic.local.bash/disk_usage/root_disk_usage/0b7b48b8/root_disk_usage.err
systemd_default_target/7a3586a1: Current Working Directory : /tmp/tmpg3p4j9in/var/tests/generic.local.bash/systemd-default-target/systemd_default_target/7a3586a1/stage
systemd_default_target/7a3586a1: Running Test via command: bash systemd_default_target_build.sh
systemd_default_target/7a3586a1: failed to submit job with returncode: 1
systemd_default_target/7a3586a1: Detected failure in running test, will attempt to retry test: 1 times
systemd_default_target/7a3586a1: Run - 1/1
systemd_default_target/7a3586a1: Running Test via command: bash systemd_default_target_build.sh
systemd_default_target/7a3586a1: failed to submit job with returncode: 1
systemd_default_target/7a3586a1: Test completed in 0.035488 seconds with returncode: 1
systemd_default_target/7a3586a1: Writing output file -  /tmp/tmpg3p4j9in/var/tests/generic.local.bash/systemd-default-target/systemd_default_target/7a3586a1/systemd_default_target.out
systemd_default_target/7a3586a1: Writing error file - /tmp/tmpg3p4j9in/var/tests/generic.local.bash/systemd-default-target/systemd_default_target/7a3586a1/systemd_default_target.err
ulimit_cputime_unlimited/eca6d25f: Current Working Directory : /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_cputime_unlimited/eca6d25f/stage
ulimit_cputime_unlimited/eca6d25f: Running Test via command: bash ulimit_cputime_unlimited_build.sh
ulimit_cputime_unlimited/eca6d25f: Test completed in 0.004769 seconds with returncode: 0
ulimit_cputime_unlimited/eca6d25f: Writing output file -  /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_cputime_unlimited/eca6d25f/ulimit_cputime_unlimited.out
ulimit_cputime_unlimited/eca6d25f: Writing error file - /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_cputime_unlimited/eca6d25f/ulimit_cputime_unlimited.err
ulimit_cputime_unlimited/eca6d25f: performing regular expression - '^unlimited$' on file: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_cputime_unlimited/eca6d25f/ulimit_cputime_unlimited.out
ulimit_cputime_unlimited/eca6d25f: Regular Expression Match - Success!
ulimit_stacksize_unlimited/dd2acf61: Current Working Directory : /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_stacksize_unlimited/dd2acf61/stage
ulimit_stacksize_unlimited/dd2acf61: Running Test via command: bash ulimit_stacksize_unlimited_build.sh
ulimit_stacksize_unlimited/dd2acf61: Test completed in 0.0068 seconds with returncode: 0
ulimit_stacksize_unlimited/dd2acf61: Writing output file -  /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_stacksize_unlimited/dd2acf61/ulimit_stacksize_unlimited.out
ulimit_stacksize_unlimited/dd2acf61: Writing error file - /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_stacksize_unlimited/dd2acf61/ulimit_stacksize_unlimited.err
ulimit_stacksize_unlimited/dd2acf61: performing regular expression - '^unlimited$' on file: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_stacksize_unlimited/dd2acf61/ulimit_stacksize_unlimited.out
ulimit_stacksize_unlimited/dd2acf61: Regular Expression Match - Failed!
ulimit_filelock_unlimited/0192b744: Current Working Directory : /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_filelock_unlimited/0192b744/stage
ulimit_filelock_unlimited/0192b744: Running Test via command: bash ulimit_filelock_unlimited_build.sh
ulimit_filelock_unlimited/0192b744: Test completed in 0.004472 seconds with returncode: 0
ulimit_filelock_unlimited/0192b744: Writing output file -  /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_filelock_unlimited/0192b744/ulimit_filelock_unlimited.out
ulimit_filelock_unlimited/0192b744: Writing error file - /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_filelock_unlimited/0192b744/ulimit_filelock_unlimited.err
ulimit_filelock_unlimited/0192b744: performing regular expression - '^unlimited$' on file: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/ulimits/ulimit_filelock_unlimited/0192b744/ulimit_filelock_unlimited.out
ulimit_filelock_unlimited/0192b744: Regular Expression Match - Success!
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ ulimit_max_user_process │ generic.local.bash │ FAIL   │ 0          │ 0.004   │
│ _2048/0c75ca60          │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ ulimit_cputime_unlimite │ generic.local.bash │ PASS   │ 0          │ 0.005   │
│ d/eca6d25f              │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ systemd_default_target/ │ generic.local.bash │ FAIL   │ 1          │ 0.035   │
│ 7a3586a1                │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ hello_world/daed79f4    │ generic.local.bash │ PASS   │ 0          │ 0.004   │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ root_disk_usage/0b7b48b │ generic.local.bash │ PASS   │ 0          │ 0.009   │
│ 8                       │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ ulimit_vmsize_unlimited │ generic.local.bash │ PASS   │ 0          │ 0.004   │
│ /21cd51e2               │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ kernel_swapusage/fcad0b │ generic.local.bash │ FAIL   │ 255        │ 0.013   │
│ e2                      │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ ulimit_stacksize_unlimi │ generic.local.bash │ FAIL   │ 0          │ 0.007   │
│ ted/dd2acf61            │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ ulimit_filedescriptor_4 │ generic.local.bash │ FAIL   │ 0          │ 0.005   │
│ 096/1f0c61f6            │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ ulimit_filelock_unlimit │ generic.local.bash │ PASS   │ 0          │ 0.004   │
│ ed/0192b744             │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 5/10 Percentage: 50.000%
Failed Tests: 5/10 Percentage: 50.000%


Adding 10 test results to report file: /tmp/tmpg3p4j9in/var/report.json
Writing Logfile to /tmp/tmpg3p4j9in/var/logs/buildtest_zyr0exvr.log

The -x option can be used to exclude buildspecs which works similar to -b where it can be a file or directory path. In the next example try running the following commands:

buildtest build -b general_tests/configuration -x general_tests/configuration/ulimits.yml
buildtest build -b general_tests/configuration -x general_tests/configuration

You will notice in the second command when buildtest has no buildspecs to build it will terminate immediately.

Buildtest supports test timeout which can be used if you don’t want to wait indefinitely for test to complete. You can use --timeout option which will terminate test if it exceeds the timelimit. The timeout is in number of seconds. Let’s try running the following example and take note of how timeout affects the test status:

buildtest build -b tutorials/sleep.yml
buildtest build -b tutorials/sleep.yml --timeout=1
buildtest build -b tutorials/sleep.yml --timeout=5

Buildtest supports test discovery based on tags via buildtest build --tags or short option -t. Let’s try running the following test, take note of the output as buildtest will show discovered buildspecs based on tag names

buildtest build -t network
$ buildtest build -t network
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-26220538-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/11/08 18:02:56                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  2.1                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.10.14                                                  │
│ Configuration File: /tmp/tmpg3p4j9in/config.yml                              │
│ Test Directory:     /tmp/tmpg3p4j9in/var/tests                               │
│ Report File:        /tmp/tmpg3p4j9in/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/tags_example.yml                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1                                                                     ║
╚══════════════════════════════════════════════════════════════════════════════╝
                           Buildspecs By Tag=network                            
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/tags_example.yml                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1                                                                     ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/tags_example.yml: VALID
Total builder objects created: 2
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ string_t │ script │ generic. │ None     │ None  │ None  │ tags can │ /home/d │
│ ag/460ae │        │ local.ba │          │       │       │ be a     │ ocs/che │
│ 484      │        │ sh       │          │       │       │ string   │ ckouts/ │
│          │        │          │          │       │       │          │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/tu │
│          │        │          │          │       │       │          │ torials │
│          │        │          │          │       │       │          │ /tags_e │
│          │        │          │          │       │       │          │ xample. │
│          │        │          │          │       │       │          │ yml     │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ list_of_ │ script │ generic. │ None     │ None  │ None  │ tags can │ /home/d │
│ strings_ │        │ local.ba │          │       │       │ be a     │ ocs/che │
│ tags/cd6 │        │ sh       │          │       │       │ list of  │ ckouts/ │
│ a8354    │        │          │          │       │       │ strings  │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/tu │
│          │        │          │          │       │       │          │ torials │
│          │        │          │          │       │       │          │ /tags_e │
│          │        │          │          │       │       │          │ xample. │
│          │        │          │          │       │       │          │ yml     │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
string_tag/460ae484: Creating Test Directory: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/tags_example/string_tag/460ae484
list_of_strings_tags/cd6a8354: Creating Test Directory: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/tags_example/list_of_strings_tags/cd6a8354
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
list_of_strings_tags/cd6a8354 does not have any dependencies adding test to queue
string_tag/460ae484 does not have any dependencies adding test to queue
    Builders Eligible to Run     
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                       ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ list_of_strings_tags/cd6a8354 │
│ string_tag/460ae484           │
└───────────────────────────────┘
list_of_strings_tags/cd6a8354: Current Working Directory : /tmp/tmpg3p4j9in/var/tests/generic.local.bash/tags_example/list_of_strings_tags/cd6a8354/stage
list_of_strings_tags/cd6a8354: Running Test via command: bash list_of_strings_tags_build.sh
list_of_strings_tags/cd6a8354: Test completed in 3.022089 seconds with returncode: 0
list_of_strings_tags/cd6a8354: Writing output file -  /tmp/tmpg3p4j9in/var/tests/generic.local.bash/tags_example/list_of_strings_tags/cd6a8354/list_of_strings_tags.out
list_of_strings_tags/cd6a8354: Writing error file - /tmp/tmpg3p4j9in/var/tests/generic.local.bash/tags_example/list_of_strings_tags/cd6a8354/list_of_strings_tags.err
string_tag/460ae484: Current Working Directory : /tmp/tmpg3p4j9in/var/tests/generic.local.bash/tags_example/string_tag/460ae484/stage
string_tag/460ae484: Running Test via command: bash string_tag_build.sh
string_tag/460ae484: Test completed in 0.005901 seconds with returncode: 0
string_tag/460ae484: Writing output file -  /tmp/tmpg3p4j9in/var/tests/generic.local.bash/tags_example/string_tag/460ae484/string_tag.out
string_tag/460ae484: Writing error file - /tmp/tmpg3p4j9in/var/tests/generic.local.bash/tags_example/string_tag/460ae484/string_tag.err
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ list_of_strings_tags/cd │ generic.local.bash │ PASS   │ 0          │ 3.022   │
│ 6a8354                  │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ string_tag/460ae484     │ generic.local.bash │ PASS   │ 0          │ 0.006   │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%


Adding 2 test results to report file: /tmp/tmpg3p4j9in/var/report.json
Writing Logfile to /tmp/tmpg3p4j9in/var/logs/buildtest_lt2d_uzv.log

To learn more about how to run tests, please refer to Building Test (buildtest build) guide.

Querying Test Report

Buildtest will keep track of all tests in a report file (JSON) that can be used to display content of all test runs and query metadata specific to test. To fetch all test runs you can use buildtest report command which will displays output in a table format.

You can fetch the path to report file by running:

buildtest report path

Buildtest can write test results to alternate report file if buildtest --report is used, you can list all report files by running:

buildtest report list

buildtest rt is an alias for buildtest report command for those that hate typing :-)

We can fetch all pass and failed tests via --fail and --pass. Let’s try running the following:

buildtest rt --fail
buildtest rt --pass

Now let’s assume you want to know total failed tests in report file, you can use --row-count option which displays total row count. Let’s run the following and see total fail count:

buildtest rt --fail --row-count

The buildtest rt summary can be useful if you want to summary of report file.

Buildtest supports paging support with buildtest rt which can be useful when you have lots of tests. To enable pagination you can run:

buildtest rt --pager

Finally we can filter test records and format table columns via --filter and --format option. Let’s try running the following command

buildtest rt --filter tags=network --format name,id,tags
$ buildtest rt --filter tags=network --format name,id,tags
  Report File: /tmp/tmpg3p4j9in/var/report.json   
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ name                 ┃ id       ┃ tags         ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ list_of_strings_tags │ cd6a8354 │ network ping │
│ string_tag           │ 460ae484 │ network      │
└──────────────────────┴──────────┴──────────────┘

The --format option are comma separated list of format fields while --filter option are key=value pair. To see list of available format and filter fields you can run:

buildtest rt --helpfilter
buildtest rt --helpformat

Inspecting Test

The buildtest inspect command can be used to query test details and display metadata for one or more test. First you will want to see all available test and their corresponding unique identifiers. Let’s run the following

buildtest inspect list
$ buildtest inspect list
                      Test Summary by id, name, buildspec                       
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ id                      ┃ name                     ┃ buildspec               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 0c06043d-fa15-4c14-9744 │ variables_bash           │ /home/docs/checkouts/re │
│ -a6baad532cbf           │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/vars.ym │
│                         │                          │ l                       │
│ 886cec4c-cd7a-4f66-8826 │ unskipped                │ /home/docs/checkouts/re │
│ -197d18f82ca6           │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/skip_te │
│                         │                          │ sts.yml                 │
│ d6959442-e397-4a5a-b2f0 │ hello_world              │ /home/docs/checkouts/re │
│ -16b1f686b3b3           │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/hello_w │
│                         │                          │ orld.yml                │
│ daed79f4-057e-4b93-8478 │ hello_world              │ /home/docs/checkouts/re │
│ -fc636f66ce70           │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/hello_w │
│                         │                          │ orld.yml                │
│ 0c75ca60-0124-4833-999f │ ulimit_max_user_process_ │ /home/docs/checkouts/re │
│ -9b802c288a7b           │ 2048                     │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/general_tests/con │
│                         │                          │ figuration/ulimits.yml  │
│ eca6d25f-6175-40f7-8913 │ ulimit_cputime_unlimited │ /home/docs/checkouts/re │
│ -2feefdc4b7d0           │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/general_tests/con │
│                         │                          │ figuration/ulimits.yml  │
│ 21cd51e2-18db-458a-a330 │ ulimit_vmsize_unlimited  │ /home/docs/checkouts/re │
│ -4a91c2ca8c26           │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/general_tests/con │
│                         │                          │ figuration/ulimits.yml  │
│ dd2acf61-219b-4f4f-9428 │ ulimit_stacksize_unlimit │ /home/docs/checkouts/re │
│ -0d00b74f776e           │ ed                       │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/general_tests/con │
│                         │                          │ figuration/ulimits.yml  │
│ 1f0c61f6-8d84-40e1-b58d │ ulimit_filedescriptor_40 │ /home/docs/checkouts/re │
│ -c4ce759ff9cc           │ 96                       │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/general_tests/con │
│                         │                          │ figuration/ulimits.yml  │
│ 0192b744-3beb-49c0-be37 │ ulimit_filelock_unlimite │ /home/docs/checkouts/re │
│ -a19317fc8ba2           │ d                        │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/general_tests/con │
│                         │                          │ figuration/ulimits.yml  │
│ 7a3586a1-bf7f-4499-8ddb │ systemd_default_target   │ /home/docs/checkouts/re │
│ -68af11b5b9f2           │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/general_tests/con │
│                         │                          │ figuration/systemd-defa │
│                         │                          │ ult-target.yml          │
│ 0b7b48b8-7962-460a-b118 │ root_disk_usage          │ /home/docs/checkouts/re │
│ -04fce806e05f           │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/general_tests/con │
│                         │                          │ figuration/disk_usage.y │
│                         │                          │ ml                      │
│ fcad0be2-836f-44c6-beb4 │ kernel_swapusage         │ /home/docs/checkouts/re │
│ -ba357dfae2dc           │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/general_tests/con │
│                         │                          │ figuration/kernel_state │
│                         │                          │ .yml                    │
│ cd6a8354-b826-48f0-b0a3 │ list_of_strings_tags     │ /home/docs/checkouts/re │
│ -bbea0891db83           │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/tags_ex │
│                         │                          │ ample.yml               │
│ 460ae484-28eb-44ce-a017 │ string_tag               │ /home/docs/checkouts/re │
│ -bcf5ac6d8599           │                          │ adthedocs.org/user_buil │
│                         │                          │ ds/buildtest/checkouts/ │
│                         │                          │ devel/tutorials/tags_ex │
│                         │                          │ ample.yml               │
└─────────────────────────┴──────────────────────────┴─────────────────────────┘

In buildtest, test are referred as builders which is in format <name>/<ID> where each test has a unique identifier separated by backslash / character. To see all builders you can run:

buildtest inspect list -b

The buildtest it command is an alias for buildtest inspect command, so we will use this command going forward. We will be using the builder notation when querying test via buildtest it name and buildtest it query. The command buildtest it name will display raw JSON record from the report file for a given test. The test names can be positional arguments so you can query multiple tests simultaneously.

Let’s run the following:

buildtest it name hello_world

The buildtest it query is used to query test records in human readable format. This command is useful once you run test via buildtest build and you want to inspect test result. buildtest can display test content, output and error file and support multiple test queries including regular expression!!

Let’s try running the following

buildtest it query -o -e -t hello_world
$ buildtest it query -o -e -t hello_world
─────────────── hello_world/daed79f4-057e-4b93-8478-fc636f66ce70 ───────────────
Executor: generic.local.bash
Description: hello world example
State: PASS
Returncode: 0
Runtime: 0.004048 sec
Starttime: 2024/11/08 18:02:54
Endtime: 2024/11/08 18:02:54
Command: bash hello_world_build.sh
Test Script: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/hello_world/hello_world/daed79f4/stage/hello_world.sh
Build Script: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/hello_world/hello_world/daed79f4/hello_world_build.sh
Output File: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/hello_world/hello_world/daed79f4/hello_world.out
Error File: /tmp/tmpg3p4j9in/var/tests/generic.local.bash/hello_world/hello_world/daed79f4/hello_world.err
Log File: /tmp/tmpg3p4j9in/var/logs/buildtest_zyr0exvr.log
─ Buildspec File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/… ─
buildspecs:                                                                     
  hello_world:                                                                  
    executor: generic.local.bash                                                
    type: script                                                                
    tags: tutorials                                                             
    description: "hello world example"                                          
    run: echo "hello world!"                                                    
maintainers:                                                                    
- "@shahzebsiddiqui"                                                            
                                                                                
────────────────────────────────────────────────────────────────────────────────
───────────────────────────────── Output File  ─────────────────────────────────
hello world!                                                                    
                                                                                
────────────────────────────────────────────────────────────────────────────────
───────────────────────────────── Error File  ──────────────────────────────────
                                                                                
────────────────────────────────────────────────────────────────────────────────
────────────────────────────────── Test File  ──────────────────────────────────
#!/usr/bin/bash                                                                 
# Content of run section                                                        
echo "hello world!"                                                             
────────────────────────────────────────────────────────────────────────────────

You can retrieve paths to given test via buildtest path that can be useful if you want to navigate to directory or list contents. By default buildtest path will retrieve root directory of test. You can retrieve output and error via buildtest path -o and buildtest path -e. Let’s try running:

buildtest path hello_world
buildtest path -o hello_world
buildtest path -e hello_world

We encourage you review Query Test Report for a detailed guide on how to query test in buildtest.

Interacting with Buildspecs

Buildtest supports several ways to interact with buildspecs, such as querying buildspec cache, validating buildspecs, showing content of buildspecs, and editing buildspecs in editor. The buildtest buildspec command contains several subcommands that we will discuss in this session. To learn more we encourage you see Buildspecs Interface for detailed guide.

The buildtest show command can be used to provide a brief help message for each subcommand and usage. This command can be useful if you want to understand how a particular command works. Let’s run buildtest show buildspec which will summarize the buildtest buildspec command.

buildtest show buildspec
$ buildtest show buildspec
                               Finding Buildspecs                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command                               ┃ Description                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest buildspec find              │ Discover and validate all buildspecs │
│                                       │ and load all validated buildspecs in │
│                                       │ cache                                │
│ buildtest buildspec find --rebuild    │ Rebuild cache file                   │
│ buildtest buildspec find --pager      │ Paginate output of buildspec cache   │
│ buildtest buildspec find --search     │ Discover buildspecs in /tmp and      │
│ /tmp --search                         │ rebuild buildspec cache              │
│ $BUILDTEST_ROOT/tutorials/sleep.yml   │                                      │
│ --rebuild                             │                                      │
│ buildtest buildspec find --quiet      │ Rebuild cache file but don't display │
│ --rebuild                             │ output of cache                      │
│ buildtest buildspec find --paths      │ Print all root directories for       │
│                                       │ buildspecs                           │
│ buildtest buildspec find --buildspec  │ List all available buildspecs from   │
│                                       │ cache                                │
│ buildtest buildspec find --executors  │ List all unique executors from cache │
│ buildtest buildspec find --filter     │ Filter buildspec cache based on      │
│ type=script,tags=pass                 │ type=script and  tags='pass'         │
│ buildtest buildspec find --filter     │ Filter cache by buildspec file       │
│ buildspec=<path>                      │                                      │
│ buildtest buildspec find --format     │ Format table columns by field:       │
│ name,description                      │ 'name', and 'description'            │
│ buildtest buildspec find              │ Group tests by tag name              │
│ --group-by-tags                       │                                      │
│ buildtest buildspec find              │ Group tests by executor name         │
│ --group-by-executor                   │                                      │
│ buildtest buildspec find --helpfilter │ Show all filter fields               │
│ buildtest buildspec find --helpformat │ Show all format fields               │
│ buildtest buildspec find --terse      │ Display output in terse format       │
│ buildtest buildspec find --row-count  │ Print total count of records from    │
│                                       │ the table                            │
│ buildtest buildspec find --count=5    │ Limit output of buildspec cache to 5 │
│                                       │ elements                             │
│ buildtest buildspec find invalid      │ Show invalid buildspecs              │
│ buildtest buildspec find invalid      │ Show invalid buildspecs with error   │
│ --error                               │ messages                             │
│ buildtest buildspec find invalid      │ Print total count of records from    │
│ --row-count                           │ the table                            │
└───────────────────────────────────────┴──────────────────────────────────────┘
                             Validating Buildspecs                              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command                               ┃ Description                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest buildspec validate -b       │ Validate a buildspec with JSON       │
│ <file>                                │ Schema                               │
│ buildtest buildspec validate -b /tmp/ │ Validate all buildspecs in directory │
│ -x /tmp/network                       │ /tmp but exclude /tmp/network        │
│ buildtest buildspec validate -t       │ Validate all buildspecs for tagname  │
│ python -t mac                         │ 'python' and 'mac'                   │
│ buildtest buildspec validate -e       │ Validate all buildspecs for executor │
│ generic.local.bash                    │ 'generic.local.bash'                 │
└───────────────────────────────────────┴──────────────────────────────────────┘
                       Additional Features of Buildspecs                        
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command                               ┃ Description                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest buildspec summary           │ Show summary of buildspec cache file │
│ buildtest buildspec summary --pager   │ Paginate the output of summary for   │
│                                       │ buildspec cache                      │
│ buildtest buildspec show python_hello │ Show content of buildspec based on   │
│                                       │ test name 'python_hello'             │
│ buildtest buildspec show python_hello │ Use color theme 'emacs' for showing  │
│ --theme emacs                         │ content of test                      │
│ buildtest buildspec show-fail         │ Show content of buildspec on all     │
│                                       │ failed tests                         │
│ buildtest buildspec show-fail         │ Show content of test 'exit1_fail'    │
│ exit1_fail                            │                                      │
│ buildtest buildspec edit-test         │ Open test 'python_hello' in editor   │
│ python_hello                          │ and validate file upon closing       │
│ buildtest buildspec edit-file         │ Open file                            │
│ $BUILDTEST_ROOT/tutorials/sleep.yml   │ $BUILDTEST_ROOT/tutorials/sleep.yml  │
│                                       │ in editor and validate file upon     │
│                                       │ closing                              │
│ buildtest buildspec maintainers find  │ Find buildspec with maintainer name  │
│ johndoe                               │ 'johndoe'                            │
│ buildtest buildspec maintainers       │ List all maintainers from buildspec  │
│                                       │ cache                                │
│ buildtest buildspec maintainers       │ Print total count of records from    │
│ --row-count                           │ the table                            │
│ buildtest buildspec maintainers       │ List all maintainers in machine      │
│ --terse --no-header                   │ readable format without header       │
│ buildtest buildspec maintainers       │ Show breakdown of maintainers by     │
│ --breakdown                           │ buildspecs                           │
└───────────────────────────────────────┴──────────────────────────────────────┘

To build the buildspec cache you will need to run the following:

buildtest buildspec find --rebuild -q

The --rebuild option will rebuild the cache and -q will supress output. If you want to see all valid buildspecs in cache you can run:

buildtest buildspec find

To retrieve all tags you can run:

buildtest buildspec find --tags

We can filter tests via --filter option which expects a key=value pair. Let’s filter by tagname python by running:

buildtest buildspec find --filter tags=python

We can format the columns using --format option where each field is comma separated. Let’s format by fields name, tags, description

buildtest buildspec find --filter tags=python --format name,tags,description

To see all filter and format fields you can use --helpfilter and --helpformat to list all fields and their description.

If you want to see a summary of the buildspec cache you can run:

buildtest buildspec summary

Buildtest has an alias buildtest bc for buildtest buildspec command so let’s use this going forward.

To validate a buildspec you can use buildtest bc validate command there are several options analogous to buildtest build for discovering buildspecs such as -b, -x, -t, -e. For instance let’s validate the following buildspecs:

buildtest bc validate -b tutorials/hello_world.yml -b general_tests/configuration
buildtest bc validate -t python

Let’s try validating an invalid buildspec so you can see what happens

buildtest bc validate -b tutorials/invalid_executor.yml
$ buildtest bc validate -b tutorials/invalid_executor.yml
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… ─
"[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/invalid_executor.yml]: Unable to find executor: badexecutor in ['generic.local.bash', 'generic.local.sh', 'generic.local.csh', 'generic.local.zsh']"


1 buildspecs failed to validate

To see content of buildspec you can use buildtest bc show which expects name of test. Note tab completion is supported.

Let’s run the following:

buildtest bc show sleep hello_world

Buildtest uses rich python library for coloring which is used extensively throughout the buildtest output. Rich supports several built-in themes that can be used for your preference. The buildtest bc show --theme <THEME> can be used select a color theme.

Currently, buildtest supports the following themes, feel free to tab complete:

   buildtest bc show --theme
abap                borland             emacs               gruvbox-dark        lovelace            native              paraiso-light       sas                 stata-dark          vs
algol               bw                  friendly            gruvbox-light       manni               nord                pastie              solarized-dark      stata-light         xcode
algol_nu            colorful            friendly_grayscale  igor                material            nord-darker         perldoc             solarized-light     tango               zenburn
arduino             default             fruity              inkpot              monokai             one-dark            rainbow_dash        staroffice          trac
autumn              dracula             github-dark         lilypond            murphy              paraiso-dark        rrt                 stata               vim

Let’s try running the same example with emacs theme:

buildtest bc show --theme emacs sleep

If you want to see list of invalid buildspecs you can run:

buildtest bc find invalid

Note, if you fix your invalid buildspec, buildtest will have no way of knowing if buildspec is valid until you rebuild the buildspec cache buildtest bc find --rebuild.

Query Details from Buildtest Configuration

In order to use buildtest, you need to configure buildtest. We will not discuss how to configure buildtest in this tutorial, but show how you can interact with configuration file via command line.

Buildtest provides a default configuration file that is sufficient to get started. To view path to configuration file you can run:

buildtest config path

We have an alias buildtest cg for buildtest config command. If you want to view content of configuration file you can run

buildtest cg view
$ buildtest cg view
───────────────────────── /tmp/tmpg3p4j9in/config.yml ──────────────────────────
    1 system:
    2   generic:
    3     # specify a list of hostnames that is a regular expression where buildtest can run.
    4     hostnames: ['.*']
    5     # system description
    6     description: Generic System
    7     # specify module system to use. Supported module systems are [lmod, environment-modules, none]
    8     moduletool: none
    9 
   10     # specify size of job pool (https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool). This will configure the number of processes that can run in parallel.
   11     # If not specified then buildtest will use all available cores on the system.
   12     poolsize: 1
   13 
   14     # maximum number of jobs that can run in parallel. If not specified, buildtest will run all jobs in parallel.
   15     #max_jobs: 1
   16 
   17     # test timeout in number of seconds
   18     # timeout: 3600
   19 
   20     # enable pagination for buildtest
   21     pager: false
   22 
   23     # options for buildtest buildspec find command
   24     buildspecs:
   25       # determine whether to rebuild buildspec cache
   26       rebuild: false
   27       # determine number of records to display
   28       count: 15
   29 
   30       # specify format fields
   31       #format: name,description
   32 
   33       # display output in terse mode
   34       terse: false
   35 
   36       # list of paths to search for buildspecs where it can be file or directory
   37       # search: ["$BUILDTEST_ROOT/tutorials/containers", "$BUILDTEST_ROOT/examples", "demo.yml"]
   38 
   39     # options for buildtest report command
   40     report:
   41       # number of records to display
   42       count: 25
   43 
   44       # specify format fields
   45       format: name,id,state,runtime,returncode
   46 
   47     # specify directory paths to search for binaries
   48     #paths:
   49       # directory path to search for slurm binaries.
   50       #slurm: "/usr/bin"
   51 
   52       # directory path to search for lsf binaries.
   53       #lsf: "/usr/bin"
   54 
   55       # directory path to search for pbs binaries.
   56       #pbs: "/usr/bin"
   57 
   58       # directory path to search for torque binaries.
   59       #torque: "/usr/bin"
   60 
   61       # path to docker directory
   62       #docker: "/usr/local/bin"
   63 
   64       # path to singularity directory
   65       #singularity: "/usr/local/bin"
   66 
   67       # path to podman directory
   68       #podman: "/usr/local/bin"
   69 
   70     # start of executor configuration
   71     executors:
   72       # local executor is used to submit jobs on local machine. In this example we have 4 executors: bash, sh, csh, zsh that will submit jobs using bash, sh, csh, zsh shell respectively.
   73       local:
   74         bash:
   75           description: submit jobs on local machine using bash shell
   76           shell: bash
   77         sh:
   78           description: submit jobs on local machine using sh shell
   79           shell: sh
   80         csh:
   81           description: submit jobs on local machine using csh shell
   82           shell: csh
   83         zsh:
   84           description: submit jobs on local machine using zsh shell
   85           shell: zsh
   86     # specify compiler declaration
   87     compilers:
   88       compiler:
   89         # declaration of all gcc compilers
   90         gcc:
   91           # name of compiler
   92           builtin_gcc:
   93             cc: gcc
   94             fc: gfortran
   95             cxx: g++
   96     # specify CDASH configuration when using 'buildtest cdash upload'
   97     cdash:
   98       # CDASH server
   99       url: https://my.cdash.org/
  100       # name of CDASH project where to push test reports
  101       project: buildtest
  102       # specify the site name which should generally by name of your HPC system. This is used by CDASH to group test results by site
  103       site: generic
  104       # specify the build name which is used to group test results by build name
  105       buildname: tutorials
  106

We also support color themes (buildtest cg view --theme <theme>) when showing content of buildtest configuration.

Buildtest configuration file defines one or more executors that are used when writing test. Every test must be run by an executor. To retrieve all executors in a flat-listing you can run the following

buildtest cg executors list
$ buildtest cg executors list
generic.local.bash
generic.local.sh
generic.local.csh
generic.local.zsh

Buildtest can show executor details in JSON and YAML format, you can fetch the details by running the following

buildtest cg executors list --json
$ buildtest cg executors list --json
{
  "executors": {
    "local": {
      "bash": {
        "description": "submit jobs on local machine using bash shell",
        "shell": "bash"
      },
      "sh": {
        "description": "submit jobs on local machine using sh shell",
        "shell": "sh"
      },
      "csh": {
        "description": "submit jobs on local machine using csh shell",
        "shell": "csh"
      },
      "zsh": {
        "description": "submit jobs on local machine using zsh shell",
        "shell": "zsh"
      }
    }
  }
}
buildtest cg executors list --yaml
$ buildtest cg executors list --yaml
executors:
  local:
    bash:
      description: submit jobs on local machine using bash shell
      shell: bash
    csh:
      description: submit jobs on local machine using csh shell
      shell: csh
    sh:
      description: submit jobs on local machine using sh shell
      shell: sh
    zsh:
      description: submit jobs on local machine using zsh shell
      shell: zsh