Buildspec Overview

What is a buildspec?

A buildspec is a YAML file that defines your test in buildtest which is validated by schema followed by building a shell script and running the generated test. Buildtest will parse the buildspec with the global schema file which defines the top-level structure of buildspec file.

Example

Let’s start off with a simple example that declares two variables X and Y and prints the sum of X+Y.

buildspecs:
  add_numbers:
    type: script
    executor: generic.local.bash
    description: Add X+Y
    tags: [tutorials]
    vars:
      X: 1
      Y: 2
    run: echo "$X+$Y=" $(($X+$Y))

buildtest will validate the entire buildspec with global.schema.json, and use one of the sub-schema to validate the test defined in buildspec section. The buildspec is where you define a test, in the example above the name of the test is add_numbers. The test requires a type field which is the sub-schema used to validate the test section. In this example type: script informs buildtest to use the Script Schema when validating test section.

Each subschema has a list of field attributes that are supported, for example the fields: type, executor, vars and run are all valid fields supported by the script schema.

Let’s look at a more interesting example, shown below is a multi line run example using the script schema with test name called systemd_default_target, shown below is the content of test:

buildspecs:
  systemd_default_target:
    executor: generic.local.bash
    type: script
    tags: [system]
    description: check if default target is multi-user.target
    run: |
      if [ "multi-user.target" == `systemctl get-default` ]; then
        echo "multi-user is the default target";
        exit 0
      fi
      echo "multi-user is not the default target";
      exit 1

The test name systemd_default_target defined in buildspec section is validated with the following pattern "^[A-Za-z_][A-Za-z0-9_]*$". This test will use the executor generic.local.bash which means it will use the Local Executor with an executor name bash defined in the buildtest settings. The default buildtest settings will provide a bash executor as follows:

 system:
   generic:
     hostnames: ["localhost"]
     executors:
       local:
         bash:
           description: submit jobs on local machine using bash shell
           shell: bash

The shell: bash indicates this executor will use bash to run the test scripts. To reference this executor use the format <system>.<type>.<name> in this case generic.local.bash refers to bash executor.

The description field is an optional key that can be used to provide a brief summary of the test. The description field is limited to 80 characters. In this example we can specify multiple commands in run section, this can be done in YAML using run: | followed by content of run section tab indented 2 spaces.

In this next example, we introduce the summary field, which can be used as an extended description of test. It has no impact on the test. Unlike the description field, the summary field has no limit on character count and one can define multi-line string using the pipe symbol |.

buildspecs:
  summary_example:
    type: script
    executor: generic.local.bash
    description: The summary field can be a multi-line string and exceed 80 char
    tags: [tutorials]
    summary: |
      This is a long description of test that
      can exceed 80 characters and be multiline
    run: hostname

Script Schema

The script schema is used for writing simple scripts (bash, sh, python) in Buildspec. To use this schema you must set type: script. The run field is responsible for writing the content of test.

Shown below is schema header for script.schema.json.

{
  "$id": "script.schema.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "script schema version",
  "description": "The script schema is of ``type: script`` in sub-schema which is used for running shell scripts",
  "type": "object",
  "required": [
    "type",
    "run",
    "executor"
  ],
  "additionalProperties": false,

The "type": "object" means sub-schema is a JSON object where we define a list of key/value pair. The "required" field specifies a list of fields that must be specified in order to validate the Buildspec. In this example, type, run, and executor are required fields. The additionalProperties: false informs schema to reject any extra properties not defined in the schema.

The executor key is required for all sub-schemas which instructs buildtest which executor to use when running the test. The executors are defined in How to configure buildtest. In our first example we define variables using the vars property which is a Key/Value pair for variable assignment. The run section is required for script schema which defines the content of the test script.

Declaring Environment Variables

You can define environment variables using the env property, this is compatible with shells: bash, sh, zsh, csh and tcsh. It does not work with shell: python. In example below we declare three tests using environment variable with default shell (bash), csh, and tcsh

buildspecs:
  bash_env_variables:
    executor: generic.local.bash
    description: Declare environment variables in default shell (bash)
    type: script
    env:
      FIRST_NAME: avocado
      LAST_NAME: dinosaur
    tags: [tutorials]
    run: |
      hostname
      whoami
      echo $USER
      printf "${FIRST_NAME} ${LAST_NAME}\n"

  csh_env_declaration:
    executor: generic.local.csh
    type: script
    description: "csh shell example to declare environment variables"
    shell: /bin/csh
    tags: [tutorials]
    env:
      SHELL_NAME: "csh"
    run: echo "This is running $SHELL_NAME"

  tcsh_env_declaration:
    executor: generic.local.csh
    type: script
    description: "tcsh shell example to declare environment variables"
    shell: /bin/tcsh
    tags: [tutorials]
    env:
      path: "/usr/local/bin:$PATH"
    run: echo $path

This test can be run by issuing the following command: buildtest build -b tutorials/environment.yml. If we inspect one of the test script we will see that buildtest generates a build script that invokes the test using the shell wrapper /bin/csh for the csh test and gets the returncode.

#!/bin/bash


############# START VARIABLE DECLARATION ########################
export BUILDTEST_TEST_NAME=csh_env_declaration
export BUILDTEST_TEST_ROOT=/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.csh/environment/csh_env_declaration/0
export BUILDTEST_BUILDSPEC_DIR=/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials
export BUILDTEST_STAGE_DIR=/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.csh/environment/csh_env_declaration/0/stage
export BUILDTEST_TEST_ID=501ec5d3-e614-4ae8-9c1e-4849ce340c76
############# END VARIABLE DECLARATION   ########################


# source executor startup script
source /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/executor/generic.local.csh/before_script.sh
# Run generated script
/bin/csh /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.csh/environment/csh_env_declaration/0/stage/csh_env_declaration.csh
# Get return code
returncode=$?
# Exit with return code
exit $returncode

This generated test looks something like this

#!/bin/csh
# Declare environment variables
setenv SHELL_NAME csh


# Content of run section
echo "This is running $SHELL_NAME"

Environment variables are defined using export in bash, sh, zsh while csh and tcsh use setenv.

Declaring Variables

Variables can be defined using vars property, this is compatible with all shells except for python. The variables are defined slightly different in csh, tcsh as pose to bash, sh, and zsh. In example below we define tests with bash and csh.

In YAML strings can be specified with or without quotes however in bash, variables need to be enclosed in quotes " if you are defining a multi word string (name="First Last").

If you need define a literal string it is recommended to use the literal block | that is a special character in YAML. If you want to specify " or ' in string you can use the escape character \ followed by any of the special character. In example below we define several variables such as X, Y that contain numbers, variable literalstring is a literal string processed by YAML. The variable singlequote and doublequote defines a variable with the special character ' and ". The variables current_user and num_files store result of a shell command. This can be done using var=$(<command>) or var=`<command>` where <command> is a Linux command.

buildspecs:
  variables_bash:
    type: script
    executor: generic.local.bash
    description: Declare shell variables in bash
    tags: [tutorials]
    vars:
      X: 1
      Y: 2
      literalstring: this is a literal string
      singlequote: \'singlequote\'
      doublequote: \"doublequote\"
      current_user: "$(whoami)"
      num_files: "`find $HOME -type f -maxdepth 1 | wc -l`"
      multiline_string: |
        Hello my name is Bob \n
        I am 30 years old


    run: |
      echo "$X+$Y="$(($X+$Y))
      echo $literalstring
      echo $singlequote
      echo $doublequote
      echo "current user:" $current_user
      echo "number of files:" $num_files
      echo -e $multiline_string

Next we build this test by running buildtest build -b $BUILDTEST_ROOT/tutorials/vars.yml.

buildtest build -b $BUILDTEST_ROOT/tutorials/vars.yml

$ buildtest build -b $BUILDTEST_ROOT/tutorials/vars.yml
Buildspec Paths: ['/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/latest/tutorials', '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/latest/general_tests']
Updating buildspec cache file: /tmp/tmpiusjpuuk/var/buildspecs/cache.json
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-24049534-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/04/12 16:37:40                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.8                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpiusjpuuk/config.yml                              │
│ Test Directory:     /tmp/tmpiusjpuuk/var/tests                               │
│ Report File:        /tmp/tmpiusjpuuk/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/latest/ ║
║ tutorials/vars.yml                                                           ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1                                                                     ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/latest/tutorials/vars.yml: VALID
Total builder objects created: 1
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ variable │ script │ generic. │ None     │ None  │ None  │ Declare  │ /home/d │
│ s_bash/e │        │ local.ba │          │       │       │ shell    │ ocs/che │
│ 2c3d944  │        │ sh       │          │       │       │ variable │ ckouts/ │
│          │        │          │          │       │       │ s in     │ readthe │
│          │        │          │          │       │       │ bash     │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/l │
│          │        │          │          │       │       │          │ atest/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/vars. │
│          │        │          │          │       │       │          │ yml     │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
variables_bash/e2c3d944: Creating Test Directory: /tmp/tmpiusjpuuk/var/tests/generic.local.bash/vars/variables_bash/e2c3d944
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
variables_bash/e2c3d944 does not have any dependencies adding test to queue
 Builders Eligible to Run  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ variables_bash/e2c3d944 │
└─────────────────────────┘
variables_bash/e2c3d944: Current Working Directory : /tmp/tmpiusjpuuk/var/tests/generic.local.bash/vars/variables_bash/e2c3d944/stage
variables_bash/e2c3d944: Running Test via command: bash variables_bash_build.sh
variables_bash/e2c3d944: Test completed in 0.011928 seconds with returncode: 0
variables_bash/e2c3d944: Writing output file -  /tmp/tmpiusjpuuk/var/tests/generic.local.bash/vars/variables_bash/e2c3d944/variables_bash.out
variables_bash/e2c3d944: Writing error file - /tmp/tmpiusjpuuk/var/tests/generic.local.bash/vars/variables_bash/e2c3d944/variables_bash.err
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ variables_bash/e2c3d944 │ generic.local.bash │ PASS   │ 0          │ 0.012   │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to report file: /tmp/tmpiusjpuuk/var/report.json
Writing Logfile to /tmp/tmpiusjpuuk/var/logs/buildtest_wc2ibs7o.log

Let’s check the generated script from the previous build, you can run buildtest inspect query -o variables_bash where -o refers to output file for testname variables_bash. Take note of the output file we

buildtest inspect query -o variables_bash

$ buildtest inspect query -o variables_bash
───────────── variables_bash/e2c3d944-471d-4ee5-b24e-039592ed60fa ──────────────
Executor: generic.local.bash
Description: Declare shell variables in bash
State: PASS
Returncode: 0
Runtime: 0.011928 sec
Starttime: 2024/04/12 16:37:40
Endtime: 2024/04/12 16:37:40
Command: bash variables_bash_build.sh
Test Script: /tmp/tmpiusjpuuk/var/tests/generic.local.bash/vars/variables_bash/e2c3d944/variables_bash.sh
Build Script: /tmp/tmpiusjpuuk/var/tests/generic.local.bash/vars/variables_bash/e2c3d944/variables_bash_build.sh
Output File: /tmp/tmpiusjpuuk/var/tests/generic.local.bash/vars/variables_bash/e2c3d944/variables_bash.out
Error File: /tmp/tmpiusjpuuk/var/tests/generic.local.bash/vars/variables_bash/e2c3d944/variables_bash.err
Log File: /tmp/tmpiusjpuuk/var/logs/buildtest_wc2ibs7o.log
─ Output File: /tmp/tmpiusjpuuk/var/tests/generic.local.bash/vars/variables_b… ─
1+2=3                                                                           
this is a literal string                                                        
\'singlequote\'                                                                 
"doublequote"                                                                   
current user: docs                                                              
number of files: 4                                                              
Hello my name is Bob                                                            
 I am 30 years old

Defining Tags

The tags field can be used to classify tests which can be used to organize tests or if you want to Building By Tags (buildtest build --tags) (buildtest build --tags <TAGNAME>). Tags can be defined as a string or list of strings. In this example, the test string_tag defines a tag name network while test list_of_strings_tags define a list of tags named network and ping.

buildspecs:
  string_tag:
    type: script
    executor: generic.local.bash
    description: tags can be a string
    tags: network
    run: hostname

  list_of_strings_tags:
    type: script
    executor: generic.local.bash
    description: tags can be a list of strings
    tags: [network, ping]
    run: ping -c 4 www.google.com

Each item in tags must be a string and no duplicates are allowed, for example in this test, we define a duplicate tag network which is not allowed.

buildspecs:
  duplicate_string_tags:
    type: script
    executor: generic.local.bash
    description: duplicate strings in tags list is not allowed
    tags: [network, network]
    run: hostname

If tags is a list, it must contain atleast one item.

Skipping test

By default, buildtest will run all tests defined in buildspecs section, if you want to skip a test use the skip field which expects a boolean value. Shown below is an example test.

buildspecs:
  skip:
    type: script
    executor: generic.local.bash
    description: This test is skipped
    skip: Yes
    tags: [tutorials]
    run: hostname

  unskipped:
    type: script
    executor: generic.local.bash
    description: This test is not skipped
    skip: No
    tags: [tutorials]
    run: hostname

The first test skip will be ignored by buildtest because skip: true is defined while unskipped will be processed as usual.

Note

YAML and JSON have different representation for boolean. For json schema valid values are true and false see https://json-schema.org/understanding-json-schema/reference/boolean.html however YAML has many more representation for boolean see https://yaml.org/type/bool.html. You may use any of the YAML boolean, however it’s best to stick with json schema values true and false.

Here is an example build, notice message [skip] test is skipped during the build stage

buildtest build -b tutorials/skip_tests.yml

$ buildtest build -b tutorials/skip_tests.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-24049534-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/04/12 16:37:41                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.8                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpiusjpuuk/config.yml                              │
│ Test Directory:     /tmp/tmpiusjpuuk/var/tests                               │
│ Report File:        /tmp/tmpiusjpuuk/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/latest/ ║
║ tutorials/skip_tests.yml                                                     ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1                                                                     ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
skip: skipping test due to 'skip' property.
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/latest/tutorials/skip_tests.yml: VALID
Total builder objects created: 1
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ unskippe │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ d/e09990 │        │ local.ba │          │       │       │ test is  │ ocs/che │
│ 9f       │        │ sh       │          │       │       │ not      │ ckouts/ │
│          │        │          │          │       │       │ skipped  │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/l │
│          │        │          │          │       │       │          │ atest/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/skip_ │
│          │        │          │          │       │       │          │ tests.y │
│          │        │          │          │       │       │          │ ml      │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
unskipped/e099909f: Creating Test Directory: /tmp/tmpiusjpuuk/var/tests/generic.local.bash/skip_tests/unskipped/e099909f
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
unskipped/e099909f does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━┓
┃ Builder            ┃
┡━━━━━━━━━━━━━━━━━━━━┩
│ unskipped/e099909f │
└────────────────────┘
unskipped/e099909f: Current Working Directory : /tmp/tmpiusjpuuk/var/tests/generic.local.bash/skip_tests/unskipped/e099909f/stage
unskipped/e099909f: Running Test via command: bash unskipped_build.sh
unskipped/e099909f: Test completed in 0.00736 seconds with returncode: 0
unskipped/e099909f: Writing output file -  /tmp/tmpiusjpuuk/var/tests/generic.local.bash/skip_tests/unskipped/e099909f/unskipped.out
unskipped/e099909f: Writing error file - /tmp/tmpiusjpuuk/var/tests/generic.local.bash/skip_tests/unskipped/e099909f/unskipped.err
                               Test Summary                                
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder            ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ unskipped/e099909f │ generic.local.bash │ PASS   │ 0          │ 0.007   │
└────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to report file: /tmp/tmpiusjpuuk/var/report.json
Writing Logfile to /tmp/tmpiusjpuuk/var/logs/buildtest_aau7jbnl.log

Skipping a buildspec

Sometimes you may want to skip all test in a buildspec instead of updating every test with skip property, this can be done by setting skip at the top-level. This can be useful if you are running several test in a directory such as buildtest build -b dir1/ and you don’t want to explicitly exclude file via -x option every time, instead you can hardcode this into the buildspec. A typical use-case of skipping test is when a test is broken and you don’t want to run it then its good idea to set skip: yes on the buildspec and fix it later.

In this next example we set skip: yes, buildtest will skip the buildspec and no test will be processed even if skip is set in each test.

skip: yes
buildspecs:
  skip_all_tests:
    type: script
    executor: generic.local.bash
    description: "All test in this buildspec are skipped"
    tags: [tutorials]
    run: hostname

  this_test_is_also_skipped:
    type: script
    skip: no
    executor: generic.local.bash
    description: "This test is also skipped even if skip is defined in test"
    tags: [ tutorials ]
    run: hostname

If you try building this buildspec, you will see buildtest will skip the buildspec and terminate.

buildtest build -b tutorials/skip_buildspec.yml

$ buildtest build -b tutorials/skip_buildspec.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-24049534-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/04/12 16:37:42                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.8                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpiusjpuuk/config.yml                              │
│ Test Directory:     /tmp/tmpiusjpuuk/var/tests                               │
│ Report File:        /tmp/tmpiusjpuuk/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/latest/ ║
║ tutorials/skip_buildspec.yml                                                 ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1                                                                     ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/latest/tutorials/skip_buildspec.yml: skipping all test since 'skip' is defined
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/latest/tutorials/skip_buildspec.yml: VALID
                            Buildspecs Filtered out                             
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ buildspecs                                                                   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/latest… │
└──────────────────────────────────────────────────────────────────────────────┘

buildtest is unable to create any tests because there are no valid buildspecs. 

Please see logfile: /tmp/tmpiusjpuuk/var/buildtest.log