Buildspec Overview

What is a buildspec?

A buildspec is a YAML file that defines your test in buildtest which is validated by schema followed by building a shell script and running the generated test. Buildtest will parse the buildspec with the global schema file which defines the top-level structure of buildspec file.

Example

Let’s start off with a simple example that declares two variables X and Y and prints the sum of X+Y.

buildspecs:
  add_numbers:
    type: script
    executor: generic.local.bash
    description: Add X+Y
    tags: [tutorials]
    vars:
      X: 1
      Y: 2
    run: echo "$X+$Y=" $(($X+$Y))

buildtest will validate the entire buildspec with global.schema.json, and use one of the sub-schema to validate the test defined in buildspec section. The buildspec is where you define a test, in the example above the name of the test is add_numbers. The test requires a type field which is the sub-schema used to validate the test section. In this example type: script informs buildtest to use the Script Schema when validating test section.

Each subschema has a list of field attributes that are supported, for example the fields: type, executor, vars and run are all valid fields supported by the script schema.

Let’s look at a more interesting example, shown below is a multi line run example using the script schema with test name called systemd_default_target, shown below is the content of test:

buildspecs:
  systemd_default_target:
    executor: generic.local.bash
    type: script
    tags: [system]
    description: check if default target is multi-user.target
    run: |
      if [ "multi-user.target" == `systemctl get-default` ]; then
        echo "multi-user is the default target";
        exit 0
      fi
      echo "multi-user is not the default target";
      exit 1

The test name systemd_default_target defined in buildspec section is validated with the following pattern "^[A-Za-z_][A-Za-z0-9_]*$". This test will use the executor generic.local.bash which means it will use the Local Executor with an executor name bash defined in the buildtest settings. The default buildtest settings will provide a bash executor as follows:

 system:
   generic:
     hostnames: ["localhost"]
     executors:
       local:
         bash:
           description: submit jobs on local machine using bash shell
           shell: bash

The shell: bash indicates this executor will use bash to run the test scripts. To reference this executor use the format <system>.<type>.<name> in this case generic.local.bash refers to bash executor.

The description field is an optional key that can be used to provide a brief summary of the test. The description field is limited to 80 characters. In this example we can specify multiple commands in run section, this can be done in YAML using run: | followed by content of run section tab indented 2 spaces.

In this next example, we introduce the summary field, which can be used as an extended description of test. It has no impact on the test. Unlike the description field, the summary field has no limit on character count and one can define multi-line string using the pipe symbol |.

buildspecs:
  summary_example:
    type: script
    executor: generic.local.bash
    description: The summary field can be a multi-line string and exceed 80 char
    tags: [tutorials]
    summary: |
      This is a long description of test that
      can exceed 80 characters and be multiline
    run: hostname

Script Schema

The script schema is used for writing simple scripts (bash, sh, python) in Buildspec. To use this schema you must set type: script. The run field is responsible for writing the content of test.

Shown below is schema header for script.schema.json.

{
  "$id": "script.schema.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "script schema version",
  "description": "The script schema is of ``type: script`` in sub-schema which is used for running shell scripts",
  "type": "object",
  "required": [
    "type",
    "run",
    "executor"
  ],
  "additionalProperties": false,

The "type": "object" means sub-schema is a JSON object where we define a list of key/value pair. The "required" field specifies a list of fields that must be specified in order to validate the Buildspec. In this example, type, run, and executor are required fields. The additionalProperties: false informs schema to reject any extra properties not defined in the schema.

The executor key is required for all sub-schemas which instructs buildtest which executor to use when running the test. The executors are defined in How to configure buildtest. In our first example we define variables using the vars property which is a Key/Value pair for variable assignment. The run section is required for script schema which defines the content of the test script.

Declaring Environment Variables

You can define environment variables using the env property, this is compatible with shells: bash, sh, zsh, csh and tcsh. It does not work with shell: python. In example below we declare three tests using environment variable with default shell (bash), csh, and tcsh

buildspecs:
  bash_env_variables:
    executor: generic.local.bash
    description: Declare environment variables in default shell (bash)
    type: script
    env:
      FIRST_NAME: avocado
      LAST_NAME: dinosaur
    tags: [tutorials]
    run: |
      hostname
      whoami
      echo $USER
      printf "${FIRST_NAME} ${LAST_NAME}\n"

  csh_env_declaration:
    executor: generic.local.csh
    type: script
    description: "csh shell example to declare environment variables"
    shell: /bin/csh
    tags: [tutorials]
    env:
      SHELL_NAME: "csh"
    run: echo "This is running $SHELL_NAME"

  tcsh_env_declaration:
    executor: generic.local.csh
    type: script
    description: "tcsh shell example to declare environment variables"
    shell: /bin/tcsh
    tags: [tutorials]
    env:
      path: "/usr/local/bin:$PATH"
    run: echo $path

This test can be run by issuing the following command: buildtest build -b tutorials/environment.yml. If we inspect one of the test script we will see that buildtest generates a build script that invokes the test using the shell wrapper /bin/csh for the csh test and gets the returncode.

#!/bin/bash


############# START VARIABLE DECLARATION ########################
export BUILDTEST_TEST_NAME=csh_env_declaration
export BUILDTEST_TEST_ROOT=/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.csh/environment/csh_env_declaration/0
export BUILDTEST_BUILDSPEC_DIR=/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials
export BUILDTEST_STAGE_DIR=/Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.csh/environment/csh_env_declaration/0/stage
export BUILDTEST_TEST_ID=501ec5d3-e614-4ae8-9c1e-4849ce340c76
############# END VARIABLE DECLARATION   ########################


# source executor startup script
source /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/executor/generic.local.csh/before_script.sh
# Run generated script
/bin/csh /Users/siddiq90/Documents/GitHubDesktop/buildtest/var/tests/generic.local.csh/environment/csh_env_declaration/0/stage/csh_env_declaration.csh
# Get return code
returncode=$?
# Exit with return code
exit $returncode

This generated test looks something like this

#!/bin/csh
# Declare environment variables
setenv SHELL_NAME csh


# Content of run section
echo "This is running $SHELL_NAME"

Environment variables are defined using export in bash, sh, zsh while csh and tcsh use setenv.

Declaring Variables

Variables can be defined using vars property, this is compatible with all shells except for python. The variables are defined slightly different in csh, tcsh as pose to bash, sh, and zsh. In example below we define tests with bash and csh.

In YAML strings can be specified with or without quotes however in bash, variables need to be enclosed in quotes " if you are defining a multi word string (name="First Last").

If you need define a literal string it is recommended to use the literal block | that is a special character in YAML. If you want to specify " or ' in string you can use the escape character \ followed by any of the special character. In example below we define several variables such as X, Y that contain numbers, variable literalstring is a literal string processed by YAML. The variable singlequote and doublequote defines a variable with the special character ' and ". The variables current_user and num_files store result of a shell command. This can be done using var=$(<command>) or var=`<command>` where <command> is a Linux command.

buildspecs:
  variables_bash:
    type: script
    executor: generic.local.bash
    description: Declare shell variables in bash
    tags: [tutorials]
    vars:
      X: 1
      Y: 2
      literalstring: this is a literal string
      singlequote: \'singlequote\'
      doublequote: \"doublequote\"
      current_user: "$(whoami)"
      num_files: "`find $HOME -type f -maxdepth 1 | wc -l`"
      multiline_string: |
        Hello my name is Bob \n
        I am 30 years old


    run: |
      echo "$X+$Y="$(($X+$Y))
      echo $literalstring
      echo $singlequote
      echo $doublequote
      echo "current user:" $current_user
      echo "number of files:" $num_files
      echo -e $multiline_string

Next we build this test by running buildtest build -b $BUILDTEST_ROOT/tutorials/vars.yml.

buildtest build -b $BUILDTEST_ROOT/tutorials/vars.yml
$ buildtest build -b $BUILDTEST_ROOT/tutorials/vars.yml
Buildspec Paths: ['/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials', '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/general_tests']
Updating buildspec cache file: /tmp/tmpjcb9dx2w/var/buildspecs/cache.json
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:16:28                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/vars.yml                                                           ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/vars.yml: VALID
Total builder objects created: 1
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ variable │ script │ generic. │ None     │ None  │ None  │ Declare  │ /home/d │
│ s_bash/1 │        │ local.ba │          │       │       │ shell    │ ocs/che │
│ 7e5a942  │        │ sh       │          │       │       │ variable │ ckouts/ │
│          │        │          │          │       │       │ s in     │ readthe │
│          │        │          │          │       │       │ bash     │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/vars. │
│          │        │          │          │       │       │          │ yml     │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
variables_bash/17e5a942: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
variables_bash/17e5a942 does not have any dependencies adding test to queue
 Builders Eligible to Run  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ variables_bash/17e5a942 │
└─────────────────────────┘
variables_bash/17e5a942: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942/stage
variables_bash/17e5a942: Running Test via command: bash variables_bash_build.sh
variables_bash/17e5a942: Test completed in 0.009769 seconds with returncode: 0
variables_bash/17e5a942: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942/variables_bash.out
variables_bash/17e5a942: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942/variables_bash.err
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ variables_bash/17e5a942 │ generic.local.bash │ PASS   │ 0          │ 0.010   │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_uaulb51y.log

Let’s check the generated script from the previous build, you can run buildtest inspect query -o variables_bash where -o refers to output file for testname variables_bash. Take note of the output file we

buildtest inspect query -o variables_bash
$ buildtest inspect query -o variables_bash
───────────── variables_bash/17e5a942-9efe-478e-8c89-9bc6e35ea7ec ──────────────
Executor: generic.local.bash
Description: Declare shell variables in bash
State: PASS
Returncode: 0
Runtime: 0.009769 sec
Starttime: 2024/02/14 17:16:28
Endtime: 2024/02/14 17:16:28
Command: bash variables_bash_build.sh
Test Script: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942/variables_bash.sh
Build Script: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942/variables_bash_build.sh
Output File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942/variables_bash.out
Error File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_bash/17e5a942/variables_bash.err
Log File: /tmp/tmpjcb9dx2w/var/logs/buildtest_uaulb51y.log
─ Output File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/vars/variables_b… ─
1+2=3                                                                           
this is a literal string                                                        
\'singlequote\'                                                                 
"doublequote"                                                                   
current user: docs                                                              
number of files: 4                                                              
Hello my name is Bob                                                            
 I am 30 years old

Defining Tags

The tags field can be used to classify tests which can be used to organize tests or if you want to Building By Tags (buildtest build --tags) (buildtest build --tags <TAGNAME>). Tags can be defined as a string or list of strings. In this example, the test string_tag defines a tag name network while test list_of_strings_tags define a list of tags named network and ping.

buildspecs:
  string_tag:
    type: script
    executor: generic.local.bash
    description: tags can be a string
    tags: network
    run: hostname

  list_of_strings_tags:
    type: script
    executor: generic.local.bash
    description: tags can be a list of strings
    tags: [network, ping]
    run: ping -c 4 www.google.com

Each item in tags must be a string and no duplicates are allowed, for example in this test, we define a duplicate tag network which is not allowed.

buildspecs:
  duplicate_string_tags:
    type: script
    executor: generic.local.bash
    description: duplicate strings in tags list is not allowed
    tags: [network, network]
    run: hostname

If tags is a list, it must contain atleast one item.

Test Status

buildtest will record state of each test which can be PASS or FAIL. By default a 0 exit code is PASS and everything else is a FAIL. The status property can be used to determine how test will report its state. Currently, we can match state based on the following:

Return Code Matching

buildtest can report PASS/FAIL based on returncode, by default a 0 exit code is PASS and everything else is FAIL. The returncode can be a list of exit codes to match. In this example we have four tests called exit1_fail, exit1_pass, returncode_list_mismatch and returncode_int_match. We expect exit1_fail and returncode_mismatch to FAIL while exit1_pass and returncode_int_match will PASS.

buildspecs:

  exit1_fail:
    executor: generic.local.bash
    type: script
    description: exit 1 by default is FAIL
    tags: [tutorials, fail]
    run: exit 1

  exit1_pass:
    executor: generic.local.bash
    type: script
    description: report exit 1 as PASS
    run: exit 1
    tags: [tutorials, pass]
    status:
      returncode: [1]

  returncode_list_mismatch:
    executor: generic.local.bash
    type: script
    description: exit 2 failed since it failed to match returncode 1
    run: exit 2
    tags: [tutorials, fail]
    status:
      returncode: [1, 3]

  returncode_int_match:
    executor: generic.local.bash
    type: script
    description: exit 128 matches returncode 128
    run: exit 128
    tags: [tutorials, pass]
    status:
      returncode: 128

Let’s build this test and pay close attention to the status column in output.

buildtest build -b tutorials/test_status/pass_returncode.yml
$ buildtest build -b tutorials/test_status/pass_returncode.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:16:29                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/test_status/pass_returncode.yml                                    ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/pass_returncode.yml: VALID
Total builder objects created: 4
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ exit1_fa │ script │ generic. │ None     │ None  │ None  │ exit 1   │ /home/d │
│ il/e1b85 │        │ local.ba │          │       │       │ by       │ ocs/che │
│ 944      │        │ sh       │          │       │       │ default  │ ckouts/ │
│          │        │          │          │       │       │ is FAIL  │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ pass_re │
│          │        │          │          │       │       │          │ turncod │
│          │        │          │          │       │       │          │ e.yml   │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ exit1_pa │ script │ generic. │ None     │ None  │ None  │ report   │ /home/d │
│ ss/31743 │        │ local.ba │          │       │       │ exit 1   │ ocs/che │
│ 897      │        │ sh       │          │       │       │ as PASS  │ ckouts/ │
│          │        │          │          │       │       │          │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ pass_re │
│          │        │          │          │       │       │          │ turncod │
│          │        │          │          │       │       │          │ e.yml   │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ returnco │ script │ generic. │ None     │ None  │ None  │ exit 2   │ /home/d │
│ de_list_ │        │ local.ba │          │       │       │ failed   │ ocs/che │
│ mismatch │        │ sh       │          │       │       │ since it │ ckouts/ │
│ /84a02a2 │        │          │          │       │       │ failed   │ readthe │
│ a        │        │          │          │       │       │ to match │ docs.or │
│          │        │          │          │       │       │ returnco │ g/user_ │
│          │        │          │          │       │       │ de 1     │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ pass_re │
│          │        │          │          │       │       │          │ turncod │
│          │        │          │          │       │       │          │ e.yml   │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ returnco │ script │ generic. │ None     │ None  │ None  │ exit 128 │ /home/d │
│ de_int_m │        │ local.ba │          │       │       │ matches  │ ocs/che │
│ atch/fd4 │        │ sh       │          │       │       │ returnco │ ckouts/ │
│ 2032f    │        │          │          │       │       │ de 128   │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ pass_re │
│          │        │          │          │       │       │          │ turncod │
│          │        │          │          │       │       │          │ e.yml   │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
exit1_fail/e1b85944: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/pass_returncode/exit1_fail/e1b85944
exit1_pass/31743897: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/pass_returncode/exit1_pass/31743897
returncode_list_mismatch/84a02a2a: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/pass_returncode/returncode_list_mismatch/84a02a2a
returncode_int_match/fd42032f: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/pass_returncode/returncode_int_match/fd42032f
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
returncode_list_mismatch/84a02a2a does not have any dependencies adding test to queue
exit1_fail/e1b85944 does not have any dependencies adding test to queue
returncode_int_match/fd42032f does not have any dependencies adding test to queue
exit1_pass/31743897 does not have any dependencies adding test to queue
      Builders Eligible to Run       
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ returncode_list_mismatch/84a02a2a │
│ returncode_int_match/fd42032f     │
│ exit1_fail/e1b85944               │
│ exit1_pass/31743897               │
└───────────────────────────────────┘
returncode_list_mismatch/84a02a2a: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/pass_returncode/returncode_list_mismatch/84a02a2a/stage
returncode_list_mismatch/84a02a2a: Running Test via command: bash returncode_list_mismatch_build.sh
returncode_list_mismatch/84a02a2a: failed to submit job with returncode: 2
───────────── Error Message for returncode_list_mismatch/84a02a2a ──────────────

returncode_list_mismatch/84a02a2a: Detected failure in running test, will attempt to retry test: 1 times
returncode_list_mismatch/84a02a2a: Run - 1/1
returncode_list_mismatch/84a02a2a: Running Test via command: bash returncode_list_mismatch_build.sh
returncode_list_mismatch/84a02a2a: failed to submit job with returncode: 2
returncode_list_mismatch/84a02a2a: Test completed in 0.015189 seconds with returncode: 2
returncode_list_mismatch/84a02a2a: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/pass_returncode/returncode_list_mismatch/84a02a2a/returncode_list_mismatch.out
returncode_list_mismatch/84a02a2a: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/pass_returncode/returncode_list_mismatch/84a02a2a/returncode_list_mismatch.err
returncode_list_mismatch/84a02a2a: Checking returncode - 2 is matched in list [1, 3]
returncode_int_match/fd42032f: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/pass_returncode/returncode_int_match/fd42032f/stage
returncode_int_match/fd42032f: Running Test via command: bash returncode_int_match_build.sh
returncode_int_match/fd42032f: failed to submit job with returncode: 128
─────────────── Error Message for returncode_int_match/fd42032f ────────────────

returncode_int_match/fd42032f: Detected failure in running test, will attempt to retry test: 1 times
returncode_int_match/fd42032f: Run - 1/1
returncode_int_match/fd42032f: Running Test via command: bash returncode_int_match_build.sh
returncode_int_match/fd42032f: failed to submit job with returncode: 128
returncode_int_match/fd42032f: Test completed in 0.0145 seconds with returncode: 128
returncode_int_match/fd42032f: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/pass_returncode/returncode_int_match/fd42032f/returncode_int_match.out
returncode_int_match/fd42032f: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/pass_returncode/returncode_int_match/fd42032f/returncode_int_match.err
returncode_int_match/fd42032f: Checking returncode - 128 is matched in list [128]
exit1_fail/e1b85944: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/pass_returncode/exit1_fail/e1b85944/stage
exit1_fail/e1b85944: Running Test via command: bash exit1_fail_build.sh
exit1_fail/e1b85944: failed to submit job with returncode: 1
──────────────────── Error Message for exit1_fail/e1b85944 ─────────────────────

exit1_fail/e1b85944: Detected failure in running test, will attempt to retry test: 1 times
exit1_fail/e1b85944: Run - 1/1
exit1_fail/e1b85944: Running Test via command: bash exit1_fail_build.sh
exit1_fail/e1b85944: failed to submit job with returncode: 1
exit1_fail/e1b85944: Test completed in 0.014695 seconds with returncode: 1
exit1_fail/e1b85944: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/pass_returncode/exit1_fail/e1b85944/exit1_fail.out
exit1_fail/e1b85944: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/pass_returncode/exit1_fail/e1b85944/exit1_fail.err
exit1_pass/31743897: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/pass_returncode/exit1_pass/31743897/stage
exit1_pass/31743897: Running Test via command: bash exit1_pass_build.sh
exit1_pass/31743897: failed to submit job with returncode: 1
──────────────────── Error Message for exit1_pass/31743897 ─────────────────────

exit1_pass/31743897: Detected failure in running test, will attempt to retry test: 1 times
exit1_pass/31743897: Run - 1/1
exit1_pass/31743897: Running Test via command: bash exit1_pass_build.sh
exit1_pass/31743897: failed to submit job with returncode: 1
exit1_pass/31743897: Test completed in 0.014586 seconds with returncode: 1
exit1_pass/31743897: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/pass_returncode/exit1_pass/31743897/exit1_pass.out
exit1_pass/31743897: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/pass_returncode/exit1_pass/31743897/exit1_pass.err
exit1_pass/31743897: Checking returncode - 1 is matched in list [1]
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ exit1_pass/31743897     │ generic.local.bash │ PASS   │ 1          │ 0.015   │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ returncode_list_mismatc │ generic.local.bash │ FAIL   │ 2          │ 0.015   │
│ h/84a02a2a              │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ returncode_int_match/fd │ generic.local.bash │ PASS   │ 128        │ 0.015   │
│ 42032f                  │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ exit1_fail/e1b85944     │ generic.local.bash │ FAIL   │ 1          │ 0.015   │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 2/4 Percentage: 50.000%
Failed Tests: 2/4 Percentage: 50.000%


Adding 4 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_g5rflgwz.log

The returncode field can be an integer or list of integers but it may not accept duplicate values. If you specify a list of exit codes, buildtest will check actual returncode with list of expected returncodes specified by returncode field.

Shown below are examples of invalid returncodes:

# empty list is not allowed
returncode: []

# floating point is not accepted in list
returncode: [1, 1.5]

# floating point not accepted
returncode: 1.5

# duplicates are not allowed
returncode: [1, 2, 5, 5]

Passing Test based on regular expression

buildtest can configure PASS/FAIL of test based on regular expression on output or error file. This can be useful if you are expecting a certain output from the test as pose to returncode check.

In this example we introduce, the regex field which is part of status that expects a regular expression via exp. The stream property must be stdout or stderr which indicates buildtest will read output or error file and apply regular expression. If there is a match, buildtest will record the test state as PASS otherwise it will be a FAIL. In this example, we have four tests that will apply two regular expression on output file and two regular expression on error file.

buildspecs:
  status_regex_stdout_pass:
    executor: generic.local.bash
    type: script
    tags: [system]
    description: Pass test based on regular expression
    run: echo "PASS"
    status:
      regex:
        stream: stdout
        exp: "^(PASS)$"

  status_regex_stdout_fail:
    executor: generic.local.bash
    type: script
    tags: [system]
    description: Pass test based on regular expression
    run: echo "FAIL"
    status:
      regex:
        stream: stdout
        exp: "^(123FAIL)$"

  status_regex_stderr_pass:
    executor: generic.local.bash
    type: script
    tags: [system]
    description: Pass test based on regular expression
    run: echo "PASS" 1>&2
    status:
      regex:
        stream: stderr
        exp: '^(PASS)$'

  status_regex_stderr_fail:
    executor: generic.local.bash
    type: script
    tags: [system]
    description: Pass test based on regular expression
    run: echo "FAIL" 1>&2
    status:
      regex:
        stream: stderr
        exp: "^(123FAIL)$"

Now if we run this test, we will see first test will pass while second one will fail even though the returncode is a 0. Take a close look at the status property

buildtest build -b tutorials/test_status/status_regex.yml
$ buildtest build -b tutorials/test_status/status_regex.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:16:30                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/test_status/status_regex.yml                                       ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/status_regex.yml: VALID
Total builder objects created: 4
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ status_r │ script │ generic. │ None     │ None  │ None  │ Pass     │ /home/d │
│ egex_std │        │ local.ba │          │       │       │ test     │ ocs/che │
│ out_pass │        │ sh       │          │       │       │ based on │ ckouts/ │
│ /a2573ac │        │          │          │       │       │ regular  │ readthe │
│ c        │        │          │          │       │       │ expressi │ docs.or │
│          │        │          │          │       │       │ on       │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ status_ │
│          │        │          │          │       │       │          │ regex.y │
│          │        │          │          │       │       │          │ ml      │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ status_r │ script │ generic. │ None     │ None  │ None  │ Pass     │ /home/d │
│ egex_std │        │ local.ba │          │       │       │ test     │ ocs/che │
│ out_fail │        │ sh       │          │       │       │ based on │ ckouts/ │
│ /f7bc7c4 │        │          │          │       │       │ regular  │ readthe │
│ 2        │        │          │          │       │       │ expressi │ docs.or │
│          │        │          │          │       │       │ on       │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ status_ │
│          │        │          │          │       │       │          │ regex.y │
│          │        │          │          │       │       │          │ ml      │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ status_r │ script │ generic. │ None     │ None  │ None  │ Pass     │ /home/d │
│ egex_std │        │ local.ba │          │       │       │ test     │ ocs/che │
│ err_pass │        │ sh       │          │       │       │ based on │ ckouts/ │
│ /8bc6a3a │        │          │          │       │       │ regular  │ readthe │
│ 2        │        │          │          │       │       │ expressi │ docs.or │
│          │        │          │          │       │       │ on       │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ status_ │
│          │        │          │          │       │       │          │ regex.y │
│          │        │          │          │       │       │          │ ml      │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ status_r │ script │ generic. │ None     │ None  │ None  │ Pass     │ /home/d │
│ egex_std │        │ local.ba │          │       │       │ test     │ ocs/che │
│ err_fail │        │ sh       │          │       │       │ based on │ ckouts/ │
│ /ab07907 │        │          │          │       │       │ regular  │ readthe │
│ 1        │        │          │          │       │       │ expressi │ docs.or │
│          │        │          │          │       │       │ on       │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ status_ │
│          │        │          │          │       │       │          │ regex.y │
│          │        │          │          │       │       │          │ ml      │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
status_regex_stdout_pass/a2573acc: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stdout_pass/a2573acc
status_regex_stdout_fail/f7bc7c42: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stdout_fail/f7bc7c42
status_regex_stderr_pass/8bc6a3a2: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stderr_pass/8bc6a3a2
status_regex_stderr_fail/ab079071: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stderr_fail/ab079071
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
status_regex_stdout_fail/f7bc7c42 does not have any dependencies adding test to queue
status_regex_stderr_fail/ab079071 does not have any dependencies adding test to queue
status_regex_stdout_pass/a2573acc does not have any dependencies adding test to queue
status_regex_stderr_pass/8bc6a3a2 does not have any dependencies adding test to queue
      Builders Eligible to Run       
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ status_regex_stdout_fail/f7bc7c42 │
│ status_regex_stderr_fail/ab079071 │
│ status_regex_stdout_pass/a2573acc │
│ status_regex_stderr_pass/8bc6a3a2 │
└───────────────────────────────────┘
status_regex_stdout_fail/f7bc7c42: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stdout_fail/f7bc7c42/stage
status_regex_stdout_fail/f7bc7c42: Running Test via command: bash status_regex_stdout_fail_build.sh
status_regex_stdout_fail/f7bc7c42: Test completed in 0.005988 seconds with returncode: 0
status_regex_stdout_fail/f7bc7c42: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stdout_fail/f7bc7c42/status_regex_stdout_fail.out
status_regex_stdout_fail/f7bc7c42: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stdout_fail/f7bc7c42/status_regex_stdout_fail.err
status_regex_stdout_fail/f7bc7c42: performing regular expression - '^(123FAIL)$' on file: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stdout_fail/f7bc7c42/status_regex_stdout_fail.out
status_regex_stdout_fail/f7bc7c42: Regular Expression Match - Failed!
status_regex_stderr_fail/ab079071: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stderr_fail/ab079071/stage
status_regex_stderr_fail/ab079071: Running Test via command: bash status_regex_stderr_fail_build.sh
status_regex_stderr_fail/ab079071: Test completed in 0.005836 seconds with returncode: 0
status_regex_stderr_fail/ab079071: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stderr_fail/ab079071/status_regex_stderr_fail.out
status_regex_stderr_fail/ab079071: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stderr_fail/ab079071/status_regex_stderr_fail.err
status_regex_stderr_fail/ab079071: performing regular expression - '^(123FAIL)$' on file: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stderr_fail/ab079071/status_regex_stderr_fail.err
status_regex_stderr_fail/ab079071: Regular Expression Match - Failed!
status_regex_stdout_pass/a2573acc: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stdout_pass/a2573acc/stage
status_regex_stdout_pass/a2573acc: Running Test via command: bash status_regex_stdout_pass_build.sh
status_regex_stdout_pass/a2573acc: Test completed in 0.005598 seconds with returncode: 0
status_regex_stdout_pass/a2573acc: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stdout_pass/a2573acc/status_regex_stdout_pass.out
status_regex_stdout_pass/a2573acc: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stdout_pass/a2573acc/status_regex_stdout_pass.err
status_regex_stdout_pass/a2573acc: performing regular expression - '^(PASS)$' on file: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stdout_pass/a2573acc/status_regex_stdout_pass.out
status_regex_stdout_pass/a2573acc: Regular Expression Match - Success!
status_regex_stderr_pass/8bc6a3a2: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stderr_pass/8bc6a3a2/stage
status_regex_stderr_pass/8bc6a3a2: Running Test via command: bash status_regex_stderr_pass_build.sh
status_regex_stderr_pass/8bc6a3a2: Test completed in 0.005536 seconds with returncode: 0
status_regex_stderr_pass/8bc6a3a2: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stderr_pass/8bc6a3a2/status_regex_stderr_pass.out
status_regex_stderr_pass/8bc6a3a2: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stderr_pass/8bc6a3a2/status_regex_stderr_pass.err
status_regex_stderr_pass/8bc6a3a2: performing regular expression - '^(PASS)$' on file: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/status_regex/status_regex_stderr_pass/8bc6a3a2/status_regex_stderr_pass.err
status_regex_stderr_pass/8bc6a3a2: Regular Expression Match - Success!
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ status_regex_stdout_fai │ generic.local.bash │ FAIL   │ 0          │ 0.006   │
│ l/f7bc7c42              │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ status_regex_stderr_fai │ generic.local.bash │ FAIL   │ 0          │ 0.006   │
│ l/ab079071              │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ status_regex_stdout_pas │ generic.local.bash │ PASS   │ 0          │ 0.006   │
│ s/a2573acc              │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ status_regex_stderr_pas │ generic.local.bash │ PASS   │ 0          │ 0.006   │
│ s/8bc6a3a2              │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 2/4 Percentage: 50.000%
Failed Tests: 2/4 Percentage: 50.000%


Adding 4 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_kbenvetx.log

Specify Regular Expression on arbitrary filenames

In the previous example, we applied regular expression on output or error file; however, you may want to apply regular expression on arbitrary files. This can be done by specifying file_regex property. The file_regex property is an array of assertions, where each assertion must have filename and exp property. The filename property is the path to filename and exp is the regular expression you want to apply.

In this example, we have three tests that make use of file_regex property. The first test will perform a regular expression on multiple file names. The second test will attempt to check on a directory name which is not supported since regular expression must be applied to file name. The third test will show that variable expansion and environment variable expansion is supported.

buildspecs:
  regex_on_multiple_files:
    type: script
    executor: generic.local.bash
    description: Test regex on multiple files
    run: |
      echo "Hello" > hello.txt
      buildtest --help > buildtest_help.txt
    status:
      file_regex:
        - file: hello.txt
          exp: '^(Hello)$'
        - file: buildtest_help.txt
          exp: '^(usage: buildtest)'

  regex_on_directory_not_supported:
    type: script
    executor: generic.local.bash
    description: Test regex on directory is not supported
    run: |
      mkdir -p hello
      echo "Hello" > hello/hello.txt
    status:
      file_regex:
        - file: hello
          exp: '^(Hello)$'

  file_expansion_supported:
    type: script
    executor: generic.local.bash
    description: Test regex with variable and shell expansion
    run: |
      echo "Hello" > $BUILDTEST_ROOT/hello.txt
      echo "Hello" > $HOME/hello.txt
    status:
      file_regex:
        - file: $BUILDTEST_ROOT/hello.txt
          exp: '^(Hello)$'
        - file: ~/hello.txt
          exp: '^(Hello)$'

We can build this test by running the following command:

buildtest build -b tutorials/test_status/regex_on_filename.yml
$ buildtest build -b tutorials/test_status/regex_on_filename.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:16:31                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/test_status/regex_on_filename.yml                                  ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/regex_on_filename.yml: VALID
Total builder objects created: 3
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ regex_on │ script │ generic. │ None     │ None  │ None  │ Test     │ /home/d │
│ _multipl │        │ local.ba │          │       │       │ regex on │ ocs/che │
│ e_files/ │        │ sh       │          │       │       │ multiple │ ckouts/ │
│ 4626e7a3 │        │          │          │       │       │ files    │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ regex_o │
│          │        │          │          │       │       │          │ n_filen │
│          │        │          │          │       │       │          │ ame.yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ regex_on │ script │ generic. │ None     │ None  │ None  │ Test     │ /home/d │
│ _directo │        │ local.ba │          │       │       │ regex on │ ocs/che │
│ ry_not_s │        │ sh       │          │       │       │ director │ ckouts/ │
│ upported │        │          │          │       │       │ y is not │ readthe │
│ /a36f76e │        │          │          │       │       │ supporte │ docs.or │
│ 0        │        │          │          │       │       │ d        │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ regex_o │
│          │        │          │          │       │       │          │ n_filen │
│          │        │          │          │       │       │          │ ame.yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ file_exp │ script │ generic. │ None     │ None  │ None  │ Test     │ /home/d │
│ ansion_s │        │ local.ba │          │       │       │ regex    │ ocs/che │
│ upported │        │ sh       │          │       │       │ with     │ ckouts/ │
│ /d611b29 │        │          │          │       │       │ variable │ readthe │
│ 6        │        │          │          │       │       │ and      │ docs.or │
│          │        │          │          │       │       │ shell    │ g/user_ │
│          │        │          │          │       │       │ expansio │ builds/ │
│          │        │          │          │       │       │ n        │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ regex_o │
│          │        │          │          │       │       │          │ n_filen │
│          │        │          │          │       │       │          │ ame.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
regex_on_multiple_files/4626e7a3: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/regex_on_multiple_files/4626e7a3
regex_on_directory_not_supported/a36f76e0: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/regex_on_directory_not_supported/a36f76e0
file_expansion_supported/d611b296: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/file_expansion_supported/d611b296
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
file_expansion_supported/d611b296 does not have any dependencies adding test to queue
regex_on_directory_not_supported/a36f76e0 does not have any dependencies adding test to queue
regex_on_multiple_files/4626e7a3 does not have any dependencies adding test to queue
          Builders Eligible to Run           
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                                   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ file_expansion_supported/d611b296         │
│ regex_on_directory_not_supported/a36f76e0 │
│ regex_on_multiple_files/4626e7a3          │
└───────────────────────────────────────────┘
file_expansion_supported/d611b296: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/file_expansion_supported/d611b296/stage
file_expansion_supported/d611b296: Running Test via command: bash file_expansion_supported_build.sh
file_expansion_supported/d611b296: Test completed in 0.005944 seconds with returncode: 0
file_expansion_supported/d611b296: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/file_expansion_supported/d611b296/file_expansion_supported.out
file_expansion_supported/d611b296: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/file_expansion_supported/d611b296/file_expansion_supported.err
file_expansion_supported/d611b296: Performing regex expression '^(Hello)$' on file /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/hello.txt
file_expansion_supported/d611b296: Regular expression on file /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/hello.txt is a MATCH!
file_expansion_supported/d611b296: Performing regex expression '^(Hello)$' on file /home/docs/hello.txt
file_expansion_supported/d611b296: Regular expression on file /home/docs/hello.txt is a MATCH!
regex_on_directory_not_supported/a36f76e0: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/regex_on_directory_not_supported/a36f76e0/stage
regex_on_directory_not_supported/a36f76e0: Running Test via command: bash regex_on_directory_not_supported_build.sh
regex_on_directory_not_supported/a36f76e0: Test completed in 0.006741 seconds with returncode: 0
regex_on_directory_not_supported/a36f76e0: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/regex_on_directory_not_supported/a36f76e0/regex_on_directory_not_supported.out
regex_on_directory_not_supported/a36f76e0: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/regex_on_directory_not_supported/a36f76e0/regex_on_directory_not_supported.err
regex_on_directory_not_supported/a36f76e0: File: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/regex_on_directory_not_supported/a36f76e0/stage/hello is not a file
regex_on_multiple_files/4626e7a3: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/regex_on_multiple_files/4626e7a3/stage
regex_on_multiple_files/4626e7a3: Running Test via command: bash regex_on_multiple_files_build.sh
regex_on_multiple_files/4626e7a3: Test completed in 0.475765 seconds with returncode: 0
regex_on_multiple_files/4626e7a3: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/regex_on_multiple_files/4626e7a3/regex_on_multiple_files.out
regex_on_multiple_files/4626e7a3: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/regex_on_multiple_files/4626e7a3/regex_on_multiple_files.err
regex_on_multiple_files/4626e7a3: Performing regex expression '^(Hello)$' on file /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/regex_on_multiple_files/4626e7a3/stage/hello.txt
regex_on_multiple_files/4626e7a3: Regular expression on file /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/regex_on_multiple_files/4626e7a3/stage/hello.txt is a MATCH!
regex_on_multiple_files/4626e7a3: Performing regex expression '^(usage: buildtest)' on file /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/regex_on_multiple_files/4626e7a3/stage/buildtest_help.txt
regex_on_multiple_files/4626e7a3: Regular expression on file /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/regex_on_filename/regex_on_multiple_files/4626e7a3/stage/buildtest_help.txt is a MATCH!
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ file_expansion_supporte │ generic.local.bash │ PASS   │ 0          │ 0.006   │
│ d/d611b296              │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ regex_on_multiple_files │ generic.local.bash │ PASS   │ 0          │ 0.476   │
│ /4626e7a3               │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ regex_on_directory_not_ │ generic.local.bash │ FAIL   │ 0          │ 0.007   │
│ supported/a36f76e0      │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 2/3 Percentage: 66.667%
Failed Tests: 1/3 Percentage: 33.333%


Adding 3 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_k6dqfst2.log

Passing Test based on runtime

buildtest can determine state of test based on runtime property which is part of status object. This can be used if you want to control how test PASS or FAIL based on execution time of test. In example below we have five tests that make use of runtime property for passing a test. The runtime property support min and max property that can mark test pass based on minimum and maximum runtime. A test will pass if it’s execution time is greater than min time and less than max time. If min is specified without max property the upperbound is not set, likewise max without min will pass if test is less than max time. The lower bound is not set, but test runtime will be greater than 0 sec.

In test timelimit_min, we sleep for 2 seconds and it will pass because minimum runtime is 1.0 seconds. Similarly, timelimit_max will pass because we sleep for 2 seconds with a max time of 5.0.

buildspecs:
  timelimit_min_max:
    type: script
    executor: generic.local.sh
    description: "Run a sleep job for 2 seconds and test pass if its within 1.0-3.0sec"
    tags: ["tutorials"]
    run: sleep 2
    status:
      runtime:
        min: 1.0
        max: 3.0

  timelimit_min:
    type: script
    executor: generic.local.sh
    description: "Run a sleep job for 2 seconds and test pass if its exceeds min time of 1.0 sec"
    tags: ["tutorials"]
    run: sleep 2
    status:
      runtime:
        min: 1.0

  timelimit_max:
    type: script
    executor: generic.local.sh
    description: "Run a sleep job for 2 seconds and test pass if it's within max time: 5.0 sec"
    tags: ["tutorials"]
    run: sleep 2
    status:
      runtime:
        max: 5.0

  timelimit_min_fail:
    type: script
    executor: generic.local.sh
    description: "This test fails because it runs less than mintime of 10 second"
    tags: ["tutorials"]
    run: sleep 2
    status:
      runtime:
        min: 10.0

  timelimit_max_fail:
    type: script
    executor: generic.local.sh
    description: "This test fails because it exceeds maxtime of 1.0 second"
    tags: ["tutorials"]
    run: sleep 3
    status:
      runtime:
        max: 1.0
buildtest build -b tutorials/test_status/runtime_status_test.yml
$ buildtest build -b tutorials/test_status/runtime_status_test.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:16:32                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/test_status/runtime_status_test.yml                                ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/runtime_status_test.yml: VALID
Total builder objects created: 5
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ timelimi │ script │ generic. │ None     │ None  │ None  │ Run a    │ /home/d │
│ t_min_ma │        │ local.sh │          │       │       │ sleep    │ ocs/che │
│ x/5aca79 │        │          │          │       │       │ job for  │ ckouts/ │
│ 03       │        │          │          │       │       │ 2        │ readthe │
│          │        │          │          │       │       │ seconds  │ docs.or │
│          │        │          │          │       │       │ and test │ g/user_ │
│          │        │          │          │       │       │ pass if  │ builds/ │
│          │        │          │          │       │       │ its      │ buildte │
│          │        │          │          │       │       │ within   │ st/chec │
│          │        │          │          │       │       │ 1.0-3.0s │ kouts/s │
│          │        │          │          │       │       │ ec       │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ runtime │
│          │        │          │          │       │       │          │ _status │
│          │        │          │          │       │       │          │ _test.y │
│          │        │          │          │       │       │          │ ml      │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ timelimi │ script │ generic. │ None     │ None  │ None  │ Run a    │ /home/d │
│ t_min/36 │        │ local.sh │          │       │       │ sleep    │ ocs/che │
│ c043c7   │        │          │          │       │       │ job for  │ ckouts/ │
│          │        │          │          │       │       │ 2        │ readthe │
│          │        │          │          │       │       │ seconds  │ docs.or │
│          │        │          │          │       │       │ and test │ g/user_ │
│          │        │          │          │       │       │ pass if  │ builds/ │
│          │        │          │          │       │       │ its      │ buildte │
│          │        │          │          │       │       │ exceeds  │ st/chec │
│          │        │          │          │       │       │ min time │ kouts/s │
│          │        │          │          │       │       │ of 1.0   │ table/t │
│          │        │          │          │       │       │ sec      │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ runtime │
│          │        │          │          │       │       │          │ _status │
│          │        │          │          │       │       │          │ _test.y │
│          │        │          │          │       │       │          │ ml      │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ timelimi │ script │ generic. │ None     │ None  │ None  │ Run a    │ /home/d │
│ t_max/9d │        │ local.sh │          │       │       │ sleep    │ ocs/che │
│ af7250   │        │          │          │       │       │ job for  │ ckouts/ │
│          │        │          │          │       │       │ 2        │ readthe │
│          │        │          │          │       │       │ seconds  │ docs.or │
│          │        │          │          │       │       │ and test │ g/user_ │
│          │        │          │          │       │       │ pass if  │ builds/ │
│          │        │          │          │       │       │ it's     │ buildte │
│          │        │          │          │       │       │ within   │ st/chec │
│          │        │          │          │       │       │ max      │ kouts/s │
│          │        │          │          │       │       │ time:    │ table/t │
│          │        │          │          │       │       │ 5.0 sec  │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ runtime │
│          │        │          │          │       │       │          │ _status │
│          │        │          │          │       │       │          │ _test.y │
│          │        │          │          │       │       │          │ ml      │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ timelimi │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ t_min_fa │        │ local.sh │          │       │       │ test     │ ocs/che │
│ il/fbd61 │        │          │          │       │       │ fails    │ ckouts/ │
│ e45      │        │          │          │       │       │ because  │ readthe │
│          │        │          │          │       │       │ it runs  │ docs.or │
│          │        │          │          │       │       │ less     │ g/user_ │
│          │        │          │          │       │       │ than     │ builds/ │
│          │        │          │          │       │       │ mintime  │ buildte │
│          │        │          │          │       │       │ of 10    │ st/chec │
│          │        │          │          │       │       │ second   │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ runtime │
│          │        │          │          │       │       │          │ _status │
│          │        │          │          │       │       │          │ _test.y │
│          │        │          │          │       │       │          │ ml      │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ timelimi │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ t_max_fa │        │ local.sh │          │       │       │ test     │ ocs/che │
│ il/70ed4 │        │          │          │       │       │ fails    │ ckouts/ │
│ 45f      │        │          │          │       │       │ because  │ readthe │
│          │        │          │          │       │       │ it       │ docs.or │
│          │        │          │          │       │       │ exceeds  │ g/user_ │
│          │        │          │          │       │       │ maxtime  │ builds/ │
│          │        │          │          │       │       │ of 1.0   │ buildte │
│          │        │          │          │       │       │ second   │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ runtime │
│          │        │          │          │       │       │          │ _status │
│          │        │          │          │       │       │          │ _test.y │
│          │        │          │          │       │       │          │ ml      │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
timelimit_min_max/5aca7903: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_min_max/5aca7903
timelimit_min/36c043c7: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_min/36c043c7
timelimit_max/9daf7250: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_max/9daf7250
timelimit_min_fail/fbd61e45: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_min_fail/fbd61e45
timelimit_max_fail/70ed445f: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_max_fail/70ed445f
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
timelimit_max_fail/70ed445f does not have any dependencies adding test to queue
timelimit_min/36c043c7 does not have any dependencies adding test to queue
timelimit_min_fail/fbd61e45 does not have any dependencies adding test to queue
timelimit_min_max/5aca7903 does not have any dependencies adding test to queue
timelimit_max/9daf7250 does not have any dependencies adding test to queue
   Builders Eligible to Run    
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                     ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ timelimit_max_fail/70ed445f │
│ timelimit_min/36c043c7      │
│ timelimit_min_fail/fbd61e45 │
│ timelimit_min_max/5aca7903  │
│ timelimit_max/9daf7250      │
└─────────────────────────────┘
timelimit_max_fail/70ed445f: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_max_fail/70ed445f/stage
timelimit_max_fail/70ed445f: Running Test via command: bash timelimit_max_fail_build.sh
timelimit_max_fail/70ed445f: Test completed in 3.006201 seconds with returncode: 0
timelimit_max_fail/70ed445f: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_max_fail/70ed445f/timelimit_max_fail.out
timelimit_max_fail/70ed445f: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_max_fail/70ed445f/timelimit_max_fail.err
timelimit_max_fail/70ed445f: Checking runtime < maxtime: 3.006201 < 1.0 
timelimit_min/36c043c7: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_min/36c043c7/stage
timelimit_min/36c043c7: Running Test via command: bash timelimit_min_build.sh
timelimit_min/36c043c7: Test completed in 2.006338 seconds with returncode: 0
timelimit_min/36c043c7: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_min/36c043c7/timelimit_min.out
timelimit_min/36c043c7: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_min/36c043c7/timelimit_min.err
timelimit_min/36c043c7: Checking  mintime < runtime: 1.0 < 2.006338
timelimit_min_fail/fbd61e45: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_min_fail/fbd61e45/stage
timelimit_min_fail/fbd61e45: Running Test via command: bash timelimit_min_fail_build.sh
timelimit_min_fail/fbd61e45: Test completed in 2.006525 seconds with returncode: 0
timelimit_min_fail/fbd61e45: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_min_fail/fbd61e45/timelimit_min_fail.out
timelimit_min_fail/fbd61e45: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_min_fail/fbd61e45/timelimit_min_fail.err
timelimit_min_fail/fbd61e45: Checking  mintime < runtime: 10.0 < 2.006525
timelimit_min_max/5aca7903: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_min_max/5aca7903/stage
timelimit_min_max/5aca7903: Running Test via command: bash timelimit_min_max_build.sh
timelimit_min_max/5aca7903: Test completed in 2.00634 seconds with returncode: 0
timelimit_min_max/5aca7903: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_min_max/5aca7903/timelimit_min_max.out
timelimit_min_max/5aca7903: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_min_max/5aca7903/timelimit_min_max.err
timelimit_min_max/5aca7903: Checking mintime < runtime < maxtime: 1.0 < 2.00634 < 3.0
timelimit_max/9daf7250: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_max/9daf7250/stage
timelimit_max/9daf7250: Running Test via command: bash timelimit_max_build.sh
timelimit_max/9daf7250: Test completed in 2.006543 seconds with returncode: 0
timelimit_max/9daf7250: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_max/9daf7250/timelimit_max.out
timelimit_max/9daf7250: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/runtime_status_test/timelimit_max/9daf7250/timelimit_max.err
timelimit_max/9daf7250: Checking runtime < maxtime: 2.006543 < 5.0 
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                   ┃ executor         ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ timelimit_max/9daf7250    │ generic.local.sh │ PASS   │ 0          │ 2.007   │
├───────────────────────────┼──────────────────┼────────┼────────────┼─────────┤
│ timelimit_max_fail/70ed44 │ generic.local.sh │ FAIL   │ 0          │ 3.006   │
│ 5f                        │                  │        │            │         │
├───────────────────────────┼──────────────────┼────────┼────────────┼─────────┤
│ timelimit_min/36c043c7    │ generic.local.sh │ PASS   │ 0          │ 2.006   │
├───────────────────────────┼──────────────────┼────────┼────────────┼─────────┤
│ timelimit_min_max/5aca790 │ generic.local.sh │ PASS   │ 0          │ 2.006   │
│ 3                         │                  │        │            │         │
├───────────────────────────┼──────────────────┼────────┼────────────┼─────────┤
│ timelimit_min_fail/fbd61e │ generic.local.sh │ FAIL   │ 0          │ 2.007   │
│ 45                        │                  │        │            │         │
└───────────────────────────┴──────────────────┴────────┴────────────┴─────────┘



Passed Tests: 3/5 Percentage: 60.000%
Failed Tests: 2/5 Percentage: 40.000%


Adding 5 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_o_h7mjc6.log

If we look at the test results, we expect the first three tests timelimit_min, timelimit_max, timelimit_min_max will pass while the last two tests fail because it fails to comply with runtime property.

buildtest report --filter buildspec=tutorials/test_status/runtime_status_test.yml --format name,id,state,runtime --latest
$ buildtest report --filter buildspec=tutorials/test_status/runtime_status_test.yml --format name,id,state,runtime --latest
                 Report File: /tmp/tmpjcb9dx2w/var/report.json                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ name                           ┃ id             ┃ state      ┃ runtime       ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ timelimit_max                  │ 9daf7250       │ PASS       │ 2.006543      │
├────────────────────────────────┼────────────────┼────────────┼───────────────┤
│ timelimit_max_fail             │ 70ed445f       │ FAIL       │ 3.006201      │
├────────────────────────────────┼────────────────┼────────────┼───────────────┤
│ timelimit_min                  │ 36c043c7       │ PASS       │ 2.006338      │
├────────────────────────────────┼────────────────┼────────────┼───────────────┤
│ timelimit_min_max              │ 5aca7903       │ PASS       │ 2.00634       │
├────────────────────────────────┼────────────────┼────────────┼───────────────┤
│ timelimit_min_fail             │ fbd61e45       │ FAIL       │ 2.006525      │
└────────────────────────────────┴────────────────┴────────────┴───────────────┘

Explicitly Declaring Status of Test

You can explicitly define status of test regardless of what buildtest does for checking status of test. This can be useful if you want to explicitly mark a test as PASS or FAIL regardless of how test behaves. This can be done via state property which expects one of two types PASS or FAIL. If state property is specified, buildtest will ignore any checks including returncode, regex, or runtime match.

In this next example we will demonstrate how one can use state property for marking test state. In this example we have four tests. The first test always_pass will PASS even though we have a non-zero returncode. The second test always_fail will FAIL even though it has a 0 returncode. The last two test demonstrate how one can define state regardless of what is specified for returncode match. buildtest will honor the state property even if their is a match on the returncode.

buildspecs:
  always_pass:
    type: script
    executor: 'generic.local.sh'
    description: This test will always 'PASS'
    run: exit 1
    status:
      state: PASS

  always_fail:
    type: script
    executor: 'generic.local.sh'
    description: This test will always 'FAIL'
    run: exit 0
    status:
      state: FAIL

  test_fail_returncode_match:
    type: script
    executor: 'generic.local.sh'
    description: This test will 'FAIL' even if we have returncode match
    run: exit 1
    status:
      state: FAIL
      returncode: 1

  test_pass_returncode_mismatch:
    type: script
    executor: 'generic.local.sh'
    description: This test will 'PASS' even if we have returncode mismatch
    run: exit 1
    status:
      state: PASS
      returncode: 2

If we build this test, we expect buildtest to honor the value of state property

buildtest build -b tutorials/test_status/explicit_state.yml
$ buildtest build -b tutorials/test_status/explicit_state.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:16:44                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/test_status/explicit_state.yml                                     ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/explicit_state.yml: VALID
Total builder objects created: 4
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ always_p │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ ass/d822 │        │ local.sh │          │       │       │ test     │ ocs/che │
│ d5a4     │        │          │          │       │       │ will     │ ckouts/ │
│          │        │          │          │       │       │ always   │ readthe │
│          │        │          │          │       │       │ 'PASS'   │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ explici │
│          │        │          │          │       │       │          │ t_state │
│          │        │          │          │       │       │          │ .yml    │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ always_f │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ ail/feab │        │ local.sh │          │       │       │ test     │ ocs/che │
│ 4fcf     │        │          │          │       │       │ will     │ ckouts/ │
│          │        │          │          │       │       │ always   │ readthe │
│          │        │          │          │       │       │ 'FAIL'   │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ explici │
│          │        │          │          │       │       │          │ t_state │
│          │        │          │          │       │       │          │ .yml    │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ test_fai │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ l_return │        │ local.sh │          │       │       │ test     │ ocs/che │
│ code_mat │        │          │          │       │       │ will     │ ckouts/ │
│ ch/54cad │        │          │          │       │       │ 'FAIL'   │ readthe │
│ b12      │        │          │          │       │       │ even if  │ docs.or │
│          │        │          │          │       │       │ we have  │ g/user_ │
│          │        │          │          │       │       │ returnco │ builds/ │
│          │        │          │          │       │       │ de match │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ explici │
│          │        │          │          │       │       │          │ t_state │
│          │        │          │          │       │       │          │ .yml    │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ test_pas │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ s_return │        │ local.sh │          │       │       │ test     │ ocs/che │
│ code_mis │        │          │          │       │       │ will     │ ckouts/ │
│ match/80 │        │          │          │       │       │ 'PASS'   │ readthe │
│ fb6d81   │        │          │          │       │       │ even if  │ docs.or │
│          │        │          │          │       │       │ we have  │ g/user_ │
│          │        │          │          │       │       │ returnco │ builds/ │
│          │        │          │          │       │       │ de       │ buildte │
│          │        │          │          │       │       │ mismatch │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ explici │
│          │        │          │          │       │       │          │ t_state │
│          │        │          │          │       │       │          │ .yml    │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
always_pass/d822d5a4: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/explicit_state/always_pass/d822d5a4
always_fail/feab4fcf: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/explicit_state/always_fail/feab4fcf
test_fail_returncode_match/54cadb12: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/explicit_state/test_fail_returncode_match/54cadb12
test_pass_returncode_mismatch/80fb6d81: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/explicit_state/test_pass_returncode_mismatch/80fb6d81
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
test_pass_returncode_mismatch/80fb6d81 does not have any dependencies adding test to queue
test_fail_returncode_match/54cadb12 does not have any dependencies adding test to queue
always_pass/d822d5a4 does not have any dependencies adding test to queue
always_fail/feab4fcf does not have any dependencies adding test to queue
         Builders Eligible to Run         
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                                ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ test_pass_returncode_mismatch/80fb6d81 │
│ test_fail_returncode_match/54cadb12    │
│ always_pass/d822d5a4                   │
│ always_fail/feab4fcf                   │
└────────────────────────────────────────┘
test_pass_returncode_mismatch/80fb6d81: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/explicit_state/test_pass_returncode_mismatch/80fb6d81/stage
test_pass_returncode_mismatch/80fb6d81: Running Test via command: bash test_pass_returncode_mismatch_build.sh
test_pass_returncode_mismatch/80fb6d81: failed to submit job with returncode: 1
─────────── Error Message for test_pass_returncode_mismatch/80fb6d81 ───────────

test_pass_returncode_mismatch/80fb6d81: Detected failure in running test, will attempt to retry test: 1 times
test_pass_returncode_mismatch/80fb6d81: Run - 1/1
test_pass_returncode_mismatch/80fb6d81: Running Test via command: bash test_pass_returncode_mismatch_build.sh
test_pass_returncode_mismatch/80fb6d81: failed to submit job with returncode: 1
test_pass_returncode_mismatch/80fb6d81: Test completed in 0.013634 seconds with returncode: 1
test_pass_returncode_mismatch/80fb6d81: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/explicit_state/test_pass_returncode_mismatch/80fb6d81/test_pass_returncode_mismatch.out
test_pass_returncode_mismatch/80fb6d81: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/explicit_state/test_pass_returncode_mismatch/80fb6d81/test_pass_returncode_mismatch.err
test_fail_returncode_match/54cadb12: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/explicit_state/test_fail_returncode_match/54cadb12/stage
test_fail_returncode_match/54cadb12: Running Test via command: bash test_fail_returncode_match_build.sh
test_fail_returncode_match/54cadb12: failed to submit job with returncode: 1
──────────── Error Message for test_fail_returncode_match/54cadb12 ─────────────

test_fail_returncode_match/54cadb12: Detected failure in running test, will attempt to retry test: 1 times
test_fail_returncode_match/54cadb12: Run - 1/1
test_fail_returncode_match/54cadb12: Running Test via command: bash test_fail_returncode_match_build.sh
test_fail_returncode_match/54cadb12: failed to submit job with returncode: 1
test_fail_returncode_match/54cadb12: Test completed in 0.012792 seconds with returncode: 1
test_fail_returncode_match/54cadb12: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/explicit_state/test_fail_returncode_match/54cadb12/test_fail_returncode_match.out
test_fail_returncode_match/54cadb12: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/explicit_state/test_fail_returncode_match/54cadb12/test_fail_returncode_match.err
always_pass/d822d5a4: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/explicit_state/always_pass/d822d5a4/stage
always_pass/d822d5a4: Running Test via command: bash always_pass_build.sh
always_pass/d822d5a4: failed to submit job with returncode: 1
──────────────────── Error Message for always_pass/d822d5a4 ────────────────────

always_pass/d822d5a4: Detected failure in running test, will attempt to retry test: 1 times
always_pass/d822d5a4: Run - 1/1
always_pass/d822d5a4: Running Test via command: bash always_pass_build.sh
always_pass/d822d5a4: failed to submit job with returncode: 1
always_pass/d822d5a4: Test completed in 0.012348 seconds with returncode: 1
always_pass/d822d5a4: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/explicit_state/always_pass/d822d5a4/always_pass.out
always_pass/d822d5a4: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/explicit_state/always_pass/d822d5a4/always_pass.err
always_fail/feab4fcf: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/explicit_state/always_fail/feab4fcf/stage
always_fail/feab4fcf: Running Test via command: bash always_fail_build.sh
always_fail/feab4fcf: Test completed in 0.004562 seconds with returncode: 0
always_fail/feab4fcf: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/explicit_state/always_fail/feab4fcf/always_fail.out
always_fail/feab4fcf: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.sh/explicit_state/always_fail/feab4fcf/always_fail.err
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                   ┃ executor         ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ always_pass/d822d5a4      │ generic.local.sh │ PASS   │ 1          │ 0.012   │
├───────────────────────────┼──────────────────┼────────┼────────────┼─────────┤
│ always_fail/feab4fcf      │ generic.local.sh │ FAIL   │ 0          │ 0.005   │
├───────────────────────────┼──────────────────┼────────┼────────────┼─────────┤
│ test_pass_returncode_mism │ generic.local.sh │ PASS   │ 1          │ 0.014   │
│ atch/80fb6d81             │                  │        │            │         │
├───────────────────────────┼──────────────────┼────────┼────────────┼─────────┤
│ test_fail_returncode_matc │ generic.local.sh │ FAIL   │ 1          │ 0.013   │
│ h/54cadb12                │                  │        │            │         │
└───────────────────────────┴──────────────────┴────────┴────────────┴─────────┘



Passed Tests: 2/4 Percentage: 50.000%
Failed Tests: 2/4 Percentage: 50.000%


Adding 4 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_1_lvntuw.log

File Checks

buildtest supports various file checks that can be used as means for passing test. This can include checking for File Existence, File and Directory Checks, Symbolic Link Check, and File Count.

File Existence

For instance, if you want to check for file existence, you can use exists property which expects a list of file or directory names to check. This can be useful if your test will write some output file or directory and test will pass based on existence of file/directory.

In the example below we have two tests, first test will pass, where all files exist. We check for files and directory path, note variable and shell expansion is supported.

In the second example, we expect this test to fail because filename bar does not exist.

buildspecs:
  status_exists:
    type: script
    executor: generic.local.bash
    description: status check based for file and directory
    run: |
      mkdir -p $HOME/dirA
      mkdir -p /tmp/ABC
      touch file1
    status:
      exists:
        - $HOME/dirA
        - ~/.bashrc
        - /tmp/ABC
        - file1
  status_exists_failure:
    type: script
    executor: generic.local.bash
    description: status check failure for existence
    run: touch foo
    status:
      exists:
        - bar

We can run this test by running the following, take note of the output.

buildtest build -b tutorials/test_status/exists.yml
$ buildtest build -b tutorials/test_status/exists.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:16:45                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/test_status/exists.yml                                             ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/exists.yml: VALID
Total builder objects created: 2
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ status_e │ script │ generic. │ None     │ None  │ None  │ status   │ /home/d │
│ xists/e8 │        │ local.ba │          │       │       │ check    │ ocs/che │
│ 911956   │        │ sh       │          │       │       │ based    │ ckouts/ │
│          │        │          │          │       │       │ for file │ readthe │
│          │        │          │          │       │       │ and      │ docs.or │
│          │        │          │          │       │       │ director │ g/user_ │
│          │        │          │          │       │       │ y        │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ exists. │
│          │        │          │          │       │       │          │ yml     │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ status_e │ script │ generic. │ None     │ None  │ None  │ status   │ /home/d │
│ xists_fa │        │ local.ba │          │       │       │ check    │ ocs/che │
│ ilure/ce │        │ sh       │          │       │       │ failure  │ ckouts/ │
│ ab01b7   │        │          │          │       │       │ for      │ readthe │
│          │        │          │          │       │       │ existenc │ docs.or │
│          │        │          │          │       │       │ e        │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ exists. │
│          │        │          │          │       │       │          │ yml     │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
status_exists/e8911956: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/exists/status_exists/e8911956
status_exists_failure/ceab01b7: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/exists/status_exists_failure/ceab01b7
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
status_exists/e8911956 does not have any dependencies adding test to queue
status_exists_failure/ceab01b7 does not have any dependencies adding test to queue
     Builders Eligible to Run     
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ status_exists/e8911956         │
│ status_exists_failure/ceab01b7 │
└────────────────────────────────┘
status_exists/e8911956: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/exists/status_exists/e8911956/stage
status_exists/e8911956: Running Test via command: bash status_exists_build.sh
status_exists/e8911956: Test completed in 0.009116 seconds with returncode: 0
status_exists/e8911956: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/exists/status_exists/e8911956/status_exists.out
status_exists/e8911956: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/exists/status_exists/e8911956/status_exists.err
status_exists/e8911956: Test all files:  ['$HOME/dirA', '~/.bashrc', '/tmp/ABC', 'file1']  existences
status_exists/e8911956: file: /home/docs/dirA exists
status_exists/e8911956: file: /home/docs/.bashrc exists
status_exists/e8911956: file: /tmp/ABC exists
status_exists/e8911956: file: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/exists/status_exists/e8911956/stage/file1 exists
status_exists/e8911956: Exist Check: True
status_exists_failure/ceab01b7: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/exists/status_exists_failure/ceab01b7/stage
status_exists_failure/ceab01b7: Running Test via command: bash status_exists_failure_build.sh
status_exists_failure/ceab01b7: Test completed in 0.006405 seconds with returncode: 0
status_exists_failure/ceab01b7: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/exists/status_exists_failure/ceab01b7/status_exists_failure.out
status_exists_failure/ceab01b7: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/exists/status_exists_failure/ceab01b7/status_exists_failure.err
status_exists_failure/ceab01b7: Test all files:  ['bar']  existences 
status_exists_failure/ceab01b7: file: bar does not exist
status_exists_failure/ceab01b7: Exist Check: False
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ status_exists/e8911956  │ generic.local.bash │ PASS   │ 0          │ 0.009   │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ status_exists_failure/c │ generic.local.bash │ FAIL   │ 0          │ 0.006   │
│ eab01b7                 │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 1/2 Percentage: 50.000%
Failed Tests: 1/2 Percentage: 50.000%


Adding 2 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_5_73qny1.log

Each item in the exists field must be a string, which can lead to issue in example below let’s assume we want a test to pass based on a directory name 1, if we specify as follows, this test will fail validation.

buildspecs:
  file_exists_failure:
   type: script
   executor: generic.local.bash
   description: this test will fail validation, because item must be a string
   run: mkdir -p 1
   status:
     exists: [1]

We can validate this buildspec by running the following

buildtest bc validate -b tutorials/test_status/file_exists_exception.yml
$ buildtest bc validate -b tutorials/test_status/file_exists_exception.yml
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─
1 is not of type 'string'

Failed validating 'type' in schema['properties']['status']['properties']['exists']['items']:
    {'type': 'string'}

On instance['status']['exists'][0]:
    1


1 buildspecs failed to validate

In order to run this test, we need to enclose each item in quotes. Shown below is the same test with quotations.

buildspecs:
  file_exists_pass:
    type: script
    executor: generic.local.bash
    description: this test will pass
    run: mkdir -p 1
    status:
      exists: [ '1' ]

Let’s validate and build this test.

buildtest bc validate -b tutorials/test_status/file_exists_with_number.yml
$ buildtest bc validate -b tutorials/test_status/file_exists_with_number.yml
buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/file_exists_with_number.yml is valid
All buildspecs passed validation!!!
buildtest build -b tutorials/test_status/file_exists_with_number.yml
$ buildtest build -b tutorials/test_status/file_exists_with_number.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:16:47                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/test_status/file_exists_with_number.yml                            ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/file_exists_with_number.yml: VALID
Total builder objects created: 1
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ file_exi │ script │ generic. │ None     │ None  │ None  │ this     │ /home/d │
│ sts_pass │        │ local.ba │          │       │       │ test     │ ocs/che │
│ /4eddc83 │        │ sh       │          │       │       │ will     │ ckouts/ │
│ 7        │        │          │          │       │       │ pass     │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ file_ex │
│          │        │          │          │       │       │          │ ists_wi │
│          │        │          │          │       │       │          │ th_numb │
│          │        │          │          │       │       │          │ er.yml  │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
file_exists_pass/4eddc837: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/4eddc837
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
file_exists_pass/4eddc837 does not have any dependencies adding test to queue
  Builders Eligible to Run   
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ file_exists_pass/4eddc837 │
└───────────────────────────┘
file_exists_pass/4eddc837: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/4eddc837/stage
file_exists_pass/4eddc837: Running Test via command: bash file_exists_pass_build.sh
file_exists_pass/4eddc837: Test completed in 0.007166 seconds with returncode: 0
file_exists_pass/4eddc837: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/4eddc837/file_exists_pass.out
file_exists_pass/4eddc837: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/4eddc837/file_exists_pass.err
file_exists_pass/4eddc837: Test all files:  ['1']  existences 
file_exists_pass/4eddc837: file: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_exists_with_number/file_exists_pass/4eddc837/stage/1 exists
file_exists_pass/4eddc837: Exist Check: True
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ file_exists_pass/4eddc8 │ generic.local.bash │ PASS   │ 0          │ 0.007   │
│ 37                      │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_y9d1_en0.log

File and Directory Checks

In the next example, we introduce checks for files and directory via is_file and is_dir property, which behaves similar to exists except they will check if each item is a file or directory. We expect the first test to fail, because $HOME/.bashrc is not a directory but a file. The second test will incorporate the same test and use is_file for status check.

buildspecs:
  file_and_dir_checks:
    type: script
    executor: generic.local.bash
    description: status check for files and directories
    run: hostname
    status:
      is_dir:
        - $HOME
        - $HOME/.bashrc
        - /tmp
  combined_file_and_dir_checks:
    type: script
    executor: generic.local.bash
    description: status check for files and directories
    run: hostname
    status:
      is_dir:
        - $HOME
        - /tmp
      is_file:
        - $HOME/.bashrc

Let’s build the test and see the output.

buildtest build -b tutorials/test_status/file_and_dir_check.yml
$ buildtest build -b tutorials/test_status/file_and_dir_check.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:16:47                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/test_status/file_and_dir_check.yml                                 ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/file_and_dir_check.yml: VALID
Total builder objects created: 2
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ file_and │ script │ generic. │ None     │ None  │ None  │ status   │ /home/d │
│ _dir_che │        │ local.ba │          │       │       │ check    │ ocs/che │
│ cks/128a │        │ sh       │          │       │       │ for      │ ckouts/ │
│ c80e     │        │          │          │       │       │ files    │ readthe │
│          │        │          │          │       │       │ and      │ docs.or │
│          │        │          │          │       │       │ director │ g/user_ │
│          │        │          │          │       │       │ ies      │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ file_an │
│          │        │          │          │       │       │          │ d_dir_c │
│          │        │          │          │       │       │          │ heck.ym │
│          │        │          │          │       │       │          │ l       │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ combined │ script │ generic. │ None     │ None  │ None  │ status   │ /home/d │
│ _file_an │        │ local.ba │          │       │       │ check    │ ocs/che │
│ d_dir_ch │        │ sh       │          │       │       │ for      │ ckouts/ │
│ ecks/ed7 │        │          │          │       │       │ files    │ readthe │
│ 80719    │        │          │          │       │       │ and      │ docs.or │
│          │        │          │          │       │       │ director │ g/user_ │
│          │        │          │          │       │       │ ies      │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ file_an │
│          │        │          │          │       │       │          │ d_dir_c │
│          │        │          │          │       │       │          │ heck.ym │
│          │        │          │          │       │       │          │ l       │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
file_and_dir_checks/128ac80e: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_and_dir_check/file_and_dir_checks/128ac80e
combined_file_and_dir_checks/ed780719: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_and_dir_check/combined_file_and_dir_checks/ed780719
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
combined_file_and_dir_checks/ed780719 does not have any dependencies adding test to queue
file_and_dir_checks/128ac80e does not have any dependencies adding test to queue
        Builders Eligible to Run         
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ combined_file_and_dir_checks/ed780719 │
│ file_and_dir_checks/128ac80e          │
└───────────────────────────────────────┘
combined_file_and_dir_checks/ed780719: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_and_dir_check/combined_file_and_dir_checks/ed780719/stage
combined_file_and_dir_checks/ed780719: Running Test via command: bash combined_file_and_dir_checks_build.sh
combined_file_and_dir_checks/ed780719: Test completed in 0.006658 seconds with returncode: 0
combined_file_and_dir_checks/ed780719: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_and_dir_check/combined_file_and_dir_checks/ed780719/combined_file_and_dir_checks.out
combined_file_and_dir_checks/ed780719: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_and_dir_check/combined_file_and_dir_checks/ed780719/combined_file_and_dir_checks.err
combined_file_and_dir_checks/ed780719: Test all files:  ['$HOME', '/tmp']  existences
combined_file_and_dir_checks/ed780719: file: /home/docs is a directory 
combined_file_and_dir_checks/ed780719: file: /tmp is a directory 
combined_file_and_dir_checks/ed780719: Directory Existence Check: True
combined_file_and_dir_checks/ed780719: Test all files:  ['$HOME/.bashrc']  existences
combined_file_and_dir_checks/ed780719: file: /home/docs/.bashrc is a file 
combined_file_and_dir_checks/ed780719: File Existence Check: True
file_and_dir_checks/128ac80e: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_and_dir_check/file_and_dir_checks/128ac80e/stage
file_and_dir_checks/128ac80e: Running Test via command: bash file_and_dir_checks_build.sh
file_and_dir_checks/128ac80e: Test completed in 0.006296 seconds with returncode: 0
file_and_dir_checks/128ac80e: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_and_dir_check/file_and_dir_checks/128ac80e/file_and_dir_checks.out
file_and_dir_checks/128ac80e: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_and_dir_check/file_and_dir_checks/128ac80e/file_and_dir_checks.err
file_and_dir_checks/128ac80e: Test all files:  ['$HOME', '$HOME/.bashrc', '/tmp']  existences
file_and_dir_checks/128ac80e: file: /home/docs is a directory 
file_and_dir_checks/128ac80e: file: $HOME/.bashrc is not a directory
file_and_dir_checks/128ac80e: file: /tmp is a directory 
file_and_dir_checks/128ac80e: Directory Existence Check: False
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ file_and_dir_checks/128 │ generic.local.bash │ FAIL   │ 0          │ 0.006   │
│ ac80e                   │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ combined_file_and_dir_c │ generic.local.bash │ PASS   │ 0          │ 0.007   │
│ hecks/ed780719          │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 1/2 Percentage: 50.000%
Failed Tests: 1/2 Percentage: 50.000%


Adding 2 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_8hlckkrb.log

File Count

buildtest can check for number of files in a directory. This can be useful if your test writes number of files and you want to check if the number of files is as expected. You can use the file_count property to perform file count. This property is a list of assertion, where each item is an object. The dir and count are required keys.

The dir is the path to directory to perform directory traversal, and count key is the number of expected files that will be used for comparison. In the first test, we perform a directory walk and expect 5 files in the directory. We can perform directory search based on file extension by using ext key. The ext property can be a string or a list. The second test will perform directory walk on directory named foo and search for file extensions .sh, .py, .txt. The depth property controls the maximum depth for directory traversal, this must be of an integer type. The depth property is optional and if not specified, we will perform full directory traversal.

buildspecs:
  file_count_on_directory:
    type: script
    executor: generic.local.bash
    description: file count check in directory
    run: |
      mkdir -p foo
      touch foo/{1..5}
    status:
      file_count:
        - dir: foo
          count: 5
  file_count_by_extension:
    type: script
    executor: generic.local.bash
    description: file count by extension
    run: |
      mkdir -p foo/bar
      touch foo/{1..5}.sh
      touch foo/bar/{1..3}.py foo/bar/{1..3}.txt
    status:
      file_count:
        - dir: foo
          ext: '.sh'
          depth: 1
          count: 5
        - dir: foo/bar
          ext: ['.py', '.txt']
          count: 6

We can run this test by running the following.

buildtest build -b tutorials/test_status/file_count.yml
$ buildtest build -b tutorials/test_status/file_count.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:16:48                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/test_status/file_count.yml                                         ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/file_count.yml: VALID
Total builder objects created: 2
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ file_cou │ script │ generic. │ None     │ None  │ None  │ file     │ /home/d │
│ nt_on_di │        │ local.ba │          │       │       │ count    │ ocs/che │
│ rectory/ │        │ sh       │          │       │       │ check in │ ckouts/ │
│ 0ab8f3b6 │        │          │          │       │       │ director │ readthe │
│          │        │          │          │       │       │ y        │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ file_co │
│          │        │          │          │       │       │          │ unt.yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ file_cou │ script │ generic. │ None     │ None  │ None  │ file     │ /home/d │
│ nt_by_ex │        │ local.ba │          │       │       │ count by │ ocs/che │
│ tension/ │        │ sh       │          │       │       │ extensio │ ckouts/ │
│ 37ccb038 │        │          │          │       │       │ n        │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ file_co │
│          │        │          │          │       │       │          │ unt.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
file_count_on_directory/0ab8f3b6: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count/file_count_on_directory/0ab8f3b6
file_count_by_extension/37ccb038: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count/file_count_by_extension/37ccb038
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
file_count_on_directory/0ab8f3b6 does not have any dependencies adding test to queue
file_count_by_extension/37ccb038 does not have any dependencies adding test to queue
      Builders Eligible to Run      
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ file_count_on_directory/0ab8f3b6 │
│ file_count_by_extension/37ccb038 │
└──────────────────────────────────┘
file_count_on_directory/0ab8f3b6: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count/file_count_on_directory/0ab8f3b6/stage
file_count_on_directory/0ab8f3b6: Running Test via command: bash file_count_on_directory_build.sh
file_count_on_directory/0ab8f3b6: Test completed in 0.007998 seconds with returncode: 0
file_count_on_directory/0ab8f3b6: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count/file_count_on_directory/0ab8f3b6/file_count_on_directory.out
file_count_on_directory/0ab8f3b6: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count/file_count_on_directory/0ab8f3b6/file_count_on_directory.err
file_count_on_directory/0ab8f3b6: Found 5 file in directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count/file_count_on_directory/0ab8f3b6/stage/foo. Comparing with reference count: 5. Comparison check is 5 == 5 which evaluates to True
file_count_on_directory/0ab8f3b6: File Count Check: True
file_count_by_extension/37ccb038: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count/file_count_by_extension/37ccb038/stage
file_count_by_extension/37ccb038: Running Test via command: bash file_count_by_extension_build.sh
file_count_by_extension/37ccb038: Test completed in 0.008717 seconds with returncode: 0
file_count_by_extension/37ccb038: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count/file_count_by_extension/37ccb038/file_count_by_extension.out
file_count_by_extension/37ccb038: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count/file_count_by_extension/37ccb038/file_count_by_extension.err
file_count_by_extension/37ccb038: Found 5 file in directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count/file_count_by_extension/37ccb038/stage/foo. Comparing with reference count: 5. Comparison check is 5 == 5 which evaluates to True
file_count_by_extension/37ccb038: Found 6 file in directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count/file_count_by_extension/37ccb038/stage/foo/bar. Comparing with reference count: 6. Comparison check is 6 == 6 which evaluates to True
file_count_by_extension/37ccb038: File Count Check: True
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ file_count_by_extension │ generic.local.bash │ PASS   │ 0          │ 0.009   │
│ /37ccb038               │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ file_count_on_directory │ generic.local.bash │ PASS   │ 0          │ 0.008   │
│ /0ab8f3b6               │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%


Adding 2 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_vo6r9toj.log

In the next example, we introduce filepattern property which allows you to check for files based on a pattern. The filepattern property is a regular expression which is compiled via re.compile and applied to a directory path. Please note the regular expression must be valid, otherwise buildtest will not return any files during directory traversal.

You can use filepattern and ext property together to search for files. If both are specified, then we will search for files with both methods and join the two list prior to performing comparison.

buildspecs:
  file_count_by_expression:
    type: script
    executor: generic.local.bash
    description: file count by expression
    run: |
      mkdir -p /tmp
      touch /tmp/foo{1..5}.txt
      ls -l /tmp/foo*.txt
      filenames=$(find $BUILDTEST_ROOT/buildtest -type f  \( -name "defaults.py" -o -name "main.py" \) -maxdepth 1)
      totalfiles=$(find $BUILDTEST_ROOT/buildtest -type f  \( -name "defaults.py" -o -name "main.py" \) -maxdepth 1 | wc -l)
      echo "Filenames: $filenames"
      echo "Total files: $totalfiles"
    status:
      file_count:
        - dir: /tmp
          filepattern: 'foo[1-5].txt$'
          count: 5
        - dir: '$BUILDTEST_ROOT/buildtest'
          filepattern: '(defaults|main).py$'
          count: 2
          depth: 1

  file_extension_and_filepattern:
    type: script
    executor: generic.local.bash
    description: file count by file extension and file pattern
    run: |
      touch foo{1..5}.txt
      touch {conf,main}.py
      ls -l
    status:
      file_count:
        - dir: .
          ext: '.txt'
          filepattern: '(conf|main).py$'
          count: 7

Let’s build this test by running the following:

buildtest build -b tutorials/test_status/file_count_pattern.yml
$ buildtest build -b tutorials/test_status/file_count_pattern.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:16:49                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/test_status/file_count_pattern.yml                                 ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/file_count_pattern.yml: VALID
Total builder objects created: 2
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ file_cou │ script │ generic. │ None     │ None  │ None  │ file     │ /home/d │
│ nt_by_ex │        │ local.ba │          │       │       │ count by │ ocs/che │
│ pression │        │ sh       │          │       │       │ expressi │ ckouts/ │
│ /212f013 │        │          │          │       │       │ on       │ readthe │
│ d        │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ file_co │
│          │        │          │          │       │       │          │ unt_pat │
│          │        │          │          │       │       │          │ tern.ym │
│          │        │          │          │       │       │          │ l       │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ file_ext │ script │ generic. │ None     │ None  │ None  │ file     │ /home/d │
│ ension_a │        │ local.ba │          │       │       │ count by │ ocs/che │
│ nd_filep │        │ sh       │          │       │       │ file     │ ckouts/ │
│ attern/f │        │          │          │       │       │ extensio │ readthe │
│ 16c6d6b  │        │          │          │       │       │ n and    │ docs.or │
│          │        │          │          │       │       │ file     │ g/user_ │
│          │        │          │          │       │       │ pattern  │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ file_co │
│          │        │          │          │       │       │          │ unt_pat │
│          │        │          │          │       │       │          │ tern.ym │
│          │        │          │          │       │       │          │ l       │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
file_count_by_expression/212f013d: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_pattern/file_count_by_expression/212f013d
file_extension_and_filepattern/f16c6d6b: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_pattern/file_extension_and_filepattern/f16c6d6b
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
file_count_by_expression/212f013d does not have any dependencies adding test to queue
file_extension_and_filepattern/f16c6d6b does not have any dependencies adding test to queue
         Builders Eligible to Run          
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                                 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ file_count_by_expression/212f013d       │
│ file_extension_and_filepattern/f16c6d6b │
└─────────────────────────────────────────┘
file_count_by_expression/212f013d: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_pattern/file_count_by_expression/212f013d/stage
file_count_by_expression/212f013d: Running Test via command: bash file_count_by_expression_build.sh
file_count_by_expression/212f013d: Test completed in 0.013648 seconds with returncode: 0
file_count_by_expression/212f013d: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_pattern/file_count_by_expression/212f013d/file_count_by_expression.out
file_count_by_expression/212f013d: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_pattern/file_count_by_expression/212f013d/file_count_by_expression.err
file_count_by_expression/212f013d: Found 5 file in directory: /tmp. Comparing with reference count: 5. Comparison check is 5 == 5 which evaluates to True
file_count_by_expression/212f013d: Found 2 file in directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/buildtest. Comparing with reference count: 2. Comparison check is 2 == 2 which evaluates to True
file_count_by_expression/212f013d: File Count Check: True
file_extension_and_filepattern/f16c6d6b: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_pattern/file_extension_and_filepattern/f16c6d6b/stage
file_extension_and_filepattern/f16c6d6b: Running Test via command: bash file_extension_and_filepattern_build.sh
file_extension_and_filepattern/f16c6d6b: Test completed in 0.00903 seconds with returncode: 0
file_extension_and_filepattern/f16c6d6b: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_pattern/file_extension_and_filepattern/f16c6d6b/file_extension_and_filepattern.out
file_extension_and_filepattern/f16c6d6b: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_pattern/file_extension_and_filepattern/f16c6d6b/file_extension_and_filepattern.err
file_extension_and_filepattern/f16c6d6b: Found 7 file in directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_pattern/file_extension_and_filepattern/f16c6d6b/stage. Comparing with reference count: 7. Comparison check is 7 == 7 which evaluates to True
file_extension_and_filepattern/f16c6d6b: File Count Check: True
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ file_extension_and_file │ generic.local.bash │ PASS   │ 0          │ 0.009   │
│ pattern/f16c6d6b        │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ file_count_by_expressio │ generic.local.bash │ PASS   │ 0          │ 0.014   │
│ n/212f013d              │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%


Adding 2 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_hudwh2ic.log

In the next example, we will introduce filetype property that can be used to filter directory search based on file type. The filetype property can one of the following values file, dir, symlink. Once set, the directory traversal will seek out files based on the file type. Note that when filetype is set to dir we will return the parent directory and all sub-directories. This test will create a few subdirectories and create symbolic link, next we will perform directory search by directory and symbolic link. We expect this test to pass as we will find 3 directories and 2 symbolic links.

buildspecs:
  file_count_by_filetype:
    type: script
    executor: generic.local.bash
    description: Count the number of directories and symbolic links
    run: |
      mkdir -p foo/{bar,baz}
      find foo -type dir
      ln -s foo/bar foo/bar.link
      ln -s foo/baz foo/baz.link
    status:
      file_count:
        - dir: foo
          count: 3
          filetype: 'dir'
        - dir: foo
          count: 2
          filetype: 'symlink'

Let’s build this test by running the following:

buildtest build -b tutorials/test_status/file_count_filetype.yml
$ buildtest build -b tutorials/test_status/file_count_filetype.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:16:50                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/test_status/file_count_filetype.yml                                ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/file_count_filetype.yml: VALID
Total builder objects created: 1
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ file_cou │ script │ generic. │ None     │ None  │ None  │ Count    │ /home/d │
│ nt_by_fi │        │ local.ba │          │       │       │ the      │ ocs/che │
│ letype/b │        │ sh       │          │       │       │ number   │ ckouts/ │
│ c445933  │        │          │          │       │       │ of       │ readthe │
│          │        │          │          │       │       │ director │ docs.or │
│          │        │          │          │       │       │ ies and  │ g/user_ │
│          │        │          │          │       │       │ symbolic │ builds/ │
│          │        │          │          │       │       │ links    │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ file_co │
│          │        │          │          │       │       │          │ unt_fil │
│          │        │          │          │       │       │          │ etype.y │
│          │        │          │          │       │       │          │ ml      │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
file_count_by_filetype/bc445933: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_filetype/file_count_by_filetype/bc445933
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
file_count_by_filetype/bc445933 does not have any dependencies adding test to queue
     Builders Eligible to Run      
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                         ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ file_count_by_filetype/bc445933 │
└─────────────────────────────────┘
file_count_by_filetype/bc445933: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_filetype/file_count_by_filetype/bc445933/stage
file_count_by_filetype/bc445933: Running Test via command: bash file_count_by_filetype_build.sh
file_count_by_filetype/bc445933: failed to submit job with returncode: 1
────────────── Error Message for file_count_by_filetype/bc445933 ───────────────
find: Must separate multiple arguments to -type using: ','

file_count_by_filetype/bc445933: Detected failure in running test, will attempt to retry test: 1 times
file_count_by_filetype/bc445933: Run - 1/1
file_count_by_filetype/bc445933: Running Test via command: bash file_count_by_filetype_build.sh
file_count_by_filetype/bc445933: failed to submit job with returncode: 1
file_count_by_filetype/bc445933: Test completed in 0.020411 seconds with returncode: 1
file_count_by_filetype/bc445933: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_filetype/file_count_by_filetype/bc445933/file_count_by_filetype.out
file_count_by_filetype/bc445933: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_filetype/file_count_by_filetype/bc445933/file_count_by_filetype.err
file_count_by_filetype/bc445933: Found 3 file in directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_filetype/file_count_by_filetype/bc445933/stage/foo. Comparing with reference count: 3. Comparison check is 3 == 3 which evaluates to True
file_count_by_filetype/bc445933: Found 0 file in directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_filetype/file_count_by_filetype/bc445933/stage/foo. Comparing with reference count: 2. Comparison check is 0 == 2 which evaluates to False
file_count_by_filetype/bc445933: File Count Check: False
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ file_count_by_filetype/ │ generic.local.bash │ FAIL   │ 1          │ 0.020   │
│ bc445933                │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 0/1 Percentage: 0.000%
Failed Tests: 1/1 Percentage: 100.000%


Adding 1 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_uvcyyy1u.log

Buildtest will perform a directory walk when using file_count, which can run into performance issues if you have a large directory. We have added a safety check during directory traversal to a maximum of 999999 files. You have the option to configure the directory traversal limit using file_traversal_limit which is an integer, the default value is 10000 if not specified. The minimum value and maximum value can be 1 and 999999 respectively.

In this next example, we will illustrate how this feature works. We will create 99 .txt files in directory foo. We will perform two assertions with different values for file_traversal_limit. In the first example, we will set this to 50 and expect 50 files returned. We expect this check to be True. In the next example, we will set file_traversal_limit to 20 and set count: 10 where we should expect a failure. In principle, we should retrieve 20 files but this will mismatch the comparison check.

buildspecs:
  file_traverse_limit:
    type: script
    executor: generic.local.bash
    description: Use of file_traverse_limit to limit number of files searched in a directory
    run: |
      mkdir foo
      touch foo/{1..99}.txt
    status:
      file_count:
        - dir: foo
          count: 50
          file_traverse_limit: 50
        - dir: foo
          count: 10
          file_traverse_limit: 20

We can try building this test by running the following:

buildtest build -b tutorials/test_status/file_count_file_traverse_limit.yml
$ buildtest build -b tutorials/test_status/file_count_file_traverse_limit.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:16:50                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/test_status/file_count_file_traverse_limit.yml                     ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/file_count_file_traverse_limit.yml: VALID
Total builder objects created: 1
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ file_tra │ script │ generic. │ None     │ None  │ None  │ Use of   │ /home/d │
│ verse_li │        │ local.ba │          │       │       │ file_tra │ ocs/che │
│ mit/1de5 │        │ sh       │          │       │       │ verse_li │ ckouts/ │
│ 1e1a     │        │          │          │       │       │ mit to   │ readthe │
│          │        │          │          │       │       │ limit    │ docs.or │
│          │        │          │          │       │       │ number   │ g/user_ │
│          │        │          │          │       │       │ of files │ builds/ │
│          │        │          │          │       │       │ searched │ buildte │
│          │        │          │          │       │       │ in a     │ st/chec │
│          │        │          │          │       │       │ director │ kouts/s │
│          │        │          │          │       │       │ y        │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ file_co │
│          │        │          │          │       │       │          │ unt_fil │
│          │        │          │          │       │       │          │ e_trave │
│          │        │          │          │       │       │          │ rse_lim │
│          │        │          │          │       │       │          │ it.yml  │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
file_traverse_limit/1de51e1a: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_file_traverse_limit/file_traverse_limit/1de51e1a
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
file_traverse_limit/1de51e1a does not have any dependencies adding test to queue
    Builders Eligible to Run    
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                      ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ file_traverse_limit/1de51e1a │
└──────────────────────────────┘
file_traverse_limit/1de51e1a: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_file_traverse_limit/file_traverse_limit/1de51e1a/stage
file_traverse_limit/1de51e1a: Running Test via command: bash file_traverse_limit_build.sh
file_traverse_limit/1de51e1a: Test completed in 0.010123 seconds with returncode: 0
file_traverse_limit/1de51e1a: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_file_traverse_limit/file_traverse_limit/1de51e1a/file_traverse_limit.out
file_traverse_limit/1de51e1a: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_file_traverse_limit/file_traverse_limit/1de51e1a/file_traverse_limit.err
file_traverse_limit/1de51e1a: Found 50 file in directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_file_traverse_limit/file_traverse_limit/1de51e1a/stage/foo. Comparing with reference count: 50. Comparison check is 50 == 50 which evaluates to True
file_traverse_limit/1de51e1a: Found 20 file in directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/file_count_file_traverse_limit/file_traverse_limit/1de51e1a/stage/foo. Comparing with reference count: 10. Comparison check is 20 == 10 which evaluates to False
file_traverse_limit/1de51e1a: File Count Check: False
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ file_traverse_limit/1de │ generic.local.bash │ FAIL   │ 0          │ 0.010   │
│ 51e1a                   │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 0/1 Percentage: 0.000%
Failed Tests: 1/1 Percentage: 100.000%


Adding 1 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_564ofagq.log

Status Mode

By default, the status check performed by buildtest is a logical OR, where if any of the status check is True, then the test will PASS. However, if you want to change this behavior to logical AND, you can use the mode property. The valid values can be [AND, and, OR, or]. In the example below, we have two tests that illustrate the use of mode.

buildspecs:
  status_logical_and:
    type: script
    executor: 'generic.local.bash'
    description: 'Using logical AND to check status'
    run: |
      echo "This is a test"
      exit 1
    status:
      mode: and
      returncode: 1
      regex:
        stream: stdout
        exp: 'This is a test'

  status_logical_or:
    type: script
    executor: 'generic.local.bash'
    description: 'Using logical OR to check status'
    run: |
      echo "This is a test"
      exit 1
    status:
      mode: or
      returncode: 0
      regex:
        stream: stdout
        exp: 'This is a test'

The first test uses mode: and which implies all status check are evaluated as logical AND, we expect this test to PASS. In the second test, we use mode: or where status check are evaluated as logical OR which is the default behavior. Note if mode is not specified, it is equivalent to mode: or. In second test, we expect this to pass because regex check will PASS however, the returncode check will fail due to mismatch in returncode. If we changed this to mode: and then we would expect this test to fail.

Shown below is the output of running this test.

buildtest build -b tutorials/test_status/mode.yml
$ buildtest build -b tutorials/test_status/mode.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:16:51                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/test_status/mode.yml                                               ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/mode.yml: VALID
Total builder objects created: 2
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ status_l │ script │ generic. │ None     │ None  │ None  │ Using    │ /home/d │
│ ogical_a │        │ local.ba │          │       │       │ logical  │ ocs/che │
│ nd/6b18b │        │ sh       │          │       │       │ AND to   │ ckouts/ │
│ c25      │        │          │          │       │       │ check    │ readthe │
│          │        │          │          │       │       │ status   │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ mode.ym │
│          │        │          │          │       │       │          │ l       │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ status_l │ script │ generic. │ None     │ None  │ None  │ Using    │ /home/d │
│ ogical_o │        │ local.ba │          │       │       │ logical  │ ocs/che │
│ r/4b0d0d │        │ sh       │          │       │       │ OR to    │ ckouts/ │
│ 09       │        │          │          │       │       │ check    │ readthe │
│          │        │          │          │       │       │ status   │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/test_ │
│          │        │          │          │       │       │          │ status/ │
│          │        │          │          │       │       │          │ mode.ym │
│          │        │          │          │       │       │          │ l       │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
status_logical_and/6b18bc25: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/mode/status_logical_and/6b18bc25
status_logical_or/4b0d0d09: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/mode/status_logical_or/4b0d0d09
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
status_logical_or/4b0d0d09 does not have any dependencies adding test to queue
status_logical_and/6b18bc25 does not have any dependencies adding test to queue
   Builders Eligible to Run    
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                     ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ status_logical_or/4b0d0d09  │
│ status_logical_and/6b18bc25 │
└─────────────────────────────┘
status_logical_or/4b0d0d09: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/mode/status_logical_or/4b0d0d09/stage
status_logical_or/4b0d0d09: Running Test via command: bash status_logical_or_build.sh
status_logical_or/4b0d0d09: failed to submit job with returncode: 1
───────────────── Error Message for status_logical_or/4b0d0d09 ─────────────────

status_logical_or/4b0d0d09: Detected failure in running test, will attempt to retry test: 1 times
status_logical_or/4b0d0d09: Run - 1/1
status_logical_or/4b0d0d09: Running Test via command: bash status_logical_or_build.sh
status_logical_or/4b0d0d09: failed to submit job with returncode: 1
status_logical_or/4b0d0d09: Test completed in 0.014795 seconds with returncode: 1
status_logical_or/4b0d0d09: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/mode/status_logical_or/4b0d0d09/status_logical_or.out
status_logical_or/4b0d0d09: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/mode/status_logical_or/4b0d0d09/status_logical_or.err
status_logical_or/4b0d0d09: Checking returncode - 1 is matched in list [0]
status_logical_or/4b0d0d09: performing regular expression - 'This is a test' on file: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/mode/status_logical_or/4b0d0d09/status_logical_or.out
status_logical_or/4b0d0d09: Regular Expression Match - Success!
status_logical_and/6b18bc25: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/mode/status_logical_and/6b18bc25/stage
status_logical_and/6b18bc25: Running Test via command: bash status_logical_and_build.sh
status_logical_and/6b18bc25: failed to submit job with returncode: 1
──────────────── Error Message for status_logical_and/6b18bc25 ─────────────────

status_logical_and/6b18bc25: Detected failure in running test, will attempt to retry test: 1 times
status_logical_and/6b18bc25: Run - 1/1
status_logical_and/6b18bc25: Running Test via command: bash status_logical_and_build.sh
status_logical_and/6b18bc25: failed to submit job with returncode: 1
status_logical_and/6b18bc25: Test completed in 0.014419 seconds with returncode: 1
status_logical_and/6b18bc25: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/mode/status_logical_and/6b18bc25/status_logical_and.out
status_logical_and/6b18bc25: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/mode/status_logical_and/6b18bc25/status_logical_and.err
status_logical_and/6b18bc25: Checking returncode - 1 is matched in list [1]
status_logical_and/6b18bc25: performing regular expression - 'This is a test' on file: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/mode/status_logical_and/6b18bc25/status_logical_and.out
status_logical_and/6b18bc25: Regular Expression Match - Success!
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ status_logical_or/4b0d0 │ generic.local.bash │ PASS   │ 1          │ 0.015   │
│ d09                     │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ status_logical_and/6b18 │ generic.local.bash │ PASS   │ 1          │ 0.014   │
│ bc25                    │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%


Adding 2 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_ye74ea9q.log

Skipping test

By default, buildtest will run all tests defined in buildspecs section, if you want to skip a test use the skip field which expects a boolean value. Shown below is an example test.

buildspecs:
  skip:
    type: script
    executor: generic.local.bash
    description: This test is skipped
    skip: Yes
    tags: [tutorials]
    run: hostname

  unskipped:
    type: script
    executor: generic.local.bash
    description: This test is not skipped
    skip: No
    tags: [tutorials]
    run: hostname

The first test skip will be ignored by buildtest because skip: true is defined while unskipped will be processed as usual.

Note

YAML and JSON have different representation for boolean. For json schema valid values are true and false see https://json-schema.org/understanding-json-schema/reference/boolean.html however YAML has many more representation for boolean see https://yaml.org/type/bool.html. You may use any of the YAML boolean, however it’s best to stick with json schema values true and false.

Here is an example build, notice message [skip] test is skipped during the build stage

buildtest build -b tutorials/skip_tests.yml
$ buildtest build -b tutorials/skip_tests.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:16:52                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/skip_tests.yml                                                     ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
skip: skipping test due to 'skip' property.
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/skip_tests.yml: VALID
Total builder objects created: 1
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ unskippe │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ d/09a35b │        │ local.ba │          │       │       │ test is  │ ocs/che │
│ 9d       │        │ sh       │          │       │       │ not      │ ckouts/ │
│          │        │          │          │       │       │ skipped  │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/skip_ │
│          │        │          │          │       │       │          │ tests.y │
│          │        │          │          │       │       │          │ ml      │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
unskipped/09a35b9d: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/skip_tests/unskipped/09a35b9d
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
unskipped/09a35b9d does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━┓
┃ Builder            ┃
┡━━━━━━━━━━━━━━━━━━━━┩
│ unskipped/09a35b9d │
└────────────────────┘
unskipped/09a35b9d: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/skip_tests/unskipped/09a35b9d/stage
unskipped/09a35b9d: Running Test via command: bash unskipped_build.sh
unskipped/09a35b9d: Test completed in 0.00658 seconds with returncode: 0
unskipped/09a35b9d: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/skip_tests/unskipped/09a35b9d/unskipped.out
unskipped/09a35b9d: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/skip_tests/unskipped/09a35b9d/unskipped.err
                               Test Summary                                
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder            ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ unskipped/09a35b9d │ generic.local.bash │ PASS   │ 0          │ 0.007   │
└────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_bt2uhm69.log

Skipping a buildspec

Sometimes you may want to skip all test in a buildspec instead of updating every test with skip property, this can be done by setting skip at the top-level. This can be useful if you are running several test in a directory such as buildtest build -b dir1/ and you don’t want to explicitly exclude file via -x option every time, instead you can hardcode this into the buildspec. A typical use-case of skipping test is when a test is broken and you don’t want to run it then its good idea to set skip: yes on the buildspec and fix it later.

In this next example we set skip: yes, buildtest will skip the buildspec and no test will be processed even if skip is set in each test.

skip: yes
buildspecs:
  skip_all_tests:
    type: script
    executor: generic.local.bash
    description: "All test in this buildspec are skipped"
    tags: [tutorials]
    run: hostname

  this_test_is_also_skipped:
    type: script
    skip: no
    executor: generic.local.bash
    description: "This test is also skipped even if skip is defined in test"
    tags: [ tutorials ]
    run: hostname

If you try building this buildspec, you will see buildtest will skip the buildspec and terminate.

buildtest build -b tutorials/skip_buildspec.yml
$ buildtest build -b tutorials/skip_buildspec.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:16:53                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/skip_buildspec.yml                                                 ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/skip_buildspec.yml: skipping all test since 'skip' is defined
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/skip_buildspec.yml: VALID
                            Buildspecs Filtered out                             
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ buildspecs                                                                   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… │
└──────────────────────────────────────────────────────────────────────────────┘

buildtest is unable to create any tests because there are no valid buildspecs. 

Please see logfile: /tmp/tmpjcb9dx2w/var/buildtest.log