Test Dependency

Note

This feature is subject to change

Buildtest can support test dependencies which allows one to specify condition before running a test. Let’s take a look at this next example, we have a buildspec with three tests jobA, jobB, and jobC. The test jobA will run immediately but now we introduce a new keyword needs which is a list of test names as dependency. We want test jobB to run after jobA is complete, and jobC to run once jobA and jobB is complete.

buildspecs:
  jobA:
    type: script
    executor: generic.local.bash
    description: no job dependency
    run: |
      echo "This job has no dependency"
      sleep 5

  jobB:
    type: script
    executor: generic.local.bash
    description: job dependency on jobA
    needs: [jobA]
    run: |
      echo "This job depends on jobA"
      sleep 2

  jobC:
    type: script
    executor: generic.local.bash
    description: job dependency on jobA and jobB
    needs: [jobA, jobB]
    run: |
      echo "This job depends on jobA and jobB"
      sleep 2

The needs property expects a list of strings, and values must match name of test. If you specify an invalid test name in needs property then buildtest will ignore the value. If multiple tests are specified in needs property then all test must finish prior to running test.

Let’s run this test, and take a note that buildtest will run test jobA, followed by jobB then jobC.

buildtest build -b tutorials/job_dependency/ex1.yml
$ buildtest build -b tutorials/job_dependency/ex1.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:19:49                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/job_dependency/ex1.yml                                             ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/job_dependency/ex1.yml: VALID
Total builder objects created: 3
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ jobA/98f │ script │ generic. │ None     │ None  │ None  │ no job   │ /home/d │
│ 86743    │        │ local.ba │          │       │       │ dependen │ ocs/che │
│          │        │ sh       │          │       │       │ cy       │ ckouts/ │
│          │        │          │          │       │       │          │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/job_d │
│          │        │          │          │       │       │          │ ependen │
│          │        │          │          │       │       │          │ cy/ex1. │
│          │        │          │          │       │       │          │ yml     │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ jobB/587 │ script │ generic. │ None     │ None  │ None  │ job      │ /home/d │
│ 98039    │        │ local.ba │          │       │       │ dependen │ ocs/che │
│          │        │ sh       │          │       │       │ cy on    │ ckouts/ │
│          │        │          │          │       │       │ jobA     │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/job_d │
│          │        │          │          │       │       │          │ ependen │
│          │        │          │          │       │       │          │ cy/ex1. │
│          │        │          │          │       │       │          │ yml     │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ jobC/8f5 │ script │ generic. │ None     │ None  │ None  │ job      │ /home/d │
│ 5b448    │        │ local.ba │          │       │       │ dependen │ ocs/che │
│          │        │ sh       │          │       │       │ cy on    │ ckouts/ │
│          │        │          │          │       │       │ jobA and │ readthe │
│          │        │          │          │       │       │ jobB     │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/job_d │
│          │        │          │          │       │       │          │ ependen │
│          │        │          │          │       │       │          │ cy/ex1. │
│          │        │          │          │       │       │          │ yml     │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
jobA/98f86743: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex1/jobA/98f86743
jobB/58798039: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex1/jobB/58798039
jobC/8f55b448: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex1/jobC/8f55b448
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
jobC/8f55b448 Skipping job because it has job dependency on jobA/98f86743 which is in state PENDING
jobA/98f86743 does not have any dependencies adding test to queue
jobB/58798039 Skipping job because it has job dependency on jobA/98f86743 which is in state PENDING
Builders Eligible to Run
┏━━━━━━━━━━━━━━━┓
┃ Builder       ┃
┡━━━━━━━━━━━━━━━┩
│ jobA/98f86743 │
└───────────────┘
jobA/98f86743: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex1/jobA/98f86743/stage
jobA/98f86743: Running Test via command: bash jobA_build.sh
jobA/98f86743: Test completed in 5.007491 seconds with returncode: 0
jobA/98f86743: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex1/jobA/98f86743/jobA.out
jobA/98f86743: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex1/jobA/98f86743/jobA.err
───────────────────────────────── Iteration 2 ──────────────────────────────────
jobC/8f55b448 Skipping job because it has job dependency on jobB/58798039 which is in state PENDING
Builders Eligible to Run
┏━━━━━━━━━━━━━━━┓
┃ Builder       ┃
┡━━━━━━━━━━━━━━━┩
│ jobB/58798039 │
└───────────────┘
jobB/58798039: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex1/jobB/58798039/stage
jobB/58798039: Running Test via command: bash jobB_build.sh
jobB/58798039: Test completed in 2.007787 seconds with returncode: 0
jobB/58798039: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex1/jobB/58798039/jobB.out
jobB/58798039: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex1/jobB/58798039/jobB.err
───────────────────────────────── Iteration 3 ──────────────────────────────────
Builders Eligible to Run
┏━━━━━━━━━━━━━━━┓
┃ Builder       ┃
┡━━━━━━━━━━━━━━━┩
│ jobC/8f55b448 │
└───────────────┘
jobC/8f55b448: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex1/jobC/8f55b448/stage
jobC/8f55b448: Running Test via command: bash jobC_build.sh
jobC/8f55b448: Test completed in 2.007635 seconds with returncode: 0
jobC/8f55b448: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex1/jobC/8f55b448/jobC.out
jobC/8f55b448: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex1/jobC/8f55b448/jobC.err
                             Test Summary                             
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder       ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ jobC/8f55b448 │ generic.local.bash │ PASS   │ 0          │ 2.008   │
├───────────────┼────────────────────┼────────┼────────────┼─────────┤
│ jobA/98f86743 │ generic.local.bash │ PASS   │ 0          │ 5.007   │
├───────────────┼────────────────────┼────────┼────────────┼─────────┤
│ jobB/58798039 │ generic.local.bash │ PASS   │ 0          │ 2.008   │
└───────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 3/3 Percentage: 100.000%
Failed Tests: 0/3 Percentage: 0.000%


Adding 3 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_fu9aj3cb.log

Test Dependency by returncode

In this next example, we can control behavior of job dependency based on returncode for a given test. This test has three tests: test1, test2 and test3. The first test will exit with returncode 1 but this test will pass because we have set state: PASS to override the status check. The next test test2 requires test1 to have a returncode of 1 in order to satisfy dependency. The returncode property expects a valid returncode and it can be a list of returncode similar to how one specify returncode under the status property see Return Code Matching. The needs property can support multiple test with returncode, in test3 we require test1 to have returncode 1 while test2 has a returncode of 2. We expect test2 to return a returncode of 2 because of exit 2 statement so we expect all three tests to run.

buildspecs:
  test1:
    type: script
    executor: generic.local.bash
    description: This test will pass with exit 1
    run: exit 1
    status:
      state: PASS

  test2:
    type: script
    executor: generic.local.bash
    description: This test will run if test1 has returncode 1
    run: exit 2
    status:
      state: PASS
    needs:
      - test1:
          returncode: 1

  test3:
    type: script
    executor: generic.local.bash
    description: This test will run if test1 has returncode 1 and test2 has returncode 2
    run: exit 1
    status:
      state: PASS
    needs:
      - test1:
          returncode: 1
      - test2:
          returncode: 2

Let’s build this test and take note of execution order of test.

buildtest build -b tutorials/job_dependency/ex2.yml
$ buildtest build -b tutorials/job_dependency/ex2.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:19:59                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/job_dependency/ex2.yml                                             ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/job_dependency/ex2.yml: VALID
Total builder objects created: 3
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ test1/f6 │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ e1b5e9   │        │ local.ba │          │       │       │ test     │ ocs/che │
│          │        │ sh       │          │       │       │ will     │ ckouts/ │
│          │        │          │          │       │       │ pass     │ readthe │
│          │        │          │          │       │       │ with     │ docs.or │
│          │        │          │          │       │       │ exit 1   │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/job_d │
│          │        │          │          │       │       │          │ ependen │
│          │        │          │          │       │       │          │ cy/ex2. │
│          │        │          │          │       │       │          │ yml     │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ test2/c7 │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ a26255   │        │ local.ba │          │       │       │ test     │ ocs/che │
│          │        │ sh       │          │       │       │ will run │ ckouts/ │
│          │        │          │          │       │       │ if test1 │ readthe │
│          │        │          │          │       │       │ has      │ docs.or │
│          │        │          │          │       │       │ returnco │ g/user_ │
│          │        │          │          │       │       │ de 1     │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/job_d │
│          │        │          │          │       │       │          │ ependen │
│          │        │          │          │       │       │          │ cy/ex2. │
│          │        │          │          │       │       │          │ yml     │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ test3/bb │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ 109ee6   │        │ local.ba │          │       │       │ test     │ ocs/che │
│          │        │ sh       │          │       │       │ will run │ ckouts/ │
│          │        │          │          │       │       │ if test1 │ readthe │
│          │        │          │          │       │       │ has      │ docs.or │
│          │        │          │          │       │       │ returnco │ g/user_ │
│          │        │          │          │       │       │ de 1 and │ builds/ │
│          │        │          │          │       │       │ test2    │ buildte │
│          │        │          │          │       │       │ has      │ st/chec │
│          │        │          │          │       │       │ returnco │ kouts/s │
│          │        │          │          │       │       │ de 2     │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/job_d │
│          │        │          │          │       │       │          │ ependen │
│          │        │          │          │       │       │          │ cy/ex2. │
│          │        │          │          │       │       │          │ yml     │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
test1/f6e1b5e9: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex2/test1/f6e1b5e9
test2/c7a26255: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex2/test2/c7a26255
test3/bb109ee6: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex2/test3/bb109ee6
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
test3/bb109ee6 Skipping job because it has job dependency on test1/f6e1b5e9 
test2/c7a26255 Skipping job because it has job dependency on test1/f6e1b5e9 
test1/f6e1b5e9 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━┓
┃ Builder        ┃
┡━━━━━━━━━━━━━━━━┩
│ test1/f6e1b5e9 │
└────────────────┘
test1/f6e1b5e9: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex2/test1/f6e1b5e9/stage
test1/f6e1b5e9: Running Test via command: bash test1_build.sh
test1/f6e1b5e9: failed to submit job with returncode: 1
─────────────────────── Error Message for test1/f6e1b5e9 ───────────────────────

test1/f6e1b5e9: Detected failure in running test, will attempt to retry test: 1 times
test1/f6e1b5e9: Run - 1/1
test1/f6e1b5e9: Running Test via command: bash test1_build.sh
test1/f6e1b5e9: failed to submit job with returncode: 1
test1/f6e1b5e9: Test completed in 0.014774 seconds with returncode: 1
test1/f6e1b5e9: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex2/test1/f6e1b5e9/test1.out
test1/f6e1b5e9: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex2/test1/f6e1b5e9/test1.err
───────────────────────────────── Iteration 2 ──────────────────────────────────
test3/bb109ee6 Skipping job because it has job dependency on test2/c7a26255 
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━┓
┃ Builder        ┃
┡━━━━━━━━━━━━━━━━┩
│ test2/c7a26255 │
└────────────────┘
test2/c7a26255: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex2/test2/c7a26255/stage
test2/c7a26255: Running Test via command: bash test2_build.sh
test2/c7a26255: failed to submit job with returncode: 2
─────────────────────── Error Message for test2/c7a26255 ───────────────────────

test2/c7a26255: Detected failure in running test, will attempt to retry test: 1 times
test2/c7a26255: Run - 1/1
test2/c7a26255: Running Test via command: bash test2_build.sh
test2/c7a26255: failed to submit job with returncode: 2
test2/c7a26255: Test completed in 0.014043 seconds with returncode: 2
test2/c7a26255: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex2/test2/c7a26255/test2.out
test2/c7a26255: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex2/test2/c7a26255/test2.err
───────────────────────────────── Iteration 3 ──────────────────────────────────
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━┓
┃ Builder        ┃
┡━━━━━━━━━━━━━━━━┩
│ test3/bb109ee6 │
└────────────────┘
test3/bb109ee6: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex2/test3/bb109ee6/stage
test3/bb109ee6: Running Test via command: bash test3_build.sh
test3/bb109ee6: failed to submit job with returncode: 1
─────────────────────── Error Message for test3/bb109ee6 ───────────────────────

test3/bb109ee6: Detected failure in running test, will attempt to retry test: 1 times
test3/bb109ee6: Run - 1/1
test3/bb109ee6: Running Test via command: bash test3_build.sh
test3/bb109ee6: failed to submit job with returncode: 1
test3/bb109ee6: Test completed in 0.014203 seconds with returncode: 1
test3/bb109ee6: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex2/test3/bb109ee6/test3.out
test3/bb109ee6: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex2/test3/bb109ee6/test3.err
                             Test Summary                              
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder        ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ test3/bb109ee6 │ generic.local.bash │ PASS   │ 1          │ 0.014   │
├────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ test1/f6e1b5e9 │ generic.local.bash │ PASS   │ 1          │ 0.015   │
├────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ test2/c7a26255 │ generic.local.bash │ PASS   │ 2          │ 0.014   │
└────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 3/3 Percentage: 100.000%
Failed Tests: 0/3 Percentage: 0.000%


Adding 3 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_735nzh4n.log

Test Dependency by state

You can specify state as a property to check for test state when specify test dependency. In this next example, we have have four tests pass_test, fail_test, pass_and_fail_test, and final_test. The first test will be a PASS because we have state: PASS. The test fail_test depends on pass_test only if it has state: PASS, if value is mismatch then test will be skipped. Note that buildtest will skip test until next iteration if test is not executed, however if test is complete then buildtest will cancel dependent test. We can specify multiple test dependencies with state property such as test pass_and_fail_test which expects pass_test to have state: PASS and fail_test to have state: FAIL. In test final_test, shows how you can combine the format, the needs property is a list of object where each element is name of test. If no properties are associated with test name then buildtest will wait until job is complete to execute test. In this example, the test expects both pass_test and fail_test to run while pass_and_fail_test must have returncode of 1.

buildspecs:
  pass_test:
    type: script
    executor: generic.local.bash
    description: This test will always pass
    status:
      state: PASS
    run: |
      echo "This test will pass"
      sleep 2

  fail_test:
    type: script
    executor: generic.local.bash
    description: This test will run if test 'pass_test' is in state 'PASS'
    status:
      state: FAIL
    needs:
      - pass_test:
          state: PASS
    run: |
      echo "This test will fail"
      sleep 2

  pass_and_fail_test:
    type: script
    executor: generic.local.bash
    description: This test will run if pass_test is 'PASS' and fail_test is 'FAIL'
    needs:
      - pass_test:
          state: PASS
      - fail_test:
          state: FAIL
    run: |
      sleep 2
      exit 1
    status:
      returncode: 1

  final_test:
    type: script
    executor: generic.local.bash
    description: Test will run after 'pass_test', 'fail_test', and 'pass_and_fail_test'
    needs:
      - pass_test
      - fail_test
      - pass_and_fail_test:
          returncode: 1
    run: |
      sleep 2
      echo "Hello world"

Let’s build this test and take note all tests are run.

buildtest build -b tutorials/job_dependency/ex3.yml
$ buildtest build -b tutorials/job_dependency/ex3.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:20:00                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/job_dependency/ex3.yml                                             ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/job_dependency/ex3.yml: VALID
Total builder objects created: 4
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ pass_tes │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ t/86d60a │        │ local.ba │          │       │       │ test     │ ocs/che │
│ b4       │        │ sh       │          │       │       │ will     │ ckouts/ │
│          │        │          │          │       │       │ always   │ readthe │
│          │        │          │          │       │       │ pass     │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/job_d │
│          │        │          │          │       │       │          │ ependen │
│          │        │          │          │       │       │          │ cy/ex3. │
│          │        │          │          │       │       │          │ yml     │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ fail_tes │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ t/47732e │        │ local.ba │          │       │       │ test     │ ocs/che │
│ 55       │        │ sh       │          │       │       │ will run │ ckouts/ │
│          │        │          │          │       │       │ if test  │ readthe │
│          │        │          │          │       │       │ 'pass_te │ docs.or │
│          │        │          │          │       │       │ st' is   │ g/user_ │
│          │        │          │          │       │       │ in state │ builds/ │
│          │        │          │          │       │       │ 'PASS'   │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/job_d │
│          │        │          │          │       │       │          │ ependen │
│          │        │          │          │       │       │          │ cy/ex3. │
│          │        │          │          │       │       │          │ yml     │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ pass_and │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ _fail_te │        │ local.ba │          │       │       │ test     │ ocs/che │
│ st/909e8 │        │ sh       │          │       │       │ will run │ ckouts/ │
│ d37      │        │          │          │       │       │ if       │ readthe │
│          │        │          │          │       │       │ pass_tes │ docs.or │
│          │        │          │          │       │       │ t is     │ g/user_ │
│          │        │          │          │       │       │ 'PASS'   │ builds/ │
│          │        │          │          │       │       │ and      │ buildte │
│          │        │          │          │       │       │ fail_tes │ st/chec │
│          │        │          │          │       │       │ t is     │ kouts/s │
│          │        │          │          │       │       │ 'FAIL'   │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/job_d │
│          │        │          │          │       │       │          │ ependen │
│          │        │          │          │       │       │          │ cy/ex3. │
│          │        │          │          │       │       │          │ yml     │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ final_te │ script │ generic. │ None     │ None  │ None  │ Test     │ /home/d │
│ st/61fca │        │ local.ba │          │       │       │ will run │ ocs/che │
│ 2ef      │        │ sh       │          │       │       │ after    │ ckouts/ │
│          │        │          │          │       │       │ 'pass_te │ readthe │
│          │        │          │          │       │       │ st',     │ docs.or │
│          │        │          │          │       │       │ 'fail_te │ g/user_ │
│          │        │          │          │       │       │ st', and │ builds/ │
│          │        │          │          │       │       │ 'pass_an │ buildte │
│          │        │          │          │       │       │ d_fail_t │ st/chec │
│          │        │          │          │       │       │ est'     │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/job_d │
│          │        │          │          │       │       │          │ ependen │
│          │        │          │          │       │       │          │ cy/ex3. │
│          │        │          │          │       │       │          │ yml     │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
pass_test/86d60ab4: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex3/pass_test/86d60ab4
fail_test/47732e55: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex3/fail_test/47732e55
pass_and_fail_test/909e8d37: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex3/pass_and_fail_test/909e8d37
final_test/61fca2ef: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex3/final_test/61fca2ef
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
final_test/61fca2ef Skipping job because it has job dependency on pass_test/86d60ab4 which is in state PENDING
fail_test/47732e55 Skipping job because it has job dependency on pass_test/86d60ab4
pass_test/86d60ab4 does not have any dependencies adding test to queue
pass_and_fail_test/909e8d37 Skipping job because it has job dependency on pass_test/86d60ab4
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━┓
┃ Builder            ┃
┡━━━━━━━━━━━━━━━━━━━━┩
│ pass_test/86d60ab4 │
└────────────────────┘
pass_test/86d60ab4: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex3/pass_test/86d60ab4/stage
pass_test/86d60ab4: Running Test via command: bash pass_test_build.sh
pass_test/86d60ab4: Test completed in 2.007785 seconds with returncode: 0
pass_test/86d60ab4: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex3/pass_test/86d60ab4/pass_test.out
pass_test/86d60ab4: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex3/pass_test/86d60ab4/pass_test.err
───────────────────────────────── Iteration 2 ──────────────────────────────────
pass_and_fail_test/909e8d37 Skipping job because it has job dependency on fail_test/47732e55
final_test/61fca2ef Skipping job because it has job dependency on fail_test/47732e55 which is in state PENDING
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━┓
┃ Builder            ┃
┡━━━━━━━━━━━━━━━━━━━━┩
│ fail_test/47732e55 │
└────────────────────┘
fail_test/47732e55: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex3/fail_test/47732e55/stage
fail_test/47732e55: Running Test via command: bash fail_test_build.sh
fail_test/47732e55: Test completed in 2.00806 seconds with returncode: 0
fail_test/47732e55: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex3/fail_test/47732e55/fail_test.out
fail_test/47732e55: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex3/fail_test/47732e55/fail_test.err
───────────────────────────────── Iteration 3 ──────────────────────────────────
final_test/61fca2ef Skipping job because it has job dependency on pass_and_fail_test/909e8d37
   Builders Eligible to Run    
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                     ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ pass_and_fail_test/909e8d37 │
└─────────────────────────────┘
pass_and_fail_test/909e8d37: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex3/pass_and_fail_test/909e8d37/stage
pass_and_fail_test/909e8d37: Running Test via command: bash pass_and_fail_test_build.sh
pass_and_fail_test/909e8d37: failed to submit job with returncode: 1
──────────────── Error Message for pass_and_fail_test/909e8d37 ─────────────────

pass_and_fail_test/909e8d37: Detected failure in running test, will attempt to retry test: 1 times
pass_and_fail_test/909e8d37: Run - 1/1
pass_and_fail_test/909e8d37: Running Test via command: bash pass_and_fail_test_build.sh
pass_and_fail_test/909e8d37: failed to submit job with returncode: 1
pass_and_fail_test/909e8d37: Test completed in 4.01982 seconds with returncode: 1
pass_and_fail_test/909e8d37: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex3/pass_and_fail_test/909e8d37/pass_and_fail_test.out
pass_and_fail_test/909e8d37: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex3/pass_and_fail_test/909e8d37/pass_and_fail_test.err
pass_and_fail_test/909e8d37: Checking returncode - 1 is matched in list [1]
───────────────────────────────── Iteration 4 ──────────────────────────────────
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder             ┃
┡━━━━━━━━━━━━━━━━━━━━━┩
│ final_test/61fca2ef │
└─────────────────────┘
final_test/61fca2ef: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex3/final_test/61fca2ef/stage
final_test/61fca2ef: Running Test via command: bash final_test_build.sh
final_test/61fca2ef: Test completed in 2.00769 seconds with returncode: 0
final_test/61fca2ef: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex3/final_test/61fca2ef/final_test.out
final_test/61fca2ef: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex3/final_test/61fca2ef/final_test.err
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ pass_test/86d60ab4      │ generic.local.bash │ PASS   │ 0          │ 2.008   │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ final_test/61fca2ef     │ generic.local.bash │ PASS   │ 0          │ 2.008   │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ fail_test/47732e55      │ generic.local.bash │ FAIL   │ 0          │ 2.008   │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ pass_and_fail_test/909e │ generic.local.bash │ PASS   │ 1          │ 4.020   │
│ 8d37                    │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 3/4 Percentage: 75.000%
Failed Tests: 1/4 Percentage: 25.000%


Adding 4 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_rsyaoazm.log

In this next example, we have three tests the first test will runtime_test will sleep for 5 seconds but it will fail due to runtime requirement of 2sec. The next two tests runtime_test_pass and runtime_test_fail both depend on runtime_test however due to condition only one of them can be run because runtime_test_pass expects runtime_test to have state: PASS while runtime_test_fail expects runtime_test to have state: FAIL. This type of workflow can be used if you want to run a set of test based on one condition while running a different set of test based on the negative condition.

buildspecs:
  runtime_test:
    type: script
    executor: generic.local.bash
    description: This test will sleep 5 second but will fail due to runtime 2sec
    status:
      runtime:
        min: 2.0
    run: sleep 5

  runtime_test_pass:
    type: script
    executor: generic.local.bash
    description: This test will run when runtime_test_pass is PASS
    needs:
      - runtime_test:
          state: PASS
    run: echo "Performing some action when test PASS"

  runtime_test_fail:
    type: script
    executor: generic.local.bash
    description: This test will run when runtime_test_pass is FAIL
    needs:
      - runtime_test:
          state: FAIL
    run: echo "Performing some action when test FAIL"

Let’s build this test and take note that we only run two tests and runtime_test_fail was skipped because test runtime_test has a state: PASS.

buildtest build -b tutorials/job_dependency/ex4.yml
$ buildtest build -b tutorials/job_dependency/ex4.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23444606-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/14 17:20:11                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.7                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpjcb9dx2w/config.yml                              │
│ Test Directory:     /tmp/tmpjcb9dx2w/var/tests                               │
│ Report File:        /tmp/tmpjcb9dx2w/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/job_dependency/ex4.yml                                             ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/job_dependency/ex4.yml: VALID
Total builder objects created: 3
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ runtime_ │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ test/870 │        │ local.ba │          │       │       │ test     │ ocs/che │
│ 50b95    │        │ sh       │          │       │       │ will     │ ckouts/ │
│          │        │          │          │       │       │ sleep 5  │ readthe │
│          │        │          │          │       │       │ second   │ docs.or │
│          │        │          │          │       │       │ but will │ g/user_ │
│          │        │          │          │       │       │ fail due │ builds/ │
│          │        │          │          │       │       │ to       │ buildte │
│          │        │          │          │       │       │ runtime  │ st/chec │
│          │        │          │          │       │       │ 2sec     │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/job_d │
│          │        │          │          │       │       │          │ ependen │
│          │        │          │          │       │       │          │ cy/ex4. │
│          │        │          │          │       │       │          │ yml     │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ runtime_ │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ test_pas │        │ local.ba │          │       │       │ test     │ ocs/che │
│ s/dbbe9c │        │ sh       │          │       │       │ will run │ ckouts/ │
│ 1f       │        │          │          │       │       │ when     │ readthe │
│          │        │          │          │       │       │ runtime_ │ docs.or │
│          │        │          │          │       │       │ test_pas │ g/user_ │
│          │        │          │          │       │       │ s is     │ builds/ │
│          │        │          │          │       │       │ PASS     │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/job_d │
│          │        │          │          │       │       │          │ ependen │
│          │        │          │          │       │       │          │ cy/ex4. │
│          │        │          │          │       │       │          │ yml     │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ runtime_ │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ test_fai │        │ local.ba │          │       │       │ test     │ ocs/che │
│ l/736c3a │        │ sh       │          │       │       │ will run │ ckouts/ │
│ fc       │        │          │          │       │       │ when     │ readthe │
│          │        │          │          │       │       │ runtime_ │ docs.or │
│          │        │          │          │       │       │ test_pas │ g/user_ │
│          │        │          │          │       │       │ s is     │ builds/ │
│          │        │          │          │       │       │ FAIL     │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/s │
│          │        │          │          │       │       │          │ table/t │
│          │        │          │          │       │       │          │ utorial │
│          │        │          │          │       │       │          │ s/job_d │
│          │        │          │          │       │       │          │ ependen │
│          │        │          │          │       │       │          │ cy/ex4. │
│          │        │          │          │       │       │          │ yml     │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
runtime_test/87050b95: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex4/runtime_test/87050b95
runtime_test_pass/dbbe9c1f: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex4/runtime_test_pass/dbbe9c1f
runtime_test_fail/736c3afc: Creating Test Directory: /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex4/runtime_test_fail/736c3afc
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
runtime_test_fail/736c3afc Skipping job because it has job dependency on runtime_test/87050b95
runtime_test/87050b95 does not have any dependencies adding test to queue
runtime_test_pass/dbbe9c1f Skipping job because it has job dependency on runtime_test/87050b95
Builders Eligible to Run 
┏━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━┩
│ runtime_test/87050b95 │
└───────────────────────┘
runtime_test/87050b95: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex4/runtime_test/87050b95/stage
runtime_test/87050b95: Running Test via command: bash runtime_test_build.sh
runtime_test/87050b95: Test completed in 5.007468 seconds with returncode: 0
runtime_test/87050b95: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex4/runtime_test/87050b95/runtime_test.out
runtime_test/87050b95: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex4/runtime_test/87050b95/runtime_test.err
runtime_test/87050b95: Checking  mintime < runtime: 2.0 < 5.007468
───────────────────────────────── Iteration 2 ──────────────────────────────────
runtime_test_fail/736c3afc is cancelled because it depends on runtime_test/87050b95 to have state: FAIL but actual value is PASS
   Builders Eligible to Run   
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ runtime_test_pass/dbbe9c1f │
└────────────────────────────┘
runtime_test_pass/dbbe9c1f: Current Working Directory : /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex4/runtime_test_pass/dbbe9c1f/stage
runtime_test_pass/dbbe9c1f: Running Test via command: bash runtime_test_pass_build.sh
runtime_test_pass/dbbe9c1f: Test completed in 0.006045 seconds with returncode: 0
runtime_test_pass/dbbe9c1f: Writing output file -  /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex4/runtime_test_pass/dbbe9c1f/runtime_test_pass.out
runtime_test_pass/dbbe9c1f: Writing error file - /tmp/tmpjcb9dx2w/var/tests/generic.local.bash/ex4/runtime_test_pass/dbbe9c1f/runtime_test_pass.err
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ runtime_test/87050b95   │ generic.local.bash │ PASS   │ 0          │ 5.007   │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ runtime_test_pass/dbbe9 │ generic.local.bash │ PASS   │ 0          │ 0.006   │
│ c1f                     │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%


Adding 2 test results to /tmp/tmpjcb9dx2w/var/report.json
Writing Logfile to /tmp/tmpjcb9dx2w/var/logs/buildtest_ejlw72f2.log