Performance Checks

Defining Metrics

buildtest provides a method to define test metrics in the buildspecs which can be used to store arbitrary content from the output/error file or an arbitrary file into named metric. The metrics property is used to define a list of metric names using regular expression to assign a value to the metric. In this example, we have two tests that define metrics hpcg_rate_stream, hpcg_state_stream in the first test and hpcg_rate_file, hpcg_state_file in the second test. The stream property is used to read from stdout/stderr and apply the regular expression defined by exp, whereas file_regex is used to define metrics from an arbitrary file where file is the path to file.

buildspecs:
  metric_regex_example:
    executor: generic.local.bash
    type: script
    description: capture result metric from output
    run: echo "HPCG result is VALID with a GFLOP/s rating of=63.6515"
    tags: tutorials
    metrics:
      hpcg_rating_stream:
        type: float
        regex:
          exp: '(\d+\.\d+)$'
          stream: stdout
      hpcg_state_stream:
        type: str
        regex:
          exp: '(VALID)'
          stream: stdout

  metric_file_regex:
    executor: generic.local.bash
    type: script
    description: capture result metric from file path
    run: echo "HPCG result is VALID with a GFLOP/s rating of=63.6515" > hpcg.txt
    tags: tutorials
    metrics:
      hpcg_rating_file:
        type: float
        file_regex:
          exp: '(\d+\.\d+)$'
          file: hpcg.txt
      hpcg_state_file:
        type: str
        file_regex:
          exp: '(VALID)'
          file: hpcg.txt

The metrics can be used with Comparison Operators for performing more sophisticated status checks. By default, a metric will be an empty dictionary if there is no metrics property. If we fail to match a regular expression, the metric will be defined as an empty string ('').

Note

If your regular expression contains an escape character \ you must surround your string in single quotes ' as pose to double quotes "

Let’s build this test.

buildtest build -b tutorials/metrics/metrics_regex.yml
$ buildtest build -b tutorials/metrics/metrics_regex.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23532485-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/22 20:32:51                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.8                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpq07_ocbs/config.yml                              │
│ Test Directory:     /tmp/tmpq07_ocbs/var/tests                               │
│ Report File:        /tmp/tmpq07_ocbs/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/metrics/metrics_regex.yml                                           ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/metrics/metrics_regex.yml: VALID
Total builder objects created: 2
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ metric_r │ script │ generic. │ None     │ None  │ None  │ capture  │ /home/d │
│ egex_exa │        │ local.ba │          │       │       │ result   │ ocs/che │
│ mple/bc7 │        │ sh       │          │       │       │ metric   │ ckouts/ │
│ 9f00b    │        │          │          │       │       │ from     │ readthe │
│          │        │          │          │       │       │ output   │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/tu │
│          │        │          │          │       │       │          │ torials │
│          │        │          │          │       │       │          │ /metric │
│          │        │          │          │       │       │          │ s/metri │
│          │        │          │          │       │       │          │ cs_rege │
│          │        │          │          │       │       │          │ x.yml   │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ metric_f │ script │ generic. │ None     │ None  │ None  │ capture  │ /home/d │
│ ile_rege │        │ local.ba │          │       │       │ result   │ ocs/che │
│ x/6a968c │        │ sh       │          │       │       │ metric   │ ckouts/ │
│ 4a       │        │          │          │       │       │ from     │ readthe │
│          │        │          │          │       │       │ file     │ docs.or │
│          │        │          │          │       │       │ path     │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/tu │
│          │        │          │          │       │       │          │ torials │
│          │        │          │          │       │       │          │ /metric │
│          │        │          │          │       │       │          │ s/metri │
│          │        │          │          │       │       │          │ cs_rege │
│          │        │          │          │       │       │          │ x.yml   │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
metric_regex_example/bc79f00b: Creating Test Directory: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/metrics_regex/metric_regex_example/bc79f00b
metric_file_regex/6a968c4a: Creating Test Directory: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/metrics_regex/metric_file_regex/6a968c4a
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
metric_file_regex/6a968c4a does not have any dependencies adding test to queue
metric_regex_example/bc79f00b does not have any dependencies adding test to queue
    Builders Eligible to Run     
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                       ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ metric_file_regex/6a968c4a    │
│ metric_regex_example/bc79f00b │
└───────────────────────────────┘
metric_file_regex/6a968c4a: Current Working Directory : /tmp/tmpq07_ocbs/var/tests/generic.local.bash/metrics_regex/metric_file_regex/6a968c4a/stage
metric_file_regex/6a968c4a: Running Test via command: bash metric_file_regex_build.sh
metric_file_regex/6a968c4a: Test completed in 0.006139 seconds with returncode: 0
metric_file_regex/6a968c4a: Writing output file -  /tmp/tmpq07_ocbs/var/tests/generic.local.bash/metrics_regex/metric_file_regex/6a968c4a/metric_file_regex.out
metric_file_regex/6a968c4a: Writing error file - /tmp/tmpq07_ocbs/var/tests/generic.local.bash/metrics_regex/metric_file_regex/6a968c4a/metric_file_regex.err
metric_regex_example/bc79f00b: Current Working Directory : /tmp/tmpq07_ocbs/var/tests/generic.local.bash/metrics_regex/metric_regex_example/bc79f00b/stage
metric_regex_example/bc79f00b: Running Test via command: bash metric_regex_example_build.sh
metric_regex_example/bc79f00b: Test completed in 0.005692 seconds with returncode: 0
metric_regex_example/bc79f00b: Writing output file -  /tmp/tmpq07_ocbs/var/tests/generic.local.bash/metrics_regex/metric_regex_example/bc79f00b/metric_regex_example.out
metric_regex_example/bc79f00b: Writing error file - /tmp/tmpq07_ocbs/var/tests/generic.local.bash/metrics_regex/metric_regex_example/bc79f00b/metric_regex_example.err
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ metric_regex_example/bc │ generic.local.bash │ PASS   │ 0          │ 0.006   │
│ 79f00b                  │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ metric_file_regex/6a968 │ generic.local.bash │ PASS   │ 0          │ 0.006   │
│ c4a                     │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%


Adding 2 test results to /tmp/tmpq07_ocbs/var/report.json
Writing Logfile to /tmp/tmpq07_ocbs/var/logs/buildtest_v8442ikf.log

The metrics are captured in the test report which can be queried via buildtest report or buildtest inspect query. Metrics can be seen in the test metadata, for instance you can run buildtest inspect query and you will see metrics shown in table output.

buildtest inspect query metric_regex_example metric_file_regex
$ buildtest inspect query metric_regex_example metric_file_regex
────────── metric_regex_example/bc79f00b-662c-4dba-b14d-897127449c3b ───────────
Executor: generic.local.bash
Description: capture result metric from output
State: PASS
Returncode: 0
Runtime: 0.005692 sec
Starttime: 2024/02/22 20:32:51
Endtime: 2024/02/22 20:32:51
Command: bash metric_regex_example_build.sh
Test Script: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/metrics_regex/metric_regex_example/bc79f00b/metric_regex_example.sh
Build Script: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/metrics_regex/metric_regex_example/bc79f00b/metric_regex_example_build.sh
Output File: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/metrics_regex/metric_regex_example/bc79f00b/metric_regex_example.out
Error File: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/metrics_regex/metric_regex_example/bc79f00b/metric_regex_example.err
Log File: /tmp/tmpq07_ocbs/var/logs/buildtest_v8442ikf.log
            Metrics             
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Name               ┃ Value   ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ hpcg_rating_stream │ 63.6515 │
│ hpcg_state_stream  │ VALID   │
└────────────────────┴─────────┘
──────────── metric_file_regex/6a968c4a-ba38-4a4b-9763-457c41956f8c ────────────
Executor: generic.local.bash
Description: capture result metric from file path
State: PASS
Returncode: 0
Runtime: 0.006139 sec
Starttime: 2024/02/22 20:32:51
Endtime: 2024/02/22 20:32:51
Command: bash metric_file_regex_build.sh
Test Script: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/metrics_regex/metric_file_regex/6a968c4a/metric_file_regex.sh
Build Script: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/metrics_regex/metric_file_regex/6a968c4a/metric_file_regex_build.sh
Output File: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/metrics_regex/metric_file_regex/6a968c4a/metric_file_regex.out
Error File: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/metrics_regex/metric_file_regex/6a968c4a/metric_file_regex.err
Log File: /tmp/tmpq07_ocbs/var/logs/buildtest_v8442ikf.log
           Metrics            
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Name             ┃ Value   ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ hpcg_rating_file │ 63.6515 │
│ hpcg_state_file  │ VALID   │
└──────────────────┴─────────┘

We can query the metrics via buildtest report which will display all metrics as a comma seperated Key/Value pair. We can use buildtest report --format metrics to extract all metrics for a test. Internally, we store the metrics as a dictionary but when we print them out via buildtest report we join them together into a single string. Shown below is the metrics for the previous build.

buildtest report --filter buildspec=tutorials/metrics/metrics_regex.yml --format name,metrics
$ buildtest report --filter buildspec=tutorials/metrics/metrics_regex.yml --format name,metrics
                 Report File: /tmp/tmpq07_ocbs/var/report.json                  
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name                  ┃ metrics                                              ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ metric_regex_example  │ hpcg_rating_stream=63.6515,hpcg_state_stream=VALID   │
├───────────────────────┼──────────────────────────────────────────────────────┤
│ metric_file_regex     │ hpcg_rating_file=63.6515,hpcg_state_file=VALID       │
└───────────────────────┴──────────────────────────────────────────────────────┘

Invalid Metrics

We will discuss a few edge-cases when defining metrics that can lead to validation error. The file_regex and regex property can’t be declared at the same time when defining a metric. In example below we have defined a metric named hello that uses both regex and file_regex.

buildspecs:
  metrics_with_regex_and_file_regex_not_allowed:
    type: script
    executor: generic.local.bash
    description: This is an invalid metric because 'file_regex' and 'regex' are both specified
    run: echo "hello" > file.txt
    metrics:
      hello:
        type: str
        file_regex:
          file: file.txt
          exp: "BAR"
        regex:
          stream: stdout
          exp: "BAR"

If we try to validate this buildspec, we will get an error message that regex and file_regex can’t be specified at the same time.

buildtest buildspec validate  -b tutorials/metrics/invalid_metrics.yml
$ buildtest buildspec validate  -b tutorials/metrics/invalid_metrics.yml
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… ─
{'type': 'str', 'file_regex': {'file': 'file.txt', 'exp': 'BAR'}, 'regex': {'stream': 'stdout', 'exp': 'BAR'}} is valid under each of {'required': ['file_regex']}, {'required': ['regex']}

Failed validating 'oneOf' in schema['properties']['metrics']['additionalProperties']:
    {'additionalProperties': False,
     'oneOf': [{'required': ['regex']}, {'required': ['file_regex']}],
     'properties': {'file_regex': {'$ref': '#/definitions/file_regex_in_metrics'},
                    'regex': {'$ref': '#/definitions/regex'},
                    'type': {'description': 'Specify python data-type '
                                            '(str, int, float) to convert '
                                            'metric. ',
                             'enum': ['str', 'int', 'float'],
                             'type': 'string'}},
     'required': ['type'],
     'type': 'object'}

On instance['metrics']['hello']:
    {'file_regex': {'exp': 'BAR', 'file': 'file.txt'},
     'regex': {'exp': 'BAR', 'stream': 'stdout'},
     'type': 'str'}


1 buildspecs failed to validate

When defining a metrics, you must specify regex or file_regex property in order to capture metric. If its not specified, you will run into validation error. In this example, we define a metrics named foo, but we don’t specify the regex or file_regex property therefore, this metric is invalid.

buildspecs:
  required_property_in_metrics:
    type: script
    executor: generic.local.bash
    description: metrics must have a file_regex or regex property to define how to capture metric
    run: echo "hello"
    metrics:
      foo:
        type: str

The metrics must follow a pattern, this is typically alphanumeric characters including dot (.), hypen (-) and underscore (_). In this example below, we have an invalid metric that doesn’t conform to pattern.

buildspecs:
  invalid_metrics_name:
    type: script
    executor: generic.local.bash
    description: "Metrics name does not follow pattern"
    run: echo "hello" > file.txt
    metrics:
      (foo-bar:
        type: str
        regex:
          stream: stdout
          exp: "BAR"

Let’s try validating the buildspec to see the error message.

buildtest buildspec validate  -b tutorials/metrics/invalid_metric_name.yml
$ buildtest buildspec validate  -b tutorials/metrics/invalid_metric_name.yml
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… ─
'(foo-bar' does not match '^[A-Za-z0-9_.-]+$'

Failed validating 'pattern' in schema['properties']['metrics']['propertyNames']:
    {'pattern': '^[A-Za-z0-9_.-]+$'}

On instance['metrics']:
    '(foo-bar'


1 buildspecs failed to validate

Comparison Operators

buildtest supports several comparison operators as part of status check such as >, >=, <=, <, ==, !=. Each metric is compared with a reference value that can be useful when running performance checks. In this section we will cover the following comparison:

Greater Equal

buildtest can determine status check based on performance check. In this next example, we will run the STREAM memory benchmark and capture metrics named copy, scale add and triad from the output and perform an Assertion Greater Equal (assert_ge) with a reference value.

The assert_ge contains a list of assertions in the comparisons property where each metric name is referenced via name that is compared with the reference value defined by ref property. The comparison is metric_value >= ref, where metric_value is the value assigned to the metric name captured by the regular expression. The type field in the metric section is used for the type conversion which can be float, int, or string. The item is a numeric field used in match.group to retrieve the output from the regular expression search. The item must be non-negative number.

 1buildspecs:
 2  stream_test:
 3    type: script
 4    executor: generic.local.bash
 5    description: Run stream test with metrics example using assert greater equal
 6    env:
 7      OMP_NUM_THREADS: 4
 8    run: |
 9      wget https://raw.githubusercontent.com/jeffhammond/STREAM/master/stream.c
10      gcc -openmp -o stream stream.c
11      ./stream
12    metrics:
13      copy:
14        type: float
15        regex:
16          exp: 'Copy:\s+(\S+)\s+.*'
17          stream: stdout
18          item: 1
19      scale:
20        type: float
21        regex:
22          exp: 'Scale:\s+(\S+)\s+.*'
23          stream: stdout
24          item: 1
25      add:
26        type: float
27        regex:
28          exp: 'Add:\s+(\S+)\s+.*'
29          stream: stdout
30          item: 1
31      triad:
32        type: float
33        regex:
34          exp: 'Triad:\s+(\S+)\s+.*'
35          stream: stdout
36          item: 1
37    status:
38      assert_ge:
39        mode: and
40        comparisons:
41        - name: copy
42          ref: 5000
43        - name: scale
44          ref: 5500
45        - name: add
46          ref: 6000
47        - name: triad
48          ref: 6500

buildtest will evaluate each assertion in the list and use a logical AND to determine the final status of assert_ge. The keyword mode is used to determine whether to perform a logical OR / AND operation when evaluating the final expression. The mode can be any of the values: [AND, OR, and, or]. If mode is ommitted the default is logical AND.

Let’s build this test, take a close look at the output of buildtest build and take note of the assertion statement.

buildtest build -b tutorials/perf_checks/assert_ge.yml
$ buildtest build -b tutorials/perf_checks/assert_ge.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23532485-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/22 20:32:54                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.8                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpq07_ocbs/config.yml                              │
│ Test Directory:     /tmp/tmpq07_ocbs/var/tests                               │
│ Report File:        /tmp/tmpq07_ocbs/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/perf_checks/assert_ge.yml                                           ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/perf_checks/assert_ge.yml: VALID
Total builder objects created: 1
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ stream_t │ script │ generic. │ None     │ None  │ None  │ Run      │ /home/d │
│ est/e910 │        │ local.ba │          │       │       │ stream   │ ocs/che │
│ a3f8     │        │ sh       │          │       │       │ test     │ ckouts/ │
│          │        │          │          │       │       │ with     │ readthe │
│          │        │          │          │       │       │ metrics  │ docs.or │
│          │        │          │          │       │       │ example  │ g/user_ │
│          │        │          │          │       │       │ using    │ builds/ │
│          │        │          │          │       │       │ assert   │ buildte │
│          │        │          │          │       │       │ greater  │ st/chec │
│          │        │          │          │       │       │ equal    │ kouts/d │
│          │        │          │          │       │       │          │ evel/tu │
│          │        │          │          │       │       │          │ torials │
│          │        │          │          │       │       │          │ /perf_c │
│          │        │          │          │       │       │          │ hecks/a │
│          │        │          │          │       │       │          │ ssert_g │
│          │        │          │          │       │       │          │ e.yml   │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
stream_test/e910a3f8: Creating Test Directory: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_ge/stream_test/e910a3f8
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
stream_test/e910a3f8 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder              ┃
┡━━━━━━━━━━━━━━━━━━━━━━┩
│ stream_test/e910a3f8 │
└──────────────────────┘
stream_test/e910a3f8: Current Working Directory : /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_ge/stream_test/e910a3f8/stage
stream_test/e910a3f8: Running Test via command: bash stream_test_build.sh
stream_test/e910a3f8: Test completed in 1.470633 seconds with returncode: 0
stream_test/e910a3f8: Writing output file -  /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_ge/stream_test/e910a3f8/stream_test.out
stream_test/e910a3f8: Writing error file - /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_ge/stream_test/e910a3f8/stream_test.err
stream_test/e910a3f8: testing metric: copy if 5890.2 >= 5000.0 - Check: True
stream_test/e910a3f8: testing metric: scale if 6652.8 >= 5500.0 - Check: True
stream_test/e910a3f8: testing metric: add if 8832.4 >= 6000.0 - Check: True
stream_test/e910a3f8: testing metric: triad if 8456.3 >= 6500.0 - Check: True
stream_test/e910a3f8: ge check: True
                                Test Summary                                 
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder              ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ stream_test/e910a3f8 │ generic.local.bash │ PASS   │ 0          │ 1.471   │
└──────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to /tmp/tmpq07_ocbs/var/report.json
Writing Logfile to /tmp/tmpq07_ocbs/var/logs/buildtest_xgzgbb2z.log

Let’s run buildtest inspect query -o stream_test to retrieve the test details and output of STREAM test.

buildtest inspect query -o stream_test
$ buildtest inspect query -o stream_test
─────────────── stream_test/e910a3f8-ff99-4f8e-86ab-472bc48090b5 ───────────────
Executor: generic.local.bash
Description: Run stream test with metrics example using assert greater equal
State: PASS
Returncode: 0
Runtime: 1.470633 sec
Starttime: 2024/02/22 20:32:54
Endtime: 2024/02/22 20:32:56
Command: bash stream_test_build.sh
Test Script: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_ge/stream_test/e910a3f8/stream_test.sh
Build Script: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_ge/stream_test/e910a3f8/stream_test_build.sh
Output File: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_ge/stream_test/e910a3f8/stream_test.out
Error File: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_ge/stream_test/e910a3f8/stream_test.err
Log File: /tmp/tmpq07_ocbs/var/logs/buildtest_xgzgbb2z.log
     Metrics      
┏━━━━━━━┳━━━━━━━━┓
┃ Name  ┃ Value  ┃
┡━━━━━━━╇━━━━━━━━┩
│ copy  │ 5890.2 │
│ scale │ 6652.8 │
│ add   │ 8832.4 │
│ triad │ 8456.3 │
└───────┴────────┘
─ Output File: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_ge/stream… ─
-------------------------------------------------------------                   
STREAM version $Revision: 5.10 $                                                
-------------------------------------------------------------                   
This system uses 8 bytes per array element.                                     
-------------------------------------------------------------                   
Array size = 10000000 (elements), Offset = 0 (elements)                         
Memory per array = 76.3 MiB (= 0.1 GiB).                                        
Total memory required = 228.9 MiB (= 0.2 GiB).                                  
Each kernel will be executed 10 times.                                          
 The *best* time for each kernel (excluding the first iteration)                
 will be used to compute the reported bandwidth.                                
-------------------------------------------------------------                   
Your clock granularity/precision appears to be 1 microseconds.                  
Each test below will take on the order of 24875 microseconds.                   
   (= 24875 clock ticks)                                                        
Increase the size of the arrays if this shows that                              
you are not getting at least 20 clock ticks per test.                           
-------------------------------------------------------------                   
WARNING -- The above is only a rough guideline.                                 
For best results, please be sure you know the                                   
precision of your system timer.                                                 
-------------------------------------------------------------                   
Function    Best Rate MB/s  Avg time     Min time     Max time                  
Copy:            5890.2     0.027293     0.027164     0.027457                  
Scale:           6652.8     0.024241     0.024050     0.024600                  
Add:             8832.4     0.027487     0.027173     0.027629                  
Triad:           8456.3     0.028595     0.028381     0.029107                  
-------------------------------------------------------------                   
Solution Validates: avg error less than 1.000000e-13 on all three arrays        
-------------------------------------------------------------

Greater Than

In this example, we perform a > operation, this can be done via assert_gt property

 1buildspecs:
 2  assert_gt_example:
 3    type: script
 4    executor: generic.local.bash
 5    description: Run stream test with metrics example using assert greater than.
 6    env:
 7      OMP_NUM_THREADS: 4
 8    run: |
 9      wget https://raw.githubusercontent.com/jeffhammond/STREAM/master/stream.c
10      gcc -openmp -o stream stream.c
11      ./stream
12    metrics:
13      copy:
14        type: float
15        regex:
16          exp: 'Copy:\s+(\S+)\s+.*'
17          stream: stdout
18          item: 1
19      scale:
20        type: float
21        regex:
22          exp: 'Scale:\s+(\S+)\s+.*'
23          stream: stdout
24          item: 1
25      add:
26        type: float
27        regex:
28          exp: 'Add:\s+(\S+)\s+.*'
29          stream: stdout
30          item: 1
31      triad:
32        type: float
33        regex:
34          exp: 'Triad:\s+(\S+)\s+.*'
35          stream: stdout
36          item: 1
37    status:
38      assert_gt:
39        comparisons:
40        - name: copy
41          ref: 5000
42        - name: scale
43          ref: 5500
44        - name: add
45          ref: 6000
46        - name: triad
47          ref: 6500

Less Than Equal

In this example, we perform a <= operation, this can be done via assert_le property

 1buildspecs:
 2  assert_le_example:
 3    type: script
 4    executor: generic.local.bash
 5    description: Run stream test with metrics example using assert less than equal
 6    env:
 7      OMP_NUM_THREADS: 4
 8    run: |
 9      wget https://raw.githubusercontent.com/jeffhammond/STREAM/master/stream.c
10      gcc -openmp -o stream stream.c
11      ./stream
12    metrics:
13      copy:
14        type: float
15        regex:
16          exp: 'Copy:\s+(\S+)\s+.*'
17          stream: stdout
18          item: 1
19      scale:
20        type: float
21        regex:
22          exp: 'Scale:\s+(\S+)\s+.*'
23          stream: stdout
24          item: 1
25      add:
26        type: float
27        regex:
28          exp: 'Add:\s+(\S+)\s+.*'
29          stream: stdout
30          item: 1
31      triad:
32        type: float
33        regex:
34          exp: 'Triad:\s+(\S+)\s+.*'
35          stream: stdout
36          item: 1
37    status:
38      assert_le:
39        comparisons:
40        - name: copy
41          ref: 5000
42        - name: scale
43          ref: 5500
44        - name: add
45          ref: 6000
46        - name: triad
47          ref: 6500

Less Than

In this example, we perform a < operation, this can be done via assert_lt property

 1buildspecs:
 2  assert_lt_example:
 3    type: script
 4    executor: generic.local.bash
 5    description: Run stream test with metrics example using assert less than
 6    env:
 7      OMP_NUM_THREADS: 4
 8    run: |
 9      wget https://raw.githubusercontent.com/jeffhammond/STREAM/master/stream.c
10      gcc -openmp -o stream stream.c
11      ./stream
12    metrics:
13      copy:
14        type: float
15        regex:
16          exp: 'Copy:\s+(\S+)\s+.*'
17          stream: stdout
18          item: 1
19      scale:
20        type: float
21        regex:
22          exp: 'Scale:\s+(\S+)\s+.*'
23          stream: stdout
24          item: 1
25      add:
26        type: float
27        regex:
28          exp: 'Add:\s+(\S+)\s+.*'
29          stream: stdout
30          item: 1
31      triad:
32        type: float
33        regex:
34          exp: 'Triad:\s+(\S+)\s+.*'
35          stream: stdout
36          item: 1
37    status:
38      assert_lt:
39        comparisons:
40        - name: copy
41          ref: 5000
42        - name: scale
43          ref: 5500
44        - name: add
45          ref: 6000
46        - name: triad
47          ref: 6500

Assert Equal

buildtest can perform assert equality check with metrics to determine status of test. In this next example, we define four metrics x, y, first, and last which will be compared with its reference value. We introduce a new property assert_eq which is composed of list of assertions. Each reference is converted to its appropriate type (int, float, str).

 1buildspecs:
 2  assert_eq_example:
 3    type: script
 4    executor: generic.local.bash
 5    description: Test for assert equality
 6    vars:
 7      X: 1
 8      Y: 1.5
 9      first: John
10      last: Smith
11    run: |
12      echo "X: $X"
13      echo "Y: $Y"
14      echo "Name: $first $last"
15    metrics:
16      x:
17        type: int
18        regex:
19          stream: stdout
20          exp: 'X:\s+(\S+)\s+.*'
21          item: 1
22      y:
23        type: float
24        regex:
25          stream: stdout
26          exp: 'Y:\s+(\S+)\s+.*'
27          item: 1
28      first:
29        type: str
30        regex:
31          stream: stdout
32          exp: 'Name:\s+(\S+)\s+.*'
33          item: 1
34      last:
35        type: str
36        regex:
37          stream: stdout
38          exp: '(Smith)$'
39          item: 1
40    status:
41      assert_eq:
42        comparisons:
43        - name: x
44          ref: 1
45        - name: y
46          ref: 1.5
47        - name: first
48          ref: John
49        - name: last
50          ref: Smith

This test is expected to pass where all assertions are True. Let’s build the test and see the output

buildtest build -b tutorials/perf_checks/assert_eq.yml
$ buildtest build -b tutorials/perf_checks/assert_eq.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23532485-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/22 20:32:57                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.8                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpq07_ocbs/config.yml                              │
│ Test Directory:     /tmp/tmpq07_ocbs/var/tests                               │
│ Report File:        /tmp/tmpq07_ocbs/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/perf_checks/assert_eq.yml                                           ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/perf_checks/assert_eq.yml: VALID
Total builder objects created: 1
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ assert_e │ script │ generic. │ None     │ None  │ None  │ Test for │ /home/d │
│ q_exampl │        │ local.ba │          │       │       │ assert   │ ocs/che │
│ e/282065 │        │ sh       │          │       │       │ equality │ ckouts/ │
│ ac       │        │          │          │       │       │          │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/tu │
│          │        │          │          │       │       │          │ torials │
│          │        │          │          │       │       │          │ /perf_c │
│          │        │          │          │       │       │          │ hecks/a │
│          │        │          │          │       │       │          │ ssert_e │
│          │        │          │          │       │       │          │ q.yml   │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
assert_eq_example/282065ac: Creating Test Directory: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq/assert_eq_example/282065ac
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
assert_eq_example/282065ac does not have any dependencies adding test to queue
   Builders Eligible to Run   
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ assert_eq_example/282065ac │
└────────────────────────────┘
assert_eq_example/282065ac: Current Working Directory : /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq/assert_eq_example/282065ac/stage
assert_eq_example/282065ac: Running Test via command: bash assert_eq_example_build.sh
assert_eq_example/282065ac: Test completed in 0.006102 seconds with returncode: 0
assert_eq_example/282065ac: Writing output file -  /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq/assert_eq_example/282065ac/assert_eq_example.out
assert_eq_example/282065ac: Writing error file - /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq/assert_eq_example/282065ac/assert_eq_example.err
assert_eq_example/282065ac: testing metric: x if 1 == 1 - Check: True
assert_eq_example/282065ac: testing metric: y if 1.5 == 1.5 - Check: True
assert_eq_example/282065ac: testing metric: first if John == John - Check: True
assert_eq_example/282065ac: testing metric: last if Smith == Smith - Check: True
assert_eq_example/282065ac: eq check: True
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ assert_eq_example/28206 │ generic.local.bash │ PASS   │ 0          │ 0.006   │
│ 5ac                     │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to /tmp/tmpq07_ocbs/var/report.json
Writing Logfile to /tmp/tmpq07_ocbs/var/logs/buildtest_ucxbhmku.log
buildtest inspect query -o assert_eq_example
$ buildtest inspect query -o assert_eq_example
──────────── assert_eq_example/282065ac-58ac-4179-92d6-aa975cc981f2 ────────────
Executor: generic.local.bash
Description: Test for assert equality
State: PASS
Returncode: 0
Runtime: 0.006102 sec
Starttime: 2024/02/22 20:32:57
Endtime: 2024/02/22 20:32:57
Command: bash assert_eq_example_build.sh
Test Script: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq/assert_eq_example/282065ac/assert_eq_example.sh
Build Script: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq/assert_eq_example/282065ac/assert_eq_example_build.sh
Output File: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq/assert_eq_example/282065ac/assert_eq_example.out
Error File: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq/assert_eq_example/282065ac/assert_eq_example.err
Log File: /tmp/tmpq07_ocbs/var/logs/buildtest_ucxbhmku.log
     Metrics     
┏━━━━━━━┳━━━━━━━┓
┃ Name  ┃ Value ┃
┡━━━━━━━╇━━━━━━━┩
│ x     │ 1     │
│ y     │ 1.5   │
│ first │ John  │
│ last  │ Smith │
└───────┴───────┘
─ Output File: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq/assert… ─
X: 1                                                                            
Y: 1.5                                                                          
Name: John Smith

In the next example, we have two tests to highlight some exceptions. In the first test, we define an invalid metric name invalid_metric in assert_eq since this metric was not defined in metrics field, therefore this test will fail. The second test will fail because we have a mismatch in value captured by metric x which is 1 however the reference value is 2.

 1buildspecs:
 2  assert_eq_invalid_metric:
 3    type: script
 4    executor: generic.local.bash
 5    description: An invalid metric name will cause failure
 6    vars:
 7      X: 1
 8    run: |
 9      echo "X: $X"
10    metrics:
11      x:
12        type: int
13        regex:
14          stream: stdout
15          exp: 'X:\s+(\S+)\s+.*'
16          item: 1
17    status:
18      assert_eq:
19        comparisons:
20        - name: x
21          ref: 1
22        - name: invalid_metric
23          ref: 'hello'
24  assert_eq_mismatch:
25    type: script
26    executor: generic.local.bash
27    description: This test will fail because there is a mismatch in metric x assert equality
28    vars:
29      X: 1
30    run: |
31      echo "X: $X"
32    metrics:
33      x:
34        type: int
35        regex:
36          stream: stdout
37          exp: 'X:\s+(\S+)\s+.*'
38          item: 1
39    status:
40      assert_eq:
41        comparisons:
42        - name: x
43          ref: 2

Let’s build this test and see the output.

buildtest build -b tutorials/perf_checks/assert_eq_exceptions.yml
$ buildtest build -b tutorials/perf_checks/assert_eq_exceptions.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23532485-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/22 20:32:58                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.8                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpq07_ocbs/config.yml                              │
│ Test Directory:     /tmp/tmpq07_ocbs/var/tests                               │
│ Report File:        /tmp/tmpq07_ocbs/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/perf_checks/assert_eq_exceptions.yml                                ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/perf_checks/assert_eq_exceptions.yml: VALID
Total builder objects created: 2
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ assert_e │ script │ generic. │ None     │ None  │ None  │ An       │ /home/d │
│ q_invali │        │ local.ba │          │       │       │ invalid  │ ocs/che │
│ d_metric │        │ sh       │          │       │       │ metric   │ ckouts/ │
│ /92a0c06 │        │          │          │       │       │ name     │ readthe │
│ 5        │        │          │          │       │       │ will     │ docs.or │
│          │        │          │          │       │       │ cause    │ g/user_ │
│          │        │          │          │       │       │ failure  │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/tu │
│          │        │          │          │       │       │          │ torials │
│          │        │          │          │       │       │          │ /perf_c │
│          │        │          │          │       │       │          │ hecks/a │
│          │        │          │          │       │       │          │ ssert_e │
│          │        │          │          │       │       │          │ q_excep │
│          │        │          │          │       │       │          │ tions.y │
│          │        │          │          │       │       │          │ ml      │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ assert_e │ script │ generic. │ None     │ None  │ None  │ This     │ /home/d │
│ q_mismat │        │ local.ba │          │       │       │ test     │ ocs/che │
│ ch/a5f24 │        │ sh       │          │       │       │ will     │ ckouts/ │
│ 55d      │        │          │          │       │       │ fail     │ readthe │
│          │        │          │          │       │       │ because  │ docs.or │
│          │        │          │          │       │       │ there is │ g/user_ │
│          │        │          │          │       │       │ a        │ builds/ │
│          │        │          │          │       │       │ mismatch │ buildte │
│          │        │          │          │       │       │ in       │ st/chec │
│          │        │          │          │       │       │ metric x │ kouts/d │
│          │        │          │          │       │       │ assert   │ evel/tu │
│          │        │          │          │       │       │ equality │ torials │
│          │        │          │          │       │       │          │ /perf_c │
│          │        │          │          │       │       │          │ hecks/a │
│          │        │          │          │       │       │          │ ssert_e │
│          │        │          │          │       │       │          │ q_excep │
│          │        │          │          │       │       │          │ tions.y │
│          │        │          │          │       │       │          │ ml      │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
assert_eq_invalid_metric/92a0c065: Creating Test Directory: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_invalid_metric/92a0c065
assert_eq_mismatch/a5f2455d: Creating Test Directory: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_mismatch/a5f2455d
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
assert_eq_mismatch/a5f2455d does not have any dependencies adding test to queue
assert_eq_invalid_metric/92a0c065 does not have any dependencies adding test to queue
      Builders Eligible to Run       
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ assert_eq_invalid_metric/92a0c065 │
│ assert_eq_mismatch/a5f2455d       │
└───────────────────────────────────┘
assert_eq_invalid_metric/92a0c065: Current Working Directory : /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_invalid_metric/92a0c065/stage
assert_eq_invalid_metric/92a0c065: Running Test via command: bash assert_eq_invalid_metric_build.sh
assert_eq_invalid_metric/92a0c065: Test completed in 0.00627 seconds with returncode: 0
assert_eq_invalid_metric/92a0c065: Writing output file -  /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_invalid_metric/92a0c065/assert_eq_invalid_metric.out
assert_eq_invalid_metric/92a0c065: Writing error file - /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_invalid_metric/92a0c065/assert_eq_invalid_metric.err
assert_eq_invalid_metric/92a0c065: testing metric: x if 1 == 1 - Check: True
assert_eq_invalid_metric/92a0c065: Unable to find metric: invalid_metric. List of valid metrics are the following: ['x']
assert_eq_invalid_metric/92a0c065: eq check: False
assert_eq_mismatch/a5f2455d: Current Working Directory : /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_mismatch/a5f2455d/stage
assert_eq_mismatch/a5f2455d: Running Test via command: bash assert_eq_mismatch_build.sh
assert_eq_mismatch/a5f2455d: Test completed in 0.005928 seconds with returncode: 0
assert_eq_mismatch/a5f2455d: Writing output file -  /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_mismatch/a5f2455d/assert_eq_mismatch.out
assert_eq_mismatch/a5f2455d: Writing error file - /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_mismatch/a5f2455d/assert_eq_mismatch.err
assert_eq_mismatch/a5f2455d: testing metric: x if 1 == 2 - Check: False
assert_eq_mismatch/a5f2455d: eq check: False
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ assert_eq_mismatch/a5f2 │ generic.local.bash │ FAIL   │ 0          │ 0.006   │
│ 455d                    │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ assert_eq_invalid_metri │ generic.local.bash │ FAIL   │ 0          │ 0.006   │
│ c/92a0c065              │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 0/2 Percentage: 0.000%
Failed Tests: 2/2 Percentage: 100.000%


Adding 2 test results to /tmp/tmpq07_ocbs/var/report.json
Writing Logfile to /tmp/tmpq07_ocbs/var/logs/buildtest_ttzw22wd.log

Assert Not Equal

In this section, we will discuss the inverse equality operation Not Equal check (!=) with reference value.

We can use assert_ne property to perform != check, it works similar to assert_eq with data types int, float and str. In this example, we check the metrics x, y, first and last and each metric should pass. The reference value is converted to the data-type (type field) for each metrics

 1buildspecs:
 2  assert_ne_example:
 3    type: script
 4    executor: generic.local.bash
 5    description: Test for assert not equal
 6    vars:
 7      X: 1
 8      Y: 1.5
 9      first: John
10      last: Smith
11    run: |
12      echo "X: $X"
13      echo "Y: $Y"
14      echo "Name: $first $last"
15    metrics:
16      x:
17        type: int
18        regex:
19          stream: stdout
20          exp: 'X:\s+(\S+)\s+.*'
21          item: 1
22      y:
23        type: float
24        regex:
25          stream: stdout
26          exp: 'Y:\s+(\S+)\s+.*'
27          item: 1
28      first:
29        type: str
30        regex:
31          stream: stdout
32          exp: 'Name:\s+(\S+)\s+.*'
33          item: 1
34      last:
35        type: str
36        regex:
37          stream: stdout
38          exp: '(Smith)$'
39          item: 1
40    status:
41      assert_ne:
42        comparisons:
43        - name: x
44          ref: 2
45        - name: y
46          ref: 2.5
47        - name: first
48          ref: Robert
49        - name: last
50          ref: Brown

We expect this test to pass. In order to run this test, you can do the following

buildtest build -b tutorials/perf_checks/assert_ne.yml
$ buildtest build -b tutorials/perf_checks/assert_ne.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23532485-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/22 20:32:59                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.8                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpq07_ocbs/config.yml                              │
│ Test Directory:     /tmp/tmpq07_ocbs/var/tests                               │
│ Report File:        /tmp/tmpq07_ocbs/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/perf_checks/assert_ne.yml                                           ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/perf_checks/assert_ne.yml: VALID
Total builder objects created: 1
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ assert_n │ script │ generic. │ None     │ None  │ None  │ Test for │ /home/d │
│ e_exampl │        │ local.ba │          │       │       │ assert   │ ocs/che │
│ e/e6dc99 │        │ sh       │          │       │       │ not      │ ckouts/ │
│ 90       │        │          │          │       │       │ equal    │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/tu │
│          │        │          │          │       │       │          │ torials │
│          │        │          │          │       │       │          │ /perf_c │
│          │        │          │          │       │       │          │ hecks/a │
│          │        │          │          │       │       │          │ ssert_n │
│          │        │          │          │       │       │          │ e.yml   │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
assert_ne_example/e6dc9990: Creating Test Directory: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_ne/assert_ne_example/e6dc9990
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
assert_ne_example/e6dc9990 does not have any dependencies adding test to queue
   Builders Eligible to Run   
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                    ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ assert_ne_example/e6dc9990 │
└────────────────────────────┘
assert_ne_example/e6dc9990: Current Working Directory : /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_ne/assert_ne_example/e6dc9990/stage
assert_ne_example/e6dc9990: Running Test via command: bash assert_ne_example_build.sh
assert_ne_example/e6dc9990: Test completed in 0.006184 seconds with returncode: 0
assert_ne_example/e6dc9990: Writing output file -  /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_ne/assert_ne_example/e6dc9990/assert_ne_example.out
assert_ne_example/e6dc9990: Writing error file - /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_ne/assert_ne_example/e6dc9990/assert_ne_example.err
assert_ne_example/e6dc9990: testing metric: x if 1 != 2 - Check: True
assert_ne_example/e6dc9990: testing metric: y if 1.5 != 2.5 - Check: True
assert_ne_example/e6dc9990: testing metric: first if John != Robert - Check: True
assert_ne_example/e6dc9990: testing metric: last if Smith != Brown - Check: True
assert_ne_example/e6dc9990: ne check: True
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ assert_ne_example/e6dc9 │ generic.local.bash │ PASS   │ 0          │ 0.006   │
│ 990                     │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to /tmp/tmpq07_ocbs/var/report.json
Writing Logfile to /tmp/tmpq07_ocbs/var/logs/buildtest_m94cc39a.log

Assert Range

The assert_range property can be used to test performance for a metric given a lower and upper bound. This property expects one to specify lower and upper field which must be an integer or floating point number to perform comparison. buildtest will perform an assertion, if metric value is in the range specified by lower and upper, then test will pass. Shown below is an example using the assert_range property with stream benchmark.

 1buildspecs:
 2  assert_range_ex:
 3    type: script
 4    executor: generic.local.bash
 5    description: Example on assert_range
 6    env:
 7      OMP_NUM_THREADS: 4
 8    run: |
 9      wget https://raw.githubusercontent.com/jeffhammond/STREAM/master/stream.c
10      gcc -openmp -o stream stream.c
11      ./stream
12    metrics:
13      copy:
14        type: float
15        regex:
16          exp: 'Copy:\s+(\S+)\s+.*'
17          stream: stdout
18          item: 1
19      scale:
20        type: float
21        regex:
22          exp: 'Scale:\s+(\S+)\s+.*'
23          stream: stdout
24          item: 1
25      add:
26        type: float
27        regex:
28          exp: 'Add:\s+(\S+)\s+.*'
29          stream: stdout
30          item: 1
31      triad:
32        type: float
33        regex:
34          exp: 'Triad:\s+(\S+)\s+.*'
35          stream: stdout
36          item: 1
37    status:
38      assert_range:
39        comparisons:
40        - name: copy
41          lower: 5000
42          upper: 20000
43        - name: scale
44          lower: 4500
45          upper: 20000
46        - name: add
47          lower: 4300
48          upper: 20000
49        - name: triad
50          lower: 5600
51          upper: 20000

Let’s build this test and see the output

buildtest build -b tutorials/perf_checks/assert_range.yml
$ buildtest build -b tutorials/perf_checks/assert_range.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23532485-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/22 20:32:59                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.8                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpq07_ocbs/config.yml                              │
│ Test Directory:     /tmp/tmpq07_ocbs/var/tests                               │
│ Report File:        /tmp/tmpq07_ocbs/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/perf_checks/assert_range.yml                                        ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/perf_checks/assert_range.yml: VALID
Total builder objects created: 1
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ assert_r │ script │ generic. │ None     │ None  │ None  │ Example  │ /home/d │
│ ange_ex/ │        │ local.ba │          │       │       │ on       │ ocs/che │
│ d5e025dd │        │ sh       │          │       │       │ assert_r │ ckouts/ │
│          │        │          │          │       │       │ ange     │ readthe │
│          │        │          │          │       │       │          │ docs.or │
│          │        │          │          │       │       │          │ g/user_ │
│          │        │          │          │       │       │          │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/tu │
│          │        │          │          │       │       │          │ torials │
│          │        │          │          │       │       │          │ /perf_c │
│          │        │          │          │       │       │          │ hecks/a │
│          │        │          │          │       │       │          │ ssert_r │
│          │        │          │          │       │       │          │ ange.ym │
│          │        │          │          │       │       │          │ l       │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
assert_range_ex/d5e025dd: Creating Test Directory: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_range/assert_range_ex/d5e025dd
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
assert_range_ex/d5e025dd does not have any dependencies adding test to queue
  Builders Eligible to Run  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ assert_range_ex/d5e025dd │
└──────────────────────────┘
assert_range_ex/d5e025dd: Current Working Directory : /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_range/assert_range_ex/d5e025dd/stage
assert_range_ex/d5e025dd: Running Test via command: bash assert_range_ex_build.sh
assert_range_ex/d5e025dd: Test completed in 1.404004 seconds with returncode: 0
assert_range_ex/d5e025dd: Writing output file -  /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_range/assert_range_ex/d5e025dd/assert_range_ex.out
assert_range_ex/d5e025dd: Writing error file - /tmp/tmpq07_ocbs/var/tests/generic.local.bash/assert_range/assert_range_ex/d5e025dd/assert_range_ex.err
assert_range_ex/d5e025dd: testing metric: copy if 5000.0 <= 5910.9 <= 20000.0 - Check: True
assert_range_ex/d5e025dd: testing metric: scale if 4500.0 <= 6648.3 <= 20000.0 - Check: True
assert_range_ex/d5e025dd: testing metric: add if 4300.0 <= 8882.0 <= 20000.0 - Check: True
assert_range_ex/d5e025dd: testing metric: triad if 5600.0 <= 8483.0 <= 20000.0 - Check: True
assert_range_ex/d5e025dd: Range Check: True
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ assert_range_ex/d5e025d │ generic.local.bash │ PASS   │ 0          │ 1.404   │
│ d                       │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to /tmp/tmpq07_ocbs/var/report.json
Writing Logfile to /tmp/tmpq07_ocbs/var/logs/buildtest_oowi89mq.log

Note that performance results may vary on your system and depending on the metric value you may want to adjust the lower and upper bound to match your requirement.

Contains and Not Contains

Buildtest can perform status check with a list of reference values and check if metrics value is in the list. The property contains and not_contains can be used to perform this type of check. The ref property is a list of reference values that a metric must have to pass metrics check.

In example below we have two tests, the first test perform contains and not_contains on metrics x. We expect both status check will pass. The second test is expected to fail because metric x will store integer value 1 but the list has string equivalent ‘1’.

 1buildspecs:
 2  contains_and_not_contains:
 3    type: script
 4    executor: generic.local.bash
 5    description: Status check based on contains and not contains where test pass
 6    vars:
 7      X: 1
 8    run: |
 9      echo "X: $X"
10    metrics:
11      x:
12        type: int
13        regex:
14          stream: stdout
15          exp: 'X:\s+(\S+)\s+.*'
16          item: 1
17    status:
18      contains:
19        comparisons:
20        - name: x
21          ref: [1, 2, 4, 8]
22      not_contains:
23        comparisons:
24        - name: x
25          ref: [2, 4]
26  assert_contains_fail:
27    type: script
28    executor: generic.local.bash
29    description: Status check based on contains where test fails
30    vars:
31      X: 1
32    run: |
33      echo "X: $X"
34    metrics:
35      x:
36        type: int
37        regex:
38          stream: stdout
39          exp: 'X:\s+(\S+)\s+.*'
40          item: 1
41    status:
42      contains:
43        comparisons:
44        - name: x
45          ref: ['1', 2, 4, 8]

You can run this test, by running the following command

buildtest build -b tutorials/perf_checks/contains.yml
$ buildtest build -b tutorials/perf_checks/contains.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│                                                                              │
│ User:               docs                                                     │
│ Hostname:           build-23532485-project-280831-buildtest                  │
│ Platform:           Linux                                                    │
│ Current Time:       2024/02/22 20:33:01                                      │
│ buildtest path:     /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version:  1.8                                                      │
│ python path:        /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version:     3.8.18                                                   │
│ Configuration File: /tmp/tmpq07_ocbs/config.yml                              │
│ Test Directory:     /tmp/tmpq07_ocbs/var/tests                               │
│ Report File:        /tmp/tmpq07_ocbs/var/report.json                         │
│ Command:            /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
───────────────────────────  Discovering Buildspecs ────────────────────────────
                             Discovered buildspecs                              
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                    ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/perf_checks/contains.yml                                            ║
╚══════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/perf_checks/contains.yml: VALID
Total builder objects created: 2
                            Builders by type=script                             
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃          ┃        ┃          ┃          ┃       ┃       ┃ descript ┃ buildsp ┃
┃ builder  ┃ type   ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion      ┃ ecs     ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ contains │ script │ generic. │ None     │ None  │ None  │ Status   │ /home/d │
│ _and_not │        │ local.ba │          │       │       │ check    │ ocs/che │
│ _contain │        │ sh       │          │       │       │ based on │ ckouts/ │
│ s/73f383 │        │          │          │       │       │ contains │ readthe │
│ 0b       │        │          │          │       │       │ and not  │ docs.or │
│          │        │          │          │       │       │ contains │ g/user_ │
│          │        │          │          │       │       │ where    │ builds/ │
│          │        │          │          │       │       │ test     │ buildte │
│          │        │          │          │       │       │ pass     │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/tu │
│          │        │          │          │       │       │          │ torials │
│          │        │          │          │       │       │          │ /perf_c │
│          │        │          │          │       │       │          │ hecks/c │
│          │        │          │          │       │       │          │ ontains │
│          │        │          │          │       │       │          │ .yml    │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ assert_c │ script │ generic. │ None     │ None  │ None  │ Status   │ /home/d │
│ ontains_ │        │ local.ba │          │       │       │ check    │ ocs/che │
│ fail/ba9 │        │ sh       │          │       │       │ based on │ ckouts/ │
│ b0632    │        │          │          │       │       │ contains │ readthe │
│          │        │          │          │       │       │ where    │ docs.or │
│          │        │          │          │       │       │ test     │ g/user_ │
│          │        │          │          │       │       │ fails    │ builds/ │
│          │        │          │          │       │       │          │ buildte │
│          │        │          │          │       │       │          │ st/chec │
│          │        │          │          │       │       │          │ kouts/d │
│          │        │          │          │       │       │          │ evel/tu │
│          │        │          │          │       │       │          │ torials │
│          │        │          │          │       │       │          │ /perf_c │
│          │        │          │          │       │       │          │ hecks/c │
│          │        │          │          │       │       │          │ ontains │
│          │        │          │          │       │       │          │ .yml    │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
contains_and_not_contains/73f3830b: Creating Test Directory: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/contains/contains_and_not_contains/73f3830b
assert_contains_fail/ba9b0632: Creating Test Directory: /tmp/tmpq07_ocbs/var/tests/generic.local.bash/contains/assert_contains_fail/ba9b0632
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
assert_contains_fail/ba9b0632 does not have any dependencies adding test to queue
contains_and_not_contains/73f3830b does not have any dependencies adding test to queue
       Builders Eligible to Run       
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ assert_contains_fail/ba9b0632      │
│ contains_and_not_contains/73f3830b │
└────────────────────────────────────┘
assert_contains_fail/ba9b0632: Current Working Directory : /tmp/tmpq07_ocbs/var/tests/generic.local.bash/contains/assert_contains_fail/ba9b0632/stage
assert_contains_fail/ba9b0632: Running Test via command: bash assert_contains_fail_build.sh
assert_contains_fail/ba9b0632: Test completed in 0.006184 seconds with returncode: 0
assert_contains_fail/ba9b0632: Writing output file -  /tmp/tmpq07_ocbs/var/tests/generic.local.bash/contains/assert_contains_fail/ba9b0632/assert_contains_fail.out
assert_contains_fail/ba9b0632: Writing error file - /tmp/tmpq07_ocbs/var/tests/generic.local.bash/contains/assert_contains_fail/ba9b0632/assert_contains_fail.err
assert_contains_fail/ba9b0632: testing metric: x if 1 in ['1', 2, 4, 8] - Check: False
assert_contains_fail/ba9b0632: Contains Check: False
contains_and_not_contains/73f3830b: Current Working Directory : /tmp/tmpq07_ocbs/var/tests/generic.local.bash/contains/contains_and_not_contains/73f3830b/stage
contains_and_not_contains/73f3830b: Running Test via command: bash contains_and_not_contains_build.sh
contains_and_not_contains/73f3830b: Test completed in 0.005891 seconds with returncode: 0
contains_and_not_contains/73f3830b: Writing output file -  /tmp/tmpq07_ocbs/var/tests/generic.local.bash/contains/contains_and_not_contains/73f3830b/contains_and_not_contains.out
contains_and_not_contains/73f3830b: Writing error file - /tmp/tmpq07_ocbs/var/tests/generic.local.bash/contains/contains_and_not_contains/73f3830b/contains_and_not_contains.err
contains_and_not_contains/73f3830b: testing metric: x if 1 in [1, 2, 4, 8] - Check: True
contains_and_not_contains/73f3830b: Contains Check: True
contains_and_not_contains/73f3830b: testing metric: x if 1 not in [2, 4] - Check: True
contains_and_not_contains/73f3830b: Not Contains Check: True
                                  Test Summary                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                 ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ contains_and_not_contai │ generic.local.bash │ PASS   │ 0          │ 0.006   │
│ ns/73f3830b             │                    │        │            │         │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ assert_contains_fail/ba │ generic.local.bash │ FAIL   │ 0          │ 0.006   │
│ 9b0632                  │                    │        │            │         │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 1/2 Percentage: 50.000%
Failed Tests: 1/2 Percentage: 50.000%


Adding 2 test results to /tmp/tmpq07_ocbs/var/report.json
Writing Logfile to /tmp/tmpq07_ocbs/var/logs/buildtest_p_gooknb.log