Performance Checks¶
Defining Metrics¶
buildtest provides a method to define test metrics in the buildspecs which can be used to
store arbitrary content from the output/error file into named metric. A metric is
defined using the metrics property where each element under the metrics property
is the name of the metric which must be a unique name. A metric can apply regular expression
on stdout, stderr like in this example below. The metrics are captured in the test report which can
be queried via buildtest report or buildtest inspect. Shown below is an example
where we define two metrics named hpcg_rating and hpcg_state.
buildspecs:
metric_regex_example:
executor: generic.local.sh
type: script
description: capture result metric from output
run: echo "HPCG result is VALID with a GFLOP/s rating of=63.6515"
tags: tutorials
metrics:
hpcg_rating:
type: float
regex:
exp: '(\d+\.\d+)$'
stream: stdout
hpcg_state:
type: str
regex:
exp: '(VALID)'
stream: stdout
The metrics will not impact behavior of test, it will only impact the test report. By default
a metric will be an empty dictionary if there is no metrics property. If we fail to match
a regular expression, the metric will be defined as an empty string.
Note
If your regular expression contains an escape character \ you must surround your
string in single quotes ' as pose to double quotes "
Let’s build this test.
buildtest build -b tutorials/metrics_regex.yml
$ buildtest build -b tutorials/metrics_regex.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19096928-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/01/06 20:22:21 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/tutorials/metrics_regex.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ metric_… │ script │ generic… │ None │ None │ None │ capture │ /home/… │
│ │ │ │ │ │ │ result │ │
│ │ │ │ │ │ │ metric │ │
│ │ │ │ │ │ │ from │ │
│ │ │ │ │ │ │ output │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
metric_regex_example/73532cd7: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.sh/metrics_regex/metric_regex_example/73532cd7
metric_regex_example/73532cd7: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.sh/metrics_regex/metric_regex_example/73532cd7/stage
metric_regex_example/73532cd7: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.sh/metrics_regex/metric_regex_example/73532cd7/metric_regex_example_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
metric_regex_example/73532cd7 does not have any dependencies adding test to queue
metric_regex_example/73532cd7: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.sh/metrics_regex/metric_regex_example/73532cd7/stage
metric_regex_example/73532cd7: Running Test via command: sh --norc --noprofile -eo pipefail metric_regex_example_build.sh
metric_regex_example/73532cd7: Test completed in 0.00329 seconds
metric_regex_example/73532cd7: Test completed with returncode: 2
metric_regex_example/73532cd7: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.sh/metrics_regex/metric_regex_example/73532cd7/metric_regex_example.out
metric_regex_example/73532cd7: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.sh/metrics_regex/metric_regex_example/73532cd7/metric_regex_example.err
In this iteration we are going to run the following tests: [metric_regex_example/73532cd7]
Test Summary
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ metric_regex_ │ generic.loc… │ FAIL │ N/A N/A N/A │ 2 │ 0.00329 │
│ example/73532 │ │ │ │ │ │
│ cd7 │ │ │ │ │ │
└───────────────┴──────────────┴────────┴───────────────┴────────────┴─────────┘
Passed Tests: 0/1 Percentage: 0.000%
Failed Tests: 1/1 Percentage: 100.000%
Adding 1 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/logs/buildtest_w3jxyg4q.log
We can query the metrics via buildtest report which will display all metrics as a comma separted
Key/Value pair. We can use buildtest report --format metrics to extract all metrics for a test.
Internally, we store the metrics as a dictionary but when we print them out via buildtest report we
join them together into a single string. Shown below is the metrics for the previous build.
buildtest report --filter buildspec=tutorials/metrics_regex.yml --format name,metrics
$ buildtest report --filter buildspec=tutorials/metrics_regex.yml --format name,metrics
Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name ┃ metrics ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ metric_regex_example │ hpcg_rating=,hpcg_state= │
└────────────────────────────────────┴─────────────────────────────────────────┘
You can define a metric based on variables or environment variables
which requires you have set vars or env property in the buildspec. The vars and
env is a property under the metric name that can be used to reference name
of variable or environment variable. If you reference an invalid name, buildtest will assign the metric an empty string.
In this next example, we define two metrics gflop and foo that are assigned to variable GFLOPS and
environment variable FOO.
Assert Greater Equal¶
buildtest can determine status check based on performance check. In this next example, we will run the
STREAM memory benchmark and capture metrics named copy, scale
add and triad from the output and perform an Assertion Greater Equal (assert_ge) with a reference value.
The assert_ge contains a list of assertions where each metric name is
referenced via name that is compared with the reference value defined by ref property. The comparison
is metric_value >= ref, where metric_value is the value assigned to the metric name captured by the regular
expression. The type field in the metric section is used for the type conversion which can be float, int, or string.
The item is a numeric field used in match.group to retrieve the output
from the regular expression search. The item must be non-negative number.
1buildspecs:
2 stream_test:
3 type: script
4 executor: generic.local.bash
5 description: Run stream test with metrics example using assert greater equal
6 env:
7 OMP_NUM_THREADS: 4
8 run: |
9 wget https://raw.githubusercontent.com/jeffhammond/STREAM/master/stream.c
10 gcc -openmp -o stream stream.c
11 ./stream
12 metrics:
13 copy:
14 type: float
15 regex:
16 exp: 'Copy:\s+(\S+)\s+.*'
17 stream: stdout
18 item: 1
19 scale:
20 type: float
21 regex:
22 exp: 'Scale:\s+(\S+)\s+.*'
23 stream: stdout
24 item: 1
25 add:
26 type: float
27 regex:
28 exp: 'Add:\s+(\S+)\s+.*'
29 stream: stdout
30 item: 1
31 triad:
32 type: float
33 regex:
34 exp: 'Triad:\s+(\S+)\s+.*'
35 stream: stdout
36 item: 1
37 status:
38 assert_ge:
39 - name: copy
40 ref: 5000
41 - name: scale
42 ref: 5500
43 - name: add
44 ref: 6000
45 - name: triad
46 ref: 6500
buildtest will evaluate each assertion in the list and use a logical AND to determine the final
status of assert_ge.
Let’s build this test, take a close look at the output of buildtest build and take note of the assertion
statement.
buildtest build -b tutorials/perf_checks/assert_ge.yml
$ buildtest build -b tutorials/perf_checks/assert_ge.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19096928-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/01/06 20:22:22 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/tutorials/perf_checks/assert_ge.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ stream_… │ script │ generic… │ None │ None │ None │ Run │ /home/… │
│ │ │ │ │ │ │ stream │ │
│ │ │ │ │ │ │ test │ │
│ │ │ │ │ │ │ with │ │
│ │ │ │ │ │ │ metrics │ │
│ │ │ │ │ │ │ example │ │
│ │ │ │ │ │ │ using │ │
│ │ │ │ │ │ │ assert │ │
│ │ │ │ │ │ │ greater │ │
│ │ │ │ │ │ │ equal │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
stream_test/b6dc4086: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_ge/stream_test/b6dc4086
stream_test/b6dc4086: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_ge/stream_test/b6dc4086/stage
stream_test/b6dc4086: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_ge/stream_test/b6dc4086/stream_test_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
stream_test/b6dc4086 does not have any dependencies adding test to queue
stream_test/b6dc4086: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_ge/stream_test/b6dc4086/stage
stream_test/b6dc4086: Running Test via command: bash --norc --noprofile -eo pipefail stream_test_build.sh
stream_test/b6dc4086: Test completed in 1.767506 seconds
stream_test/b6dc4086: Test completed with returncode: 0
stream_test/b6dc4086: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_ge/stream_test/b6dc4086/stream_test.out
stream_test/b6dc4086: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_ge/stream_test/b6dc4086/stream_test.err
stream_test/b6dc4086: testing metric: copy if 5937.4 >= 5000.0
stream_test/b6dc4086: testing metric: scale if 6671.9 >= 5500.0
stream_test/b6dc4086: testing metric: add if 8804.1 >= 6000.0
stream_test/b6dc4086: testing metric: triad if 8498.9 >= 6500.0
In this iteration we are going to run the following tests: [stream_test/b6dc4086]
Test Summary
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ stream_test/ │ generic.loc… │ PASS │ False False │ 0 │ 1.767506 │
│ b6dc4086 │ │ │ False │ │ │
└──────────────┴──────────────┴────────┴───────────────┴────────────┴──────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/logs/buildtest_q9gy3aky.log
Let’s run buildtest inspect query -o stream_test to retrieve the test details and output of STREAM test.
buildtest inspect query -o stream_test
$ buildtest inspect query -o stream_test
─────────────── stream_test/b6dc4086-712d-4666-91c1-9789c96ac1f8 ───────────────
Executor: generic.local.bash
Description: Run stream test with metrics example using assert greater equal
State: PASS
Returncode: 0
Runtime: 1.767506 sec
Starttime: 2023/01/06 20:22:22
Endtime: 2023/01/06 20:22:23
Command: bash --norc --noprofile -eo pipefail stream_test_build.sh
Test Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_ge/stream_test/b6dc4086/stream_test.sh
Build Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_ge/stream_test/b6dc4086/stream_test_build.sh
Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_ge/stream_test/b6dc4086/stream_test.out
Error File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_ge/stream_test/b6dc4086/stream_test.err
Log File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/logs/buildtest_q9gy3aky.log
Metrics
┏━━━━━━━┳━━━━━━━━┓
┃ Name ┃ Value ┃
┡━━━━━━━╇━━━━━━━━┩
│ copy │ 5937.4 │
│ scale │ 6671.9 │
│ add │ 8804.1 │
│ triad │ 8498.9 │
└───────┴────────┘
─ Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/che… ─
-------------------------------------------------------------
STREAM version $Revision: 5.10 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Array size = 10000000 (elements), Offset = 0 (elements)
Memory per array = 76.3 MiB (= 0.1 GiB).
Total memory required = 228.9 MiB (= 0.2 GiB).
Each kernel will be executed 10 times.
The *best* time for each kernel (excluding the first iteration)
will be used to compute the reported bandwidth.
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 24815 microseconds.
(= 24815 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function Best Rate MB/s Avg time Min time Max time
Copy: 5937.4 0.027043 0.026948 0.027084
Scale: 6671.9 0.024085 0.023981 0.024229
Add: 8804.1 0.027401 0.027260 0.027517
Triad: 8498.9 0.028348 0.028239 0.028545
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------
Assert Equal¶
buildtest can perform assert equality check with metrics to determine status of test. In this next example, we define
four metrics x, y, first, and last which will be compared with its reference value. We introduce a new
property assert_eq which is composed of list of assertions. Each reference is converted to its appropriate
type (int, float, str).
1buildspecs:
2 assert_eq_example:
3 type: script
4 executor: generic.local.bash
5 description: Test for assert equality
6 vars:
7 X: 1
8 Y: 1.5
9 first: John
10 last: Smith
11 run: |
12 echo "X: $X"
13 echo "Y: $Y"
14 echo "Name: $first $last"
15 metrics:
16 x:
17 type: int
18 regex:
19 stream: stdout
20 exp: 'X:\s+(\S+)\s+.*'
21 item: 1
22 y:
23 type: float
24 regex:
25 stream: stdout
26 exp: 'Y:\s+(\S+)\s+.*'
27 item: 1
28 first:
29 type: str
30 regex:
31 stream: stdout
32 exp: 'Name:\s+(\S+)\s+.*'
33 item: 1
34 last:
35 type: str
36 regex:
37 stream: stdout
38 exp: '(Smith)$'
39 item: 1
40 status:
41 assert_eq:
42 - name: x
43 ref: 1
44 - name: y
45 ref: 1.5
46 - name: first
47 ref: John
48 - name: last
49 ref: Smith
This test is expected to pass where all assertions are True. Let’s build the test and see the output
buildtest build -b tutorials/perf_checks/assert_eq.yml
$ buildtest build -b tutorials/perf_checks/assert_eq.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19096928-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/01/06 20:22:24 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/tutorials/perf_checks/assert_eq.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ assert_… │ script │ generic… │ None │ None │ None │ Test for │ /home/… │
│ │ │ │ │ │ │ assert │ │
│ │ │ │ │ │ │ equality │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
assert_eq_example/a848cf77: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq/assert_eq_example/a848cf77
assert_eq_example/a848cf77: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq/assert_eq_example/a848cf77/stage
assert_eq_example/a848cf77: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq/assert_eq_example/a848cf77/assert_eq_example_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
assert_eq_example/a848cf77 does not have any dependencies adding test to queue
assert_eq_example/a848cf77: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq/assert_eq_example/a848cf77/stage
assert_eq_example/a848cf77: Running Test via command: bash --norc --noprofile -eo pipefail assert_eq_example_build.sh
assert_eq_example/a848cf77: Test completed in 0.005614 seconds
assert_eq_example/a848cf77: Test completed with returncode: 0
assert_eq_example/a848cf77: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq/assert_eq_example/a848cf77/assert_eq_example.out
assert_eq_example/a848cf77: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq/assert_eq_example/a848cf77/assert_eq_example.err
assert_eq_example/a848cf77: testing metric: x if 1 == 1
assert_eq_example/a848cf77: testing metric: y if 1.5 == 1.5
assert_eq_example/a848cf77: testing metric: first if John == John
assert_eq_example/a848cf77: testing metric: last if Smith == Smith
In this iteration we are going to run the following tests: [assert_eq_example/a848cf77]
Test Summary
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ assert_eq_ex │ generic.loc… │ PASS │ False False │ 0 │ 0.005614 │
│ ample/a848cf │ │ │ False │ │ │
│ 77 │ │ │ │ │ │
└──────────────┴──────────────┴────────┴───────────────┴────────────┴──────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/logs/buildtest_wkp4rjap.log
buildtest inspect query -o assert_eq_example
$ buildtest inspect query -o assert_eq_example
──────────── assert_eq_example/a848cf77-8fff-425c-9e9d-69aff7fde4a8 ────────────
Executor: generic.local.bash
Description: Test for assert equality
State: PASS
Returncode: 0
Runtime: 0.005614 sec
Starttime: 2023/01/06 20:22:25
Endtime: 2023/01/06 20:22:25
Command: bash --norc --noprofile -eo pipefail assert_eq_example_build.sh
Test Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq/assert_eq_example/a848cf77/assert_eq_example.sh
Build Script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq/assert_eq_example/a848cf77/assert_eq_example_build.sh
Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq/assert_eq_example/a848cf77/assert_eq_example.out
Error File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq/assert_eq_example/a848cf77/assert_eq_example.err
Log File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/logs/buildtest_wkp4rjap.log
Metrics
┏━━━━━━━┳━━━━━━━┓
┃ Name ┃ Value ┃
┡━━━━━━━╇━━━━━━━┩
│ x │ 1 │
│ y │ 1.5 │
│ first │ John │
│ last │ Smith │
└───────┴───────┘
─ Output File: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/che… ─
X: 1
Y: 1.5
Name: John Smith
In the next example, we have two tests to highlight some exceptions. In the first test, we define an invalid metric name invalid_metric
in assert_eq since this metric was not defined in metrics field, therefore this test will fail. The second test will fail because we have
a mismatch in value captured by metric x which is 1 however the reference value is 2.
1buildspecs:
2 assert_eq_invalid_metric:
3 type: script
4 executor: generic.local.bash
5 description: An invalid metric name will cause failure
6 vars:
7 X: 1
8 run: |
9 echo "X: $X"
10 metrics:
11 x:
12 type: int
13 regex:
14 stream: stdout
15 exp: 'X:\s+(\S+)\s+.*'
16 item: 1
17 status:
18 assert_eq:
19 - name: x
20 ref: 1
21 - name: invalid_metric
22 ref: 'hello'
23 assert_eq_mismatch:
24 type: script
25 executor: generic.local.bash
26 description: This test will fail because there is a mismatch in metric x assert equality
27 vars:
28 X: 1
29 run: |
30 echo "X: $X"
31 metrics:
32 x:
33 type: int
34 regex:
35 stream: stdout
36 exp: 'X:\s+(\S+)\s+.*'
37 item: 1
38 status:
39 assert_eq:
40 - name: x
41 ref: 2
Let’s build this test and see the output.
buildtest build -b tutorials/perf_checks/assert_eq_exceptions.yml
$ buildtest build -b tutorials/perf_checks/assert_eq_exceptions.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19096928-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/01/06 20:22:26 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/tutorials/perf_checks/assert_eq_exceptions.yml: VALID
Total builder objects created: 2
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ assert_… │ script │ generic… │ None │ None │ None │ An │ /home/… │
│ │ │ │ │ │ │ invalid │ │
│ │ │ │ │ │ │ metric │ │
│ │ │ │ │ │ │ name │ │
│ │ │ │ │ │ │ will │ │
│ │ │ │ │ │ │ cause │ │
│ │ │ │ │ │ │ failure │ │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ assert_… │ script │ generic… │ None │ None │ None │ This │ /home/… │
│ │ │ │ │ │ │ test │ │
│ │ │ │ │ │ │ will │ │
│ │ │ │ │ │ │ fail │ │
│ │ │ │ │ │ │ because │ │
│ │ │ │ │ │ │ there is │ │
│ │ │ │ │ │ │ a │ │
│ │ │ │ │ │ │ mismatch │ │
│ │ │ │ │ │ │ in │ │
│ │ │ │ │ │ │ metric x │ │
│ │ │ │ │ │ │ assert │ │
│ │ │ │ │ │ │ equality │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
assert_eq_invalid_metric/91f326eb: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_invalid_metric/91f326eb
assert_eq_invalid_metric/91f326eb: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_invalid_metric/91f326eb/stage
assert_eq_invalid_metric/91f326eb: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_invalid_metric/91f326eb/assert_eq_invalid_metric_build.sh
assert_eq_mismatch/55e4421e: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_mismatch/55e4421e
assert_eq_mismatch/55e4421e: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_mismatch/55e4421e/stage
assert_eq_mismatch/55e4421e: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_mismatch/55e4421e/assert_eq_mismatch_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
assert_eq_mismatch/55e4421e does not have any dependencies adding test to queue
assert_eq_invalid_metric/91f326eb does not have any dependencies adding test to queue
assert_eq_invalid_metric/91f326eb: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_invalid_metric/91f326eb/stage
assert_eq_invalid_metric/91f326eb: Running Test via command: bash --norc --noprofile -eo pipefail assert_eq_invalid_metric_build.sh
assert_eq_invalid_metric/91f326eb: Test completed in 0.005774 seconds
assert_eq_invalid_metric/91f326eb: Test completed with returncode: 0
assert_eq_invalid_metric/91f326eb: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_invalid_metric/91f326eb/assert_eq_invalid_metric.out
assert_eq_invalid_metric/91f326eb: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_invalid_metric/91f326eb/assert_eq_invalid_metric.err
assert_eq_invalid_metric/91f326eb: testing metric: x if 1 == 1
assert_eq_invalid_metric/91f326eb: Unable to find metric: invalid_metric. List of valid metrics are the following: ['x']
assert_eq_mismatch/55e4421e: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_mismatch/55e4421e/stage
assert_eq_mismatch/55e4421e: Running Test via command: bash --norc --noprofile -eo pipefail assert_eq_mismatch_build.sh
assert_eq_mismatch/55e4421e: Test completed in 0.005438 seconds
assert_eq_mismatch/55e4421e: Test completed with returncode: 0
assert_eq_mismatch/55e4421e: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_mismatch/55e4421e/assert_eq_mismatch.out
assert_eq_mismatch/55e4421e: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_eq_exceptions/assert_eq_mismatch/55e4421e/assert_eq_mismatch.err
assert_eq_mismatch/55e4421e: testing metric: x if 1 == 2
In this iteration we are going to run the following tests: [assert_eq_invalid_metric/91f326eb, assert_eq_mismatch/55e4421e]
Test Summary
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ assert_eq_in │ generic.loc… │ FAIL │ False False │ 0 │ 0.005774 │
│ valid_metric │ │ │ False │ │ │
│ /91f326eb │ │ │ │ │ │
├──────────────┼──────────────┼────────┼───────────────┼────────────┼──────────┤
│ assert_eq_mi │ generic.loc… │ FAIL │ False False │ 0 │ 0.005438 │
│ smatch/55e44 │ │ │ False │ │ │
│ 21e │ │ │ │ │ │
└──────────────┴──────────────┴────────┴───────────────┴────────────┴──────────┘
Passed Tests: 0/2 Percentage: 0.000%
Failed Tests: 2/2 Percentage: 100.000%
Adding 2 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/logs/buildtest_j_yqor7q.log
Assert Range¶
The assert_range property can be used to test performance for a metric given a lower and upper bound. This property expects
one to specify lower and upper field which must be an integer or floating point number to perform comparison. buildtest will
perform an assertion, if metric value is in the range specified by lower and upper, then test will pass. Shown below
is an example using the assert_range property with stream benchmark.
1buildspecs:
2 assert_range_ex:
3 type: script
4 executor: generic.local.bash
5 description: Example on assert_range
6 env:
7 OMP_NUM_THREADS: 4
8 run: |
9 wget https://raw.githubusercontent.com/jeffhammond/STREAM/master/stream.c
10 gcc -openmp -o stream stream.c
11 ./stream
12 metrics:
13 copy:
14 type: float
15 regex:
16 exp: 'Copy:\s+(\S+)\s+.*'
17 stream: stdout
18 item: 1
19 scale:
20 type: float
21 regex:
22 exp: 'Scale:\s+(\S+)\s+.*'
23 stream: stdout
24 item: 1
25 add:
26 type: float
27 regex:
28 exp: 'Add:\s+(\S+)\s+.*'
29 stream: stdout
30 item: 1
31 triad:
32 type: float
33 regex:
34 exp: 'Triad:\s+(\S+)\s+.*'
35 stream: stdout
36 item: 1
37 status:
38 assert_range:
39 - name: copy
40 lower: 5000
41 upper: 20000
42 - name: scale
43 lower: 4500
44 upper: 20000
45 - name: add
46 lower: 4300
47 upper: 20000
48 - name: triad
49 lower: 5600
50 upper: 20000
Let’s build this test and see the output
buildtest build -b tutorials/perf_checks/assert_range.yml
$ buildtest build -b tutorials/perf_checks/assert_range.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-19096928-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2023/01/06 20:22:26 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 1.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.7.15 │
│ Configuration File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Test Directory: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Report File: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/t… ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/tutorials/perf_checks/assert_range.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descrip… ┃ builds… ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ assert_… │ script │ generic… │ None │ None │ None │ Example │ /home/… │
│ │ │ │ │ │ │ on │ │
│ │ │ │ │ │ │ assert_… │ │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
assert_range_ex/70156c70: Creating test directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_range/assert_range_ex/70156c70
assert_range_ex/70156c70: Creating the stage directory: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_range/assert_range_ex/70156c70/stage
assert_range_ex/70156c70: Writing build script: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_range/assert_range_ex/70156c70/assert_range_ex_build.sh
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
assert_range_ex/70156c70 does not have any dependencies adding test to queue
assert_range_ex/70156c70: Current Working Directory : /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_range/assert_range_ex/70156c70/stage
assert_range_ex/70156c70: Running Test via command: bash --norc --noprofile -eo pipefail assert_range_ex_build.sh
assert_range_ex/70156c70: Test completed in 1.417391 seconds
assert_range_ex/70156c70: Test completed with returncode: 0
assert_range_ex/70156c70: Writing output file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_range/assert_range_ex/70156c70/assert_range_ex.out
assert_range_ex/70156c70: Writing error file - /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/tests/generic.local.bash/assert_range/assert_range_ex/70156c70/assert_range_ex.err
assert_range_ex/70156c70: testing metric: copy if 5000.0 <= 5938.0 <= 20000.0
assert_range_ex/70156c70: testing metric: scale if 4500.0 <= 6680.0 <= 20000.0
assert_range_ex/70156c70: testing metric: add if 4300.0 <= 8903.1 <= 20000.0
assert_range_ex/70156c70: testing metric: triad if 5600.0 <= 8486.8 <= 20000.0
In this iteration we are going to run the following tests: [assert_range_ex/70156c70]
Test Summary
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ┃ ┃ ┃ checks ┃ ┃ ┃
┃ ┃ ┃ ┃ (ReturnCode, ┃ ┃ ┃
┃ ┃ ┃ ┃ Regex, ┃ ┃ ┃
┃ builder ┃ executor ┃ status ┃ Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ assert_range │ generic.loc… │ PASS │ False False │ 0 │ 1.417391 │
│ _ex/70156c70 │ │ │ False │ │ │
└──────────────┴──────────────┴────────┴───────────────┴────────────┴──────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/report.json
Writing Logfile to: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/v1.1/var/logs/buildtest_omqrxr53.log
Note that performance results may vary on your system and depending on the metric value you may want to adjust the lower and upper bound to match your requirement.