Defining Metrics
buildtest provides a method to define test metrics in the buildspecs which can be used to
store arbitrary content from the output/error file or an arbitrary file into named metric. The
metrics property is used to define a list of metric names using regular expression to assign a value
to the metric. In this example, we have two tests that define metrics hpcg_rate_stream, hpcg_state_stream
in the first test and hpcg_rate_file, hpcg_state_file in the second test. The stream property is used
to read from stdout/stderr and apply the regular expression defined by exp, whereas file_regex is used
to define metrics from an arbitrary file where file is the path to file.
buildspecs:
metric_regex_example:
executor: generic.local.bash
type: script
description: capture result metric from output
run: echo "HPCG result is VALID with a GFLOP/s rating of=63.6515"
tags: tutorials
metrics:
hpcg_rating_stream:
type: float
regex:
exp: '(\d+\.\d+)$'
stream: stdout
hpcg_state_stream:
type: str
regex:
exp: '(VALID)'
stream: stdout
metric_file_regex:
executor: generic.local.bash
type: script
description: capture result metric from file path
run: echo "HPCG result is VALID with a GFLOP/s rating of=63.6515" > hpcg.txt
tags: tutorials
metrics:
hpcg_rating_file:
type: float
file_regex:
exp: '(\d+\.\d+)$'
file: hpcg.txt
hpcg_state_file:
type: str
file_regex:
exp: '(VALID)'
file: hpcg.txt
The metrics can be used with Comparison Operators for performing more sophisticated status checks.
By default, a metric will be an empty dictionary if there is no metrics property. If we fail to match
a regular expression, the metric will be defined as an empty string ('').
Note
If your regular expression contains an escape character \ you must surround your
string in single quotes ' as pose to double quotes "
Let’s build this test.
buildtest build -b tutorials/metrics/metrics_regex.yml
$ buildtest build -b tutorials/metrics/metrics_regex.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-25534835-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/09/05 15:37:52 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.9.19 │
│ Configuration File: /tmp/tmpr88132k2/config.yml │
│ Test Directory: /tmp/tmpr88132k2/var/tests │
│ Report File: /tmp/tmpr88132k2/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/metrics/metrics_regex.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/metrics/metrics_regex.yml: VALID
Total builder objects created: 2
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ metric_r │ script │ generic. │ None │ None │ None │ capture │ /home/d │
│ egex_exa │ │ local.ba │ │ │ │ result │ ocs/che │
│ mple/de7 │ │ sh │ │ │ │ metric │ ckouts/ │
│ abe5f │ │ │ │ │ │ from │ readthe │
│ │ │ │ │ │ │ output │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/s │
│ │ │ │ │ │ │ │ table/t │
│ │ │ │ │ │ │ │ utorial │
│ │ │ │ │ │ │ │ s/metri │
│ │ │ │ │ │ │ │ cs/metr │
│ │ │ │ │ │ │ │ ics_reg │
│ │ │ │ │ │ │ │ ex.yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ metric_f │ script │ generic. │ None │ None │ None │ capture │ /home/d │
│ ile_rege │ │ local.ba │ │ │ │ result │ ocs/che │
│ x/d4b90d │ │ sh │ │ │ │ metric │ ckouts/ │
│ 87 │ │ │ │ │ │ from │ readthe │
│ │ │ │ │ │ │ file │ docs.or │
│ │ │ │ │ │ │ path │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/s │
│ │ │ │ │ │ │ │ table/t │
│ │ │ │ │ │ │ │ utorial │
│ │ │ │ │ │ │ │ s/metri │
│ │ │ │ │ │ │ │ cs/metr │
│ │ │ │ │ │ │ │ ics_reg │
│ │ │ │ │ │ │ │ ex.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
metric_regex_example/de7abe5f: Creating Test Directory: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex/metric_regex_example/de7abe5f
metric_file_regex/d4b90d87: Creating Test Directory: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex/metric_file_regex/d4b90d87
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
metric_file_regex/d4b90d87 does not have any dependencies adding test to queue
metric_regex_example/de7abe5f does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ metric_file_regex/d4b90d87 │
│ metric_regex_example/de7abe5f │
└───────────────────────────────┘
metric_file_regex/d4b90d87: Current Working Directory : /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex/metric_file_regex/d4b90d87/stage
metric_file_regex/d4b90d87: Running Test via command: bash metric_file_regex_build.sh
metric_file_regex/d4b90d87: Test completed in 0.008771 seconds with returncode: 0
metric_file_regex/d4b90d87: Writing output file - /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex/metric_file_regex/d4b90d87/metric_file_regex.out
metric_file_regex/d4b90d87: Writing error file - /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex/metric_file_regex/d4b90d87/metric_file_regex.err
metric_regex_example/de7abe5f: Current Working Directory : /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex/metric_regex_example/de7abe5f/stage
metric_regex_example/de7abe5f: Running Test via command: bash metric_regex_example_build.sh
metric_regex_example/de7abe5f: Test completed in 0.008798 seconds with returncode: 0
metric_regex_example/de7abe5f: Writing output file - /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex/metric_regex_example/de7abe5f/metric_regex_example.out
metric_regex_example/de7abe5f: Writing error file - /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex/metric_regex_example/de7abe5f/metric_regex_example.err
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ metric_regex_example/de │ generic.local.bash │ PASS │ 0 │ 0.009 │
│ 7abe5f │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ metric_file_regex/d4b90 │ generic.local.bash │ PASS │ 0 │ 0.009 │
│ d87 │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%
Adding 2 test results to report file: /tmp/tmpr88132k2/var/report.json
Writing Logfile to /tmp/tmpr88132k2/var/logs/buildtest_92d4a4xs.log
The metrics are captured in the test report which can
be queried via buildtest report or buildtest inspect query. Metrics can be seen in the test metadata,
for instance you can run buildtest inspect query and you will see metrics shown in table output.
buildtest inspect query metric_regex_example metric_file_regex
$ buildtest inspect query metric_regex_example metric_file_regex
──────────── metric_file_regex/d4b90d87-973d-4d4d-848d-cdeadf63b195 ────────────
Executor: generic.local.bash
Description: capture result metric from file path
State: PASS
Returncode: 0
Runtime: 0.008771 sec
Starttime: 2024/09/05 15:37:52
Endtime: 2024/09/05 15:37:52
Command: bash metric_file_regex_build.sh
Test Script: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex/metric_file_regex/d4b90d87/metric_file_regex.sh
Build Script: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex/metric_file_regex/d4b90d87/metric_file_regex_build.sh
Output File: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex/metric_file_regex/d4b90d87/metric_file_regex.out
Error File: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex/metric_file_regex/d4b90d87/metric_file_regex.err
Log File: /tmp/tmpr88132k2/var/logs/buildtest_92d4a4xs.log
Metrics
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Name ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ hpcg_rating_file │ 63.6515 │
│ hpcg_state_file │ VALID │
└──────────────────┴─────────┘
────────── metric_regex_example/de7abe5f-7a6a-4d6d-bd58-94f399d04b72 ───────────
Executor: generic.local.bash
Description: capture result metric from output
State: PASS
Returncode: 0
Runtime: 0.008798 sec
Starttime: 2024/09/05 15:37:52
Endtime: 2024/09/05 15:37:52
Command: bash metric_regex_example_build.sh
Test Script: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex/metric_regex_example/de7abe5f/metric_regex_example.sh
Build Script: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex/metric_regex_example/de7abe5f/metric_regex_example_build.sh
Output File: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex/metric_regex_example/de7abe5f/metric_regex_example.out
Error File: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex/metric_regex_example/de7abe5f/metric_regex_example.err
Log File: /tmp/tmpr88132k2/var/logs/buildtest_92d4a4xs.log
Metrics
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Name ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ hpcg_rating_stream │ 63.6515 │
│ hpcg_state_stream │ VALID │
└────────────────────┴─────────┘
We can query the metrics via buildtest report which will display all metrics as a comma seperated
Key/Value pair. We can use buildtest report --format metrics to extract all metrics for a test.
Internally, we store the metrics as a dictionary but when we print them out via buildtest report we
join them together into a single string. Shown below is the metrics for the previous build.
buildtest report --filter buildspec=tutorials/metrics/metrics_regex.yml --format name,metrics
$ buildtest report --filter buildspec=tutorials/metrics/metrics_regex.yml --format name,metrics
Report File: /tmp/tmpr88132k2/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name ┃ metrics ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ metric_regex_example │ hpcg_rating_stream=63.6515,hpcg_state_stream=VALID │
│ metric_file_regex │ hpcg_rating_file=63.6515,hpcg_state_file=VALID │
└──────────────────────┴────────────────────────────────────────────────────┘
Metrics with Regex Type via ‘re’
Building on the previous example, we will use the re property specify the regular expression type to use. By default, buildtest will
use re.search if re is not specified; however you can specify re to use re.match,
re.fullmatch, or re.search.
In this example, we will define 4 metrics hpcg_text, hpcg_result, hpcg_file_text, hpcg_file_result. The first two
metrics will capture from stdout using the regex property while the last two will capture from a file using the file_regex property.
The re.match will be used to capture the text HPCG result is VALID and HPCG result is INVALID from stdout and file, whereas
the re.search will be used to capture the test result 63.6515 and 28.1215 from stdout and file.
Finally, we will use the comparison operator assert_eq: Equal to compare the metrics with reference value.
buildspecs:
metric_regex_example_with_re:
executor: generic.local.bash
type: script
description: capture metric with different regex types
tags: tutorials
run: |
echo "HPCG result is VALID with a GFLOP/s rating of=63.6515"
echo "HPCG result is INVALID with a GFLOP/s rating of=28.1215" > hpcg.txt
metrics:
hpcg_result:
type: float
regex:
re: "re.search"
exp: '(\d+\.\d+)$'
stream: stdout
hpcg_text:
type: str
regex:
re: "re.match"
exp: '^HPCG result is VALID'
stream: stdout
hpcg_file_text:
type: str
file_regex:
re: "re.match"
exp: '^HPCG result is INVALID'
file: hpcg.txt
hpcg_file_result:
type: float
file_regex:
re: "re.search"
exp: '(\d+\.\d+)$'
file: hpcg.txt
status:
assert_eq:
comparisons:
- name: hpcg_text
ref: "HPCG result is VALID"
- name: hpcg_result
ref: 63.6515
- name: hpcg_file_text
ref: "HPCG result is INVALID"
- name: hpcg_file_result
ref: 28.1215
Let’s attempt to build this test
buildtest build -b tutorials/metrics/metrics_with_regex_type.yml
$ buildtest build -b tutorials/metrics/metrics_with_regex_type.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-25534835-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/09/05 15:37:55 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.9.19 │
│ Configuration File: /tmp/tmpr88132k2/config.yml │
│ Test Directory: /tmp/tmpr88132k2/var/tests │
│ Report File: /tmp/tmpr88132k2/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/metrics/metrics_with_regex_type.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/metrics/metrics_with_regex_type.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ metric_r │ script │ generic. │ None │ None │ None │ capture │ /home/d │
│ egex_exa │ │ local.ba │ │ │ │ metric │ ocs/che │
│ mple_wit │ │ sh │ │ │ │ with │ ckouts/ │
│ h_re/e9f │ │ │ │ │ │ differen │ readthe │
│ 8d5f0 │ │ │ │ │ │ t regex │ docs.or │
│ │ │ │ │ │ │ types │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/s │
│ │ │ │ │ │ │ │ table/t │
│ │ │ │ │ │ │ │ utorial │
│ │ │ │ │ │ │ │ s/metri │
│ │ │ │ │ │ │ │ cs/metr │
│ │ │ │ │ │ │ │ ics_wit │
│ │ │ │ │ │ │ │ h_regex │
│ │ │ │ │ │ │ │ _type.y │
│ │ │ │ │ │ │ │ ml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
metric_regex_example_with_re/e9f8d5f0: Creating Test Directory: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_with_regex_type/metric_regex_example_with_re/e9f8d5f0
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
metric_regex_example_with_re/e9f8d5f0 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ metric_regex_example_with_re/e9f8d5f0 │
└───────────────────────────────────────┘
metric_regex_example_with_re/e9f8d5f0: Current Working Directory : /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_with_regex_type/metric_regex_example_with_re/e9f8d5f0/stage
metric_regex_example_with_re/e9f8d5f0: Running Test via command: bash metric_regex_example_with_re_build.sh
metric_regex_example_with_re/e9f8d5f0: Test completed in 0.008932 seconds with returncode: 0
metric_regex_example_with_re/e9f8d5f0: Writing output file - /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_with_regex_type/metric_regex_example_with_re/e9f8d5f0/metric_regex_example_with_re.out
metric_regex_example_with_re/e9f8d5f0: Writing error file - /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_with_regex_type/metric_regex_example_with_re/e9f8d5f0/metric_regex_example_with_re.err
metric_regex_example_with_re/e9f8d5f0: testing metric: hpcg_text if HPCG result is VALID == HPCG result is VALID - Check: True
metric_regex_example_with_re/e9f8d5f0: testing metric: hpcg_result if 63.6515 == 63.6515 - Check: True
metric_regex_example_with_re/e9f8d5f0: testing metric: hpcg_file_text if HPCG result is INVALID == HPCG result is INVALID - Check: True
metric_regex_example_with_re/e9f8d5f0: testing metric: hpcg_file_result if 28.1215 == 28.1215 - Check: True
metric_regex_example_with_re/e9f8d5f0: eq check: True
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ metric_regex_example_wi │ generic.local.bash │ PASS │ 0 │ 0.009 │
│ th_re/e9f8d5f0 │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to report file: /tmp/tmpr88132k2/var/report.json
Writing Logfile to /tmp/tmpr88132k2/var/logs/buildtest_mdffofnd.log
Upon completion, lets take a look at the metrics for this test, we can see this by running buildtest inspect query
which shows the name of captured metrics and its corresponding values.
buildtest inspect query metric_regex_example_with_re
$ buildtest inspect query metric_regex_example_with_re
────── metric_regex_example_with_re/e9f8d5f0-cce1-4bc8-b7d8-24ce016cea3b ───────
Executor: generic.local.bash
Description: capture metric with different regex types
State: PASS
Returncode: 0
Runtime: 0.008932 sec
Starttime: 2024/09/05 15:37:55
Endtime: 2024/09/05 15:37:55
Command: bash metric_regex_example_with_re_build.sh
Test Script: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_with_regex_type/metric_regex_example_with_re/e9f8d5f0/metric_regex_example_with_re.sh
Build Script: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_with_regex_type/metric_regex_example_with_re/e9f8d5f0/metric_regex_example_with_re_build.sh
Output File: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_with_regex_type/metric_regex_example_with_re/e9f8d5f0/metric_regex_example_with_re.out
Error File: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_with_regex_type/metric_regex_example_with_re/e9f8d5f0/metric_regex_example_with_re.err
Log File: /tmp/tmpr88132k2/var/logs/buildtest_mdffofnd.log
Metrics
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩
│ hpcg_result │ 63.6515 │
│ hpcg_text │ HPCG result is VALID │
│ hpcg_file_text │ HPCG result is INVALID │
│ hpcg_file_result │ 28.1215 │
└──────────────────┴────────────────────────┘
Extract line numbers in Metrics
We can specify linenum property in metrics to capture text from a specific line.
In the first example, we will define 2 metrics second_line and without_linenum using regex
The former metrics captures the text This is line: 2, whereas the latter captures the text This is line: 1.
buildspecs:
metric_regex_with_linenum_example:
executor: generic.local.bash
type: script
description: capture result metric from output
run: |
echo "This is line: 1"
echo "This is line: 2"
echo "This is line: 3"
echo "This is line: 4"
tags: tutorials
metrics:
second_line:
type: str
regex:
exp: 'This is line: \d'
stream: stdout
linenum: 1
without_linenum:
type: str
regex:
exp: 'This is line: \d'
stream: stdout
status:
assert_eq:
comparisons:
- name: second_line
ref: "This is line: 2"
- name: without_linenum
ref: "This is line: 1"
Let’s build the test and take a look at the metrics for this test.
buildtest build -b tutorials/metrics/metrics_regex_with_linenum.yml
$ buildtest build -b tutorials/metrics/metrics_regex_with_linenum.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-25534835-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/09/05 15:37:57 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.9.19 │
│ Configuration File: /tmp/tmpr88132k2/config.yml │
│ Test Directory: /tmp/tmpr88132k2/var/tests │
│ Report File: /tmp/tmpr88132k2/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/metrics/metrics_regex_with_linenum.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/metrics/metrics_regex_with_linenum.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ metric_r │ script │ generic. │ None │ None │ None │ capture │ /home/d │
│ egex_wit │ │ local.ba │ │ │ │ result │ ocs/che │
│ h_linenu │ │ sh │ │ │ │ metric │ ckouts/ │
│ m_exampl │ │ │ │ │ │ from │ readthe │
│ e/963d8f │ │ │ │ │ │ output │ docs.or │
│ d5 │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/s │
│ │ │ │ │ │ │ │ table/t │
│ │ │ │ │ │ │ │ utorial │
│ │ │ │ │ │ │ │ s/metri │
│ │ │ │ │ │ │ │ cs/metr │
│ │ │ │ │ │ │ │ ics_reg │
│ │ │ │ │ │ │ │ ex_with │
│ │ │ │ │ │ │ │ _linenu │
│ │ │ │ │ │ │ │ m.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
metric_regex_with_linenum_example/963d8fd5: Creating Test Directory: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex_with_linenum/metric_regex_with_linenum_example/963d8fd5
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
metric_regex_with_linenum_example/963d8fd5 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ metric_regex_with_linenum_example/963d8fd5 │
└────────────────────────────────────────────┘
metric_regex_with_linenum_example/963d8fd5: Current Working Directory : /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex_with_linenum/metric_regex_with_linenum_example/963d8fd5/stage
metric_regex_with_linenum_example/963d8fd5: Running Test via command: bash metric_regex_with_linenum_example_build.sh
metric_regex_with_linenum_example/963d8fd5: Test completed in 0.009031 seconds with returncode: 0
metric_regex_with_linenum_example/963d8fd5: Writing output file - /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex_with_linenum/metric_regex_with_linenum_example/963d8fd5/metric_regex_with_linenum_example.out
metric_regex_with_linenum_example/963d8fd5: Writing error file - /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex_with_linenum/metric_regex_with_linenum_example/963d8fd5/metric_regex_with_linenum_example.err
metric_regex_with_linenum_example/963d8fd5: testing metric: second_line if This is line: 2 == This is line: 2 - Check: True
metric_regex_with_linenum_example/963d8fd5: testing metric: without_linenum if This is line: 1 == This is line: 1 - Check: True
metric_regex_with_linenum_example/963d8fd5: eq check: True
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ metric_regex_with_linen │ generic.local.bash │ PASS │ 0 │ 0.009 │
│ um_example/963d8fd5 │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to report file: /tmp/tmpr88132k2/var/report.json
Writing Logfile to /tmp/tmpr88132k2/var/logs/buildtest_tnzstshs.log
buildtest inspect query metric_regex_with_linenum_example
$ buildtest inspect query metric_regex_with_linenum_example
──── metric_regex_with_linenum_example/963d8fd5-3d6d-4b5a-a900-7235d40512db ────
Executor: generic.local.bash
Description: capture result metric from output
State: PASS
Returncode: 0
Runtime: 0.009031 sec
Starttime: 2024/09/05 15:37:57
Endtime: 2024/09/05 15:37:57
Command: bash metric_regex_with_linenum_example_build.sh
Test Script: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex_with_linenum/metric_regex_with_linenum_example/963d8fd5/metric_regex_with_linenum_example.sh
Build Script: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex_with_linenum/metric_regex_with_linenum_example/963d8fd5/metric_regex_with_linenum_example_build.sh
Output File: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex_with_linenum/metric_regex_with_linenum_example/963d8fd5/metric_regex_with_linenum_example.out
Error File: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_regex_with_linenum/metric_regex_with_linenum_example/963d8fd5/metric_regex_with_linenum_example.err
Log File: /tmp/tmpr88132k2/var/logs/buildtest_tnzstshs.log
Metrics
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Name ┃ Value ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ second_line │ This is line: 2 │
│ without_linenum │ This is line: 1 │
└─────────────────┴─────────────────┘
linenum property can also be applied with file regular expression (file_regex). In the example below,
we define 2 metrics with the same regular expression defined via exp. Its worth noting that the regular expression
will attempt to find the first match. When linenum is specified the content from file is extracted by line first
and regular expression is applied afterwards.
buildspecs:
metric_file_regex_with_linenum_example:
executor: generic.local.bash
type: script
description: capture result metric from file path
run: |
echo -e "HPCG result is INVALID with a GFLOP/s rating of=28.1215" > hpcg.txt
echo -e "HPCG result is VALID with a GFLOP/s rating of=68.9888" >> hpcg.txt
tags: tutorials
metrics:
last_line:
type: float
file_regex:
exp: '(\d+\.\d+)$'
linenum: -1
file: hpcg.txt
without_linenum:
type: float
file_regex:
exp: '(\d+\.\d+)$'
file: hpcg.txt
status:
assert_eq:
comparisons:
- name: last_line
ref: 68.9888
- name: without_linenum
ref: 28.1215
Let’s build the test and take a look at the captured metrics.
buildtest build -b tutorials/metrics/metrics_file_regex_with_linenum.yml
$ buildtest build -b tutorials/metrics/metrics_file_regex_with_linenum.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-25534835-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/09/05 15:37:59 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.1 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.9.19 │
│ Configuration File: /tmp/tmpr88132k2/config.yml │
│ Test Directory: /tmp/tmpr88132k2/var/tests │
│ Report File: /tmp/tmpr88132k2/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ ║
║ tutorials/metrics/metrics_file_regex_with_linenum.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/metrics/metrics_file_regex_with_linenum.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ metric_f │ script │ generic. │ None │ None │ None │ capture │ /home/d │
│ ile_rege │ │ local.ba │ │ │ │ result │ ocs/che │
│ x_with_l │ │ sh │ │ │ │ metric │ ckouts/ │
│ inenum_e │ │ │ │ │ │ from │ readthe │
│ xample/2 │ │ │ │ │ │ file │ docs.or │
│ 352a61e │ │ │ │ │ │ path │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/s │
│ │ │ │ │ │ │ │ table/t │
│ │ │ │ │ │ │ │ utorial │
│ │ │ │ │ │ │ │ s/metri │
│ │ │ │ │ │ │ │ cs/metr │
│ │ │ │ │ │ │ │ ics_fil │
│ │ │ │ │ │ │ │ e_regex │
│ │ │ │ │ │ │ │ _with_l │
│ │ │ │ │ │ │ │ inenum. │
│ │ │ │ │ │ │ │ yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
metric_file_regex_with_linenum_example/2352a61e: Creating Test Directory: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_file_regex_with_linenum/metric_file_regex_with_linenum_example/2352a61e
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
metric_file_regex_with_linenum_example/2352a61e does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ metric_file_regex_with_linenum_example/2352a61e │
└─────────────────────────────────────────────────┘
metric_file_regex_with_linenum_example/2352a61e: Current Working Directory : /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_file_regex_with_linenum/metric_file_regex_with_linenum_example/2352a61e/stage
metric_file_regex_with_linenum_example/2352a61e: Running Test via command: bash metric_file_regex_with_linenum_example_build.sh
metric_file_regex_with_linenum_example/2352a61e: Test completed in 0.008989 seconds with returncode: 0
metric_file_regex_with_linenum_example/2352a61e: Writing output file - /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_file_regex_with_linenum/metric_file_regex_with_linenum_example/2352a61e/metric_file_regex_with_linenum_example.out
metric_file_regex_with_linenum_example/2352a61e: Writing error file - /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_file_regex_with_linenum/metric_file_regex_with_linenum_example/2352a61e/metric_file_regex_with_linenum_example.err
metric_file_regex_with_linenum_example/2352a61e: testing metric: last_line if 68.9888 == 68.9888 - Check: True
metric_file_regex_with_linenum_example/2352a61e: testing metric: without_linenum if 28.1215 == 28.1215 - Check: True
metric_file_regex_with_linenum_example/2352a61e: eq check: True
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ metric_file_regex_with_ │ generic.local.bash │ PASS │ 0 │ 0.009 │
│ linenum_example/2352a61 │ │ │ │ │
│ e │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to report file: /tmp/tmpr88132k2/var/report.json
Writing Logfile to /tmp/tmpr88132k2/var/logs/buildtest_ch4y7k13.log
buildtest inspect query metric_file_regex_with_linenum_example
$ buildtest inspect query metric_file_regex_with_linenum_example
─ metric_file_regex_with_linenum_example/2352a61e-6a03-4911-9d7e-d4d19ef148fc ──
Executor: generic.local.bash
Description: capture result metric from file path
State: PASS
Returncode: 0
Runtime: 0.008989 sec
Starttime: 2024/09/05 15:37:59
Endtime: 2024/09/05 15:37:59
Command: bash metric_file_regex_with_linenum_example_build.sh
Test Script: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_file_regex_with_linenum/metric_file_regex_with_linenum_example/2352a61e/metric_file_regex_with_linenum_example.sh
Build Script: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_file_regex_with_linenum/metric_file_regex_with_linenum_example/2352a61e/metric_file_regex_with_linenum_example_build.sh
Output File: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_file_regex_with_linenum/metric_file_regex_with_linenum_example/2352a61e/metric_file_regex_with_linenum_example.out
Error File: /tmp/tmpr88132k2/var/tests/generic.local.bash/metrics_file_regex_with_linenum/metric_file_regex_with_linenum_example/2352a61e/metric_file_regex_with_linenum_example.err
Log File: /tmp/tmpr88132k2/var/logs/buildtest_ch4y7k13.log
Metrics
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Name ┃ Value ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ last_line │ 68.9888 │
│ without_linenum │ 28.1215 │
└─────────────────┴─────────┘
Invalid Metrics
We will discuss a few edge-cases when defining metrics that can lead to validation error. The file_regex and
regex property can’t be declared at the same time when defining a metric. In example below
we have defined a metric named hello that uses both regex and file_regex.
buildspecs:
metrics_with_regex_and_file_regex_not_allowed:
type: script
executor: generic.local.bash
description: This is an invalid metric because 'file_regex' and 'regex' are both specified
run: echo "hello" > file.txt
metrics:
hello:
type: str
file_regex:
file: file.txt
exp: "BAR"
regex:
stream: stdout
exp: "BAR"
If we try to validate this buildspec, we will get an error message that regex and file_regex can’t be specified
at the same time.
buildtest buildspec validate -b tutorials/metrics/invalid_metrics.yml
$ buildtest buildspec validate -b tutorials/metrics/invalid_metrics.yml
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─
{'type': 'str', 'file_regex': {'file': 'file.txt', 'exp': 'BAR'}, 'regex': {'stream': 'stdout', 'exp': 'BAR'}} is valid under each of {'required': ['file_regex']}, {'required': ['regex']}
Failed validating 'oneOf' in schema['properties']['metrics']['additionalProperties']:
{'type': 'object',
'additionalProperties': False,
'required': ['type'],
'properties': {'type': {'type': 'string',
'description': 'Specify python data-type '
'(str, int, float) to convert '
'metric. ',
'enum': ['str', 'int', 'float']},
'regex': {'$ref': '#/definitions/regex'},
'file_regex': {'$ref': '#/definitions/file_regex_in_metrics'}},
'oneOf': [{'required': ['regex']}, {'required': ['file_regex']}]}
On instance['metrics']['hello']:
{'type': 'str',
'file_regex': {'file': 'file.txt', 'exp': 'BAR'},
'regex': {'stream': 'stdout', 'exp': 'BAR'}}
1 buildspecs failed to validate
When defining a metrics, you must specify regex or file_regex property in order to capture metric. If its not
specified, you will run into validation error. In this example, we define a metrics named foo, but we don’t
specify the regex or file_regex property therefore, this metric is invalid.
buildspecs:
required_property_in_metrics:
type: script
executor: generic.local.bash
description: metrics must have a file_regex or regex property to define how to capture metric
run: echo "hello"
metrics:
foo:
type: str
The metrics must follow a pattern, this is typically alphanumeric characters including dot (.), hypen (-)
and underscore (_). In this example below, we have an invalid metric that doesn’t conform to pattern.
buildspecs:
invalid_metrics_name:
type: script
executor: generic.local.bash
description: "Metrics name does not follow pattern"
run: echo "hello" > file.txt
metrics:
(foo-bar:
type: str
regex:
stream: stdout
exp: "BAR"
Let’s try validating the buildspec to see the error message.
buildtest buildspec validate -b tutorials/metrics/invalid_metric_name.yml
$ buildtest buildspec validate -b tutorials/metrics/invalid_metric_name.yml
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─
'(foo-bar' does not match '^[A-Za-z0-9_.-]+$'
Failed validating 'pattern' in schema['properties']['metrics']['propertyNames']:
{'pattern': '^[A-Za-z0-9_.-]+$'}
On instance['metrics']:
'(foo-bar'
1 buildspecs failed to validate