Defining Metrics
buildtest provides a method to define test metrics in the buildspecs which can be used to
store arbitrary content from the output/error file or an arbitrary file into named metric. The
metrics property is used to define a list of metric names using regular expression to assign a value
to the metric. In this example, we have two tests that define metrics hpcg_rate_stream, hpcg_state_stream
in the first test and hpcg_rate_file, hpcg_state_file in the second test. The stream property is used
to read from stdout/stderr and apply the regular expression defined by exp, whereas file_regex is used
to define metrics from an arbitrary file where file is the path to file.
buildspecs:
metric_regex_example:
executor: generic.local.bash
type: script
description: capture result metric from output
run: echo "HPCG result is VALID with a GFLOP/s rating of=63.6515"
tags: tutorials
metrics:
hpcg_rating_stream:
type: float
regex:
exp: '(\d+\.\d+)$'
stream: stdout
hpcg_state_stream:
type: str
regex:
exp: '(VALID)'
stream: stdout
metric_file_regex:
executor: generic.local.bash
type: script
description: capture result metric from file path
run: echo "HPCG result is VALID with a GFLOP/s rating of=63.6515" > hpcg.txt
tags: tutorials
metrics:
hpcg_rating_file:
type: float
file_regex:
exp: '(\d+\.\d+)$'
file: hpcg.txt
hpcg_state_file:
type: str
file_regex:
exp: '(VALID)'
file: hpcg.txt
The metrics can be used with Comparison Operators for performing more sophisticated status checks.
By default, a metric will be an empty dictionary if there is no metrics property. If we fail to match
a regular expression, the metric will be defined as an empty string ('').
Note
If your regular expression contains an escape character \ you must surround your
string in single quotes ' as pose to double quotes "
Let’s build this test.
buildtest build -b tutorials/metrics/metrics_regex.yml
$ buildtest build -b tutorials/metrics/metrics_regex.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:44:48 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/metrics/metrics_regex.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/metrics/metrics_regex.yml: VALID
Total builder objects created: 2
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ metric_r │ script │ generic. │ None │ None │ None │ capture │ /home/d │
│ egex_exa │ │ local.ba │ │ │ │ result │ ocs/che │
│ mple/314 │ │ sh │ │ │ │ metric │ ckouts/ │
│ 44365 │ │ │ │ │ │ from │ readthe │
│ │ │ │ │ │ │ output │ docs.or │
│ │ │ │ │ │ │ │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /metric │
│ │ │ │ │ │ │ │ s/metri │
│ │ │ │ │ │ │ │ cs_rege │
│ │ │ │ │ │ │ │ x.yml │
├──────────┼────────┼──────────┼──────────┼───────┼───────┼──────────┼─────────┤
│ metric_f │ script │ generic. │ None │ None │ None │ capture │ /home/d │
│ ile_rege │ │ local.ba │ │ │ │ result │ ocs/che │
│ x/9cf094 │ │ sh │ │ │ │ metric │ ckouts/ │
│ 1f │ │ │ │ │ │ from │ readthe │
│ │ │ │ │ │ │ file │ docs.or │
│ │ │ │ │ │ │ path │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /metric │
│ │ │ │ │ │ │ │ s/metri │
│ │ │ │ │ │ │ │ cs_rege │
│ │ │ │ │ │ │ │ x.yml │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
metric_regex_example/31444365: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_regex/metric_regex_example/31444365
metric_file_regex/9cf0941f: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_regex/metric_file_regex/9cf0941f
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
metric_regex_example/31444365 does not have any dependencies adding test to queue
metric_file_regex/9cf0941f does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ metric_regex_example/31444365 │
│ metric_file_regex/9cf0941f │
└───────────────────────────────┘
metric_regex_example/31444365: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_regex/metric_regex_example/31444365/stage
metric_regex_example/31444365: Running Test via command: bash metric_regex_example_build.sh
metric_regex_example/31444365: Test completed in 0.006204 seconds with returncode: 0
metric_regex_example/31444365: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_regex/metric_regex_example/31444365/metric_regex_example.out
metric_regex_example/31444365: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_regex/metric_regex_example/31444365/metric_regex_example.err
metric_file_regex/9cf0941f: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_regex/metric_file_regex/9cf0941f/stage
metric_file_regex/9cf0941f: Running Test via command: bash metric_file_regex_build.sh
metric_file_regex/9cf0941f: Test completed in 0.005759 seconds with returncode: 0
metric_file_regex/9cf0941f: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_regex/metric_file_regex/9cf0941f/metric_file_regex.out
metric_file_regex/9cf0941f: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_regex/metric_file_regex/9cf0941f/metric_file_regex.err
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ metric_regex_example/31 │ generic.local.bash │ PASS │ 0 │ 0.006 │
│ 444365 │ │ │ │ │
├─────────────────────────┼────────────────────┼────────┼────────────┼─────────┤
│ metric_file_regex/9cf09 │ generic.local.bash │ PASS │ 0 │ 0.006 │
│ 41f │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%
Adding 2 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_4orp7ihk.log
The metrics are captured in the test report which can
be queried via buildtest report or buildtest inspect query. Metrics can be seen in the test metadata,
for instance you can run buildtest inspect query and you will see metrics shown in table output.
buildtest inspect query metric_regex_example metric_file_regex
$ buildtest inspect query metric_regex_example metric_file_regex
──────────── metric_file_regex/9cf0941f-ea22-4b7e-9969-fa3da5b8b6c0 ────────────
Executor: generic.local.bash
Description: capture result metric from file path
State: PASS
Returncode: 0
Runtime: 0.005759 sec
Starttime: 2024/04/25 15:44:48
Endtime: 2024/04/25 15:44:48
Command: bash metric_file_regex_build.sh
Test Script: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_regex/metric_file_regex/9cf0941f/metric_file_regex.sh
Build Script: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_regex/metric_file_regex/9cf0941f/metric_file_regex_build.sh
Output File: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_regex/metric_file_regex/9cf0941f/metric_file_regex.out
Error File: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_regex/metric_file_regex/9cf0941f/metric_file_regex.err
Log File: /tmp/tmp0ns2hqkt/var/logs/buildtest_4orp7ihk.log
Metrics
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Name ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ hpcg_rating_file │ 63.6515 │
│ hpcg_state_file │ VALID │
└──────────────────┴─────────┘
────────── metric_regex_example/31444365-c94c-49e8-b05e-052c7215de61 ───────────
Executor: generic.local.bash
Description: capture result metric from output
State: PASS
Returncode: 0
Runtime: 0.006204 sec
Starttime: 2024/04/25 15:44:48
Endtime: 2024/04/25 15:44:48
Command: bash metric_regex_example_build.sh
Test Script: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_regex/metric_regex_example/31444365/metric_regex_example.sh
Build Script: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_regex/metric_regex_example/31444365/metric_regex_example_build.sh
Output File: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_regex/metric_regex_example/31444365/metric_regex_example.out
Error File: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_regex/metric_regex_example/31444365/metric_regex_example.err
Log File: /tmp/tmp0ns2hqkt/var/logs/buildtest_4orp7ihk.log
Metrics
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Name ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ hpcg_rating_stream │ 63.6515 │
│ hpcg_state_stream │ VALID │
└────────────────────┴─────────┘
We can query the metrics via buildtest report which will display all metrics as a comma seperated
Key/Value pair. We can use buildtest report --format metrics to extract all metrics for a test.
Internally, we store the metrics as a dictionary but when we print them out via buildtest report we
join them together into a single string. Shown below is the metrics for the previous build.
buildtest report --filter buildspec=tutorials/metrics/metrics_regex.yml --format name,metrics
$ buildtest report --filter buildspec=tutorials/metrics/metrics_regex.yml --format name,metrics
Report File: /tmp/tmp0ns2hqkt/var/report.json
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name ┃ metrics ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ metric_regex_example │ hpcg_rating_stream=63.6515,hpcg_state_stream=VALID │
├───────────────────────┼──────────────────────────────────────────────────────┤
│ metric_file_regex │ hpcg_rating_file=63.6515,hpcg_state_file=VALID │
└───────────────────────┴──────────────────────────────────────────────────────┘
Metrics with Regex Type via ‘re’
Building on the previous example, we will use the re property specify the regular expression type to use. By default, buildtest will
use re.search if re is not specified; however you can specify re to use re.match,
re.fullmatch, or re.search.
In this example, we will define 4 metrics hpcg_text, hpcg_result, hpcg_file_text, hpcg_file_result. The first two
metrics will capture from stdout using the regex property while the last two will capture from a file using the file_regex property.
The re.match will be used to capture the text HPCG result is VALID and HPCG result is INVALID from stdout and file, whereas
the re.search will be used to capture the test result 63.6515 and 28.1215 from stdout and file.
Finally, we will use the comparison operator assert_eq: Equal to compare the metrics with reference value.
buildspecs:
metric_regex_example_with_re:
executor: generic.local.bash
type: script
description: capture metric with different regex types
tags: tutorials
run: |
echo "HPCG result is VALID with a GFLOP/s rating of=63.6515"
echo "HPCG result is INVALID with a GFLOP/s rating of=28.1215" > hpcg.txt
metrics:
hpcg_result:
type: float
regex:
re: "re.search"
exp: '(\d+\.\d+)$'
stream: stdout
hpcg_text:
type: str
regex:
re: "re.match"
exp: '^HPCG result is VALID'
stream: stdout
hpcg_file_text:
type: str
file_regex:
re: "re.match"
exp: '^HPCG result is INVALID'
file: hpcg.txt
hpcg_file_result:
type: float
file_regex:
re: "re.search"
exp: '(\d+\.\d+)$'
file: hpcg.txt
status:
assert_eq:
comparisons:
- name: hpcg_text
ref: "HPCG result is VALID"
- name: hpcg_result
ref: 63.6515
- name: hpcg_file_text
ref: "HPCG result is INVALID"
- name: hpcg_file_result
ref: 28.1215
Let’s attempt to build this test
buildtest build -b tutorials/metrics/metrics_with_regex_type.yml
$ buildtest build -b tutorials/metrics/metrics_with_regex_type.yml
╭───────────────────────────── buildtest summary ──────────────────────────────╮
│ │
│ User: docs │
│ Hostname: build-24186293-project-280831-buildtest │
│ Platform: Linux │
│ Current Time: 2024/04/25 15:44:50 │
│ buildtest path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ buildtest version: 2.0 │
│ python path: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ python version: 3.8.18 │
│ Configuration File: /tmp/tmp0ns2hqkt/config.yml │
│ Test Directory: /tmp/tmp0ns2hqkt/var/tests │
│ Report File: /tmp/tmp0ns2hqkt/var/report.json │
│ Command: /home/docs/checkouts/readthedocs.org/user_builds/buildte │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
─────────────────────────── Discovering Buildspecs ────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/t ║
║ utorials/metrics/metrics_with_regex_type.yml ║
╟──────────────────────────────────────────────────────────────────────────────╢
║ Total: 1 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────── Parsing Buildspecs ──────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/tutorials/metrics/metrics_with_regex_type.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ ┃ descript ┃ buildsp ┃
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ ion ┃ ecs ┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ metric_r │ script │ generic. │ None │ None │ None │ capture │ /home/d │
│ egex_exa │ │ local.ba │ │ │ │ metric │ ocs/che │
│ mple_wit │ │ sh │ │ │ │ with │ ckouts/ │
│ h_re/2df │ │ │ │ │ │ differen │ readthe │
│ f422f │ │ │ │ │ │ t regex │ docs.or │
│ │ │ │ │ │ │ types │ g/user_ │
│ │ │ │ │ │ │ │ builds/ │
│ │ │ │ │ │ │ │ buildte │
│ │ │ │ │ │ │ │ st/chec │
│ │ │ │ │ │ │ │ kouts/d │
│ │ │ │ │ │ │ │ evel/tu │
│ │ │ │ │ │ │ │ torials │
│ │ │ │ │ │ │ │ /metric │
│ │ │ │ │ │ │ │ s/metri │
│ │ │ │ │ │ │ │ cs_with │
│ │ │ │ │ │ │ │ _regex_ │
│ │ │ │ │ │ │ │ type.ym │
│ │ │ │ │ │ │ │ l │
└──────────┴────────┴──────────┴──────────┴───────┴───────┴──────────┴─────────┘
──────────────────────────────── Building Test ─────────────────────────────────
metric_regex_example_with_re/2dff422f: Creating Test Directory: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_with_regex_type/metric_regex_example_with_re/2dff422f
──────────────────────────────── Running Tests ─────────────────────────────────
Spawning 1 processes for processing builders
───────────────────────────────── Iteration 1 ──────────────────────────────────
metric_regex_example_with_re/2dff422f does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ metric_regex_example_with_re/2dff422f │
└───────────────────────────────────────┘
metric_regex_example_with_re/2dff422f: Current Working Directory : /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_with_regex_type/metric_regex_example_with_re/2dff422f/stage
metric_regex_example_with_re/2dff422f: Running Test via command: bash metric_regex_example_with_re_build.sh
metric_regex_example_with_re/2dff422f: Test completed in 0.005996 seconds with returncode: 0
metric_regex_example_with_re/2dff422f: Writing output file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_with_regex_type/metric_regex_example_with_re/2dff422f/metric_regex_example_with_re.out
metric_regex_example_with_re/2dff422f: Writing error file - /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_with_regex_type/metric_regex_example_with_re/2dff422f/metric_regex_example_with_re.err
metric_regex_example_with_re/2dff422f: testing metric: hpcg_text if HPCG result is VALID == HPCG result is VALID - Check: True
metric_regex_example_with_re/2dff422f: testing metric: hpcg_result if 63.6515 == 63.6515 - Check: True
metric_regex_example_with_re/2dff422f: testing metric: hpcg_file_text if HPCG result is INVALID == HPCG result is INVALID - Check: True
metric_regex_example_with_re/2dff422f: testing metric: hpcg_file_result if 28.1215 == 28.1215 - Check: True
metric_regex_example_with_re/2dff422f: eq check: True
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ metric_regex_example_wi │ generic.local.bash │ PASS │ 0 │ 0.006 │
│ th_re/2dff422f │ │ │ │ │
└─────────────────────────┴────────────────────┴────────┴────────────┴─────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to report file: /tmp/tmp0ns2hqkt/var/report.json
Writing Logfile to /tmp/tmp0ns2hqkt/var/logs/buildtest_pwnbgdmv.log
Upon completion, lets take a look at the metrics for this test, we can see this by running buildtest inspect query
which shows the name of captured metrics and its corresponding values.
buildtest inspect query metric_regex_example_with_re
$ buildtest inspect query metric_regex_example_with_re
────── metric_regex_example_with_re/2dff422f-754b-409d-87be-00bae0b5eb9b ───────
Executor: generic.local.bash
Description: capture metric with different regex types
State: PASS
Returncode: 0
Runtime: 0.005996 sec
Starttime: 2024/04/25 15:44:50
Endtime: 2024/04/25 15:44:50
Command: bash metric_regex_example_with_re_build.sh
Test Script: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_with_regex_type/metric_regex_example_with_re/2dff422f/metric_regex_example_with_re.sh
Build Script: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_with_regex_type/metric_regex_example_with_re/2dff422f/metric_regex_example_with_re_build.sh
Output File: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_with_regex_type/metric_regex_example_with_re/2dff422f/metric_regex_example_with_re.out
Error File: /tmp/tmp0ns2hqkt/var/tests/generic.local.bash/metrics_with_regex_type/metric_regex_example_with_re/2dff422f/metric_regex_example_with_re.err
Log File: /tmp/tmp0ns2hqkt/var/logs/buildtest_pwnbgdmv.log
Metrics
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩
│ hpcg_result │ 63.6515 │
│ hpcg_text │ HPCG result is VALID │
│ hpcg_file_text │ HPCG result is INVALID │
│ hpcg_file_result │ 28.1215 │
└──────────────────┴────────────────────────┘
Invalid Metrics
We will discuss a few edge-cases when defining metrics that can lead to validation error. The file_regex and
regex property can’t be declared at the same time when defining a metric. In example below
we have defined a metric named hello that uses both regex and file_regex.
buildspecs:
metrics_with_regex_and_file_regex_not_allowed:
type: script
executor: generic.local.bash
description: This is an invalid metric because 'file_regex' and 'regex' are both specified
run: echo "hello" > file.txt
metrics:
hello:
type: str
file_regex:
file: file.txt
exp: "BAR"
regex:
stream: stdout
exp: "BAR"
If we try to validate this buildspec, we will get an error message that regex and file_regex can’t be specified
at the same time.
buildtest buildspec validate -b tutorials/metrics/invalid_metrics.yml
$ buildtest buildspec validate -b tutorials/metrics/invalid_metrics.yml
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… ─
{'type': 'str', 'file_regex': {'file': 'file.txt', 'exp': 'BAR'}, 'regex': {'stream': 'stdout', 'exp': 'BAR'}} is valid under each of {'required': ['file_regex']}, {'required': ['regex']}
Failed validating 'oneOf' in schema['properties']['metrics']['additionalProperties']:
{'additionalProperties': False,
'oneOf': [{'required': ['regex']}, {'required': ['file_regex']}],
'properties': {'file_regex': {'$ref': '#/definitions/file_regex_in_metrics'},
'regex': {'$ref': '#/definitions/regex'},
'type': {'description': 'Specify python data-type '
'(str, int, float) to convert '
'metric. ',
'enum': ['str', 'int', 'float'],
'type': 'string'}},
'required': ['type'],
'type': 'object'}
On instance['metrics']['hello']:
{'file_regex': {'exp': 'BAR', 'file': 'file.txt'},
'regex': {'exp': 'BAR', 'stream': 'stdout'},
'type': 'str'}
1 buildspecs failed to validate
When defining a metrics, you must specify regex or file_regex property in order to capture metric. If its not
specified, you will run into validation error. In this example, we define a metrics named foo, but we don’t
specify the regex or file_regex property therefore, this metric is invalid.
buildspecs:
required_property_in_metrics:
type: script
executor: generic.local.bash
description: metrics must have a file_regex or regex property to define how to capture metric
run: echo "hello"
metrics:
foo:
type: str
The metrics must follow a pattern, this is typically alphanumeric characters including dot (.), hypen (-)
and underscore (_). In this example below, we have an invalid metric that doesn’t conform to pattern.
buildspecs:
invalid_metrics_name:
type: script
executor: generic.local.bash
description: "Metrics name does not follow pattern"
run: echo "hello" > file.txt
metrics:
(foo-bar:
type: str
regex:
stream: stdout
exp: "BAR"
Let’s try validating the buildspec to see the error message.
buildtest buildspec validate -b tutorials/metrics/invalid_metric_name.yml
$ buildtest buildspec validate -b tutorials/metrics/invalid_metric_name.yml
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/devel/… ─
'(foo-bar' does not match '^[A-Za-z0-9_.-]+$'
Failed validating 'pattern' in schema['properties']['metrics']['propertyNames']:
{'pattern': '^[A-Za-z0-9_.-]+$'}
On instance['metrics']:
'(foo-bar'
1 buildspecs failed to validate