Note
Please see Tutorials Setup before you proceed with this section
Using Compiler Schema
The compiler schema is used for compilation of programs, currently we support
single source file compilation. In order to use the compiler schema you must set type: compiler in your
test. For more details see compiler schema docs
We assume the reader has basic understanding of Global Schema validation. Shown below is the schema header definition for compiler.schema.json:
{
"$id": "compiler.schema.json",
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "compiler schema",
"description": "The compiler schema is of ``type: compiler`` in sub-schema which is used for compiling and running programs",
"type": "object",
"required": [
"type",
"source",
"compilers",
"executor"
],
The required fields for compiler schema are type, compilers, source and executor.
Compilation of Hello World
We will start out with compilation a Hello World program in Fortran using the GNU compiler.
In this example we have a test called hello_f. The type: compiler is set to signify this
test will be validated with compiler.schema.json.
The source property is used to specify the source code to compile, this can be a
relative path to buildspec file or an absolute path.
In this example the source file src/hello.f90 is relative path to where buildspec file is located.
The compilers section declares compiler configuration, the name
property is required that is used to search compiler names from our buildtest configuration via regular
expression. In this example we use the gcc_7.5.0 compiler specified in the regular expression. The
default section specifies default compiler configuration applicable to a specific compiler group like gcc.
Within each compiler group we can specify options like cflags, fflags, cxxflags, ldflags to customize compilation line.
buildspecs:
hello_f:
type: compiler
description: "Hello World Fortran Compilation"
executor: generic.local.bash
tags: [tutorials, compile]
source: "src/hello.f90"
compilers:
name: ["^(gcc_7.5.0)$"]
default:
gcc:
fflags: -Wall
Shown below is an example build for this test.
buildtest build -b /home/spack/buildtest/examples/compilers/gnu_hello_fortran.yml
$ buildtest build -b /home/spack/buildtest/examples/compilers/gnu_hello_fortran.yml
╭─────────────────────────────────────────────────────── buildtest summary ───────────────────────────────────────────────────────╮
│ │
│ User: spack │
│ Hostname: 651e70126b2d │
│ Platform: Linux │
│ Current Time: 2023/11/09 19:34:52 │
│ buildtest path: /home/spack/buildtest/bin/buildtest │
│ buildtest version: 1.6 │
│ python path: /home/spack/pyenv/buildtest/bin/python3 │
│ python version: 3.8.6 │
│ Configuration File: /home/spack/buildtest/buildtest/settings/spack_container.yml │
│ Test Directory: /home/spack/runs │
│ Report File: /home/spack/buildtest/var/report.json │
│ Command: /home/spack/buildtest/bin/buildtest build -b /home/spack/buildtest/examples/compilers/gnu_hello_fortran.yml │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
─────────────────────────────────────────────────────────────────────────────────────────────────────── Discovering Buildspecs ────────────────────────────────────────────────────────────────────────────────────────────────────────
Discovered buildspecs
╔════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟────────────────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/compilers/gnu_hello_fortran.yml ║
╚════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ──────────────────────────────────────────────────────────────────────────────────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/compilers/gnu_hello_fortran.yml: VALID
Total builder objects created: 1
Builders by type=compiler
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ description ┃ buildspecs ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ hello_f/d8438739 │ compiler │ generic.local.bash │ gcc_7.5.0 │ None │ None │ Hello World Fortran Compilation │ /home/spack/buildtest/examples/compilers/gnu_hello_fortran.yml │
└──────────────────┴──────────┴────────────────────┴───────────┴───────┴───────┴─────────────────────────────────┴────────────────────────────────────────────────────────────────┘
──────────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
hello_f/d8438739: Creating test directory: /home/spack/runs/generic.local.bash/gnu_hello_fortran/hello_f/d8438739
hello_f/d8438739: Creating the stage directory: /home/spack/runs/generic.local.bash/gnu_hello_fortran/hello_f/d8438739/stage
hello_f/d8438739: Writing build script: /home/spack/runs/generic.local.bash/gnu_hello_fortran/hello_f/d8438739/hello_f_build.sh
──────────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
Spawning 4 processes for processing builders
───────────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
hello_f/d8438739 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━┩
│ hello_f/d8438739 │
└──────────────────┘
hello_f/d8438739: Current Working Directory : /home/spack/runs/generic.local.bash/gnu_hello_fortran/hello_f/d8438739/stage
hello_f/d8438739: Running Test via command: bash --norc --noprofile -eo pipefail hello_f_build.sh
hello_f/d8438739: Test completed in 4.905627 seconds
hello_f/d8438739: Test completed with returncode: 0
hello_f/d8438739: Writing output file - /home/spack/runs/generic.local.bash/gnu_hello_fortran/hello_f/d8438739/hello_f.out
hello_f/d8438739: Writing error file - /home/spack/runs/generic.local.bash/gnu_hello_fortran/hello_f/d8438739/hello_f.err
Test Summary
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ hello_f/d8438739 │ generic.local.bash │ PASS │ None None None │ 0 │ 4.905627 │
└──────────────────┴────────────────────┴────────┴─────────────────────────────────────┴────────────┴──────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to /home/spack/buildtest/var/report.json
Writing Logfile to: /home/spack/buildtest/var/logs/buildtest_9xiwai6f.log
The generated test for test name hello_f is the following:
buildtest inspect query -t hello_f
$ buildtest inspect query -t hello_f
───────────────────────────────────────────────────────────────────────────────────────────── hello_f/d8438739-5c79-4b4b-ab26-f86b849af7e8 ─────────────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: Hello World Fortran Compilation
State: PASS
Returncode: 0
Runtime: 4.905627 sec
Starttime: 2023/11/09 19:34:52
Endtime: 2023/11/09 19:34:57
Command: bash --norc --noprofile -eo pipefail hello_f_build.sh
Test Script: /home/spack/runs/generic.local.bash/gnu_hello_fortran/hello_f/d8438739/hello_f.sh
Build Script: /home/spack/runs/generic.local.bash/gnu_hello_fortran/hello_f/d8438739/hello_f_build.sh
Output File: /home/spack/runs/generic.local.bash/gnu_hello_fortran/hello_f/d8438739/hello_f.out
Error File: /home/spack/runs/generic.local.bash/gnu_hello_fortran/hello_f/d8438739/hello_f.err
Log File: /home/spack/buildtest/var/logs/buildtest_9xiwai6f.log
───────────────────────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/gnu_hello_fortran/hello_f/d8438739/hello_f.sh ─────────────────────────────────────────────────────────────────────
#!/bin/bash
set -eo pipefail
# name of executable
_EXEC=hello.f90.exe
# Compilation Line
/usr/bin/gfortran -Wall -o $_EXEC /home/spack/buildtest/examples/compilers/src/hello.f90
# Run executable
./$_EXEC
How does buildtest detect programming language?
buildtest will detect the file extension of source file based specified by source property
to detect programming language and generate the appropriate C, C++, or Fortran compilation line based
on language detected.
Shown below is the file extension table buildtest uses for determining the programming language.
Language |
File Extension |
|---|---|
C |
.c |
C++ |
.cc .cxx .cpp .c++ |
Fortran |
.f90 .F90 .f95 .f .F .FOR .for .FTN .ftn |
How does buildtest search for compiler
Compilers are defined in your configuration file that is used by buildtest to search compilers.
In configuring compilers you will learn how one can define compiler, each compiler
instance will have a unique name and be under one of the compiler groups like gcc, intel, cray, etc…
You can see the compiler declaration from our configuration file by running buildtest config compilers -y
which will display compiler settings in YAML format. Note that for each compiler instance one can define name of compiler
and path to cc, fc, cxx wrapper. In addition one can specify module property to map compiler instance
to modulefile. If module property is defined you can specify list of modules to load via load property and buildtest will
automatically load these modules when using the compiler.
buildtest config compilers list -y
$ buildtest config compilers list -y
gcc:
clang_6.0.0:
cc: /usr/bin/clang
cxx: /usr/bin/clang++
fc: None
gcc_6.5.0:
cc: gcc
cxx: g++
fc: gfortran
module:
load:
- gcc/6.5.0-gcc-7.5.0
gcc_7.5.0:
cc: /usr/bin/gcc
cxx: /usr/bin/g++
fc: /usr/bin/gfortran
gcc_8.3.0:
cc: gcc
cxx: g++
fc: gfortran
module:
load:
- gcc/8.3.0-gcc-7.5.0
buildtest selects compiler based on name property which is a list of regular expression
applied for available compilers defined in buildtest configuration. In this next example, we will
compile an OpenACC code that will compute vector addition. We specify all gcc compilers
are used for building this test. The -fopenacc compiler flag enables GNU compilers to compile OpenACC code,
we can set this via fflags property. We can specify linker flags via ldflags during compilation,
this code requires we specify -lm flag to link with math library.
buildspecs:
vecadd_gnu:
type: compiler
description: Vector Addition example with GNU compiler
tags: [tutorials, compile]
executor: generic.local.bash
source: src/vecAdd.c
compilers:
name: ["^(gcc)"]
default:
gcc:
cflags: -fopenacc
ldflags: -lm
We expect buildtest to generate one test per gcc compiler as you can see below.
buildtest build -b /home/spack/buildtest/examples/compilers/vecadd.yml
$ buildtest build -b /home/spack/buildtest/examples/compilers/vecadd.yml
╭───────────────────────────────────────────────── buildtest summary ──────────────────────────────────────────────────╮
│ │
│ User: spack │
│ Hostname: 651e70126b2d │
│ Platform: Linux │
│ Current Time: 2023/11/09 19:35:00 │
│ buildtest path: /home/spack/buildtest/bin/buildtest │
│ buildtest version: 1.6 │
│ python path: /home/spack/pyenv/buildtest/bin/python3 │
│ python version: 3.8.6 │
│ Configuration File: /home/spack/buildtest/buildtest/settings/spack_container.yml │
│ Test Directory: /home/spack/runs │
│ Report File: /home/spack/buildtest/var/report.json │
│ Command: /home/spack/buildtest/bin/buildtest build -b /home/spack/buildtest/examples/compilers/vecadd.yml │
│ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
─────────────────────────────────────────────────────────────────────────────────────────────────────── Discovering Buildspecs ────────────────────────────────────────────────────────────────────────────────────────────────────────
Discovered buildspecs
╔═════════════════════════════════════════════════════╗
║ buildspec ║
╟─────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/compilers/vecadd.yml ║
╚═════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ──────────────────────────────────────────────────────────────────────────────────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/compilers/vecadd.yml: VALID
Total builder objects created: 3
Builders by type=compiler
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ description ┃ buildspecs ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ vecadd_gnu/25714aca │ compiler │ generic.local.bash │ gcc_7.5.0 │ None │ None │ Vector Addition example with GNU compiler │ /home/spack/buildtest/examples/compilers/vecadd.yml │
├─────────────────────┼──────────┼────────────────────┼───────────┼───────┼───────┼───────────────────────────────────────────┼─────────────────────────────────────────────────────┤
│ vecadd_gnu/30b30668 │ compiler │ generic.local.bash │ gcc_6.5.0 │ None │ None │ Vector Addition example with GNU compiler │ /home/spack/buildtest/examples/compilers/vecadd.yml │
├─────────────────────┼──────────┼────────────────────┼───────────┼───────┼───────┼───────────────────────────────────────────┼─────────────────────────────────────────────────────┤
│ vecadd_gnu/6d2fde9b │ compiler │ generic.local.bash │ gcc_8.3.0 │ None │ None │ Vector Addition example with GNU compiler │ /home/spack/buildtest/examples/compilers/vecadd.yml │
└─────────────────────┴──────────┴────────────────────┴───────────┴───────┴───────┴───────────────────────────────────────────┴─────────────────────────────────────────────────────┘
──────────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
vecadd_gnu/25714aca: Creating test directory: /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/25714aca
vecadd_gnu/25714aca: Creating the stage directory: /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/25714aca/stage
vecadd_gnu/25714aca: Writing build script: /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/25714aca/vecadd_gnu_build.sh
vecadd_gnu/30b30668: Creating test directory: /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/30b30668
vecadd_gnu/30b30668: Creating the stage directory: /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/30b30668/stage
vecadd_gnu/30b30668: Writing build script: /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/30b30668/vecadd_gnu_build.sh
vecadd_gnu/6d2fde9b: Creating test directory: /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/6d2fde9b
vecadd_gnu/6d2fde9b: Creating the stage directory: /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/6d2fde9b/stage
vecadd_gnu/6d2fde9b: Writing build script: /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/6d2fde9b/vecadd_gnu_build.sh
──────────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
Spawning 4 processes for processing builders
───────────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
vecadd_gnu/25714aca does not have any dependencies adding test to queue
vecadd_gnu/6d2fde9b does not have any dependencies adding test to queue
vecadd_gnu/30b30668 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━┩
│ vecadd_gnu/30b30668 │
│ vecadd_gnu/25714aca │
│ vecadd_gnu/6d2fde9b │
└─────────────────────┘
vecadd_gnu/30b30668: Current Working Directory : /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/30b30668/stage
vecadd_gnu/25714aca: Current Working Directory : /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/25714aca/stage
vecadd_gnu/6d2fde9b: Current Working Directory : /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/6d2fde9b/stage
vecadd_gnu/30b30668: Running Test via command: bash --norc --noprofile -eo pipefail vecadd_gnu_build.sh
vecadd_gnu/25714aca: Running Test via command: bash --norc --noprofile -eo pipefail vecadd_gnu_build.sh
vecadd_gnu/6d2fde9b: Running Test via command: bash --norc --noprofile -eo pipefail vecadd_gnu_build.sh
vecadd_gnu/25714aca: Test completed in 4.606659 seconds
vecadd_gnu/25714aca: Test completed with returncode: 0
vecadd_gnu/25714aca: Writing output file - /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/25714aca/vecadd_gnu.out
vecadd_gnu/25714aca: Writing error file - /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/25714aca/vecadd_gnu.err
vecadd_gnu/6d2fde9b: Test completed in 4.688952 seconds
vecadd_gnu/6d2fde9b: Test completed with returncode: 0
vecadd_gnu/6d2fde9b: Writing output file - /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/6d2fde9b/vecadd_gnu.out
vecadd_gnu/6d2fde9b: Writing error file - /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/6d2fde9b/vecadd_gnu.err
vecadd_gnu/30b30668: Test completed in 4.804224 seconds
vecadd_gnu/30b30668: Test completed with returncode: 0
vecadd_gnu/30b30668: Writing output file - /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/30b30668/vecadd_gnu.out
vecadd_gnu/30b30668: Writing error file - /home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/30b30668/vecadd_gnu.err
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ vecadd_gnu/6d2fde9b │ generic.local.bash │ PASS │ None None None │ 0 │ 4.688952 │
├─────────────────────┼────────────────────┼────────┼─────────────────────────────────────┼────────────┼──────────┤
│ vecadd_gnu/25714aca │ generic.local.bash │ PASS │ None None None │ 0 │ 4.606659 │
├─────────────────────┼────────────────────┼────────┼─────────────────────────────────────┼────────────┼──────────┤
│ vecadd_gnu/30b30668 │ generic.local.bash │ PASS │ None None None │ 0 │ 4.804224 │
└─────────────────────┴────────────────────┴────────┴─────────────────────────────────────┴────────────┴──────────┘
Passed Tests: 3/3 Percentage: 100.000%
Failed Tests: 0/3 Percentage: 0.000%
Adding 3 test results to /home/spack/buildtest/var/report.json
Writing Logfile to: /home/spack/buildtest/var/logs/buildtest_hi5qmt8i.log
Customize Compiler Option
We can specify custom compiler options per compiler instance using the config property.
In this next example, we have a Hello World C example that will specify different cflags
based on compiler name. We can specify default compiler setting via default property. In this
example the default is -O1. The config section can be used to specify custom compiler
options for each compiler name.
buildspecs:
hello_c:
type: compiler
description: "Hello World C Compilation"
executor: generic.local.bash
tags: [tutorials, compile]
source: "src/hello.c"
compilers:
name: ["^(gcc)"]
default:
gcc:
cflags: -O1
config:
gcc_6.5.0:
cflags: -O2
gcc_8.3.0:
cflags: -O3
Let’s build this test, we will see there is one builder instance for each compiler.
buildtest build -b /home/spack/buildtest/examples/compilers/gnu_hello_c.yml
$ buildtest build -b /home/spack/buildtest/examples/compilers/gnu_hello_c.yml
╭──────────────────────────────────────────────────── buildtest summary ────────────────────────────────────────────────────╮
│ │
│ User: spack │
│ Hostname: 651e70126b2d │
│ Platform: Linux │
│ Current Time: 2023/11/09 19:35:06 │
│ buildtest path: /home/spack/buildtest/bin/buildtest │
│ buildtest version: 1.6 │
│ python path: /home/spack/pyenv/buildtest/bin/python3 │
│ python version: 3.8.6 │
│ Configuration File: /home/spack/buildtest/buildtest/settings/spack_container.yml │
│ Test Directory: /home/spack/runs │
│ Report File: /home/spack/buildtest/var/report.json │
│ Command: /home/spack/buildtest/bin/buildtest build -b /home/spack/buildtest/examples/compilers/gnu_hello_c.yml │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
─────────────────────────────────────────────────────────────────────────────────────────────────────── Discovering Buildspecs ────────────────────────────────────────────────────────────────────────────────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/compilers/gnu_hello_c.yml ║
╚══════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ──────────────────────────────────────────────────────────────────────────────────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/compilers/gnu_hello_c.yml: VALID
Total builder objects created: 3
Builders by type=compiler
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ description ┃ buildspecs ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ hello_c/eab51033 │ compiler │ generic.local.bash │ gcc_7.5.0 │ None │ None │ Hello World C Compilation │ /home/spack/buildtest/examples/compilers/gnu_hello_c.yml │
├──────────────────┼──────────┼────────────────────┼───────────┼───────┼───────┼───────────────────────────┼──────────────────────────────────────────────────────────┤
│ hello_c/607e8431 │ compiler │ generic.local.bash │ gcc_6.5.0 │ None │ None │ Hello World C Compilation │ /home/spack/buildtest/examples/compilers/gnu_hello_c.yml │
├──────────────────┼──────────┼────────────────────┼───────────┼───────┼───────┼───────────────────────────┼──────────────────────────────────────────────────────────┤
│ hello_c/d674237a │ compiler │ generic.local.bash │ gcc_8.3.0 │ None │ None │ Hello World C Compilation │ /home/spack/buildtest/examples/compilers/gnu_hello_c.yml │
└──────────────────┴──────────┴────────────────────┴───────────┴───────┴───────┴───────────────────────────┴──────────────────────────────────────────────────────────┘
──────────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
hello_c/eab51033: Creating test directory: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/eab51033
hello_c/eab51033: Creating the stage directory: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/eab51033/stage
hello_c/eab51033: Writing build script: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/eab51033/hello_c_build.sh
hello_c/607e8431: Creating test directory: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/607e8431
hello_c/607e8431: Creating the stage directory: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/607e8431/stage
hello_c/607e8431: Writing build script: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/607e8431/hello_c_build.sh
hello_c/d674237a: Creating test directory: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/d674237a
hello_c/d674237a: Creating the stage directory: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/d674237a/stage
hello_c/d674237a: Writing build script: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/d674237a/hello_c_build.sh
──────────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
Spawning 4 processes for processing builders
───────────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
hello_c/d674237a does not have any dependencies adding test to queue
hello_c/607e8431 does not have any dependencies adding test to queue
hello_c/eab51033 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━┩
│ hello_c/eab51033 │
│ hello_c/607e8431 │
│ hello_c/d674237a │
└──────────────────┘
hello_c/eab51033: Current Working Directory : /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/eab51033/stage
hello_c/607e8431: Current Working Directory : /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/607e8431/stage
hello_c/eab51033: Running Test via command: bash --norc --noprofile -eo pipefail hello_c_build.sh
hello_c/d674237a: Current Working Directory : /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/d674237a/stage
hello_c/607e8431: Running Test via command: bash --norc --noprofile -eo pipefail hello_c_build.sh
hello_c/d674237a: Running Test via command: bash --norc --noprofile -eo pipefail hello_c_build.sh
hello_c/eab51033: Test completed in 5.470192 seconds
hello_c/eab51033: Test completed with returncode: 0
hello_c/eab51033: Writing output file - /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/eab51033/hello_c.out
hello_c/eab51033: Writing error file - /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/eab51033/hello_c.err
hello_c/d674237a: Test completed in 5.574394 seconds
hello_c/d674237a: Test completed with returncode: 0
hello_c/d674237a: Writing output file - /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/d674237a/hello_c.out
hello_c/d674237a: Writing error file - /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/d674237a/hello_c.err
hello_c/607e8431: Test completed in 5.605869 seconds
hello_c/607e8431: Test completed with returncode: 0
hello_c/607e8431: Writing output file - /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/607e8431/hello_c.out
hello_c/607e8431: Writing error file - /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/607e8431/hello_c.err
Test Summary
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ hello_c/eab51033 │ generic.local.bash │ PASS │ None None None │ 0 │ 5.470192 │
├──────────────────┼────────────────────┼────────┼─────────────────────────────────────┼────────────┼──────────┤
│ hello_c/607e8431 │ generic.local.bash │ PASS │ None None None │ 0 │ 5.605869 │
├──────────────────┼────────────────────┼────────┼─────────────────────────────────────┼────────────┼──────────┤
│ hello_c/d674237a │ generic.local.bash │ PASS │ None None None │ 0 │ 5.574394 │
└──────────────────┴────────────────────┴────────┴─────────────────────────────────────┴────────────┴──────────┘
Passed Tests: 3/3 Percentage: 100.000%
Failed Tests: 0/3 Percentage: 0.000%
Adding 3 test results to /home/spack/buildtest/var/report.json
Writing Logfile to: /home/spack/buildtest/var/logs/buildtest_1z1hkb8x.log
If we inspect the following test, we see each test has its own compiler flags. The default cflag
is -O1 while gcc_6.5.0 will use -O2 and gcc_8.3.0 will use -O3.
buildtest inspect query -t hello_c/
$ buildtest inspect query -t hello_c/
───────────────────────────────────────────────────────────────────────────────────────────── hello_c/eab51033-4a38-4a71-ae56-425781b7a00b ─────────────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: Hello World C Compilation
State: PASS
Returncode: 0
Runtime: 5.470192 sec
Starttime: 2023/11/09 19:35:06
Endtime: 2023/11/09 19:35:11
Command: bash --norc --noprofile -eo pipefail hello_c_build.sh
Test Script: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/eab51033/hello_c.sh
Build Script: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/eab51033/hello_c_build.sh
Output File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/eab51033/hello_c.out
Error File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/eab51033/hello_c.err
Log File: /home/spack/buildtest/var/logs/buildtest_1z1hkb8x.log
──────────────────────────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/eab51033/hello_c.sh ────────────────────────────────────────────────────────────────────────
#!/bin/bash
set -eo pipefail
# name of executable
_EXEC=hello.c.exe
# Compilation Line
/usr/bin/gcc -O1 -o $_EXEC /home/spack/buildtest/examples/compilers/src/hello.c
# Run executable
./$_EXEC
───────────────────────────────────────────────────────────────────────────────────────────── hello_c/607e8431-e5c1-4811-93d3-a2c35e1fa852 ─────────────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: Hello World C Compilation
State: PASS
Returncode: 0
Runtime: 5.605869 sec
Starttime: 2023/11/09 19:35:06
Endtime: 2023/11/09 19:35:12
Command: bash --norc --noprofile -eo pipefail hello_c_build.sh
Test Script: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/607e8431/hello_c.sh
Build Script: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/607e8431/hello_c_build.sh
Output File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/607e8431/hello_c.out
Error File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/607e8431/hello_c.err
Log File: /home/spack/buildtest/var/logs/buildtest_1z1hkb8x.log
──────────────────────────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/607e8431/hello_c.sh ────────────────────────────────────────────────────────────────────────
#!/bin/bash
set -eo pipefail
# name of executable
_EXEC=hello.c.exe
# Loading modules
module load gcc/6.5.0-gcc-7.5.0
# Compilation Line
gcc -O2 -o $_EXEC /home/spack/buildtest/examples/compilers/src/hello.c
# Run executable
./$_EXEC
───────────────────────────────────────────────────────────────────────────────────────────── hello_c/d674237a-519a-49db-81a4-d330211c66a0 ─────────────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: Hello World C Compilation
State: PASS
Returncode: 0
Runtime: 5.574394 sec
Starttime: 2023/11/09 19:35:06
Endtime: 2023/11/09 19:35:12
Command: bash --norc --noprofile -eo pipefail hello_c_build.sh
Test Script: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/d674237a/hello_c.sh
Build Script: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/d674237a/hello_c_build.sh
Output File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/d674237a/hello_c.out
Error File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/d674237a/hello_c.err
Log File: /home/spack/buildtest/var/logs/buildtest_1z1hkb8x.log
──────────────────────────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/d674237a/hello_c.sh ────────────────────────────────────────────────────────────────────────
#!/bin/bash
set -eo pipefail
# name of executable
_EXEC=hello.c.exe
# Loading modules
module load gcc/8.3.0-gcc-7.5.0
# Compilation Line
gcc -O3 -o $_EXEC /home/spack/buildtest/examples/compilers/src/hello.c
# Run executable
./$_EXEC
Excluding Compilers
The exclude property allows one to exclude compilers upon discovery which is a list of compiler
names that will be removed prior to building test. buildtest will exclude any compilers if they were
found based on regular expression via name property. In this next example, we will exclude
gcc_6.5.0 compiler from building
buildspecs:
vecadd_gnu_exclude:
type: compiler
description: Vector Addition example with GNU compilers but exclude gcc_6.5.0
tags: [tutorials, compile]
executor: generic.local.bash
source: src/vecAdd.c
compilers:
name: ["^(gcc)"]
exclude: [gcc_6.5.0]
default:
gcc:
cflags: -fopenacc
ldflags: -lm
Now if we run this test, we will notice that there is only one build for this test even though buildtest
discovered both gcc_6.5.0 and gcc_8.3.0 compilers.
buildtest build -b /home/spack/buildtest/examples/compilers/compiler_exclude.yml
$ buildtest build -b /home/spack/buildtest/examples/compilers/compiler_exclude.yml
╭────────────────────────────────────────────────────── buildtest summary ───────────────────────────────────────────────────────╮
│ │
│ User: spack │
│ Hostname: 651e70126b2d │
│ Platform: Linux │
│ Current Time: 2023/11/09 19:35:14 │
│ buildtest path: /home/spack/buildtest/bin/buildtest │
│ buildtest version: 1.6 │
│ python path: /home/spack/pyenv/buildtest/bin/python3 │
│ python version: 3.8.6 │
│ Configuration File: /home/spack/buildtest/buildtest/settings/spack_container.yml │
│ Test Directory: /home/spack/runs │
│ Report File: /home/spack/buildtest/var/report.json │
│ Command: /home/spack/buildtest/bin/buildtest build -b /home/spack/buildtest/examples/compilers/compiler_exclude.yml │
│ │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
─────────────────────────────────────────────────────────────────────────────────────────────────────── Discovering Buildspecs ────────────────────────────────────────────────────────────────────────────────────────────────────────
Discovered buildspecs
╔═══════════════════════════════════════════════════════════════╗
║ buildspec ║
╟───────────────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/compilers/compiler_exclude.yml ║
╚═══════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ──────────────────────────────────────────────────────────────────────────────────────────────────────────
vecadd_gnu_exclude: Excluding compiler gcc_6.5.0 during test generation
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/compilers/compiler_exclude.yml: VALID
Total builder objects created: 2
Builders by type=compiler
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ description ┃ buildspecs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ vecadd_gnu_exclude/e5f18587 │ compiler │ generic.local.bash │ gcc_7.5.0 │ None │ None │ Vector Addition example with GNU compilers but exclude gcc_6.5.0 │ /home/spack/buildtest/examples/compilers/compiler_exclude.yml │
├─────────────────────────────┼──────────┼────────────────────┼───────────┼───────┼───────┼──────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ vecadd_gnu_exclude/62ef4394 │ compiler │ generic.local.bash │ gcc_8.3.0 │ None │ None │ Vector Addition example with GNU compilers but exclude gcc_6.5.0 │ /home/spack/buildtest/examples/compilers/compiler_exclude.yml │
└─────────────────────────────┴──────────┴────────────────────┴───────────┴───────┴───────┴──────────────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────┘
──────────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
vecadd_gnu_exclude/e5f18587: Creating test directory: /home/spack/runs/generic.local.bash/compiler_exclude/vecadd_gnu_exclude/e5f18587
vecadd_gnu_exclude/e5f18587: Creating the stage directory: /home/spack/runs/generic.local.bash/compiler_exclude/vecadd_gnu_exclude/e5f18587/stage
vecadd_gnu_exclude/e5f18587: Writing build script: /home/spack/runs/generic.local.bash/compiler_exclude/vecadd_gnu_exclude/e5f18587/vecadd_gnu_exclude_build.sh
vecadd_gnu_exclude/62ef4394: Creating test directory: /home/spack/runs/generic.local.bash/compiler_exclude/vecadd_gnu_exclude/62ef4394
vecadd_gnu_exclude/62ef4394: Creating the stage directory: /home/spack/runs/generic.local.bash/compiler_exclude/vecadd_gnu_exclude/62ef4394/stage
vecadd_gnu_exclude/62ef4394: Writing build script: /home/spack/runs/generic.local.bash/compiler_exclude/vecadd_gnu_exclude/62ef4394/vecadd_gnu_exclude_build.sh
──────────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
Spawning 4 processes for processing builders
───────────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
vecadd_gnu_exclude/e5f18587 does not have any dependencies adding test to queue
vecadd_gnu_exclude/62ef4394 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ vecadd_gnu_exclude/e5f18587 │
│ vecadd_gnu_exclude/62ef4394 │
└─────────────────────────────┘
vecadd_gnu_exclude/e5f18587: Current Working Directory : /home/spack/runs/generic.local.bash/compiler_exclude/vecadd_gnu_exclude/e5f18587/stage
vecadd_gnu_exclude/62ef4394: Current Working Directory : /home/spack/runs/generic.local.bash/compiler_exclude/vecadd_gnu_exclude/62ef4394/stage
vecadd_gnu_exclude/e5f18587: Running Test via command: bash --norc --noprofile -eo pipefail vecadd_gnu_exclude_build.sh
vecadd_gnu_exclude/62ef4394: Running Test via command: bash --norc --noprofile -eo pipefail vecadd_gnu_exclude_build.sh
vecadd_gnu_exclude/e5f18587: Test completed in 4.395005 seconds
vecadd_gnu_exclude/e5f18587: Test completed with returncode: 0
vecadd_gnu_exclude/e5f18587: Writing output file - /home/spack/runs/generic.local.bash/compiler_exclude/vecadd_gnu_exclude/e5f18587/vecadd_gnu_exclude.out
vecadd_gnu_exclude/e5f18587: Writing error file - /home/spack/runs/generic.local.bash/compiler_exclude/vecadd_gnu_exclude/e5f18587/vecadd_gnu_exclude.err
vecadd_gnu_exclude/62ef4394: Test completed in 4.481865 seconds
vecadd_gnu_exclude/62ef4394: Test completed with returncode: 0
vecadd_gnu_exclude/62ef4394: Writing output file - /home/spack/runs/generic.local.bash/compiler_exclude/vecadd_gnu_exclude/62ef4394/vecadd_gnu_exclude.out
vecadd_gnu_exclude/62ef4394: Writing error file - /home/spack/runs/generic.local.bash/compiler_exclude/vecadd_gnu_exclude/62ef4394/vecadd_gnu_exclude.err
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ vecadd_gnu_exclude/62ef4394 │ generic.local.bash │ PASS │ None None None │ 0 │ 4.481865 │
├─────────────────────────────┼────────────────────┼────────┼─────────────────────────────────────┼────────────┼──────────┤
│ vecadd_gnu_exclude/e5f18587 │ generic.local.bash │ PASS │ None None None │ 0 │ 4.395005 │
└─────────────────────────────┴────────────────────┴────────┴─────────────────────────────────────┴────────────┴──────────┘
Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%
Adding 2 test results to /home/spack/buildtest/var/report.json
Writing Logfile to: /home/spack/buildtest/var/logs/buildtest_7wc6uto7.log
Setting environment variables
We can define environment variables using env property which is a list of
key/value pair where key is environment name and value is a string assigned to the environment.
The env property can be used in default or config section within a compiler instance.
In this next example we have an OpenMP Hello World example which defines environment variable
OMP_NUM_THREADS that controls number of OpenMP threads to use when running program. In this example
we will set OMP_NUM_THREADS=2
buildspecs:
openmp_hello_c_example:
type: compiler
description: OpenMP Hello World C example
executor: generic.local.bash
tags: [tutorials, compile]
source: "src/hello_omp.c"
compilers:
name: ["^(gcc_7.5.0)"]
default:
gcc:
cflags: -fopenmp
env:
OMP_NUM_THREADS: 2
Now let’s build this test.
buildtest build -b /home/spack/buildtest/examples/compilers/openmp_hello.yml
$ buildtest build -b /home/spack/buildtest/examples/compilers/openmp_hello.yml
╭──────────────────────────────────────────────────── buildtest summary ─────────────────────────────────────────────────────╮
│ │
│ User: spack │
│ Hostname: 651e70126b2d │
│ Platform: Linux │
│ Current Time: 2023/11/09 19:35:19 │
│ buildtest path: /home/spack/buildtest/bin/buildtest │
│ buildtest version: 1.6 │
│ python path: /home/spack/pyenv/buildtest/bin/python3 │
│ python version: 3.8.6 │
│ Configuration File: /home/spack/buildtest/buildtest/settings/spack_container.yml │
│ Test Directory: /home/spack/runs │
│ Report File: /home/spack/buildtest/var/report.json │
│ Command: /home/spack/buildtest/bin/buildtest build -b /home/spack/buildtest/examples/compilers/openmp_hello.yml │
│ │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
─────────────────────────────────────────────────────────────────────────────────────────────────────── Discovering Buildspecs ────────────────────────────────────────────────────────────────────────────────────────────────────────
Discovered buildspecs
╔═══════════════════════════════════════════════════════════╗
║ buildspec ║
╟───────────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/compilers/openmp_hello.yml ║
╚═══════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ──────────────────────────────────────────────────────────────────────────────────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/compilers/openmp_hello.yml: VALID
Total builder objects created: 1
Builders by type=compiler
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ description ┃ buildspecs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ openmp_hello_c_example/a442f53e │ compiler │ generic.local.bash │ gcc_7.5.0 │ None │ None │ OpenMP Hello World C example │ /home/spack/buildtest/examples/compilers/openmp_hello.yml │
└─────────────────────────────────┴──────────┴────────────────────┴───────────┴───────┴───────┴──────────────────────────────┴───────────────────────────────────────────────────────────┘
──────────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
openmp_hello_c_example/a442f53e: Creating test directory: /home/spack/runs/generic.local.bash/openmp_hello/openmp_hello_c_example/a442f53e
openmp_hello_c_example/a442f53e: Creating the stage directory: /home/spack/runs/generic.local.bash/openmp_hello/openmp_hello_c_example/a442f53e/stage
openmp_hello_c_example/a442f53e: Writing build script: /home/spack/runs/generic.local.bash/openmp_hello/openmp_hello_c_example/a442f53e/openmp_hello_c_example_build.sh
──────────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
Spawning 4 processes for processing builders
───────────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
openmp_hello_c_example/a442f53e does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ openmp_hello_c_example/a442f53e │
└─────────────────────────────────┘
openmp_hello_c_example/a442f53e: Current Working Directory : /home/spack/runs/generic.local.bash/openmp_hello/openmp_hello_c_example/a442f53e/stage
openmp_hello_c_example/a442f53e: Running Test via command: bash --norc --noprofile -eo pipefail openmp_hello_c_example_build.sh
openmp_hello_c_example/a442f53e: Test completed in 3.888445 seconds
openmp_hello_c_example/a442f53e: Test completed with returncode: 0
openmp_hello_c_example/a442f53e: Writing output file - /home/spack/runs/generic.local.bash/openmp_hello/openmp_hello_c_example/a442f53e/openmp_hello_c_example.out
openmp_hello_c_example/a442f53e: Writing error file - /home/spack/runs/generic.local.bash/openmp_hello/openmp_hello_c_example/a442f53e/openmp_hello_c_example.err
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ openmp_hello_c_example/a442f53e │ generic.local.bash │ PASS │ None None None │ 0 │ 3.888445 │
└─────────────────────────────────┴────────────────────┴────────┴─────────────────────────────────────┴────────────┴──────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to /home/spack/buildtest/var/report.json
Writing Logfile to: /home/spack/buildtest/var/logs/buildtest_3xv21iaw.log
We can see the generated test using buildtest inspect query given the name of test. Take a close
look at the export OMP_NUM_THREADS in the generated test.
buildtest inspect query -t openmp_hello_c_example
$ buildtest inspect query -t openmp_hello_c_example
───────────────────────────────────────────────────────────────────────────────────── openmp_hello_c_example/a442f53e-3cc6-4a42-bf07-604708ba6192 ──────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: OpenMP Hello World C example
State: PASS
Returncode: 0
Runtime: 3.888445 sec
Starttime: 2023/11/09 19:35:19
Endtime: 2023/11/09 19:35:23
Command: bash --norc --noprofile -eo pipefail openmp_hello_c_example_build.sh
Test Script: /home/spack/runs/generic.local.bash/openmp_hello/openmp_hello_c_example/a442f53e/openmp_hello_c_example.sh
Build Script: /home/spack/runs/generic.local.bash/openmp_hello/openmp_hello_c_example/a442f53e/openmp_hello_c_example_build.sh
Output File: /home/spack/runs/generic.local.bash/openmp_hello/openmp_hello_c_example/a442f53e/openmp_hello_c_example.out
Error File: /home/spack/runs/generic.local.bash/openmp_hello/openmp_hello_c_example/a442f53e/openmp_hello_c_example.err
Log File: /home/spack/buildtest/var/logs/buildtest_3xv21iaw.log
──────────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/openmp_hello/openmp_hello_c_example/a442f53e/openmp_hello_c_example.sh ─────────────────────────────────────────────────────────
#!/bin/bash
set -eo pipefail
# name of executable
_EXEC=hello_omp.c.exe
export OMP_NUM_THREADS="2"
# Compilation Line
/usr/bin/gcc -fopenmp -o $_EXEC /home/spack/buildtest/examples/compilers/src/hello_omp.c
# Run executable
./$_EXEC
We can define environment variables per compiler instance. buildtest will automatically override
any key in defaults with one matched under config for the compiler name. In this next example,
we will define OMP_NUM_THREADS to 4 for gcc_8.3.0 while the default is 2 for all gcc compilers.
buildspecs:
override_environmentvars:
type: compiler
description: override default environment variables
executor: generic.local.bash
tags: [tutorials, compile]
source: "src/hello_omp.c"
compilers:
name: ['^(gcc_7.5.0|gcc_8.3.0)']
default:
gcc:
cflags: -fopenmp
env:
OMP_NUM_THREADS: 2
config:
gcc_8.3.0:
env:
OMP_NUM_THREADS: 4
We can build this test by running:
buildtest build -b $BUILDTEST_ROOT/examples/compilers/envvar_override.yml
Next, let’s see the generated test by running buildtest inspect query -t override_environmentvars. Take a note that we have
export OMP_NUM_THREADS=4 for gcc_8.3.0 test and export OMP_NUM_THREADS=2 for system gcc.
buildtest inspect query -t override_environmentvars/
$ buildtest inspect query -t override_environmentvars/
──────────────────────────────────────────────────────────────────────────────────── override_environmentvars/140e17b4-a948-4aba-aa1c-44c45542a62c ─────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: override default environment variables
State: PASS
Returncode: 0
Runtime: 3.995772 sec
Starttime: 2023/11/09 19:35:25
Endtime: 2023/11/09 19:35:29
Command: bash --norc --noprofile -eo pipefail override_environmentvars_build.sh
Test Script: /home/spack/runs/generic.local.bash/envvar_override/override_environmentvars/140e17b4/override_environmentvars.sh
Build Script: /home/spack/runs/generic.local.bash/envvar_override/override_environmentvars/140e17b4/override_environmentvars_build.sh
Output File: /home/spack/runs/generic.local.bash/envvar_override/override_environmentvars/140e17b4/override_environmentvars.out
Error File: /home/spack/runs/generic.local.bash/envvar_override/override_environmentvars/140e17b4/override_environmentvars.err
Log File: /home/spack/buildtest/var/logs/buildtest_idc1k412.log
───────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/envvar_override/override_environmentvars/140e17b4/override_environmentvars.sh ─────────────────────────────────────────────────────
#!/bin/bash
set -eo pipefail
# name of executable
_EXEC=hello_omp.c.exe
export OMP_NUM_THREADS="4"
# Loading modules
module load gcc/8.3.0-gcc-7.5.0
# Compilation Line
gcc -fopenmp -o $_EXEC /home/spack/buildtest/examples/compilers/src/hello_omp.c
# Run executable
./$_EXEC
──────────────────────────────────────────────────────────────────────────────────── override_environmentvars/b84ea77e-59d5-4740-aa0e-4146e09f4a8c ─────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: override default environment variables
State: PASS
Returncode: 0
Runtime: 4.063811 sec
Starttime: 2023/11/09 19:35:25
Endtime: 2023/11/09 19:35:29
Command: bash --norc --noprofile -eo pipefail override_environmentvars_build.sh
Test Script: /home/spack/runs/generic.local.bash/envvar_override/override_environmentvars/b84ea77e/override_environmentvars.sh
Build Script: /home/spack/runs/generic.local.bash/envvar_override/override_environmentvars/b84ea77e/override_environmentvars_build.sh
Output File: /home/spack/runs/generic.local.bash/envvar_override/override_environmentvars/b84ea77e/override_environmentvars.out
Error File: /home/spack/runs/generic.local.bash/envvar_override/override_environmentvars/b84ea77e/override_environmentvars.err
Log File: /home/spack/buildtest/var/logs/buildtest_idc1k412.log
───────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/envvar_override/override_environmentvars/b84ea77e/override_environmentvars.sh ─────────────────────────────────────────────────────
#!/bin/bash
set -eo pipefail
# name of executable
_EXEC=hello_omp.c.exe
export OMP_NUM_THREADS="2"
# Compilation Line
/usr/bin/gcc -fopenmp -o $_EXEC /home/spack/buildtest/examples/compilers/src/hello_omp.c
# Run executable
./$_EXEC
Tweak how test are passed
The status property is used to determine how buildtest will determine status of test (PASS/FAIL).
By default, an exitcode 0 is a PASS and anything else is FAIL.
Sometimes, it may be useful check output of test to determine using regular expression. This
can be done via regex property. In this example, we define two tests, the first one defines status
property in the default gcc group. This means all compilers that belong to gcc
group will be matched based on returncode.
In second test we override the status property for a given compiler instance under the config section.
The test is expected to produce output of final result: 1.000000 but we will specify a different value in order to
show this test will fail.
buildspecs:
default_status_returncode:
type: compiler
description: status check based on returncode
tags: [tutorials, compile]
executor: generic.local.bash
source: src/vecAdd.c
compilers:
name: ["^(gcc)"]
default:
gcc:
cflags: -fopenacc
ldflags: -lm
status:
returncode: 0
override_status_regex:
type: compiler
description: override 'status' property in compiler gcc_8.3.0 instance
tags: [tutorials, compile]
executor: generic.local.bash
source: src/vecAdd.c
compilers:
name: ["^(gcc)"]
default:
gcc:
cflags: -fopenacc
ldflags: -lm
status:
returncode: 1
config:
gcc_8.3.0:
status:
regex:
stream: stdout
exp: "^final result: 0.99$"
If we build this test, we should expect the first test example should pass based on
returncode 0. If the returncode doesn’t match buildtest will report failure. For the second test example,
we have have specified test will pass if we get a returncode 1 based on default property however for
gcc_8.3.0 compiler test we have defined a status property to check based on regular expression.
We will expect both tests to fail since we will have a mismatch on returncode and regular expression
buildtest build -b /home/spack/buildtest/examples/compilers/compiler_status_regex.yml
$ buildtest build -b /home/spack/buildtest/examples/compilers/compiler_status_regex.yml
╭───────────────────────────────────────────────────────── buildtest summary ─────────────────────────────────────────────────────────╮
│ │
│ User: spack │
│ Hostname: cb5c73bf02b8 │
│ Platform: Linux │
│ Current Time: 2023/11/09 20:33:26 │
│ buildtest path: /home/spack/buildtest/bin/buildtest │
│ buildtest version: 1.6 │
│ python path: /home/spack/pyenv/buildtest/bin/python3 │
│ python version: 3.8.6 │
│ Configuration File: /home/spack/buildtest/buildtest/settings/spack_container.yml │
│ Test Directory: /home/spack/runs │
│ Report File: /home/spack/buildtest/var/report.json │
│ Command: /home/spack/buildtest/bin/buildtest build -b /home/spack/buildtest/examples/compilers/compiler_status_regex.yml │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
─────────────────────────────────────────────────────────────────────────────────────────────────────── Discovering Buildspecs ────────────────────────────────────────────────────────────────────────────────────────────────────────
Discovered buildspecs
╔════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟────────────────────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/compilers/compiler_status_regex.yml ║
╚════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ──────────────────────────────────────────────────────────────────────────────────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/compilers/compiler_status_regex.yml: VALID
Total builder objects created: 6
Builders by type=compiler
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ description ┃ buildspecs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ default_status_returncode/54a37b17 │ compiler │ generic.local.bash │ gcc_7.5.0 │ None │ None │ status check based on returncode │ /home/spack/buildtest/examples/compilers/compiler_status_regex.yml │
├────────────────────────────────────┼──────────┼────────────────────┼───────────┼───────┼───────┼───────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ default_status_returncode/f7feb30b │ compiler │ generic.local.bash │ gcc_6.5.0 │ None │ None │ status check based on returncode │ /home/spack/buildtest/examples/compilers/compiler_status_regex.yml │
├────────────────────────────────────┼──────────┼────────────────────┼───────────┼───────┼───────┼───────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ default_status_returncode/6a85c0d1 │ compiler │ generic.local.bash │ gcc_8.3.0 │ None │ None │ status check based on returncode │ /home/spack/buildtest/examples/compilers/compiler_status_regex.yml │
├────────────────────────────────────┼──────────┼────────────────────┼───────────┼───────┼───────┼───────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ override_status_regex/dfdd02f3 │ compiler │ generic.local.bash │ gcc_7.5.0 │ None │ None │ override 'status' property in compiler gcc_8.3.0 instance │ /home/spack/buildtest/examples/compilers/compiler_status_regex.yml │
├────────────────────────────────────┼──────────┼────────────────────┼───────────┼───────┼───────┼───────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ override_status_regex/a2b67d4f │ compiler │ generic.local.bash │ gcc_6.5.0 │ None │ None │ override 'status' property in compiler gcc_8.3.0 instance │ /home/spack/buildtest/examples/compilers/compiler_status_regex.yml │
├────────────────────────────────────┼──────────┼────────────────────┼───────────┼───────┼───────┼───────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ override_status_regex/c3687c2b │ compiler │ generic.local.bash │ gcc_8.3.0 │ None │ None │ override 'status' property in compiler gcc_8.3.0 instance │ /home/spack/buildtest/examples/compilers/compiler_status_regex.yml │
└────────────────────────────────────┴──────────┴────────────────────┴───────────┴───────┴───────┴───────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────┘
──────────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
default_status_returncode/54a37b17: Creating test directory: /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/54a37b17
default_status_returncode/54a37b17: Creating the stage directory: /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/54a37b17/stage
default_status_returncode/54a37b17: Writing build script: /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/54a37b17/default_status_returncode_build.sh
default_status_returncode/f7feb30b: Creating test directory: /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/f7feb30b
default_status_returncode/f7feb30b: Creating the stage directory: /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/f7feb30b/stage
default_status_returncode/f7feb30b: Writing build script: /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/f7feb30b/default_status_returncode_build.sh
default_status_returncode/6a85c0d1: Creating test directory: /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/6a85c0d1
default_status_returncode/6a85c0d1: Creating the stage directory: /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/6a85c0d1/stage
default_status_returncode/6a85c0d1: Writing build script: /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/6a85c0d1/default_status_returncode_build.sh
override_status_regex/dfdd02f3: Creating test directory: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/dfdd02f3
override_status_regex/dfdd02f3: Creating the stage directory: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/dfdd02f3/stage
override_status_regex/dfdd02f3: Writing build script: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/dfdd02f3/override_status_regex_build.sh
override_status_regex/a2b67d4f: Creating test directory: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/a2b67d4f
override_status_regex/a2b67d4f: Creating the stage directory: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/a2b67d4f/stage
override_status_regex/a2b67d4f: Writing build script: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/a2b67d4f/override_status_regex_build.sh
override_status_regex/c3687c2b: Creating test directory: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/c3687c2b
override_status_regex/c3687c2b: Creating the stage directory: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/c3687c2b/stage
override_status_regex/c3687c2b: Writing build script: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/c3687c2b/override_status_regex_build.sh
──────────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
Spawning 4 processes for processing builders
───────────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
default_status_returncode/6a85c0d1 does not have any dependencies adding test to queue
default_status_returncode/54a37b17 does not have any dependencies adding test to queue
override_status_regex/c3687c2b does not have any dependencies adding test to queue
default_status_returncode/f7feb30b does not have any dependencies adding test to queue
override_status_regex/dfdd02f3 does not have any dependencies adding test to queue
override_status_regex/a2b67d4f does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ default_status_returncode/6a85c0d1 │
│ default_status_returncode/54a37b17 │
│ override_status_regex/c3687c2b │
│ default_status_returncode/f7feb30b │
│ override_status_regex/dfdd02f3 │
│ override_status_regex/a2b67d4f │
└────────────────────────────────────┘
default_status_returncode/6a85c0d1: Current Working Directory : /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/6a85c0d1/stage
override_status_regex/c3687c2b: Current Working Directory : /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/c3687c2b/stage
default_status_returncode/f7feb30b: Current Working Directory : /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/f7feb30b/stage
default_status_returncode/54a37b17: Current Working Directory : /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/54a37b17/stage
default_status_returncode/6a85c0d1: Running Test via command: bash --norc --noprofile -eo pipefail default_status_returncode_build.sh
override_status_regex/c3687c2b: Running Test via command: bash --norc --noprofile -eo pipefail override_status_regex_build.sh
default_status_returncode/f7feb30b: Running Test via command: bash --norc --noprofile -eo pipefail default_status_returncode_build.sh
default_status_returncode/54a37b17: Running Test via command: bash --norc --noprofile -eo pipefail default_status_returncode_build.sh
default_status_returncode/54a37b17: Test completed in 7.303727 seconds
default_status_returncode/54a37b17: Test completed with returncode: 0
default_status_returncode/54a37b17: Writing output file - /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/54a37b17/default_status_returncode.out
default_status_returncode/54a37b17: Writing error file - /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/54a37b17/default_status_returncode.err
default_status_returncode/54a37b17: Checking returncode - 0 is matched in list [0]
override_status_regex/dfdd02f3: Current Working Directory : /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/dfdd02f3/stage
override_status_regex/dfdd02f3: Running Test via command: bash --norc --noprofile -eo pipefail override_status_regex_build.sh
default_status_returncode/f7feb30b: Test completed in 7.571007 seconds
default_status_returncode/f7feb30b: Test completed with returncode: 0
default_status_returncode/f7feb30b: Writing output file - /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/f7feb30b/default_status_returncode.out
default_status_returncode/f7feb30b: Writing error file - /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/f7feb30b/default_status_returncode.err
default_status_returncode/f7feb30b: Checking returncode - 0 is matched in list [0]
override_status_regex/a2b67d4f: Current Working Directory : /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/a2b67d4f/stage
override_status_regex/a2b67d4f: Running Test via command: bash --norc --noprofile -eo pipefail override_status_regex_build.sh
override_status_regex/c3687c2b: Test completed in 7.669674 seconds
override_status_regex/c3687c2b: Test completed with returncode: 0
override_status_regex/c3687c2b: Writing output file - /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/c3687c2b/override_status_regex.out
override_status_regex/c3687c2b: Writing error file - /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/c3687c2b/override_status_regex.err
override_status_regex/c3687c2b: performing regular expression - '^final result: 0.99$' on file: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/c3687c2b/override_status_regex.out
override_status_regex/c3687c2b: Regular Expression Match - Failed!
default_status_returncode/6a85c0d1: Test completed in 7.771137 seconds
default_status_returncode/6a85c0d1: Test completed with returncode: 0
default_status_returncode/6a85c0d1: Writing output file - /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/6a85c0d1/default_status_returncode.out
default_status_returncode/6a85c0d1: Writing error file - /home/spack/runs/generic.local.bash/compiler_status_regex/default_status_returncode/6a85c0d1/default_status_returncode.err
default_status_returncode/6a85c0d1: Checking returncode - 0 is matched in list [0]
override_status_regex/dfdd02f3: Test completed in 7.473231 seconds
override_status_regex/dfdd02f3: Test completed with returncode: 0
override_status_regex/dfdd02f3: Writing output file - /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/dfdd02f3/override_status_regex.out
override_status_regex/dfdd02f3: Writing error file - /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/dfdd02f3/override_status_regex.err
override_status_regex/dfdd02f3: Checking returncode - 0 is matched in list [1]
override_status_regex/a2b67d4f: Test completed in 7.457885 seconds
override_status_regex/a2b67d4f: Test completed with returncode: 0
override_status_regex/a2b67d4f: Writing output file - /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/a2b67d4f/override_status_regex.out
override_status_regex/a2b67d4f: Writing error file - /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/a2b67d4f/override_status_regex.err
override_status_regex/a2b67d4f: Checking returncode - 0 is matched in list [1]
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ override_status_regex/dfdd02f3 │ generic.local.bash │ FAIL │ False None None │ 0 │ 7.473231 │
├────────────────────────────────────┼────────────────────┼────────┼─────────────────────────────────────┼────────────┼──────────┤
│ default_status_returncode/f7feb30b │ generic.local.bash │ PASS │ True None None │ 0 │ 7.571007 │
├────────────────────────────────────┼────────────────────┼────────┼─────────────────────────────────────┼────────────┼──────────┤
│ override_status_regex/a2b67d4f │ generic.local.bash │ FAIL │ False None None │ 0 │ 7.457885 │
├────────────────────────────────────┼────────────────────┼────────┼─────────────────────────────────────┼────────────┼──────────┤
│ override_status_regex/c3687c2b │ generic.local.bash │ FAIL │ None False None │ 0 │ 7.669674 │
├────────────────────────────────────┼────────────────────┼────────┼─────────────────────────────────────┼────────────┼──────────┤
│ default_status_returncode/6a85c0d1 │ generic.local.bash │ PASS │ True None None │ 0 │ 7.771137 │
├────────────────────────────────────┼────────────────────┼────────┼─────────────────────────────────────┼────────────┼──────────┤
│ default_status_returncode/54a37b17 │ generic.local.bash │ PASS │ True None None │ 0 │ 7.303727 │
└────────────────────────────────────┴────────────────────┴────────┴─────────────────────────────────────┴────────────┴──────────┘
Passed Tests: 3/6 Percentage: 50.000%
Failed Tests: 3/6 Percentage: 50.000%
Adding 6 test results to /home/spack/buildtest/var/report.json
Writing Logfile to: /home/spack/buildtest/var/logs/buildtest_gr_129l1.log
For the second test, we see the generated output is final result: 1.000000 but our regular expression has a different expected value therefore this test will fail even though we have a exitcode of 0.
buildtest inspect query -o override_status_regex/
$ buildtest inspect query -o override_status_regex/
────────────────────────────────────────────────────────────────────────────────────── override_status_regex/a2b67d4f-3144-4ee4-8229-55da8fe0c7c3 ──────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: override 'status' property in compiler gcc_8.3.0 instance
State: FAIL
Returncode: 0
Runtime: 7.457885 sec
Starttime: 2023/11/09 20:33:34
Endtime: 2023/11/09 20:33:42
Command: bash --norc --noprofile -eo pipefail override_status_regex_build.sh
Test Script: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/a2b67d4f/override_status_regex.sh
Build Script: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/a2b67d4f/override_status_regex_build.sh
Output File: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/a2b67d4f/override_status_regex.out
Error File: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/a2b67d4f/override_status_regex.err
Log File: /home/spack/buildtest/var/logs/buildtest_gr_129l1.log
─────────────────────────────────────────────────── Output File: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/a2b67d4f/override_status_regex.out ────────────────────────────────────────────────────
==> Regenerating tcl module files
final result: 1.000000
────────────────────────────────────────────────────────────────────────────────────── override_status_regex/c3687c2b-99d5-4ea6-9d3c-87e1b166c8c0 ──────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: override 'status' property in compiler gcc_8.3.0 instance
State: FAIL
Returncode: 0
Runtime: 7.669674 sec
Starttime: 2023/11/09 20:33:26
Endtime: 2023/11/09 20:33:34
Command: bash --norc --noprofile -eo pipefail override_status_regex_build.sh
Test Script: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/c3687c2b/override_status_regex.sh
Build Script: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/c3687c2b/override_status_regex_build.sh
Output File: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/c3687c2b/override_status_regex.out
Error File: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/c3687c2b/override_status_regex.err
Log File: /home/spack/buildtest/var/logs/buildtest_gr_129l1.log
─────────────────────────────────────────────────── Output File: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/c3687c2b/override_status_regex.out ────────────────────────────────────────────────────
==> Regenerating tcl module files
final result: 1.000000
────────────────────────────────────────────────────────────────────────────────────── override_status_regex/dfdd02f3-5d08-4f2c-a3a6-52d459809603 ──────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: override 'status' property in compiler gcc_8.3.0 instance
State: FAIL
Returncode: 0
Runtime: 7.473231 sec
Starttime: 2023/11/09 20:33:34
Endtime: 2023/11/09 20:33:41
Command: bash --norc --noprofile -eo pipefail override_status_regex_build.sh
Test Script: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/dfdd02f3/override_status_regex.sh
Build Script: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/dfdd02f3/override_status_regex_build.sh
Output File: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/dfdd02f3/override_status_regex.out
Error File: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/dfdd02f3/override_status_regex.err
Log File: /home/spack/buildtest/var/logs/buildtest_gr_129l1.log
─────────────────────────────────────────────────── Output File: /home/spack/runs/generic.local.bash/compiler_status_regex/override_status_regex/dfdd02f3/override_status_regex.out ────────────────────────────────────────────────────
==> Regenerating tcl module files
final result: 1.000000
Customize Run Line
buildtest will define variable _EXEC in the job script that can be used to reference
the generated binary. By default, buildtest will run the program standalone, but sometimes you
may want to customize how job is run. This may include passing arguments or running
binary through a job/mpi launcher. The run property can be used to configure how program is executed.
The compiled executable will be present in local directory which can be accessed via ./$_EXEC. In example below
we pass arguments 1 3 for builtin_gcc compiler and 100 200 for gcc_8.3.0 compiler.
buildspecs:
custom_run_by_compilers:
type: compiler
description: Customize binary launch based on compiler
executor: generic.local.bash
tags: [tutorials, compile]
source: "src/argc.c"
compilers:
name: ["^(gcc_7.5.0|gcc_8.3.0)"]
config:
gcc_7.5.0:
run: ./$_EXEC 1 3
gcc_8.3.0:
run: ./$_EXEC 100 120
You can build this test by running the following:
buildtest build -b $BUILDTEST_ROOT/examples/compilers/custom_run.yml
Once test is complete let’s inspect the generated test. We see that buildtest will insert the line specified
by run property after compilation and run the executable.
buildtest inspect query -b -t custom_run_by_compilers/
$ buildtest inspect query -b -t custom_run_by_compilers/
───────────────────────────────────────────────────────────────────────────────────── custom_run_by_compilers/2ff1acec-5d97-480a-aaa3-79148dcdcbc1 ─────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: Customize binary launch based on compiler
State: PASS
Returncode: 0
Runtime: 4.296552 sec
Starttime: 2023/11/09 19:35:45
Endtime: 2023/11/09 19:35:49
Command: bash --norc --noprofile -eo pipefail custom_run_by_compilers_build.sh
Test Script: /home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/2ff1acec/custom_run_by_compilers.sh
Build Script: /home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/2ff1acec/custom_run_by_compilers_build.sh
Output File: /home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/2ff1acec/custom_run_by_compilers.out
Error File: /home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/2ff1acec/custom_run_by_compilers.err
Log File: /home/spack/buildtest/var/logs/buildtest_xqbefz7a.log
──────────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/2ff1acec/custom_run_by_compilers.sh ─────────────────────────────────────────────────────────
#!/bin/bash
set -eo pipefail
# name of executable
_EXEC=argc.c.exe
# Loading modules
module load gcc/8.3.0-gcc-7.5.0
# Compilation Line
gcc -o $_EXEC /home/spack/buildtest/examples/compilers/src/argc.c
# Run executable
./$_EXEC 100 120
───────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/2ff1acec/custom_run_by_compilers_build.sh ──────────────────────────────────────────────────────
#!/bin/bash
# Function to handle all signals and perform cleanup
function cleanup() {
echo "Signal trapped. Performing cleanup before exiting."
exitcode=$?
echo "buildtest: command \`$BASH_COMMAND' failed (exit code: $exitcode)"
exit $exitcode
}
# Trap all signals and call the cleanup function
trap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU
export BUILDTEST_TEST_NAME=custom_run_by_compilers
export BUILDTEST_TEST_ROOT=/home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/2ff1acec
export BUILDTEST_BUILDSPEC_DIR=/home/spack/buildtest/examples/compilers
export BUILDTEST_STAGE_DIR=/home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/2ff1acec/stage
# source executor startup script
source /home/spack/buildtest/var/executor/generic.local.bash/before_script.sh
# Run generated script
bash --norc --noprofile -eo pipefail /home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/2ff1acec/stage/custom_run_by_compilers.sh
# Get return code
returncode=$?
# Exit with return code
exit $returncode
───────────────────────────────────────────────────────────────────────────────────── custom_run_by_compilers/51def0b0-ed0e-4825-b8d8-f4133a9b1634 ─────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: Customize binary launch based on compiler
State: PASS
Returncode: 0
Runtime: 4.197142 sec
Starttime: 2023/11/09 19:35:45
Endtime: 2023/11/09 19:35:49
Command: bash --norc --noprofile -eo pipefail custom_run_by_compilers_build.sh
Test Script: /home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/51def0b0/custom_run_by_compilers.sh
Build Script: /home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/51def0b0/custom_run_by_compilers_build.sh
Output File: /home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/51def0b0/custom_run_by_compilers.out
Error File: /home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/51def0b0/custom_run_by_compilers.err
Log File: /home/spack/buildtest/var/logs/buildtest_xqbefz7a.log
──────────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/51def0b0/custom_run_by_compilers.sh ─────────────────────────────────────────────────────────
#!/bin/bash
set -eo pipefail
# name of executable
_EXEC=argc.c.exe
# Compilation Line
/usr/bin/gcc -o $_EXEC /home/spack/buildtest/examples/compilers/src/argc.c
# Run executable
./$_EXEC 1 3
───────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/51def0b0/custom_run_by_compilers_build.sh ──────────────────────────────────────────────────────
#!/bin/bash
# Function to handle all signals and perform cleanup
function cleanup() {
echo "Signal trapped. Performing cleanup before exiting."
exitcode=$?
echo "buildtest: command \`$BASH_COMMAND' failed (exit code: $exitcode)"
exit $exitcode
}
# Trap all signals and call the cleanup function
trap cleanup SIGINT SIGTERM SIGHUP SIGQUIT SIGABRT SIGKILL SIGALRM SIGPIPE SIGTERM SIGTSTP SIGTTIN SIGTTOU
export BUILDTEST_TEST_NAME=custom_run_by_compilers
export BUILDTEST_TEST_ROOT=/home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/51def0b0
export BUILDTEST_BUILDSPEC_DIR=/home/spack/buildtest/examples/compilers
export BUILDTEST_STAGE_DIR=/home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/51def0b0/stage
# source executor startup script
source /home/spack/buildtest/var/executor/generic.local.bash/before_script.sh
# Run generated script
bash --norc --noprofile -eo pipefail /home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/51def0b0/stage/custom_run_by_compilers.sh
# Get return code
returncode=$?
# Exit with return code
exit $returncode
Pre/Post sections for build and run section
We can specify arbitrary shell commands before/after compilation or running binary via
pre_build, post_build, pre_run and post_run property. This can be useful
if you want to specify additional commands required to compile or run executable that are not
generated automatically
This next example illustrates how one can use these properties to have more control over the generated test.
buildspecs:
pre_post_build_run:
type: compiler
description: example using pre_build, post_build, pre_run, post_run example
executor: generic.local.bash
tags: [tutorials, compile]
source: "src/hello.c"
compilers:
name: ["^(gcc_7.5.0)$"]
default:
gcc:
cflags: -Wall
pre_build: |
echo "These are commands run before compilation"
post_build: |
echo "These are commands run after compilation"
pre_run: |
echo "These are commands run before running script"
post_run:
echo "These are commands run after running script"
The format of the test structure is as follows.
#!{shebang path} -- defaults to #!/bin/bash depends on executor name (local.bash, local.sh)
{job directives} -- sbatch or bsub field
{environment variables} -- env field
{variable declaration} -- vars field
{module commands} -- modules field
{pre build commands} -- pre_build field
{compile program} -- build field
{post build commands} -- post_build field
{pre run commands} -- pre_run field
{run executable} -- run field
{post run commands} -- post_run field
You can run this example by running the following command:
buildtest build -b $BUILDTEST_ROOT/examples/compilers/pre_post_build_run.yml
If we inspect the content of test we see that buildtest will insert the shell commands
for pre_build, post_build, pre_run and post_run in its corresponding section.
buildtest inspect query -t pre_post_build_run
$ buildtest inspect query -t pre_post_build_run
─────────────────────────────────────────────────────────────────────────────────────── pre_post_build_run/7576dea5-edd8-413c-a709-f3fffe022961 ────────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: example using pre_build, post_build, pre_run, post_run example
State: PASS
Returncode: 0
Runtime: 3.844583 sec
Starttime: 2023/11/09 19:35:51
Endtime: 2023/11/09 19:35:55
Command: bash --norc --noprofile -eo pipefail pre_post_build_run_build.sh
Test Script: /home/spack/runs/generic.local.bash/pre_post_build_run/pre_post_build_run/7576dea5/pre_post_build_run.sh
Build Script: /home/spack/runs/generic.local.bash/pre_post_build_run/pre_post_build_run/7576dea5/pre_post_build_run_build.sh
Output File: /home/spack/runs/generic.local.bash/pre_post_build_run/pre_post_build_run/7576dea5/pre_post_build_run.out
Error File: /home/spack/runs/generic.local.bash/pre_post_build_run/pre_post_build_run/7576dea5/pre_post_build_run.err
Log File: /home/spack/buildtest/var/logs/buildtest_qnzqfuun.log
───────────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/pre_post_build_run/pre_post_build_run/7576dea5/pre_post_build_run.sh ──────────────────────────────────────────────────────────
#!/bin/bash
set -eo pipefail
# name of executable
_EXEC=hello.c.exe
### START OF PRE BUILD SECTION ###
echo "These are commands run before compilation"
### END OF PRE BUILD SECTION ###
# Compilation Line
/usr/bin/gcc -Wall -o $_EXEC /home/spack/buildtest/examples/compilers/src/hello.c
### START OF POST BUILD SECTION ###
echo "These are commands run after compilation"
### END OF POST BUILD SECTION ###
### START OF PRE RUN SECTION ###
echo "These are commands run before running script"
### END OF PRE RUN SECTION ###
# Run executable
./$_EXEC
### START OF POST RUN SECTION ###
echo "These are commands run after running script"
### END OF POST RUN SECTION ###
Running Stream Benchmark with multiple compilers
In this example, we will show how one can run STREAM benchmark
with multiple compilers with script schema. The compilers property can be set in the script schema which
is used to search for compilers to create separate test for each discovered compiler. buildtest will set the following
variables in the script that is mapped to each compiler
BUILDTEST_CC: Path to C compiler
BUILDTEST_CXX: Path to C++ compiler
BUILDTEST_FC: Path to Fortran compiler
BUILDTEST_CFLAGS: C compiler flags
BUILDTEST_CXXFLAGS: C++ compiler flags
BUILDTEST_FFLAGS: Fortran compiler flags
BUILDTEST_CPPFLAGS: C++ Pre Preprocessor flags
BUILDTEST_LDFLAGS: Linker Flags
In the run section we can reference these variables to compile stream.c to build the code and run it. We will
set environment OMP_NUM_THREADS to control number of OpenMP threads when running the benchmark.
buildspecs:
stream_openmp_c:
executor: generic.local.bash
type: script
description: "STREAM Microbenchmark C Test with OpenMP"
tags: ["benchmark"]
compilers:
name: ['^(gcc)']
default:
gcc:
cflags: -fopenmp -O2
env:
OMP_NUM_THREADS: 8
run: |
wget https://www.cs.virginia.edu/stream/FTP/Code/stream.c
$BUILDTEST_CC $BUILDTEST_CFLAGS stream.c -o stream
./stream
If we build this test we see one test is created per compiler instance defined in our configuration file.
buildtest build -b /home/spack/buildtest/examples/compilers/stream_example.yml
$ buildtest build -b /home/spack/buildtest/examples/compilers/stream_example.yml
╭───────────────────────────────────────────────────── buildtest summary ──────────────────────────────────────────────────────╮
│ │
│ User: spack │
│ Hostname: 651e70126b2d │
│ Platform: Linux │
│ Current Time: 2023/11/09 19:35:56 │
│ buildtest path: /home/spack/buildtest/bin/buildtest │
│ buildtest version: 1.6 │
│ python path: /home/spack/pyenv/buildtest/bin/python3 │
│ python version: 3.8.6 │
│ Configuration File: /home/spack/buildtest/buildtest/settings/spack_container.yml │
│ Test Directory: /home/spack/runs │
│ Report File: /home/spack/buildtest/var/report.json │
│ Command: /home/spack/buildtest/bin/buildtest build -b /home/spack/buildtest/examples/compilers/stream_example.yml │
│ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
─────────────────────────────────────────────────────────────────────────────────────────────────────── Discovering Buildspecs ────────────────────────────────────────────────────────────────────────────────────────────────────────
Discovered buildspecs
╔═════════════════════════════════════════════════════════════╗
║ buildspec ║
╟─────────────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/compilers/stream_example.yml ║
╚═════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ──────────────────────────────────────────────────────────────────────────────────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/compilers/stream_example.yml: VALID
Total builder objects created: 3
Builders by type=script
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ description ┃ buildspecs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ stream_openmp_c/ddbeefba │ script │ generic.local.bash │ gcc_7.5.0 │ None │ None │ STREAM Microbenchmark C Test with OpenMP │ /home/spack/buildtest/examples/compilers/stream_example.yml │
├──────────────────────────┼────────┼────────────────────┼───────────┼───────┼───────┼──────────────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ stream_openmp_c/21d29876 │ script │ generic.local.bash │ gcc_6.5.0 │ None │ None │ STREAM Microbenchmark C Test with OpenMP │ /home/spack/buildtest/examples/compilers/stream_example.yml │
├──────────────────────────┼────────┼────────────────────┼───────────┼───────┼───────┼──────────────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ stream_openmp_c/dee137b4 │ script │ generic.local.bash │ gcc_8.3.0 │ None │ None │ STREAM Microbenchmark C Test with OpenMP │ /home/spack/buildtest/examples/compilers/stream_example.yml │
└──────────────────────────┴────────┴────────────────────┴───────────┴───────┴───────┴──────────────────────────────────────────┴─────────────────────────────────────────────────────────────┘
──────────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
stream_openmp_c/ddbeefba: Creating test directory: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/ddbeefba
stream_openmp_c/ddbeefba: Creating the stage directory: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/ddbeefba/stage
stream_openmp_c/ddbeefba: Writing build script: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/ddbeefba/stream_openmp_c_build.sh
stream_openmp_c/21d29876: Creating test directory: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/21d29876
stream_openmp_c/21d29876: Creating the stage directory: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/21d29876/stage
stream_openmp_c/21d29876: Writing build script: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/21d29876/stream_openmp_c_build.sh
stream_openmp_c/dee137b4: Creating test directory: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/dee137b4
stream_openmp_c/dee137b4: Creating the stage directory: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/dee137b4/stage
stream_openmp_c/dee137b4: Writing build script: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/dee137b4/stream_openmp_c_build.sh
──────────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
Spawning 4 processes for processing builders
───────────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
stream_openmp_c/dee137b4 does not have any dependencies adding test to queue
stream_openmp_c/ddbeefba does not have any dependencies adding test to queue
stream_openmp_c/21d29876 does not have any dependencies adding test to queue
Builders Eligible to Run
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ stream_openmp_c/dee137b4 │
│ stream_openmp_c/ddbeefba │
│ stream_openmp_c/21d29876 │
└──────────────────────────┘
stream_openmp_c/dee137b4: Current Working Directory : /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/dee137b4/stage
stream_openmp_c/ddbeefba: Current Working Directory : /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/ddbeefba/stage
stream_openmp_c/21d29876: Current Working Directory : /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/21d29876/stage
stream_openmp_c/dee137b4: Running Test via command: bash --norc --noprofile -eo pipefail stream_openmp_c_build.sh
stream_openmp_c/ddbeefba: Running Test via command: bash --norc --noprofile -eo pipefail stream_openmp_c_build.sh
stream_openmp_c/21d29876: Running Test via command: bash --norc --noprofile -eo pipefail stream_openmp_c_build.sh
stream_openmp_c/21d29876: Test completed in 8.217116 seconds
stream_openmp_c/21d29876: Test completed with returncode: 0
stream_openmp_c/21d29876: Writing output file - /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/21d29876/stream_openmp_c.out
stream_openmp_c/21d29876: Writing error file - /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/21d29876/stream_openmp_c.err
stream_openmp_c/ddbeefba: Test completed in 8.240962 seconds
stream_openmp_c/ddbeefba: Test completed with returncode: 0
stream_openmp_c/ddbeefba: Writing output file - /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/ddbeefba/stream_openmp_c.out
stream_openmp_c/ddbeefba: Writing error file - /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/ddbeefba/stream_openmp_c.err
stream_openmp_c/dee137b4: Test completed in 8.257345 seconds
stream_openmp_c/dee137b4: Test completed with returncode: 0
stream_openmp_c/dee137b4: Writing output file - /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/dee137b4/stream_openmp_c.out
stream_openmp_c/dee137b4: Writing error file - /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/dee137b4/stream_openmp_c.err
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ stream_openmp_c/21d29876 │ generic.local.bash │ PASS │ None None None │ 0 │ 8.217116 │
├──────────────────────────┼────────────────────┼────────┼─────────────────────────────────────┼────────────┼──────────┤
│ stream_openmp_c/ddbeefba │ generic.local.bash │ PASS │ None None None │ 0 │ 8.240962 │
├──────────────────────────┼────────────────────┼────────┼─────────────────────────────────────┼────────────┼──────────┤
│ stream_openmp_c/dee137b4 │ generic.local.bash │ PASS │ None None None │ 0 │ 8.257345 │
└──────────────────────────┴────────────────────┴────────┴─────────────────────────────────────┴────────────┴──────────┘
Passed Tests: 3/3 Percentage: 100.000%
Failed Tests: 0/3 Percentage: 0.000%
Adding 3 test results to /home/spack/buildtest/var/report.json
Writing Logfile to: /home/spack/buildtest/var/logs/buildtest_x2eeaaak.log
Next, let’s see the generated test using buildtest inspect query, we notice buildtest will set variables BUILDTEST_* for
each test to map to each compiler.
buildtest inspect query -t stream_openmp_c/
$ buildtest inspect query -t stream_openmp_c/
───────────────────────────────────────────────────────────────────────────────────────── stream_openmp_c/dee137b4-9a8f-4809-a3f9-8ab8280cdaa9 ─────────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: STREAM Microbenchmark C Test with OpenMP
State: PASS
Returncode: 0
Runtime: 8.257345 sec
Starttime: 2023/11/09 19:35:56
Endtime: 2023/11/09 19:36:05
Command: bash --norc --noprofile -eo pipefail stream_openmp_c_build.sh
Test Script: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/dee137b4/stream_openmp_c.sh
Build Script: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/dee137b4/stream_openmp_c_build.sh
Output File: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/dee137b4/stream_openmp_c.out
Error File: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/dee137b4/stream_openmp_c.err
Log File: /home/spack/buildtest/var/logs/buildtest_x2eeaaak.log
────────────────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/dee137b4/stream_openmp_c.sh ───────────────────────────────────────────────────────────────
#!/bin/bash
set -eo pipefail
BUILDTEST_CC="gcc"
BUILDTEST_CXX="g++"
BUILDTEST_FC="gfortran"
BUILDTEST_CFLAGS="-fopenmp -O2"
BUILDTEST_CXXFLAGS=
BUILDTEST_FFLAGS=
BUILDTEST_CPPFLAGS=
BUILDTEST_LDFLAGS=
export OMP_NUM_THREADS="8"
module load gcc/8.3.0-gcc-7.5.0
# Content of run section
wget https://www.cs.virginia.edu/stream/FTP/Code/stream.c
$BUILDTEST_CC $BUILDTEST_CFLAGS stream.c -o stream
./stream
───────────────────────────────────────────────────────────────────────────────────────── stream_openmp_c/21d29876-2b7e-4bd4-999e-83f1234344e7 ─────────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: STREAM Microbenchmark C Test with OpenMP
State: PASS
Returncode: 0
Runtime: 8.217116 sec
Starttime: 2023/11/09 19:35:56
Endtime: 2023/11/09 19:36:05
Command: bash --norc --noprofile -eo pipefail stream_openmp_c_build.sh
Test Script: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/21d29876/stream_openmp_c.sh
Build Script: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/21d29876/stream_openmp_c_build.sh
Output File: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/21d29876/stream_openmp_c.out
Error File: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/21d29876/stream_openmp_c.err
Log File: /home/spack/buildtest/var/logs/buildtest_x2eeaaak.log
────────────────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/21d29876/stream_openmp_c.sh ───────────────────────────────────────────────────────────────
#!/bin/bash
set -eo pipefail
BUILDTEST_CC="gcc"
BUILDTEST_CXX="g++"
BUILDTEST_FC="gfortran"
BUILDTEST_CFLAGS="-fopenmp -O2"
BUILDTEST_CXXFLAGS=
BUILDTEST_FFLAGS=
BUILDTEST_CPPFLAGS=
BUILDTEST_LDFLAGS=
export OMP_NUM_THREADS="8"
module load gcc/6.5.0-gcc-7.5.0
# Content of run section
wget https://www.cs.virginia.edu/stream/FTP/Code/stream.c
$BUILDTEST_CC $BUILDTEST_CFLAGS stream.c -o stream
./stream
───────────────────────────────────────────────────────────────────────────────────────── stream_openmp_c/ddbeefba-7873-4663-ac24-9a1b7cfc0f06 ─────────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: STREAM Microbenchmark C Test with OpenMP
State: PASS
Returncode: 0
Runtime: 8.240962 sec
Starttime: 2023/11/09 19:35:56
Endtime: 2023/11/09 19:36:05
Command: bash --norc --noprofile -eo pipefail stream_openmp_c_build.sh
Test Script: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/ddbeefba/stream_openmp_c.sh
Build Script: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/ddbeefba/stream_openmp_c_build.sh
Output File: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/ddbeefba/stream_openmp_c.out
Error File: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/ddbeefba/stream_openmp_c.err
Log File: /home/spack/buildtest/var/logs/buildtest_x2eeaaak.log
────────────────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/stream_example/stream_openmp_c/ddbeefba/stream_openmp_c.sh ───────────────────────────────────────────────────────────────
#!/bin/bash
set -eo pipefail
BUILDTEST_CC="/usr/bin/gcc"
BUILDTEST_CXX="/usr/bin/g++"
BUILDTEST_FC="/usr/bin/gfortran"
BUILDTEST_CFLAGS="-fopenmp -O2"
BUILDTEST_CXXFLAGS=
BUILDTEST_FFLAGS=
BUILDTEST_CPPFLAGS=
BUILDTEST_LDFLAGS=
export OMP_NUM_THREADS="8"
# Content of run section
wget https://www.cs.virginia.edu/stream/FTP/Code/stream.c
$BUILDTEST_CC $BUILDTEST_CFLAGS stream.c -o stream
./stream
In the next example, we will run STREAM benchmark and define metrics from test results and assign them to metrics name
copy, add, scale, triad. Each metrics will be searched using regular expression against stdout stream
buildspecs:
stream_openmp_metrics:
executor: generic.local.bash
type: script
description: "STREAM Microbenchmark C Test with OpenMP"
tags: ["benchmark"]
compilers:
name: ['^(gcc_7.5.0)']
default:
gcc:
cflags: -fopenmp -O2
env:
OMP_NUM_THREADS: 16
run: |
wget https://www.cs.virginia.edu/stream/FTP/Code/stream.c
$BUILDTEST_CC $BUILDTEST_CFLAGS stream.c -o stream
./stream
metrics:
copy:
type: float
regex:
stream: stdout
exp: 'Copy:\s+(\S+)\s+.*'
item: 1
scale:
type: float
regex:
stream: stdout
exp: 'Scale:\s+(\S+)\s+.*'
item: 1
add:
type: float
regex:
stream: stdout
exp: 'Add:\s+(\S+)\s+.*'
item: 1
triad:
type: float
regex:
stream: stdout
exp: 'Triad:\s+(\S+)\s+.*'
item: 1
buildtest will record the metrics in the test report and buildtest inspect query will display metrics
if found in test. Shown below we see the output of the metrics and its corresponding values.
buildtest inspect query -o stream_openmp_metrics/
$ buildtest inspect query -o stream_openmp_metrics/
────────────────────────────────────────────────────────────────────────────────────── stream_openmp_metrics/b9659dd9-5112-47e3-bd0d-2ac976de3b71 ──────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: STREAM Microbenchmark C Test with OpenMP
State: PASS
Returncode: 0
Runtime: 6.233122 sec
Starttime: 2023/11/09 19:36:07
Endtime: 2023/11/09 19:36:14
Command: bash --norc --noprofile -eo pipefail stream_openmp_metrics_build.sh
Test Script: /home/spack/runs/generic.local.bash/stream_example_metrics/stream_openmp_metrics/b9659dd9/stream_openmp_metrics.sh
Build Script: /home/spack/runs/generic.local.bash/stream_example_metrics/stream_openmp_metrics/b9659dd9/stream_openmp_metrics_build.sh
Output File: /home/spack/runs/generic.local.bash/stream_example_metrics/stream_openmp_metrics/b9659dd9/stream_openmp_metrics.out
Error File: /home/spack/runs/generic.local.bash/stream_example_metrics/stream_openmp_metrics/b9659dd9/stream_openmp_metrics.err
Log File: /home/spack/buildtest/var/logs/buildtest_nx_trqpd.log
Metrics
┏━━━━━━━┳━━━━━━━━━┓
┃ Name ┃ Value ┃
┡━━━━━━━╇━━━━━━━━━┩
│ copy │ 20984.0 │
│ scale │ -7665.5 │
│ add │ 18479.8 │
│ triad │ 18977.0 │
└───────┴─────────┘
─────────────────────────────────────────────────── Output File: /home/spack/runs/generic.local.bash/stream_example_metrics/stream_openmp_metrics/b9659dd9/stream_openmp_metrics.out ───────────────────────────────────────────────────
==> Regenerating tcl module files
-------------------------------------------------------------
STREAM version $Revision: 5.10 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Array size = 10000000 (elements), Offset = 0 (elements)
Memory per array = 76.3 MiB (= 0.1 GiB).
Total memory required = 228.9 MiB (= 0.2 GiB).
Each kernel will be executed 10 times.
The *best* time for each kernel (excluding the first iteration)
will be used to compute the reported bandwidth.
-------------------------------------------------------------
Number of Threads requested = 16
Number of Threads counted = 16
-------------------------------------------------------------
Your clock granularity/precision appears to be 10 microseconds.
Each test below will take on the order of 8573 microseconds.
(= 857 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function Best Rate MB/s Avg time Min time Max time
Copy: 20984.0 0.009370 0.007625 0.011476
Scale: -7665.5 0.006650 -0.020873 0.011797
Add: 18479.8 0.015770 0.012987 0.018503
Triad: 18977.0 0.015389 0.012647 0.018043
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------