Note

Please see Tutorials Setup before you proceed with this section

Using Compiler Schema

The compiler schema is used for compilation of programs, currently we support single source file compilation. In order to use the compiler schema you must set type: compiler in your test. For more details see compiler schema docs

We assume the reader has basic understanding of Global Schema validation. Shown below is the schema header definition for compiler.schema.json:

{
  "$id": "compiler.schema.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "compiler schema",
  "description": "The compiler schema is of ``type: compiler`` in sub-schema which is used for compiling and running programs",
  "type": "object",
  "required": [
    "type",
    "source",
    "compilers",
    "executor"
  ],

The required fields for compiler schema are type, compilers, source and executor.

Compilation of Hello World

We will start out with compilation a Hello World program in Fortran using the GNU compiler. In this example we have a test called hello_f. The type: compiler is set to signify this test will be validated with compiler.schema.json.

The source property is used to specify the source code to compile, this can be a relative path to buildspec file or an absolute path. In this example the source file src/hello.f90 is relative path to where buildspec file is located. The compilers section declares compiler configuration, the name property is required that is used to search compiler names from our buildtest configuration via regular expression. In this example we use the builtin_gcc compiler as regular expression which is the system gcc compiler provided by buildtest. The default section specifies default compiler configuration applicable to a specific compiler group like gcc. Within each compiler group we can specify options like cflags, fflags, cxxflags, ldflags to customize compilation line.

buildspecs:
  hello_f:
    type: compiler
    description: "Hello World Fortran Compilation"
    executor: generic.local.bash
    tags: [tutorials, compile]
    source: "src/hello.f90"
    compilers:
      name: ["^(gcc_7.5.0)$"]
      default:
        gcc:
          fflags: -Wall

Shown below is an example build for this test.

buildtest build -b /home/spack/buildtest/examples/compilers/gnu_hello_fortran.yml
$ buildtest build -b /home/spack/buildtest/examples/compilers/gnu_hello_fortran.yml 
╭────────────────────────── buildtest summary ──────────────────────────╮
│                                                                       │
│ User:               spack                                             │
│ Hostname:           356ea13b3433                                      │
│ Platform:           Linux                                             │
│ Current Time:       2022/06/29 17:27:22                               │
│ buildtest path:     /home/spack/buildtest/bin/buildtest               │
│ buildtest version:  0.14.0                                            │
│ python path:        /home/spack/pyenv/buildtest/bin/python3           │
│ python version:     3.8.6                                             │
│ Configuration File:                                                   │
│ /home/spack/buildtest/buildtest/settings/spack_container.yml          │
│ Test Directory:     /home/spack/runs                                  │
│ Report File:        /home/spack/buildtest/var/report.json             │
│ Command:            /home/spack/buildtest/bin/buildtest build -b      │
│ /home/spack/buildtest/examples/compilers/gnu_hello_fortran.yml        │
│                                                                       │
╰───────────────────────────────────────────────────────────────────────╯
────────────────────────  Discovering Buildspecs ────────────────────────
                      Discovered buildspecs                       
╔════════════════════════════════════════════════════════════════╗
║ buildspec                                                      ║
╟────────────────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/compilers/gnu_hello_fortran.yml ║
╚════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────── Parsing Buildspecs ───────────────────────────
Buildtest will parse 1 buildspecs
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/compilers/gnu_hello_fortran.yml: VALID
Total builder objects created: 1
Total compiler builder: 1
Total script builder: 0
Total spack builder: 0
                        Compiler Builder Details                         
┏━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder  ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descript… ┃ buildsp… ┃
┡━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━┩
│ hello_f… │ generic… │ gcc_7.5… │ None  │ None  │ Hello     │ /home/s… │
│          │          │          │       │       │ World     │          │
│          │          │          │       │       │ Fortran   │          │
│          │          │          │       │       │ Compilat… │          │
└──────────┴──────────┴──────────┴───────┴───────┴───────────┴──────────┘
───────────────────────────── Building Test ─────────────────────────────
hello_f/2fbc7f44: Creating test directory: 
/home/spack/runs/generic.local.bash/gnu_hello_fortran/hello_f/2fbc7f44
hello_f/2fbc7f44: Creating the stage directory: /home/spack/runs/generic.
local.bash/gnu_hello_fortran/hello_f/2fbc7f44/stage
hello_f/2fbc7f44: Writing build script: /home/spack/runs/generic.local.ba
sh/gnu_hello_fortran/hello_f/2fbc7f44/hello_f_build.sh
───────────────────────────── Running Tests ─────────────────────────────
Spawning 4 processes for processing builders
────────────────────────────── Iteration 1 ──────────────────────────────
hello_f/2fbc7f44 does not have any dependencies adding test to queue
hello_f/2fbc7f44: Running Test via command: bash --norc --noprofile -eo 
pipefail hello_f_build.sh
hello_f/2fbc7f44: Test completed in 0.144778 seconds
hello_f/2fbc7f44: Test completed with returncode: 0
hello_f/2fbc7f44: Writing output file -  /home/spack/runs/generic.local.b
ash/gnu_hello_fortran/hello_f/2fbc7f44/hello_f.out
hello_f/2fbc7f44: Writing error file - /home/spack/runs/generic.local.bas
h/gnu_hello_fortran/hello_f/2fbc7f44/hello_f.err
In this iteration we are going to run the following tests: [hello_f/2fbc7f44]
                              Test Summary                               
┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃            ┃            ┃        ┃ checks (Re ┃            ┃          ┃
┃            ┃            ┃        ┃ turnCode,  ┃            ┃          ┃
┃            ┃            ┃        ┃ Regex,     ┃            ┃          ┃
┃ builder    ┃ executor   ┃ status ┃ Runtime)   ┃ returnCode ┃ runtime  ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ hello_f/2f │ generic.l… │ PASS   │ N/A N/A    │ 0          │ 0.144778 │
│ bc7f44     │            │        │ N/A        │            │          │
└────────────┴────────────┴────────┴────────────┴────────────┴──────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to /home/spack/buildtest/var/report.json
Writing Logfile to: /home/spack/buildtest/var/logs/buildtest_o1jub4w4.log

The generated test for test name hello_f is the following:

buildtest inspect query -t hello_f
$ buildtest inspect query -t hello_f 
───────────── hello_f/2fbc7f44-7762-473a-b8ae-95a327219493 ──────────────
Executor: generic.local.bash
Description: Hello World Fortran Compilation
State: PASS
Returncode: 0
Runtime: 0.144778 sec
Starttime: 2022/06/29 17:27:22
Endtime: 2022/06/29 17:27:22
Command: bash --norc --noprofile -eo pipefail hello_f_build.sh
Test Script: /home/spack/runs/generic.local.bash/gnu_hello_fortran/hello_
f/2fbc7f44/hello_f.sh
Build Script: /home/spack/runs/generic.local.bash/gnu_hello_fortran/hello
_f/2fbc7f44/hello_f_build.sh
Output File: /home/spack/runs/generic.local.bash/gnu_hello_fortran/hello_
f/2fbc7f44/hello_f.out
Error File: /home/spack/runs/generic.local.bash/gnu_hello_fortran/hello_f
/2fbc7f44/hello_f.err
Log File: /home/spack/buildtest/var/logs/buildtest_o1jub4w4.log
─ Test File: /home/spack/runs/generic.local.bash/gnu_hello_fortran/hel… ─
#!/bin/bash                                                              
                                                                         
                                                                         
# name of executable                                                     
_EXEC=hello.f90.exe                                                      
# Compilation Line                                                       
/usr/bin/gfortran -Wall -o $_EXEC /home/spack/buildtest/examples/compiler
                                                                         
                                                                         
# Run executable                                                         
./$_EXEC

How does buildtest detect programming language?

buildtest will detect the file extension of source file based specified by source property to detect programming language and generate the appropriate C, C++, or Fortran compilation line based on language detected.

Shown below is the file extension table buildtest uses for determining the programming language.

File Extension Language Mapping

Language

File Extension

C

.c

C++

.cc .cxx .cpp .c++

Fortran

.f90 .F90 .f95 .f .F .FOR .for .FTN .ftn

How does buildtest search for compiler

Compilers are defined in your configuration file that is used by buildtest to search compilers. In configuring compilers you will learn how one can define compiler, each compiler instance will have a unique name and be under one of the compiler groups like gcc, intel, cray, etc…

You can see the compiler declaration from our configuration file by running buildtest config compilers -y which will display compiler settings in YAML format. Note that for each compiler instance one can define name of compiler and path to cc, fc, cxx wrapper. In addition one can specify module property to map compiler instance to modulefile. If module property is defined you can specify list of modules to load via load property and buildtest will automatically load these modules when using the compiler.

buildtest config compilers -y
$ buildtest config compilers -y 
gcc:
  clang_6.0.0:
    cc: /usr/bin/clang
    cxx: /usr/bin/clang++
    fc: None
  gcc_6.5.0:
    cc: gcc
    cxx: g++
    fc: gfortran
    module:
      load:
      - gcc/6.5.0-gcc-7.5.0
  gcc_7.5.0:
    cc: /usr/bin/gcc
    cxx: /usr/bin/g++
    fc: /usr/bin/gfortran
  gcc_8.3.0:
    cc: gcc
    cxx: g++
    fc: gfortran
    module:
      load:
      - gcc/8.3.0-gcc-7.5.0

buildtest selects compiler based on name property which is a list of regular expression applied for available compilers defined in buildtest configuration. In this next example, we will compile an OpenACC code that will compute vector addition. We specify all gcc compilers are used for building this test. The -fopenacc compiler flag enables GNU compilers to compile OpenACC code, we can set this via fflags property. We can specify linker flags via ldflags during compilation, this code requires we specify -lm flag to link with math library.

buildspecs:
  vecadd_gnu:
    type: compiler
    description: Vector Addition example with GNU compiler
    tags: [tutorials, compile]
    executor: generic.local.bash
    source: src/vecAdd.c
    compilers:
      name: ["^(gcc)"]
      default:
        gcc:
          cflags: -fopenacc
          ldflags: -lm

We expect buildtest to generate one test per gcc compiler as you can see below.

buildtest build -b /home/spack/buildtest/examples/compilers/vecadd.yml
$ buildtest build -b /home/spack/buildtest/examples/compilers/vecadd.yml 
╭────────────────────────── buildtest summary ──────────────────────────╮
│                                                                       │
│ User:               spack                                             │
│ Hostname:           356ea13b3433                                      │
│ Platform:           Linux                                             │
│ Current Time:       2022/06/29 17:27:24                               │
│ buildtest path:     /home/spack/buildtest/bin/buildtest               │
│ buildtest version:  0.14.0                                            │
│ python path:        /home/spack/pyenv/buildtest/bin/python3           │
│ python version:     3.8.6                                             │
│ Configuration File:                                                   │
│ /home/spack/buildtest/buildtest/settings/spack_container.yml          │
│ Test Directory:     /home/spack/runs                                  │
│ Report File:        /home/spack/buildtest/var/report.json             │
│ Command:            /home/spack/buildtest/bin/buildtest build -b      │
│ /home/spack/buildtest/examples/compilers/vecadd.yml                   │
│                                                                       │
╰───────────────────────────────────────────────────────────────────────╯
────────────────────────  Discovering Buildspecs ────────────────────────
                 Discovered buildspecs                 
╔═════════════════════════════════════════════════════╗
║ buildspec                                           ║
╟─────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/compilers/vecadd.yml ║
╚═════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────── Parsing Buildspecs ───────────────────────────
Buildtest will parse 1 buildspecs
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/compilers/vecadd.yml: VALID
Total builder objects created: 3
Total compiler builder: 3
Total script builder: 0
Total spack builder: 0
                        Compiler Builder Details                         
┏━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder  ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descript… ┃ buildsp… ┃
┡━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━┩
│ vecadd_… │ generic… │ gcc_7.5… │ None  │ None  │ Vector    │ /home/s… │
│          │          │          │       │       │ Addition  │          │
│          │          │          │       │       │ example   │          │
│          │          │          │       │       │ with GNU  │          │
│          │          │          │       │       │ compiler  │          │
├──────────┼──────────┼──────────┼───────┼───────┼───────────┼──────────┤
│ vecadd_… │ generic… │ gcc_6.5… │ None  │ None  │ Vector    │ /home/s… │
│          │          │          │       │       │ Addition  │          │
│          │          │          │       │       │ example   │          │
│          │          │          │       │       │ with GNU  │          │
│          │          │          │       │       │ compiler  │          │
├──────────┼──────────┼──────────┼───────┼───────┼───────────┼──────────┤
│ vecadd_… │ generic… │ gcc_8.3… │ None  │ None  │ Vector    │ /home/s… │
│          │          │          │       │       │ Addition  │          │
│          │          │          │       │       │ example   │          │
│          │          │          │       │       │ with GNU  │          │
│          │          │          │       │       │ compiler  │          │
└──────────┴──────────┴──────────┴───────┴───────┴───────────┴──────────┘
───────────────────────────── Building Test ─────────────────────────────
vecadd_gnu/0ff5e1ec: Creating test directory: 
/home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/0ff5e1ec
vecadd_gnu/0ff5e1ec: Creating the stage directory: 
/home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/0ff5e1ec/stage
vecadd_gnu/0ff5e1ec: Writing build script: /home/spack/runs/generic.local
.bash/vecadd/vecadd_gnu/0ff5e1ec/vecadd_gnu_build.sh
vecadd_gnu/ecf8cee7: Creating test directory: 
/home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/ecf8cee7
vecadd_gnu/ecf8cee7: Creating the stage directory: 
/home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/ecf8cee7/stage
vecadd_gnu/ecf8cee7: Writing build script: /home/spack/runs/generic.local
.bash/vecadd/vecadd_gnu/ecf8cee7/vecadd_gnu_build.sh
vecadd_gnu/6f7d8ad2: Creating test directory: 
/home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/6f7d8ad2
vecadd_gnu/6f7d8ad2: Creating the stage directory: 
/home/spack/runs/generic.local.bash/vecadd/vecadd_gnu/6f7d8ad2/stage
vecadd_gnu/6f7d8ad2: Writing build script: /home/spack/runs/generic.local
.bash/vecadd/vecadd_gnu/6f7d8ad2/vecadd_gnu_build.sh
───────────────────────────── Running Tests ─────────────────────────────
Spawning 4 processes for processing builders
────────────────────────────── Iteration 1 ──────────────────────────────
vecadd_gnu/ecf8cee7 does not have any dependencies adding test to queue
vecadd_gnu/6f7d8ad2 does not have any dependencies adding test to queue
vecadd_gnu/0ff5e1ec does not have any dependencies adding test to queue
vecadd_gnu/6f7d8ad2: Running Test via command: bash --norc --noprofile 
-eo pipefail vecadd_gnu_build.sh
vecadd_gnu/0ff5e1ec: Running Test via command: bash --norc --noprofile 
-eo pipefail vecadd_gnu_build.sh
vecadd_gnu/ecf8cee7: Running Test via command: bash --norc --noprofile 
-eo pipefail vecadd_gnu_build.sh
vecadd_gnu/0ff5e1ec: Test completed in 0.117218 seconds
vecadd_gnu/6f7d8ad2: Test completed in 0.119933 seconds
vecadd_gnu/0ff5e1ec: Test completed with returncode: 0
vecadd_gnu/ecf8cee7: Test completed in 0.116401 seconds
vecadd_gnu/6f7d8ad2: Test completed with returncode: 0
vecadd_gnu/ecf8cee7: Test completed with returncode: 0
vecadd_gnu/0ff5e1ec: Writing output file -  /home/spack/runs/generic.loca
l.bash/vecadd/vecadd_gnu/0ff5e1ec/vecadd_gnu.out
vecadd_gnu/6f7d8ad2: Writing output file -  /home/spack/runs/generic.loca
l.bash/vecadd/vecadd_gnu/6f7d8ad2/vecadd_gnu.out
vecadd_gnu/0ff5e1ec: Writing error file - /home/spack/runs/generic.local.
bash/vecadd/vecadd_gnu/0ff5e1ec/vecadd_gnu.err
vecadd_gnu/6f7d8ad2: Writing error file - /home/spack/runs/generic.local.
bash/vecadd/vecadd_gnu/6f7d8ad2/vecadd_gnu.err
vecadd_gnu/ecf8cee7: Writing output file -  /home/spack/runs/generic.loca
l.bash/vecadd/vecadd_gnu/ecf8cee7/vecadd_gnu.out
vecadd_gnu/ecf8cee7: Writing error file - /home/spack/runs/generic.local.
bash/vecadd/vecadd_gnu/ecf8cee7/vecadd_gnu.err
In this iteration we are going to run the following tests: [vecadd_gnu/ecf8cee7, vecadd_gnu/6f7d8ad2, vecadd_gnu/0ff5e1ec]
                              Test Summary                               
┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃            ┃            ┃        ┃ checks (Re ┃            ┃          ┃
┃            ┃            ┃        ┃ turnCode,  ┃            ┃          ┃
┃            ┃            ┃        ┃ Regex,     ┃            ┃          ┃
┃ builder    ┃ executor   ┃ status ┃ Runtime)   ┃ returnCode ┃ runtime  ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ vecadd_gnu │ generic.l… │ PASS   │ N/A N/A    │ 0          │ 0.116401 │
│ /ecf8cee7  │            │        │ N/A        │            │          │
├────────────┼────────────┼────────┼────────────┼────────────┼──────────┤
│ vecadd_gnu │ generic.l… │ PASS   │ N/A N/A    │ 0          │ 0.119933 │
│ /6f7d8ad2  │            │        │ N/A        │            │          │
├────────────┼────────────┼────────┼────────────┼────────────┼──────────┤
│ vecadd_gnu │ generic.l… │ PASS   │ N/A N/A    │ 0          │ 0.117218 │
│ /0ff5e1ec  │            │        │ N/A        │            │          │
└────────────┴────────────┴────────┴────────────┴────────────┴──────────┘



Passed Tests: 3/3 Percentage: 100.000%
Failed Tests: 0/3 Percentage: 0.000%


Adding 3 test results to /home/spack/buildtest/var/report.json
Writing Logfile to: /home/spack/buildtest/var/logs/buildtest_kk6d_zts.log

Customize Compiler Option

We can specify custom compiler options per compiler instance using the config property. In this next example, we have a Hello World C example that will specify different cflags based on compiler name. We can specify default compiler setting via default property. In this example the default is -O1. The config section can be used to specify custom compiler options for each compiler name.

buildspecs:
  hello_c:
    type: compiler
    description: "Hello World C Compilation"
    executor: generic.local.bash
    tags: [tutorials, compile]
    source: "src/hello.c"
    compilers:
      name: ["^(gcc)"]
      default:
        gcc:
          cflags: -O1
      config:
        gcc_6.5.0:
          cflags: -O2
        gcc_8.3.0:
          cflags: -O3

Let’s build this test, we will see there is one builder instance for each compiler.

buildtest build -b /home/spack/buildtest/examples/compilers/gnu_hello_c.yml
$ buildtest build -b /home/spack/buildtest/examples/compilers/gnu_hello_c.yml 
╭────────────────────────── buildtest summary ──────────────────────────╮
│                                                                       │
│ User:               spack                                             │
│ Hostname:           356ea13b3433                                      │
│ Platform:           Linux                                             │
│ Current Time:       2022/06/29 17:27:25                               │
│ buildtest path:     /home/spack/buildtest/bin/buildtest               │
│ buildtest version:  0.14.0                                            │
│ python path:        /home/spack/pyenv/buildtest/bin/python3           │
│ python version:     3.8.6                                             │
│ Configuration File:                                                   │
│ /home/spack/buildtest/buildtest/settings/spack_container.yml          │
│ Test Directory:     /home/spack/runs                                  │
│ Report File:        /home/spack/buildtest/var/report.json             │
│ Command:            /home/spack/buildtest/bin/buildtest build -b      │
│ /home/spack/buildtest/examples/compilers/gnu_hello_c.yml              │
│                                                                       │
╰───────────────────────────────────────────────────────────────────────╯
────────────────────────  Discovering Buildspecs ────────────────────────
                   Discovered buildspecs                    
╔══════════════════════════════════════════════════════════╗
║ buildspec                                                ║
╟──────────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/compilers/gnu_hello_c.yml ║
╚══════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────── Parsing Buildspecs ───────────────────────────
Buildtest will parse 1 buildspecs
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/compilers/gnu_hello_c.yml: VALID
Total builder objects created: 3
Total compiler builder: 3
Total script builder: 0
Total spack builder: 0
                        Compiler Builder Details                         
┏━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder  ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descript… ┃ buildsp… ┃
┡━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━┩
│ hello_c… │ generic… │ gcc_7.5… │ None  │ None  │ Hello     │ /home/s… │
│          │          │          │       │       │ World C   │          │
│          │          │          │       │       │ Compilat… │          │
├──────────┼──────────┼──────────┼───────┼───────┼───────────┼──────────┤
│ hello_c… │ generic… │ gcc_6.5… │ None  │ None  │ Hello     │ /home/s… │
│          │          │          │       │       │ World C   │          │
│          │          │          │       │       │ Compilat… │          │
├──────────┼──────────┼──────────┼───────┼───────┼───────────┼──────────┤
│ hello_c… │ generic… │ gcc_8.3… │ None  │ None  │ Hello     │ /home/s… │
│          │          │          │       │       │ World C   │          │
│          │          │          │       │       │ Compilat… │          │
└──────────┴──────────┴──────────┴───────┴───────┴───────────┴──────────┘
───────────────────────────── Building Test ─────────────────────────────
hello_c/896a99ad: Creating test directory: 
/home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/896a99ad
hello_c/896a99ad: Creating the stage directory: 
/home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/896a99ad/stage
hello_c/896a99ad: Writing build script: /home/spack/runs/generic.local.ba
sh/gnu_hello_c/hello_c/896a99ad/hello_c_build.sh
hello_c/50912101: Creating test directory: 
/home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/50912101
hello_c/50912101: Creating the stage directory: 
/home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/50912101/stage
hello_c/50912101: Writing build script: /home/spack/runs/generic.local.ba
sh/gnu_hello_c/hello_c/50912101/hello_c_build.sh
hello_c/40a1917f: Creating test directory: 
/home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/40a1917f
hello_c/40a1917f: Creating the stage directory: 
/home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/40a1917f/stage
hello_c/40a1917f: Writing build script: /home/spack/runs/generic.local.ba
sh/gnu_hello_c/hello_c/40a1917f/hello_c_build.sh
───────────────────────────── Running Tests ─────────────────────────────
Spawning 4 processes for processing builders
────────────────────────────── Iteration 1 ──────────────────────────────
hello_c/50912101 does not have any dependencies adding test to queue
hello_c/896a99ad does not have any dependencies adding test to queue
hello_c/40a1917f does not have any dependencies adding test to queue
hello_c/50912101: Running Test via command: bash --norc --noprofile -eo 
pipefail hello_c_build.sh
hello_c/896a99ad: Running Test via command: bash --norc --noprofile -eo 
pipefail hello_c_build.sh
hello_c/40a1917f: Running Test via command: bash --norc --noprofile -eo 
pipefail hello_c_build.sh
hello_c/50912101: Test completed in 0.079543 seconds
hello_c/50912101: Test completed with returncode: 0
hello_c/50912101: Writing output file -  /home/spack/runs/generic.local.b
ash/gnu_hello_c/hello_c/50912101/hello_c.out
hello_c/50912101: Writing error file - /home/spack/runs/generic.local.bas
h/gnu_hello_c/hello_c/50912101/hello_c.err
hello_c/896a99ad: Test completed in 0.086262 seconds
hello_c/896a99ad: Test completed with returncode: 0
hello_c/896a99ad: Writing output file -  /home/spack/runs/generic.local.b
ash/gnu_hello_c/hello_c/896a99ad/hello_c.out
hello_c/896a99ad: Writing error file - /home/spack/runs/generic.local.bas
h/gnu_hello_c/hello_c/896a99ad/hello_c.err
hello_c/40a1917f: Test completed in 0.095595 seconds
hello_c/40a1917f: Test completed with returncode: 0
hello_c/40a1917f: Writing output file -  /home/spack/runs/generic.local.b
ash/gnu_hello_c/hello_c/40a1917f/hello_c.out
hello_c/40a1917f: Writing error file - /home/spack/runs/generic.local.bas
h/gnu_hello_c/hello_c/40a1917f/hello_c.err
In this iteration we are going to run the following tests: [hello_c/50912101, hello_c/896a99ad, hello_c/40a1917f]
                              Test Summary                               
┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃            ┃            ┃        ┃ checks (Re ┃            ┃          ┃
┃            ┃            ┃        ┃ turnCode,  ┃            ┃          ┃
┃            ┃            ┃        ┃ Regex,     ┃            ┃          ┃
┃ builder    ┃ executor   ┃ status ┃ Runtime)   ┃ returnCode ┃ runtime  ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ hello_c/89 │ generic.l… │ PASS   │ N/A N/A    │ 0          │ 0.086262 │
│ 6a99ad     │            │        │ N/A        │            │          │
├────────────┼────────────┼────────┼────────────┼────────────┼──────────┤
│ hello_c/50 │ generic.l… │ PASS   │ N/A N/A    │ 0          │ 0.079543 │
│ 912101     │            │        │ N/A        │            │          │
├────────────┼────────────┼────────┼────────────┼────────────┼──────────┤
│ hello_c/40 │ generic.l… │ PASS   │ N/A N/A    │ 0          │ 0.095595 │
│ a1917f     │            │        │ N/A        │            │          │
└────────────┴────────────┴────────┴────────────┴────────────┴──────────┘



Passed Tests: 3/3 Percentage: 100.000%
Failed Tests: 0/3 Percentage: 0.000%


Adding 3 test results to /home/spack/buildtest/var/report.json
Writing Logfile to: /home/spack/buildtest/var/logs/buildtest_u8og58y9.log

If we inspect the following test, we see each test has its own compiler flags. The default cflag is -O1 while gcc_6.5.0 will use -O2 and gcc_8.3.0 will use -O3.

buildtest inspect query -t hello_c/
$ buildtest inspect query -t hello_c/ 
───────────── hello_c/896a99ad-ab5f-4e9e-9044-fd91b3dd5129 ──────────────
Executor: generic.local.bash
Description: Hello World C Compilation
State: PASS
Returncode: 0
Runtime: 0.086262 sec
Starttime: 2022/06/29 17:27:25
Endtime: 2022/06/29 17:27:25
Command: bash --norc --noprofile -eo pipefail hello_c_build.sh
Test Script: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/896a
99ad/hello_c.sh
Build Script: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/896
a99ad/hello_c_build.sh
Output File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/896a
99ad/hello_c.out
Error File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/896a9
9ad/hello_c.err
Log File: /home/spack/buildtest/var/logs/buildtest_u8og58y9.log
─ Test File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/8… ─
#!/bin/bash                                                              
                                                                         
                                                                         
# name of executable                                                     
_EXEC=hello.c.exe                                                        
# Compilation Line                                                       
/usr/bin/gcc -O1 -o $_EXEC /home/spack/buildtest/examples/compilers/src/h
                                                                         
                                                                         
# Run executable                                                         
./$_EXEC                                                                 
                                                                         
                                                                         
───────────── hello_c/50912101-f4da-4111-aa44-dec65ca80441 ──────────────
Executor: generic.local.bash
Description: Hello World C Compilation
State: PASS
Returncode: 0
Runtime: 0.079543 sec
Starttime: 2022/06/29 17:27:25
Endtime: 2022/06/29 17:27:25
Command: bash --norc --noprofile -eo pipefail hello_c_build.sh
Test Script: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/5091
2101/hello_c.sh
Build Script: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/509
12101/hello_c_build.sh
Output File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/5091
2101/hello_c.out
Error File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/50912
101/hello_c.err
Log File: /home/spack/buildtest/var/logs/buildtest_u8og58y9.log
─ Test File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/5… ─
#!/bin/bash                                                              
                                                                         
                                                                         
# name of executable                                                     
_EXEC=hello.c.exe                                                        
# Loading modules                                                        
module load gcc/6.5.0-gcc-7.5.0                                          
# Compilation Line                                                       
gcc -O2 -o $_EXEC /home/spack/buildtest/examples/compilers/src/hello.c   
                                                                         
                                                                         
# Run executable                                                         
./$_EXEC                                                                 
                                                                         
                                                                         
───────────── hello_c/40a1917f-8ed9-4bae-907c-0655e6a1af26 ──────────────
Executor: generic.local.bash
Description: Hello World C Compilation
State: PASS
Returncode: 0
Runtime: 0.095595 sec
Starttime: 2022/06/29 17:27:25
Endtime: 2022/06/29 17:27:25
Command: bash --norc --noprofile -eo pipefail hello_c_build.sh
Test Script: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/40a1
917f/hello_c.sh
Build Script: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/40a
1917f/hello_c_build.sh
Output File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/40a1
917f/hello_c.out
Error File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/40a19
17f/hello_c.err
Log File: /home/spack/buildtest/var/logs/buildtest_u8og58y9.log
─ Test File: /home/spack/runs/generic.local.bash/gnu_hello_c/hello_c/4… ─
#!/bin/bash                                                              
                                                                         
                                                                         
# name of executable                                                     
_EXEC=hello.c.exe                                                        
# Loading modules                                                        
module load gcc/8.3.0-gcc-7.5.0                                          
# Compilation Line                                                       
gcc -O3 -o $_EXEC /home/spack/buildtest/examples/compilers/src/hello.c   
                                                                         
                                                                         
# Run executable                                                         
./$_EXEC

Excluding Compilers

The exclude property allows one to exclude compilers upon discovery which is a list of compiler names that will be removed prior to building test. buildtest will exclude any compilers if they were found based on regular expression via name property. In this next example, we will exclude gcc_6.5.0 compiler from building

buildspecs:
  vecadd_gnu_exclude:
    type: compiler
    description: Vector Addition example with GNU compilers but exclude gcc_6.5.0
    tags: [tutorials, compile]
    executor: generic.local.bash
    source: src/vecAdd.c
    compilers:
      name: ["^(gcc)"]
      exclude: [gcc_6.5.0]
      default:
        gcc:
          cflags: -fopenacc
          ldflags: -lm

Now if we run this test, we will notice that there is only one build for this test even though buildtest discovered both gcc_6.5.0 and gcc_8.3.0 compilers.

buildtest build -b /home/spack/buildtest/examples/compilers/compiler_exclude.yml
$ buildtest build -b /home/spack/buildtest/examples/compilers/compiler_exclude.yml 
╭────────────────────────── buildtest summary ──────────────────────────╮
│                                                                       │
│ User:               spack                                             │
│ Hostname:           356ea13b3433                                      │
│ Platform:           Linux                                             │
│ Current Time:       2022/06/29 17:27:27                               │
│ buildtest path:     /home/spack/buildtest/bin/buildtest               │
│ buildtest version:  0.14.0                                            │
│ python path:        /home/spack/pyenv/buildtest/bin/python3           │
│ python version:     3.8.6                                             │
│ Configuration File:                                                   │
│ /home/spack/buildtest/buildtest/settings/spack_container.yml          │
│ Test Directory:     /home/spack/runs                                  │
│ Report File:        /home/spack/buildtest/var/report.json             │
│ Command:            /home/spack/buildtest/bin/buildtest build -b      │
│ /home/spack/buildtest/examples/compilers/compiler_exclude.yml         │
│                                                                       │
╰───────────────────────────────────────────────────────────────────────╯
────────────────────────  Discovering Buildspecs ────────────────────────
                      Discovered buildspecs                      
╔═══════════════════════════════════════════════════════════════╗
║ buildspec                                                     ║
╟───────────────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/compilers/compiler_exclude.yml ║
╚═══════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────── Parsing Buildspecs ───────────────────────────
Buildtest will parse 1 buildspecs
vecadd_gnu_exclude: Excluding compiler gcc_6.5.0 during test generation
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/compilers/compiler_exclude.yml: VALID
Total builder objects created: 2
Total compiler builder: 2
Total script builder: 0
Total spack builder: 0
                        Compiler Builder Details                         
┏━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder  ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descript… ┃ buildsp… ┃
┡━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━┩
│ vecadd_… │ generic… │ gcc_7.5… │ None  │ None  │ Vector    │ /home/s… │
│          │          │          │       │       │ Addition  │          │
│          │          │          │       │       │ example   │          │
│          │          │          │       │       │ with GNU  │          │
│          │          │          │       │       │ compilers │          │
│          │          │          │       │       │ but       │          │
│          │          │          │       │       │ exclude   │          │
│          │          │          │       │       │ gcc_6.5.0 │          │
├──────────┼──────────┼──────────┼───────┼───────┼───────────┼──────────┤
│ vecadd_… │ generic… │ gcc_8.3… │ None  │ None  │ Vector    │ /home/s… │
│          │          │          │       │       │ Addition  │          │
│          │          │          │       │       │ example   │          │
│          │          │          │       │       │ with GNU  │          │
│          │          │          │       │       │ compilers │          │
│          │          │          │       │       │ but       │          │
│          │          │          │       │       │ exclude   │          │
│          │          │          │       │       │ gcc_6.5.0 │          │
└──────────┴──────────┴──────────┴───────┴───────┴───────────┴──────────┘
───────────────────────────── Building Test ─────────────────────────────
vecadd_gnu_exclude/0c481907: Creating test directory: /home/spack/runs/ge
neric.local.bash/compiler_exclude/vecadd_gnu_exclude/0c481907
vecadd_gnu_exclude/0c481907: Creating the stage directory: /home/spack/ru
ns/generic.local.bash/compiler_exclude/vecadd_gnu_exclude/0c481907/stage
vecadd_gnu_exclude/0c481907: Writing build script: /home/spack/runs/gener
ic.local.bash/compiler_exclude/vecadd_gnu_exclude/0c481907/vecadd_gnu_exc
lude_build.sh
vecadd_gnu_exclude/22fc63e3: Creating test directory: /home/spack/runs/ge
neric.local.bash/compiler_exclude/vecadd_gnu_exclude/22fc63e3
vecadd_gnu_exclude/22fc63e3: Creating the stage directory: /home/spack/ru
ns/generic.local.bash/compiler_exclude/vecadd_gnu_exclude/22fc63e3/stage
vecadd_gnu_exclude/22fc63e3: Writing build script: /home/spack/runs/gener
ic.local.bash/compiler_exclude/vecadd_gnu_exclude/22fc63e3/vecadd_gnu_exc
lude_build.sh
───────────────────────────── Running Tests ─────────────────────────────
Spawning 4 processes for processing builders
────────────────────────────── Iteration 1 ──────────────────────────────
vecadd_gnu_exclude/22fc63e3 does not have any dependencies adding test to
queue
vecadd_gnu_exclude/0c481907 does not have any dependencies adding test to
queue
vecadd_gnu_exclude/22fc63e3: Running Test via command: bash --norc 
--noprofile -eo pipefail vecadd_gnu_exclude_build.sh
vecadd_gnu_exclude/0c481907: Running Test via command: bash --norc 
--noprofile -eo pipefail vecadd_gnu_exclude_build.sh
vecadd_gnu_exclude/0c481907: Test completed in 0.086397 seconds
vecadd_gnu_exclude/0c481907: Test completed with returncode: 0
vecadd_gnu_exclude/0c481907: Writing output file -  /home/spack/runs/gene
ric.local.bash/compiler_exclude/vecadd_gnu_exclude/0c481907/vecadd_gnu_ex
clude.out
vecadd_gnu_exclude/0c481907: Writing error file - /home/spack/runs/generi
c.local.bash/compiler_exclude/vecadd_gnu_exclude/0c481907/vecadd_gnu_excl
ude.err
vecadd_gnu_exclude/22fc63e3: Test completed in 0.095962 seconds
vecadd_gnu_exclude/22fc63e3: Test completed with returncode: 0
vecadd_gnu_exclude/22fc63e3: Writing output file -  /home/spack/runs/gene
ric.local.bash/compiler_exclude/vecadd_gnu_exclude/22fc63e3/vecadd_gnu_ex
clude.out
vecadd_gnu_exclude/22fc63e3: Writing error file - /home/spack/runs/generi
c.local.bash/compiler_exclude/vecadd_gnu_exclude/22fc63e3/vecadd_gnu_excl
ude.err
In this iteration we are going to run the following tests: [vecadd_gnu_exclude/22fc63e3, vecadd_gnu_exclude/0c481907]
                              Test Summary                               
┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃            ┃            ┃        ┃ checks (Re ┃            ┃          ┃
┃            ┃            ┃        ┃ turnCode,  ┃            ┃          ┃
┃            ┃            ┃        ┃ Regex,     ┃            ┃          ┃
┃ builder    ┃ executor   ┃ status ┃ Runtime)   ┃ returnCode ┃ runtime  ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ vecadd_gnu │ generic.l… │ PASS   │ N/A N/A    │ 0          │ 0.095962 │
│ _exclude/2 │            │        │ N/A        │            │          │
│ 2fc63e3    │            │        │            │            │          │
├────────────┼────────────┼────────┼────────────┼────────────┼──────────┤
│ vecadd_gnu │ generic.l… │ PASS   │ N/A N/A    │ 0          │ 0.086397 │
│ _exclude/0 │            │        │ N/A        │            │          │
│ c481907    │            │        │            │            │          │
└────────────┴────────────┴────────┴────────────┴────────────┴──────────┘



Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%


Adding 2 test results to /home/spack/buildtest/var/report.json
Writing Logfile to: /home/spack/buildtest/var/logs/buildtest_ha4l_7bj.log

Setting environment variables

We can define environment variables using env property which is a list of key/value pair where key is environment name and value is a string assigned to the environment. The env property can be used in default or config section within a compiler instance. In this next example we have an OpenMP Hello World example which defines environment variable OMP_NUM_THREADS that controls number of OpenMP threads to use when running program. In this example we will set OMP_NUM_THREADS=2

buildspecs:
  openmp_hello_c_example:
    type: compiler
    description: OpenMP Hello World C example
    executor: generic.local.bash
    tags: [tutorials, compile]
    source: "src/hello_omp.c"
    compilers:
      name: ["^(gcc_7.5.0)"]
      default:
        gcc:
          cflags: -fopenmp
          env:
            OMP_NUM_THREADS: 2

Now let’s build this test.

buildtest build -b /home/spack/buildtest/examples/compilers/openmp_hello.yml
$ buildtest build -b /home/spack/buildtest/examples/compilers/openmp_hello.yml 
╭────────────────────────── buildtest summary ──────────────────────────╮
│                                                                       │
│ User:               spack                                             │
│ Hostname:           356ea13b3433                                      │
│ Platform:           Linux                                             │
│ Current Time:       2022/06/29 17:27:27                               │
│ buildtest path:     /home/spack/buildtest/bin/buildtest               │
│ buildtest version:  0.14.0                                            │
│ python path:        /home/spack/pyenv/buildtest/bin/python3           │
│ python version:     3.8.6                                             │
│ Configuration File:                                                   │
│ /home/spack/buildtest/buildtest/settings/spack_container.yml          │
│ Test Directory:     /home/spack/runs                                  │
│ Report File:        /home/spack/buildtest/var/report.json             │
│ Command:            /home/spack/buildtest/bin/buildtest build -b      │
│ /home/spack/buildtest/examples/compilers/openmp_hello.yml             │
│                                                                       │
╰───────────────────────────────────────────────────────────────────────╯
────────────────────────  Discovering Buildspecs ────────────────────────
                    Discovered buildspecs                    
╔═══════════════════════════════════════════════════════════╗
║ buildspec                                                 ║
╟───────────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/compilers/openmp_hello.yml ║
╚═══════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────── Parsing Buildspecs ───────────────────────────
Buildtest will parse 1 buildspecs
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/compilers/openmp_hello.yml: VALID
Total builder objects created: 1
Total compiler builder: 1
Total script builder: 0
Total spack builder: 0
                        Compiler Builder Details                         
┏━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder  ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descript… ┃ buildsp… ┃
┡━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━┩
│ openmp_… │ generic… │ gcc_7.5… │ None  │ None  │ OpenMP    │ /home/s… │
│          │          │          │       │       │ Hello     │          │
│          │          │          │       │       │ World C   │          │
│          │          │          │       │       │ example   │          │
└──────────┴──────────┴──────────┴───────┴───────┴───────────┴──────────┘
───────────────────────────── Building Test ─────────────────────────────
openmp_hello_c_example/67ae1ee5: Creating test directory: /home/spack/run
s/generic.local.bash/openmp_hello/openmp_hello_c_example/67ae1ee5
openmp_hello_c_example/67ae1ee5: Creating the stage directory: /home/spac
k/runs/generic.local.bash/openmp_hello/openmp_hello_c_example/67ae1ee5/st
age
openmp_hello_c_example/67ae1ee5: Writing build script: /home/spack/runs/g
eneric.local.bash/openmp_hello/openmp_hello_c_example/67ae1ee5/openmp_hel
lo_c_example_build.sh
───────────────────────────── Running Tests ─────────────────────────────
Spawning 4 processes for processing builders
────────────────────────────── Iteration 1 ──────────────────────────────
openmp_hello_c_example/67ae1ee5 does not have any dependencies adding 
test to queue
openmp_hello_c_example/67ae1ee5: Running Test via command: bash --norc 
--noprofile -eo pipefail openmp_hello_c_example_build.sh
openmp_hello_c_example/67ae1ee5: Test completed in 0.062474 seconds
openmp_hello_c_example/67ae1ee5: Test completed with returncode: 0
openmp_hello_c_example/67ae1ee5: Writing output file -  /home/spack/runs/
generic.local.bash/openmp_hello/openmp_hello_c_example/67ae1ee5/openmp_he
llo_c_example.out
openmp_hello_c_example/67ae1ee5: Writing error file - /home/spack/runs/ge
neric.local.bash/openmp_hello/openmp_hello_c_example/67ae1ee5/openmp_hell
o_c_example.err
In this iteration we are going to run the following tests: [openmp_hello_c_example/67ae1ee5]
                              Test Summary                               
┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃            ┃            ┃        ┃ checks (Re ┃            ┃          ┃
┃            ┃            ┃        ┃ turnCode,  ┃            ┃          ┃
┃            ┃            ┃        ┃ Regex,     ┃            ┃          ┃
┃ builder    ┃ executor   ┃ status ┃ Runtime)   ┃ returnCode ┃ runtime  ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ openmp_hel │ generic.l… │ PASS   │ N/A N/A    │ 0          │ 0.062474 │
│ lo_c_examp │            │        │ N/A        │            │          │
│ le/67ae1ee │            │        │            │            │          │
│ 5          │            │        │            │            │          │
└────────────┴────────────┴────────┴────────────┴────────────┴──────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to /home/spack/buildtest/var/report.json
Writing Logfile to: /home/spack/buildtest/var/logs/buildtest_szn8um4h.log

We can see the generated test using buildtest inspect query given the name of test. Take a close look at the export OMP_NUM_THREADS in the generated test.

buildtest inspect query -t openmp_hello_c_example
$ buildtest inspect query -t openmp_hello_c_example 
────── openmp_hello_c_example/67ae1ee5-241c-494b-b63c-3f68ceb7039d ──────
Executor: generic.local.bash
Description: OpenMP Hello World C example
State: PASS
Returncode: 0
Runtime: 0.062474 sec
Starttime: 2022/06/29 17:27:28
Endtime: 2022/06/29 17:27:28
Command: bash --norc --noprofile -eo pipefail 
openmp_hello_c_example_build.sh
Test Script: /home/spack/runs/generic.local.bash/openmp_hello/openmp_hell
o_c_example/67ae1ee5/openmp_hello_c_example.sh
Build Script: /home/spack/runs/generic.local.bash/openmp_hello/openmp_hel
lo_c_example/67ae1ee5/openmp_hello_c_example_build.sh
Output File: /home/spack/runs/generic.local.bash/openmp_hello/openmp_hell
o_c_example/67ae1ee5/openmp_hello_c_example.out
Error File: /home/spack/runs/generic.local.bash/openmp_hello/openmp_hello
_c_example/67ae1ee5/openmp_hello_c_example.err
Log File: /home/spack/buildtest/var/logs/buildtest_szn8um4h.log
─ Test File: /home/spack/runs/generic.local.bash/openmp_hello/openmp_h… ─
#!/bin/bash                                                              
                                                                         
                                                                         
# name of executable                                                     
_EXEC=hello_omp.c.exe                                                    
export OMP_NUM_THREADS="2"                                               
# Compilation Line                                                       
/usr/bin/gcc -fopenmp -o $_EXEC /home/spack/buildtest/examples/compilers/
                                                                         
                                                                         
# Run executable                                                         
./$_EXEC

We can define environment variables per compiler instance. buildtest will automatically override any key in defaults with one matched under config for the compiler name. In this next example, we will define OMP_NUM_THREADS to 4 for gcc_8.3.0 while the default is 2 for all gcc compilers.

buildspecs:
  override_environmentvars:
    type: compiler
    description: override default environment variables
    executor: generic.local.bash
    tags: [tutorials, compile]
    source: "src/hello_omp.c"
    compilers:
      name: ['^(gcc_7.5.0|gcc_8.3.0)']
      default:
        gcc:
          cflags: -fopenmp
          env:
            OMP_NUM_THREADS: 2
      config:
        gcc_8.3.0:
          env:
            OMP_NUM_THREADS: 4

We can build this test by running:

buildtest build -b $BUILDTEST_ROOT/examples/compilers/envvar_override.yml

Next, let’s see the generated test by running buildtest inspect query -d all -t override_environmentvars. The -d all will display all test records for override_environmentvars. Take a note that we have export OMP_NUM_THREADS=4 for gcc_8.3.0 test and export OMP_NUM_THREADS=2 for system gcc.

buildtest inspect query -t override_environmentvars/
$ buildtest inspect query -t override_environmentvars/ 
───── override_environmentvars/c552e0e2-b516-48e1-9563-b208047dd11c ─────
Executor: generic.local.bash
Description: override default environment variables
State: PASS
Returncode: 0
Runtime: 0.067774 sec
Starttime: 2022/06/29 17:27:30
Endtime: 2022/06/29 17:27:30
Command: bash --norc --noprofile -eo pipefail 
override_environmentvars_build.sh
Test Script: /home/spack/runs/generic.local.bash/envvar_override/override
_environmentvars/c552e0e2/override_environmentvars.sh
Build Script: /home/spack/runs/generic.local.bash/envvar_override/overrid
e_environmentvars/c552e0e2/override_environmentvars_build.sh
Output File: /home/spack/runs/generic.local.bash/envvar_override/override
_environmentvars/c552e0e2/override_environmentvars.out
Error File: /home/spack/runs/generic.local.bash/envvar_override/override_
environmentvars/c552e0e2/override_environmentvars.err
Log File: /home/spack/buildtest/var/logs/buildtest_nhaaefle.log
─ Test File: /home/spack/runs/generic.local.bash/envvar_override/overr… ─
#!/bin/bash                                                              
                                                                         
                                                                         
# name of executable                                                     
_EXEC=hello_omp.c.exe                                                    
export OMP_NUM_THREADS="2"                                               
# Compilation Line                                                       
/usr/bin/gcc -fopenmp -o $_EXEC /home/spack/buildtest/examples/compilers/
                                                                         
                                                                         
# Run executable                                                         
./$_EXEC                                                                 
                                                                         
                                                                         
───── override_environmentvars/d3fcdf11-03aa-4bb5-bec0-f67dbf550074 ─────
Executor: generic.local.bash
Description: override default environment variables
State: PASS
Returncode: 0
Runtime: 0.096485 sec
Starttime: 2022/06/29 17:27:30
Endtime: 2022/06/29 17:27:30
Command: bash --norc --noprofile -eo pipefail 
override_environmentvars_build.sh
Test Script: /home/spack/runs/generic.local.bash/envvar_override/override
_environmentvars/d3fcdf11/override_environmentvars.sh
Build Script: /home/spack/runs/generic.local.bash/envvar_override/overrid
e_environmentvars/d3fcdf11/override_environmentvars_build.sh
Output File: /home/spack/runs/generic.local.bash/envvar_override/override
_environmentvars/d3fcdf11/override_environmentvars.out
Error File: /home/spack/runs/generic.local.bash/envvar_override/override_
environmentvars/d3fcdf11/override_environmentvars.err
Log File: /home/spack/buildtest/var/logs/buildtest_nhaaefle.log
─ Test File: /home/spack/runs/generic.local.bash/envvar_override/overr… ─
#!/bin/bash                                                              
                                                                         
                                                                         
# name of executable                                                     
_EXEC=hello_omp.c.exe                                                    
export OMP_NUM_THREADS="4"                                               
# Loading modules                                                        
module load gcc/8.3.0-gcc-7.5.0                                          
# Compilation Line                                                       
gcc -fopenmp -o $_EXEC /home/spack/buildtest/examples/compilers/src/hello
                                                                         
                                                                         
# Run executable                                                         
./$_EXEC

Tweak how test are passed

The status property is used to determine how buildtest will determine status of test (PASS/FAIL). By default, an exitcode 0 is a PASS and anything else is FAIL.

Sometimes, it may be useful check output of test to determine using regular expression. This can be done via regex property. In this example, we define two tests, the first one defines status property in the default gcc group. This means all compilers that belong to gcc group will be matched based on returncode.

In second test we override the status property for a given compiler instance under the config section. The test is expected to produce output of final result: 1.000000 but we will specify a different value in order to show this test will fail.

buildspecs:
  default_status_returncode:
    type: compiler
    description: status check based on returncode
    tags: [tutorials, compile]
    executor: generic.local.bash
    source: src/vecAdd.c
    compilers:
      name: ["^(gcc)"]
      default:
        gcc:
          cflags: -fopenacc
          ldflags: -lm
          status:
            returncode: 0

  override_status_regex:
    type: compiler
    description: override 'status' property in compiler gcc_8.3.0 instance
    tags: [tutorials, compile]
    executor: generic.local.bash
    source: src/vecAdd.c
    compilers:
      name: ["^(gcc)"]
      default:
        gcc:
          cflags: -fopenacc
          ldflags: -lm
          status:
            returncode: 1
      config:
        gcc_8.3.0:
          status:
            regex:
              stream: stdout
              exp: "^final result: 0.99$"

If we build this test, we should expect the first test example should pass based on returncode 0. If the returncode doesn’t match buildtest will report failure. For the second test example, we have have specified test will pass if we get a returncode 1 based on default property however for gcc_8.3.0 compiler test we have defined a status property to check based on regular expression. We will expect both tests to fail since we will have a mismatch on returncode and regular expression

buildtest build -b /home/spack/buildtest/examples/compilers/compiler_status_regex.yml
$ buildtest build -b /home/spack/buildtest/examples/compilers/compiler_status_regex.yml 
╭────────────────────────── buildtest summary ──────────────────────────╮
│                                                                       │
│ User:               spack                                             │
│ Hostname:           356ea13b3433                                      │
│ Platform:           Linux                                             │
│ Current Time:       2022/06/29 17:27:32                               │
│ buildtest path:     /home/spack/buildtest/bin/buildtest               │
│ buildtest version:  0.14.0                                            │
│ python path:        /home/spack/pyenv/buildtest/bin/python3           │
│ python version:     3.8.6                                             │
│ Configuration File:                                                   │
│ /home/spack/buildtest/buildtest/settings/spack_container.yml          │
│ Test Directory:     /home/spack/runs                                  │
│ Report File:        /home/spack/buildtest/var/report.json             │
│ Command:            /home/spack/buildtest/bin/buildtest build -b      │
│ /home/spack/buildtest/examples/compilers/compiler_status_regex.yml    │
│                                                                       │
╰───────────────────────────────────────────────────────────────────────╯
────────────────────────  Discovering Buildspecs ────────────────────────
                        Discovered buildspecs                         
╔════════════════════════════════════════════════════════════════════╗
║ buildspec                                                          ║
╟────────────────────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/compilers/compiler_status_regex.yml ║
╚════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────── Parsing Buildspecs ───────────────────────────
Buildtest will parse 1 buildspecs
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/compilers/compiler_status_regex.yml: VALID
Total builder objects created: 6
Total compiler builder: 6
Total script builder: 0
Total spack builder: 0
                        Compiler Builder Details                         
┏━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder  ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descript… ┃ buildsp… ┃
┡━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━┩
│ default… │ generic… │ gcc_7.5… │ None  │ None  │ status    │ /home/s… │
│          │          │          │       │       │ check     │          │
│          │          │          │       │       │ based on  │          │
│          │          │          │       │       │ returnco… │          │
├──────────┼──────────┼──────────┼───────┼───────┼───────────┼──────────┤
│ default… │ generic… │ gcc_6.5… │ None  │ None  │ status    │ /home/s… │
│          │          │          │       │       │ check     │          │
│          │          │          │       │       │ based on  │          │
│          │          │          │       │       │ returnco… │          │
├──────────┼──────────┼──────────┼───────┼───────┼───────────┼──────────┤
│ default… │ generic… │ gcc_8.3… │ None  │ None  │ status    │ /home/s… │
│          │          │          │       │       │ check     │          │
│          │          │          │       │       │ based on  │          │
│          │          │          │       │       │ returnco… │          │
├──────────┼──────────┼──────────┼───────┼───────┼───────────┼──────────┤
│ overrid… │ generic… │ gcc_7.5… │ None  │ None  │ override  │ /home/s… │
│          │          │          │       │       │ 'status'  │          │
│          │          │          │       │       │ property  │          │
│          │          │          │       │       │ in        │          │
│          │          │          │       │       │ compiler  │          │
│          │          │          │       │       │ gcc_8.3.0 │          │
│          │          │          │       │       │ instance  │          │
├──────────┼──────────┼──────────┼───────┼───────┼───────────┼──────────┤
│ overrid… │ generic… │ gcc_6.5… │ None  │ None  │ override  │ /home/s… │
│          │          │          │       │       │ 'status'  │          │
│          │          │          │       │       │ property  │          │
│          │          │          │       │       │ in        │          │
│          │          │          │       │       │ compiler  │          │
│          │          │          │       │       │ gcc_8.3.0 │          │
│          │          │          │       │       │ instance  │          │
├──────────┼──────────┼──────────┼───────┼───────┼───────────┼──────────┤
│ overrid… │ generic… │ gcc_8.3… │ None  │ None  │ override  │ /home/s… │
│          │          │          │       │       │ 'status'  │          │
│          │          │          │       │       │ property  │          │
│          │          │          │       │       │ in        │          │
│          │          │          │       │       │ compiler  │          │
│          │          │          │       │       │ gcc_8.3.0 │          │
│          │          │          │       │       │ instance  │          │
└──────────┴──────────┴──────────┴───────┴───────┴───────────┴──────────┘
───────────────────────────── Building Test ─────────────────────────────
default_status_returncode/146a17db: Creating test directory: /home/spack/
runs/generic.local.bash/compiler_status_regex/default_status_returncode/1
46a17db
default_status_returncode/146a17db: Creating the stage directory: /home/s
pack/runs/generic.local.bash/compiler_status_regex/default_status_returnc
ode/146a17db/stage
default_status_returncode/146a17db: Writing build script: /home/spack/run
s/generic.local.bash/compiler_status_regex/default_status_returncode/146a
17db/default_status_returncode_build.sh
default_status_returncode/c18e8856: Creating test directory: /home/spack/
runs/generic.local.bash/compiler_status_regex/default_status_returncode/c
18e8856
default_status_returncode/c18e8856: Creating the stage directory: /home/s
pack/runs/generic.local.bash/compiler_status_regex/default_status_returnc
ode/c18e8856/stage
default_status_returncode/c18e8856: Writing build script: /home/spack/run
s/generic.local.bash/compiler_status_regex/default_status_returncode/c18e
8856/default_status_returncode_build.sh
default_status_returncode/8f1541f1: Creating test directory: /home/spack/
runs/generic.local.bash/compiler_status_regex/default_status_returncode/8
f1541f1
default_status_returncode/8f1541f1: Creating the stage directory: /home/s
pack/runs/generic.local.bash/compiler_status_regex/default_status_returnc
ode/8f1541f1/stage
default_status_returncode/8f1541f1: Writing build script: /home/spack/run
s/generic.local.bash/compiler_status_regex/default_status_returncode/8f15
41f1/default_status_returncode_build.sh
override_status_regex/d3d57fb4: Creating test directory: /home/spack/runs
/generic.local.bash/compiler_status_regex/override_status_regex/d3d57fb4
override_status_regex/d3d57fb4: Creating the stage directory: /home/spack
/runs/generic.local.bash/compiler_status_regex/override_status_regex/d3d5
7fb4/stage
override_status_regex/d3d57fb4: Writing build script: /home/spack/runs/ge
neric.local.bash/compiler_status_regex/override_status_regex/d3d57fb4/ove
rride_status_regex_build.sh
override_status_regex/aa303b16: Creating test directory: /home/spack/runs
/generic.local.bash/compiler_status_regex/override_status_regex/aa303b16
override_status_regex/aa303b16: Creating the stage directory: /home/spack
/runs/generic.local.bash/compiler_status_regex/override_status_regex/aa30
3b16/stage
override_status_regex/aa303b16: Writing build script: /home/spack/runs/ge
neric.local.bash/compiler_status_regex/override_status_regex/aa303b16/ove
rride_status_regex_build.sh
override_status_regex/465cf5fe: Creating test directory: /home/spack/runs
/generic.local.bash/compiler_status_regex/override_status_regex/465cf5fe
override_status_regex/465cf5fe: Creating the stage directory: /home/spack
/runs/generic.local.bash/compiler_status_regex/override_status_regex/465c
f5fe/stage
override_status_regex/465cf5fe: Writing build script: /home/spack/runs/ge
neric.local.bash/compiler_status_regex/override_status_regex/465cf5fe/ove
rride_status_regex_build.sh
───────────────────────────── Running Tests ─────────────────────────────
Spawning 4 processes for processing builders
────────────────────────────── Iteration 1 ──────────────────────────────
override_status_regex/465cf5fe does not have any dependencies adding test
to queue
default_status_returncode/c18e8856 does not have any dependencies adding 
test to queue
override_status_regex/aa303b16 does not have any dependencies adding test
to queue
default_status_returncode/8f1541f1 does not have any dependencies adding 
test to queue
override_status_regex/d3d57fb4 does not have any dependencies adding test
to queue
default_status_returncode/146a17db does not have any dependencies adding 
test to queue
override_status_regex/aa303b16: Running Test via command: bash --norc 
--noprofile -eo pipefail override_status_regex_build.sh
default_status_returncode/8f1541f1: Running Test via command: bash --norc
--noprofile -eo pipefail default_status_returncode_build.sh
override_status_regex/d3d57fb4: Running Test via command: bash --norc 
--noprofile -eo pipefail override_status_regex_build.sh
override_status_regex/465cf5fe: Running Test via command: bash --norc 
--noprofile -eo pipefail override_status_regex_build.sh
override_status_regex/d3d57fb4: Test completed in 0.114215 seconds
override_status_regex/d3d57fb4: Test completed with returncode: 0
override_status_regex/aa303b16: Test completed in 0.123078 seconds
override_status_regex/aa303b16: Test completed with returncode: 0
override_status_regex/d3d57fb4: Writing output file -  /home/spack/runs/g
eneric.local.bash/compiler_status_regex/override_status_regex/d3d57fb4/ov
erride_status_regex.out
override_status_regex/d3d57fb4: Writing error file - /home/spack/runs/gen
eric.local.bash/compiler_status_regex/override_status_regex/d3d57fb4/over
ride_status_regex.err
override_status_regex/aa303b16: Writing output file -  /home/spack/runs/g
eneric.local.bash/compiler_status_regex/override_status_regex/aa303b16/ov
erride_status_regex.out
override_status_regex/aa303b16: Writing error file - /home/spack/runs/gen
eric.local.bash/compiler_status_regex/override_status_regex/aa303b16/over
ride_status_regex.err
override_status_regex/d3d57fb4: Checking returncode - 0 is matched in 
list [1]
override_status_regex/aa303b16: Checking returncode - 0 is matched in 
list [1]
override_status_regex/465cf5fe: Test completed in 0.141141 seconds
override_status_regex/465cf5fe: Test completed with returncode: 0
override_status_regex/465cf5fe: Writing output file -  /home/spack/runs/g
eneric.local.bash/compiler_status_regex/override_status_regex/465cf5fe/ov
erride_status_regex.out
default_status_returncode/146a17db: Running Test via command: bash --norc
--noprofile -eo pipefail default_status_returncode_build.sh
override_status_regex/465cf5fe: Writing error file - /home/spack/runs/gen
eric.local.bash/compiler_status_regex/override_status_regex/465cf5fe/over
ride_status_regex.err
default_status_returncode/c18e8856: Running Test via command: bash --norc
--noprofile -eo pipefail default_status_returncode_build.sh
default_status_returncode/8f1541f1: Test completed in 0.160761 seconds
default_status_returncode/8f1541f1: Test completed with returncode: 0
default_status_returncode/8f1541f1: Writing output file -  /home/spack/ru
ns/generic.local.bash/compiler_status_regex/default_status_returncode/8f1
541f1/default_status_returncode.out
override_status_regex/465cf5fe: performing regular expression - '^final 
result: 0.99$' on file: /home/spack/runs/generic.local.bash/compiler_stat
us_regex/override_status_regex/465cf5fe/override_status_regex.out
override_status_regex/465cf5fe: Regular Expression Match - Failed!
default_status_returncode/8f1541f1: Writing error file - /home/spack/runs
/generic.local.bash/compiler_status_regex/default_status_returncode/8f154
1f1/default_status_returncode.err
default_status_returncode/8f1541f1: Checking returncode - 0 is matched in
list [0]
default_status_returncode/146a17db: Test completed in 0.104225 seconds
default_status_returncode/146a17db: Test completed with returncode: 0
default_status_returncode/146a17db: Writing output file -  /home/spack/ru
ns/generic.local.bash/compiler_status_regex/default_status_returncode/146
a17db/default_status_returncode.out
default_status_returncode/146a17db: Writing error file - /home/spack/runs
/generic.local.bash/compiler_status_regex/default_status_returncode/146a1
7db/default_status_returncode.err
default_status_returncode/c18e8856: Test completed in 0.107678 seconds
default_status_returncode/146a17db: Checking returncode - 0 is matched in
list [0]
default_status_returncode/c18e8856: Test completed with returncode: 0
default_status_returncode/c18e8856: Writing output file -  /home/spack/ru
ns/generic.local.bash/compiler_status_regex/default_status_returncode/c18
e8856/default_status_returncode.out
default_status_returncode/c18e8856: Writing error file - /home/spack/runs
/generic.local.bash/compiler_status_regex/default_status_returncode/c18e8
856/default_status_returncode.err
default_status_returncode/c18e8856: Checking returncode - 0 is matched in
list [0]
In this iteration we are going to run the following tests: [override_status_regex/465cf5fe, override_status_regex/aa303b16, default_status_returncode/8f1541f1, override_status_regex/d3d57fb4, default_status_returncode/c18e8856, default_status_returncode/146a17db]
                              Test Summary                               
┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃            ┃            ┃        ┃ checks (Re ┃            ┃          ┃
┃            ┃            ┃        ┃ turnCode,  ┃            ┃          ┃
┃            ┃            ┃        ┃ Regex,     ┃            ┃          ┃
┃ builder    ┃ executor   ┃ status ┃ Runtime)   ┃ returnCode ┃ runtime  ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ override_s │ generic.l… │ FAIL   │ False      │ 0          │ 0.123078 │
│ tatus_rege │            │        │ False      │            │          │
│ x/aa303b16 │            │        │ False      │            │          │
├────────────┼────────────┼────────┼────────────┼────────────┼──────────┤
│ default_st │ generic.l… │ PASS   │ True False │ 0          │ 0.104225 │
│ atus_retur │            │        │ False      │            │          │
│ ncode/146a │            │        │            │            │          │
│ 17db       │            │        │            │            │          │
├────────────┼────────────┼────────┼────────────┼────────────┼──────────┤
│ override_s │ generic.l… │ FAIL   │ False      │ 0          │ 0.114215 │
│ tatus_rege │            │        │ False      │            │          │
│ x/d3d57fb4 │            │        │ False      │            │          │
├────────────┼────────────┼────────┼────────────┼────────────┼──────────┤
│ default_st │ generic.l… │ PASS   │ True False │ 0          │ 0.107678 │
│ atus_retur │            │        │ False      │            │          │
│ ncode/c18e │            │        │            │            │          │
│ 8856       │            │        │            │            │          │
├────────────┼────────────┼────────┼────────────┼────────────┼──────────┤
│ default_st │ generic.l… │ PASS   │ True False │ 0          │ 0.160761 │
│ atus_retur │            │        │ False      │            │          │
│ ncode/8f15 │            │        │            │            │          │
│ 41f1       │            │        │            │            │          │
├────────────┼────────────┼────────┼────────────┼────────────┼──────────┤
│ override_s │ generic.l… │ FAIL   │ False      │ 0          │ 0.141141 │
│ tatus_rege │            │        │ False      │            │          │
│ x/465cf5fe │            │        │ False      │            │          │
└────────────┴────────────┴────────┴────────────┴────────────┴──────────┘



Passed Tests: 3/6 Percentage: 50.000%
Failed Tests: 3/6 Percentage: 50.000%


Adding 6 test results to /home/spack/buildtest/var/report.json
Writing Logfile to: /home/spack/buildtest/var/logs/buildtest_lhzqv902.log

For the second test, we see the generated output is final result: 1.000000 but our regular expression has a different expected value therefore this test will fail even though we have a exitcode of 0.

buildtest inspect query -o override_status_regex/
$ buildtest inspect query -o override_status_regex/ 
────── override_status_regex/465cf5fe-e7d9-465d-baed-c258e8ae3478 ───────
Executor: generic.local.bash
Description: override 'status' property in compiler gcc_8.3.0 instance
State: FAIL
Returncode: 0
Runtime: 0.141141 sec
Starttime: 2022/06/29 17:27:32
Endtime: 2022/06/29 17:27:32
Command: bash --norc --noprofile -eo pipefail 
override_status_regex_build.sh
Test Script: /home/spack/runs/generic.local.bash/compiler_status_regex/ov
erride_status_regex/465cf5fe/override_status_regex.sh
Build Script: /home/spack/runs/generic.local.bash/compiler_status_regex/o
verride_status_regex/465cf5fe/override_status_regex_build.sh
Output File: /home/spack/runs/generic.local.bash/compiler_status_regex/ov
erride_status_regex/465cf5fe/override_status_regex.out
Error File: /home/spack/runs/generic.local.bash/compiler_status_regex/ove
rride_status_regex/465cf5fe/override_status_regex.err
Log File: /home/spack/buildtest/var/logs/buildtest_lhzqv902.log
─ Output File: /home/spack/runs/generic.local.bash/compiler_status_reg… ─
final result: 1.000000                                                   
                                                                         
────── override_status_regex/d3d57fb4-57c1-496d-8db8-67a2a058ff33 ───────
Executor: generic.local.bash
Description: override 'status' property in compiler gcc_8.3.0 instance
State: FAIL
Returncode: 0
Runtime: 0.114215 sec
Starttime: 2022/06/29 17:27:32
Endtime: 2022/06/29 17:27:32
Command: bash --norc --noprofile -eo pipefail 
override_status_regex_build.sh
Test Script: /home/spack/runs/generic.local.bash/compiler_status_regex/ov
erride_status_regex/d3d57fb4/override_status_regex.sh
Build Script: /home/spack/runs/generic.local.bash/compiler_status_regex/o
verride_status_regex/d3d57fb4/override_status_regex_build.sh
Output File: /home/spack/runs/generic.local.bash/compiler_status_regex/ov
erride_status_regex/d3d57fb4/override_status_regex.out
Error File: /home/spack/runs/generic.local.bash/compiler_status_regex/ove
rride_status_regex/d3d57fb4/override_status_regex.err
Log File: /home/spack/buildtest/var/logs/buildtest_lhzqv902.log
─ Output File: /home/spack/runs/generic.local.bash/compiler_status_reg… ─
final result: 1.000000                                                   
                                                                         
────── override_status_regex/aa303b16-c50c-4f5f-b815-c5a4c0292486 ───────
Executor: generic.local.bash
Description: override 'status' property in compiler gcc_8.3.0 instance
State: FAIL
Returncode: 0
Runtime: 0.123078 sec
Starttime: 2022/06/29 17:27:32
Endtime: 2022/06/29 17:27:32
Command: bash --norc --noprofile -eo pipefail 
override_status_regex_build.sh
Test Script: /home/spack/runs/generic.local.bash/compiler_status_regex/ov
erride_status_regex/aa303b16/override_status_regex.sh
Build Script: /home/spack/runs/generic.local.bash/compiler_status_regex/o
verride_status_regex/aa303b16/override_status_regex_build.sh
Output File: /home/spack/runs/generic.local.bash/compiler_status_regex/ov
erride_status_regex/aa303b16/override_status_regex.out
Error File: /home/spack/runs/generic.local.bash/compiler_status_regex/ove
rride_status_regex/aa303b16/override_status_regex.err
Log File: /home/spack/buildtest/var/logs/buildtest_lhzqv902.log
─ Output File: /home/spack/runs/generic.local.bash/compiler_status_reg… ─
final result: 1.000000

Customize Run Line

buildtest will define variable _EXEC in the job script that can be used to reference the generated binary. By default, buildtest will run the program standalone, but sometimes you may want to customize how job is run. This may include passing arguments or running binary through a job/mpi launcher. The run property can be used to configure how program is executed. The compiled executable will be present in local directory which can be accessed via ./$_EXEC. In example below we pass arguments 1 3 for builtin_gcc compiler and 100 200 for gcc_8.3.0 compiler.

buildspecs:
  custom_run_by_compilers:
    type: compiler
    description: Customize binary launch based on compiler
    executor: generic.local.bash
    tags: [tutorials, compile]
    source: "src/argc.c"
    compilers:
      name: ["^(gcc_7.5.0|gcc_8.3.0)"]
      config:
        gcc_7.5.0:
          run: ./$_EXEC 1 3
        gcc_8.3.0:
          run: ./$_EXEC 100 120

You can build this test by running the following:

buildtest build -b $BUILDTEST_ROOT/examples/compilers/custom_run.yml

Once test is complete let’s inspect the generated test. We see that buildtest will insert the line specified by run property after compilation and run the executable.

buildtest inspect query -b  -t custom_run_by_compilers/
$ buildtest inspect query -b  -t custom_run_by_compilers/ 
───── custom_run_by_compilers/2536736f-c0f1-4afc-9be6-dd8d323cdc4c ──────
Executor: generic.local.bash
Description: Customize binary launch based on compiler
State: PASS
Returncode: 0
Runtime: 0.456091 sec
Starttime: 2022/06/29 17:27:34
Endtime: 2022/06/29 17:27:35
Command: bash --norc --noprofile -eo pipefail 
custom_run_by_compilers_build.sh
Test Script: /home/spack/runs/generic.local.bash/custom_run/custom_run_by
_compilers/2536736f/custom_run_by_compilers.sh
Build Script: /home/spack/runs/generic.local.bash/custom_run/custom_run_b
y_compilers/2536736f/custom_run_by_compilers_build.sh
Output File: /home/spack/runs/generic.local.bash/custom_run/custom_run_by
_compilers/2536736f/custom_run_by_compilers.out
Error File: /home/spack/runs/generic.local.bash/custom_run/custom_run_by_
compilers/2536736f/custom_run_by_compilers.err
Log File: /home/spack/buildtest/var/logs/buildtest_ljkmzoae.log
─ Test File: /home/spack/runs/generic.local.bash/custom_run/custom_run… ─
#!/bin/bash                                                              
                                                                         
                                                                         
# name of executable                                                     
_EXEC=argc.c.exe                                                         
# Compilation Line                                                       
/usr/bin/gcc -o $_EXEC /home/spack/buildtest/examples/compilers/src/argc.
                                                                         
                                                                         
# Run executable                                                         
./$_EXEC 1 3                                                             
                                                                         
                                                                         
─ Test File: /home/spack/runs/generic.local.bash/custom_run/custom_run… ─
#!/bin/bash                                                              
export BUILDTEST_TEST_NAME=custom_run_by_compilers                       
export BUILDTEST_TEST_ROOT=/home/spack/runs/generic.local.bash/custom_run
export BUILDTEST_BUILDSPEC_DIR=/home/spack/buildtest/examples/compilers  
export BUILDTEST_STAGE_DIR=/home/spack/runs/generic.local.bash/custom_run
# source executor startup script                                         
source /home/spack/buildtest/var/executor/generic.local.bash/before_scrip
# Run generated script                                                   
/home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/25
# Get return code                                                        
returncode=$?                                                            
# Exit with return code                                                  
exit $returncode                                                         
───── custom_run_by_compilers/2b1553e4-975f-4282-a3dc-3e9d3f48bd1a ──────
Executor: generic.local.bash
Description: Customize binary launch based on compiler
State: PASS
Returncode: 0
Runtime: 0.525561 sec
Starttime: 2022/06/29 17:27:34
Endtime: 2022/06/29 17:27:35
Command: bash --norc --noprofile -eo pipefail 
custom_run_by_compilers_build.sh
Test Script: /home/spack/runs/generic.local.bash/custom_run/custom_run_by
_compilers/2b1553e4/custom_run_by_compilers.sh
Build Script: /home/spack/runs/generic.local.bash/custom_run/custom_run_b
y_compilers/2b1553e4/custom_run_by_compilers_build.sh
Output File: /home/spack/runs/generic.local.bash/custom_run/custom_run_by
_compilers/2b1553e4/custom_run_by_compilers.out
Error File: /home/spack/runs/generic.local.bash/custom_run/custom_run_by_
compilers/2b1553e4/custom_run_by_compilers.err
Log File: /home/spack/buildtest/var/logs/buildtest_ljkmzoae.log
─ Test File: /home/spack/runs/generic.local.bash/custom_run/custom_run… ─
#!/bin/bash                                                              
                                                                         
                                                                         
# name of executable                                                     
_EXEC=argc.c.exe                                                         
# Loading modules                                                        
module load gcc/8.3.0-gcc-7.5.0                                          
# Compilation Line                                                       
gcc -o $_EXEC /home/spack/buildtest/examples/compilers/src/argc.c        
                                                                         
                                                                         
# Run executable                                                         
./$_EXEC 100 120                                                         
                                                                         
                                                                         
─ Test File: /home/spack/runs/generic.local.bash/custom_run/custom_run… ─
#!/bin/bash                                                              
export BUILDTEST_TEST_NAME=custom_run_by_compilers                       
export BUILDTEST_TEST_ROOT=/home/spack/runs/generic.local.bash/custom_run
export BUILDTEST_BUILDSPEC_DIR=/home/spack/buildtest/examples/compilers  
export BUILDTEST_STAGE_DIR=/home/spack/runs/generic.local.bash/custom_run
# source executor startup script                                         
source /home/spack/buildtest/var/executor/generic.local.bash/before_scrip
# Run generated script                                                   
/home/spack/runs/generic.local.bash/custom_run/custom_run_by_compilers/2b
# Get return code                                                        
returncode=$?                                                            
# Exit with return code                                                  
exit $returncode

Pre/Post sections for build and run section

We can specify arbitrary shell commands before/after compilation or running binary via pre_build, post_build, pre_run and post_run property. This can be useful if you want to specify additional commands required to compile or run executable that are not generated automatically

This next example illustrates how one can use these properties to have more control over the generated test.

buildspecs:
  pre_post_build_run:
    type: compiler
    description: example using pre_build, post_build, pre_run, post_run example
    executor: generic.local.bash
    tags: [tutorials, compile]
    source: "src/hello.c"
    compilers:
      name: ["^(gcc_7.5.0)$"]
      default:
        gcc:
          cflags: -Wall
          pre_build: |
            echo "These are commands run before compilation"
          post_build: |
            echo "These are commands run after compilation"
          pre_run: |
            echo "These are commands run before running script"
          post_run:
            echo "These are commands run after running script"

The format of the test structure is as follows.

#!{shebang path} -- defaults to #!/bin/bash depends on executor name (local.bash, local.sh)
{job directives} -- sbatch or bsub field
{environment variables} -- env field
{variable declaration} -- vars field
{module commands} -- modules field

{pre build commands} -- pre_build field
{compile program} -- build field
{post build commands} -- post_build field

{pre run commands} -- pre_run field
{run executable} -- run field
{post run commands} -- post_run field

You can run this example by running the following command:

buildtest build -b $BUILDTEST_ROOT/examples/compilers/pre_post_build_run.yml

If we inspect the content of test we see that buildtest will insert the shell commands for pre_build, post_build, pre_run and post_run in its corresponding section.

buildtest inspect query -t pre_post_build_run
$ buildtest inspect query -t pre_post_build_run 
──────── pre_post_build_run/5b8d06ce-c010-4299-8349-be340b754aa5 ────────
Executor: generic.local.bash
Description: example using pre_build, post_build, pre_run, post_run 
example
State: PASS
Returncode: 0
Runtime: 0.061087 sec
Starttime: 2022/06/29 17:27:37
Endtime: 2022/06/29 17:27:38
Command: bash --norc --noprofile -eo pipefail pre_post_build_run_build.sh
Test Script: /home/spack/runs/generic.local.bash/pre_post_build_run/pre_p
ost_build_run/5b8d06ce/pre_post_build_run.sh
Build Script: /home/spack/runs/generic.local.bash/pre_post_build_run/pre_
post_build_run/5b8d06ce/pre_post_build_run_build.sh
Output File: /home/spack/runs/generic.local.bash/pre_post_build_run/pre_p
ost_build_run/5b8d06ce/pre_post_build_run.out
Error File: /home/spack/runs/generic.local.bash/pre_post_build_run/pre_po
st_build_run/5b8d06ce/pre_post_build_run.err
Log File: /home/spack/buildtest/var/logs/buildtest_emxk6zsu.log
─ Test File: /home/spack/runs/generic.local.bash/pre_post_build_run/pr… ─
#!/bin/bash                                                              
                                                                         
                                                                         
# name of executable                                                     
_EXEC=hello.c.exe                                                        
### START OF PRE BUILD SECTION ###                                       
echo "These are commands run before compilation"                         
                                                                         
### END OF PRE BUILD SECTION   ###                                       
                                                                         
                                                                         
# Compilation Line                                                       
/usr/bin/gcc -Wall -o $_EXEC /home/spack/buildtest/examples/compilers/src
                                                                         
                                                                         
### START OF POST BUILD SECTION ###                                      
echo "These are commands run after compilation"                          
                                                                         
### END OF POST BUILD SECTION ###                                        
                                                                         
                                                                         
### START OF PRE RUN SECTION ###                                         
echo "These are commands run before running script"                      
                                                                         
### END OF PRE RUN SECTION   ###                                         
                                                                         
                                                                         
# Run executable                                                         
./$_EXEC                                                                 
                                                                         
                                                                         
### START OF POST RUN SECTION ###                                        
echo "These are commands run after running script"                       
### END OF POST RUN SECTION   ###

Running Stream Benchmark with multiple compilers

In this example, we will show how one can run STREAM benchmark with multiple compilers with script schema. The compilers property can be set in the script schema which is used to search for compilers to create separate test for each discovered compiler. buildtest will set the following variables in the script that is mapped to each compiler

  • BUILDTEST_CC: Path to C compiler

  • BUILDTEST_CXX: Path to C++ compiler

  • BUILDTEST_FC: Path to Fortran compiler

  • BUILDTEST_CFLAGS: C compiler flags

  • BUILDTEST_CXXFLAGS: C++ compiler flags

  • BUILDTEST_CPPFLAGS: C++ Pre Preprocessor flags

  • BUILDTEST_LDFLAGS: Linker Flags

In the run section we can reference these variables to compile stream.c to build the code and run it. We will set environment OMP_NUM_THREADS to control number of OpenMP threads when running the benchmark.

buildspecs:
  stream_openmp_c:
    executor: generic.local.bash
    type: script
    description: "STREAM Microbenchmark C Test with OpenMP"
    tags: ["benchmark"]
    compilers:
      name: ['^(gcc)']
      default:
        gcc:
          cflags: -fopenmp -O2
          env:
           OMP_NUM_THREADS: 8
    run: |
      wget https://www.cs.virginia.edu/stream/FTP/Code/stream.c
      $BUILDTEST_CC $BUILDTEST_CFLAGS stream.c -o stream
      ./stream

If we build this test we see one test is created per compiler instance defined in our configuration file.

buildtest build -b /home/spack/buildtest/examples/compilers/stream_example.yml
$ buildtest build -b /home/spack/buildtest/examples/compilers/stream_example.yml 
╭────────────────────────── buildtest summary ──────────────────────────╮
│                                                                       │
│ User:               spack                                             │
│ Hostname:           356ea13b3433                                      │
│ Platform:           Linux                                             │
│ Current Time:       2022/06/29 17:27:39                               │
│ buildtest path:     /home/spack/buildtest/bin/buildtest               │
│ buildtest version:  0.14.0                                            │
│ python path:        /home/spack/pyenv/buildtest/bin/python3           │
│ python version:     3.8.6                                             │
│ Configuration File:                                                   │
│ /home/spack/buildtest/buildtest/settings/spack_container.yml          │
│ Test Directory:     /home/spack/runs                                  │
│ Report File:        /home/spack/buildtest/var/report.json             │
│ Command:            /home/spack/buildtest/bin/buildtest build -b      │
│ /home/spack/buildtest/examples/compilers/stream_example.yml           │
│                                                                       │
╰───────────────────────────────────────────────────────────────────────╯
────────────────────────  Discovering Buildspecs ────────────────────────
                     Discovered buildspecs                     
╔═════════════════════════════════════════════════════════════╗
║ buildspec                                                   ║
╟─────────────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/compilers/stream_example.yml ║
╚═════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────── Parsing Buildspecs ───────────────────────────
Buildtest will parse 1 buildspecs
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/compilers/stream_example.yml: VALID
Total builder objects created: 3
Total compiler builder: 0
Total script builder: 3
Total spack builder: 0
                         Script Builder Details                          
┏━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder  ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ descript… ┃ buildsp… ┃
┡━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━┩
│ stream_… │ generic… │ gcc_7.5… │ None  │ None  │ STREAM    │ /home/s… │
│          │          │          │       │       │ Microben… │          │
│          │          │          │       │       │ C Test    │          │
│          │          │          │       │       │ with      │          │
│          │          │          │       │       │ OpenMP    │          │
├──────────┼──────────┼──────────┼───────┼───────┼───────────┼──────────┤
│ stream_… │ generic… │ gcc_6.5… │ None  │ None  │ STREAM    │ /home/s… │
│          │          │          │       │       │ Microben… │          │
│          │          │          │       │       │ C Test    │          │
│          │          │          │       │       │ with      │          │
│          │          │          │       │       │ OpenMP    │          │
├──────────┼──────────┼──────────┼───────┼───────┼───────────┼──────────┤
│ stream_… │ generic… │ gcc_8.3… │ None  │ None  │ STREAM    │ /home/s… │
│          │          │          │       │       │ Microben… │          │
│          │          │          │       │       │ C Test    │          │
│          │          │          │       │       │ with      │          │
│          │          │          │       │       │ OpenMP    │          │
└──────────┴──────────┴──────────┴───────┴───────┴───────────┴──────────┘
───────────────────────────── Building Test ─────────────────────────────
stream_openmp_c/3dcccb2c: Creating test directory: /home/spack/runs/gener
ic.local.bash/stream_example/stream_openmp_c/3dcccb2c
stream_openmp_c/3dcccb2c: Creating the stage directory: /home/spack/runs/
generic.local.bash/stream_example/stream_openmp_c/3dcccb2c/stage
stream_openmp_c/3dcccb2c: Writing build script: /home/spack/runs/generic.
local.bash/stream_example/stream_openmp_c/3dcccb2c/stream_openmp_c_build.
sh
stream_openmp_c/f0e90faa: Creating test directory: /home/spack/runs/gener
ic.local.bash/stream_example/stream_openmp_c/f0e90faa
stream_openmp_c/f0e90faa: Creating the stage directory: /home/spack/runs/
generic.local.bash/stream_example/stream_openmp_c/f0e90faa/stage
stream_openmp_c/f0e90faa: Writing build script: /home/spack/runs/generic.
local.bash/stream_example/stream_openmp_c/f0e90faa/stream_openmp_c_build.
sh
stream_openmp_c/a59bb796: Creating test directory: /home/spack/runs/gener
ic.local.bash/stream_example/stream_openmp_c/a59bb796
stream_openmp_c/a59bb796: Creating the stage directory: /home/spack/runs/
generic.local.bash/stream_example/stream_openmp_c/a59bb796/stage
stream_openmp_c/a59bb796: Writing build script: /home/spack/runs/generic.
local.bash/stream_example/stream_openmp_c/a59bb796/stream_openmp_c_build.
sh
───────────────────────────── Running Tests ─────────────────────────────
Spawning 4 processes for processing builders
────────────────────────────── Iteration 1 ──────────────────────────────
stream_openmp_c/3dcccb2c does not have any dependencies adding test to 
queue
stream_openmp_c/a59bb796 does not have any dependencies adding test to 
queue
stream_openmp_c/f0e90faa does not have any dependencies adding test to 
queue
stream_openmp_c/3dcccb2c: Running Test via command: bash --norc 
--noprofile -eo pipefail stream_openmp_c_build.sh
stream_openmp_c/a59bb796: Running Test via command: bash --norc 
--noprofile -eo pipefail stream_openmp_c_build.sh
stream_openmp_c/f0e90faa: Running Test via command: bash --norc 
--noprofile -eo pipefail stream_openmp_c_build.sh
stream_openmp_c/a59bb796: Test completed in 2.444227 seconds
stream_openmp_c/a59bb796: Test completed with returncode: 0
stream_openmp_c/a59bb796: Writing output file -  /home/spack/runs/generic
.local.bash/stream_example/stream_openmp_c/a59bb796/stream_openmp_c.out
stream_openmp_c/a59bb796: Writing error file - /home/spack/runs/generic.l
ocal.bash/stream_example/stream_openmp_c/a59bb796/stream_openmp_c.err
stream_openmp_c/3dcccb2c: Test completed in 2.646666 seconds
stream_openmp_c/3dcccb2c: Test completed with returncode: 0
stream_openmp_c/3dcccb2c: Writing output file -  /home/spack/runs/generic
.local.bash/stream_example/stream_openmp_c/3dcccb2c/stream_openmp_c.out
stream_openmp_c/3dcccb2c: Writing error file - /home/spack/runs/generic.l
ocal.bash/stream_example/stream_openmp_c/3dcccb2c/stream_openmp_c.err
stream_openmp_c/f0e90faa: Test completed in 2.789559 seconds
stream_openmp_c/f0e90faa: Test completed with returncode: 0
stream_openmp_c/f0e90faa: Writing output file -  /home/spack/runs/generic
.local.bash/stream_example/stream_openmp_c/f0e90faa/stream_openmp_c.out
stream_openmp_c/f0e90faa: Writing error file - /home/spack/runs/generic.l
ocal.bash/stream_example/stream_openmp_c/f0e90faa/stream_openmp_c.err
In this iteration we are going to run the following tests: [stream_openmp_c/3dcccb2c, stream_openmp_c/a59bb796, stream_openmp_c/f0e90faa]
                              Test Summary                               
┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃            ┃            ┃        ┃ checks (Re ┃            ┃          ┃
┃            ┃            ┃        ┃ turnCode,  ┃            ┃          ┃
┃            ┃            ┃        ┃ Regex,     ┃            ┃          ┃
┃ builder    ┃ executor   ┃ status ┃ Runtime)   ┃ returnCode ┃ runtime  ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ stream_ope │ generic.l… │ PASS   │ N/A N/A    │ 0          │ 2.646666 │
│ nmp_c/3dcc │            │        │ N/A        │            │          │
│ cb2c       │            │        │            │            │          │
├────────────┼────────────┼────────┼────────────┼────────────┼──────────┤
│ stream_ope │ generic.l… │ PASS   │ N/A N/A    │ 0          │ 2.789559 │
│ nmp_c/f0e9 │            │        │ N/A        │            │          │
│ 0faa       │            │        │            │            │          │
├────────────┼────────────┼────────┼────────────┼────────────┼──────────┤
│ stream_ope │ generic.l… │ PASS   │ N/A N/A    │ 0          │ 2.444227 │
│ nmp_c/a59b │            │        │ N/A        │            │          │
│ b796       │            │        │            │            │          │
└────────────┴────────────┴────────┴────────────┴────────────┴──────────┘



Passed Tests: 3/3 Percentage: 100.000%
Failed Tests: 0/3 Percentage: 0.000%


Adding 3 test results to /home/spack/buildtest/var/report.json
Writing Logfile to: /home/spack/buildtest/var/logs/buildtest_ta936q7h.log

Next, let’s see the generated test using buildtest inspect query, we notice buildtest will set variables BUILDTEST_* for each test to map to each compiler.

buildtest inspect query -t stream_openmp_c/
$ buildtest inspect query -t stream_openmp_c/ 
───────── stream_openmp_c/a59bb796-9ae8-45e7-9d90-d70bc0c8da99 ──────────
Executor: generic.local.bash
Description: STREAM Microbenchmark C Test with OpenMP
State: PASS
Returncode: 0
Runtime: 2.444227 sec
Starttime: 2022/06/29 17:27:39
Endtime: 2022/06/29 17:27:42
Command: bash --norc --noprofile -eo pipefail stream_openmp_c_build.sh
Test Script: /home/spack/runs/generic.local.bash/stream_example/stream_op
enmp_c/a59bb796/stream_openmp_c.sh
Build Script: /home/spack/runs/generic.local.bash/stream_example/stream_o
penmp_c/a59bb796/stream_openmp_c_build.sh
Output File: /home/spack/runs/generic.local.bash/stream_example/stream_op
enmp_c/a59bb796/stream_openmp_c.out
Error File: /home/spack/runs/generic.local.bash/stream_example/stream_ope
nmp_c/a59bb796/stream_openmp_c.err
Log File: /home/spack/buildtest/var/logs/buildtest_ta936q7h.log
─ Test File: /home/spack/runs/generic.local.bash/stream_example/stream… ─
#!/bin/bash                                                              
BUILDTEST_CC="gcc"                                                       
BUILDTEST_CXX="g++"                                                      
BUILDTEST_FC="gfortran"                                                  
BUILDTEST_CFLAGS="-fopenmp -O2"                                          
BUILDTEST_CXXFLAGS=                                                      
BUILDTEST_FFLAGS=                                                        
BUILDTEST_CPPFLAGS=                                                      
BUILDTEST_LDFLAGS=                                                       
export OMP_NUM_THREADS="8"                                               
module load gcc/8.3.0-gcc-7.5.0                                          
# Content of run section                                                 
wget https://www.cs.virginia.edu/stream/FTP/Code/stream.c                
$BUILDTEST_CC $BUILDTEST_CFLAGS stream.c -o stream                       
./stream                                                                 
───────── stream_openmp_c/f0e90faa-29a8-48e7-aa07-7419c477db55 ──────────
Executor: generic.local.bash
Description: STREAM Microbenchmark C Test with OpenMP
State: PASS
Returncode: 0
Runtime: 2.789559 sec
Starttime: 2022/06/29 17:27:39
Endtime: 2022/06/29 17:27:42
Command: bash --norc --noprofile -eo pipefail stream_openmp_c_build.sh
Test Script: /home/spack/runs/generic.local.bash/stream_example/stream_op
enmp_c/f0e90faa/stream_openmp_c.sh
Build Script: /home/spack/runs/generic.local.bash/stream_example/stream_o
penmp_c/f0e90faa/stream_openmp_c_build.sh
Output File: /home/spack/runs/generic.local.bash/stream_example/stream_op
enmp_c/f0e90faa/stream_openmp_c.out
Error File: /home/spack/runs/generic.local.bash/stream_example/stream_ope
nmp_c/f0e90faa/stream_openmp_c.err
Log File: /home/spack/buildtest/var/logs/buildtest_ta936q7h.log
─ Test File: /home/spack/runs/generic.local.bash/stream_example/stream… ─
#!/bin/bash                                                              
BUILDTEST_CC="gcc"                                                       
BUILDTEST_CXX="g++"                                                      
BUILDTEST_FC="gfortran"                                                  
BUILDTEST_CFLAGS="-fopenmp -O2"                                          
BUILDTEST_CXXFLAGS=                                                      
BUILDTEST_FFLAGS=                                                        
BUILDTEST_CPPFLAGS=                                                      
BUILDTEST_LDFLAGS=                                                       
export OMP_NUM_THREADS="8"                                               
module load gcc/6.5.0-gcc-7.5.0                                          
# Content of run section                                                 
wget https://www.cs.virginia.edu/stream/FTP/Code/stream.c                
$BUILDTEST_CC $BUILDTEST_CFLAGS stream.c -o stream                       
./stream                                                                 
───────── stream_openmp_c/3dcccb2c-d6f3-40a2-b935-b06b8ee24a2b ──────────
Executor: generic.local.bash
Description: STREAM Microbenchmark C Test with OpenMP
State: PASS
Returncode: 0
Runtime: 2.646666 sec
Starttime: 2022/06/29 17:27:39
Endtime: 2022/06/29 17:27:42
Command: bash --norc --noprofile -eo pipefail stream_openmp_c_build.sh
Test Script: /home/spack/runs/generic.local.bash/stream_example/stream_op
enmp_c/3dcccb2c/stream_openmp_c.sh
Build Script: /home/spack/runs/generic.local.bash/stream_example/stream_o
penmp_c/3dcccb2c/stream_openmp_c_build.sh
Output File: /home/spack/runs/generic.local.bash/stream_example/stream_op
enmp_c/3dcccb2c/stream_openmp_c.out
Error File: /home/spack/runs/generic.local.bash/stream_example/stream_ope
nmp_c/3dcccb2c/stream_openmp_c.err
Log File: /home/spack/buildtest/var/logs/buildtest_ta936q7h.log
─ Test File: /home/spack/runs/generic.local.bash/stream_example/stream… ─
#!/bin/bash                                                              
BUILDTEST_CC="/usr/bin/gcc"                                              
BUILDTEST_CXX="/usr/bin/g++"                                             
BUILDTEST_FC="/usr/bin/gfortran"                                         
BUILDTEST_CFLAGS="-fopenmp -O2"                                          
BUILDTEST_CXXFLAGS=                                                      
BUILDTEST_FFLAGS=                                                        
BUILDTEST_CPPFLAGS=                                                      
BUILDTEST_LDFLAGS=                                                       
export OMP_NUM_THREADS="8"                                               
# Content of run section                                                 
wget https://www.cs.virginia.edu/stream/FTP/Code/stream.c                
$BUILDTEST_CC $BUILDTEST_CFLAGS stream.c -o stream                       
./stream

In the next example, we will run STREAM benchmark and extract metrics from test results and assign them to metrics name copy, add, scale, triad. Each metrics will be searched using regular expression against stdout stream

buildspecs:
  stream_openmp_metrics:
    executor: generic.local.bash
    type: script
    description: "STREAM Microbenchmark C Test with OpenMP"
    tags: ["benchmark"]
    compilers:
      name: ['^(gcc_7.5.0)']
      default:
        gcc:
          cflags: -fopenmp -O2
          env:
            OMP_NUM_THREADS: 16
    run: |
      wget https://www.cs.virginia.edu/stream/FTP/Code/stream.c
      $BUILDTEST_CC $BUILDTEST_CFLAGS stream.c -o stream
      ./stream
    metrics:
      copy:
        regex:
          stream: stdout
          exp: 'Copy:\s+(\S+)\s+.*'
      scale:
        regex:
          stream: stdout
          exp: 'Scale:\s+(\S+)\s+.*'
      add:
        regex:
          stream: stdout
          exp: 'Add:\s+(\S+)\s+.*'
      triad:
        regex:
          stream: stdout
          exp: 'Triad:\s+(\S+)\s+.*'

buildtest will record the metrics in the test report and buildtest inspect query will display metrics if found in test. Shown below we see the output of the metrics and its corresponding values.

buildtest inspect query -o stream_openmp_metrics/
$ buildtest inspect query -o stream_openmp_metrics/ 
────── stream_openmp_metrics/34a6af7b-7fa4-4874-87a5-7c2a716685d8 ───────
Executor: generic.local.bash
Description: STREAM Microbenchmark C Test with OpenMP
State: PASS
Returncode: 0
Runtime: 1.578672 sec
Starttime: 2022/06/29 17:27:46
Endtime: 2022/06/29 17:27:48
Command: bash --norc --noprofile -eo pipefail 
stream_openmp_metrics_build.sh
Test Script: /home/spack/runs/generic.local.bash/stream_example_metrics/s
tream_openmp_metrics/34a6af7b/stream_openmp_metrics.sh
Build Script: /home/spack/runs/generic.local.bash/stream_example_metrics/
stream_openmp_metrics/34a6af7b/stream_openmp_metrics_build.sh
Output File: /home/spack/runs/generic.local.bash/stream_example_metrics/s
tream_openmp_metrics/34a6af7b/stream_openmp_metrics.out
Error File: /home/spack/runs/generic.local.bash/stream_example_metrics/st
ream_openmp_metrics/34a6af7b/stream_openmp_metrics.err
Log File: /home/spack/buildtest/var/logs/buildtest_6yprzi4a.log
                                 Metrics                                 
┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name  ┃ Value                                                         ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ copy  │ Copy:           20317.5     0.010017     0.007875             │
│       │ 0.015247                                                      │
│ scale │ Scale:          14975.6     0.013843     0.010684             │
│       │ 0.018787                                                      │
│ add   │ Add:            16274.6     0.020774     0.014747             │
│       │ 0.037899                                                      │
│ triad │ Triad:          18008.7     0.018820     0.013327             │
│       │ 0.043124                                                      │
└───────┴───────────────────────────────────────────────────────────────┘
─ Output File: /home/spack/runs/generic.local.bash/stream_example_metr… ─
-------------------------------------------------------------            
STREAM version $Revision: 5.10 $                                         
-------------------------------------------------------------            
This system uses 8 bytes per array element.                              
-------------------------------------------------------------            
Array size = 10000000 (elements), Offset = 0 (elements)                  
Memory per array = 76.3 MiB (= 0.1 GiB).                                 
Total memory required = 228.9 MiB (= 0.2 GiB).                           
Each kernel will be executed 10 times.                                   
 The *best* time for each kernel (excluding the first iteration)         
 will be used to compute the reported bandwidth.                         
-------------------------------------------------------------            
Number of Threads requested = 16                                         
Number of Threads counted = 16                                           
-------------------------------------------------------------            
Your clock granularity/precision appears to be 12 microseconds.          
Each test below will take on the order of 8896 microseconds.             
   (= 741 clock ticks)                                                   
Increase the size of the arrays if this shows that                       
you are not getting at least 20 clock ticks per test.                    
-------------------------------------------------------------            
WARNING -- The above is only a rough guideline.                          
For best results, please be sure you know the                            
precision of your system timer.                                          
-------------------------------------------------------------            
Function    Best Rate MB/s  Avg time     Min time     Max time           
Copy:           20317.5     0.010017     0.007875     0.015247           
Scale:          14975.6     0.013843     0.010684     0.018787           
Add:            16274.6     0.020774     0.014747     0.037899           
Triad:          18008.7     0.018820     0.013327     0.043124           
-------------------------------------------------------------            
Solution Validates: avg error less than 1.000000e-13 on all three arrays 
-------------------------------------------------------------