Facility Test Examples

We will discuss a few test examples implemented at NERSC that are available at https://github.com/buildtesters/buildtest-nersc/.

Single Test Multiple Compilers

It’s possible to run single test across multiple compilers (gcc, intel, cray, etc…). In the next example, we will build an OpenMP reduction test using gcc, intel and cray compilers. In this test, we use name field to select compilers that start with gcc, intel and PrgEnv-cray as compiler names. The default section is organized by compiler groups which inherits compiler flags for all compilers. OpenMP flag for gcc, intel and cray differ for instance one must use -fopenmp for gcc, --qopenmp for intel and -h omp for cray.

 1 buildspecs:
 2   reduction:
 3     type: compiler
 4     executor: cori.local.bash
 5     source: src/reduction.c
 6     description: OpenMP reduction example using gcc, intel and cray compiler
 7     tags: [openmp]
 8     compilers:
 9       name: ["^(gcc|intel|PrgEnv-cray)"]
10       default:
11         all:
12           env:
13             OMP_NUM_THREADS: 4
14         gcc:
15           cflags: -fopenmp
16         intel:
17           cflags: -qopenmp
18         cray:
19           cflags: -h omp

In this example OMP_NUM_THREADS environment variable under the all section which will be used for all compiler groups. This example was built on Cori, we expect this test to run against every gcc, intel and PrgEnv-cray compiler module.

MPI Example

In this example we run a MPI Laplace code using 4 process on a KNL node using the intel/19.1.2.254 compiler. This test is run on Cori through batch queue system. We can define #SBATCH parameters using sbatch property. This program is compiled using mpiicc wrapper that can be defined using cc parameter.

buildtest would infer icc as compiler wrapper however since this is a MPI program we must specify mpiicc as the compiler wrapper to compile this MPI C program. It’s worth noting buildtest will detect the compiler wrapper under the compiler group defined under default such as intel, gnu, etc… since cc property is a property of intel this will inform buildtest to use the Intel provided wrappers.

This program is run using srun job launcher, we can control how test is executed using the run property. This test required we swap intel modules and load impi/2020 module.

 1buildspecs:
 2  laplace_mpi:
 3    type: compiler
 4    description: Laplace MPI code in C
 5    executor: slurm.knl_debug
 6    tags: ["mpi"]
 7    source: src/laplace_mpi.c
 8    compilers:
 9      name: ["^(intel/19.1.2.254)$"]
10      default:
11        all:
12          sbatch: ["-N 1", "-n 4"]
13          run: srun -n 4 $_EXEC
14        intel:
15          cc: mpiicc
16          cflags: -O3
17      config:
18        intel/19.1.2.254:
19          module:
20            load: [impi/2020.up4]
21            swap: [intel, intel/19.1.2.254]

The generated test script is show below, notice that mpiicc is used to compile the program and run via srun.

buildtest inspect query -t laplace_mpi
$ buildtest inspect query -t laplace_mpi
─────────────────────────────────────────────── laplace_mpi/1a4c7a6f-0f69-40e7-b451-f9f62843eee5 ────────────────────────────────────────────────
Executor: cori.slurm.knl_debug
Description: Laplace MPI code in C
State: PASS
Returncode: 0
Runtime: 31.496144 sec
Starttime: 2022/06/30 14:35:34
Endtime: 2022/06/30 14:36:06
Command: bash --norc --noprofile -eo pipefail laplace_mpi_build.sh
Test Script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/laplace_mpi/laplace_mpi/1a4c7a6f/laplace_mpi.sh
Build Script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/laplace_mpi/laplace_mpi/1a4c7a6f/laplace_mpi_build.sh
Output File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/laplace_mpi/laplace_mpi/1a4c7a6f/laplace_mpi.out
Error File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/laplace_mpi/laplace_mpi/1a4c7a6f/laplace_mpi.err
Log File: /global/u1/s/siddiq90/gitrepos/buildtest/var/logs/buildtest_e5cuwqhf.log
────── Test File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/laplace_mpi/laplace_mpi/1a4c7a6f/laplace_mpi.sh ───────
#!/bin/bash
#SBATCH -N 1
#SBATCH -n 4
#SBATCH --job-name=laplace_mpi
#SBATCH --output=laplace_mpi.out
#SBATCH --error=laplace_mpi.err


# name of executable
_EXEC=laplace_mpi.c.exe
# Loading modules
module swap intel intel/19.1.2.254
module load impi/2020.up4
# Compilation Line
mpiicc -O3 -o $_EXEC /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/mpi/src/laplace_mpi.c


# Run executable
srun -n 4 $_EXEC

OpenMP Hello Processor Scaling Test

In this example we have a OpenMP test that will be run across a set of processes. We define environment OMP_NUM_THREADS variable that will take value of BUILDTEST_NUMPROCS which is set when using buildtest build --procs. Our compiler of choice is PrgEnv-intel, the job will run via slurm scheduler. To compile OpenMP code for intel we use -qopenmp flag. Shown below is the source code and buildspec for this test.

// OpenMP program to print Hello World
// using C language

// OpenMP header
#include <omp.h>

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char* argv[])
{

    // Beginning of parallel region
    #pragma omp parallel
    {

        printf("Hello World... from thread = %d\n",
               omp_get_thread_num());
    }
    // Ending of parallel region
}
buildspecs:
  hello_world_openmp:
    type: compiler
    executor: cori.slurm.knl_debug
    source: src/hello.c
    description: Hello World OpenMP scaling example with processor count
    tags: [openmp]
    compilers:
      name: ["^PrgEnv-intel/6.0.5"]
      default:
        intel:
          env:
            OMP_NUM_THREADS: "$BUILDTEST_NUMPROCS"
          cflags: -qopenmp
          sbatch: ["-t 10"]

Next, we will run this test with 8, 16, and 24 processes. buildtest will create three builder objects for these test and each will run through slurm scheduler. The values will be set for OMP_NUM_THREADS. Shown below is the test execution

buildtest bd -b openmp_scale.yml --procs 8 16 24
$ buildtest bd -b openmp_scale.yml --procs 8 16 24
╭───────────────────────────────────────────────── buildtest summary ──────────────────────────────────────────────────╮
│                                                                                                                      │
│ User:               siddiq90                                                                                         │
│ Hostname:           cori10                                                                                           │
│ Platform:           Linux                                                                                            │
│ Current Time:       2022/06/30 14:39:12                                                                              │
│ buildtest path:     /global/homes/s/siddiq90/gitrepos/buildtest/bin/buildtest                                        │
│ buildtest version:  0.14.0                                                                                           │
│ python path:        /global/u1/s/siddiq90/.local/share/virtualenvs/buildtest-WqshQcL1/bin/python3                    │
│ python version:     3.9.7                                                                                            │
│ Configuration File: /global/u1/s/siddiq90/gitrepos/buildtest-nersc/config.yml                                        │
│ Test Directory:     /global/u1/s/siddiq90/gitrepos/buildtest/var/tests                                               │
│ Report File:        /global/u1/s/siddiq90/gitrepos/buildtest/var/report.json                                         │
│ Command:            /global/homes/s/siddiq90/gitrepos/buildtest/bin/buildtest bd -b openmp_scale.yml --procs 8 16 24 │
│                                                                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
────────────────────────────────────────────────────────────  Discovering Buildspecs ────────────────────────────────────────────────────────────
                                  Discovered buildspecs
╔════════════════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                              ║
╟────────────────────────────────────────────────────────────────────────────────────────╢
║ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_scale.yml ║
╚════════════════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────────────────────────────────────── Parsing Buildspecs ───────────────────────────────────────────────────────────────
Buildtest will parse 1 buildspecs
Valid Buildspecs: 1
Invalid Buildspecs: 0
/global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_scale.yml: VALID
Total builder objects created: 4
Total compiler builder: 4
Total script builder: 0
Total spack builder: 0
                                                            Compiler Builder Details
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder                   ┃ executor             ┃ compiler           ┃ nodes ┃ procs ┃ description               ┃ buildspecs                ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ hello_world_openmp/a7de0… │ cori.slurm.knl_debug │ PrgEnv-intel/6.0.5 │ None  │ None  │ Hello World OpenMP        │ /global/u1/s/siddiq90/gi… │
│                           │                      │                    │       │       │ scaling example with      │                           │
│                           │                      │                    │       │       │ processor count           │                           │
├───────────────────────────┼──────────────────────┼────────────────────┼───────┼───────┼───────────────────────────┼───────────────────────────┤
│ hello_world_openmp/ce755… │ cori.slurm.knl_debug │ PrgEnv-intel/6.0.5 │ None  │ 8     │ Hello World OpenMP        │ /global/u1/s/siddiq90/gi… │
│                           │                      │                    │       │       │ scaling example with      │                           │
│                           │                      │                    │       │       │ processor count           │                           │
├───────────────────────────┼──────────────────────┼────────────────────┼───────┼───────┼───────────────────────────┼───────────────────────────┤
│ hello_world_openmp/fa271… │ cori.slurm.knl_debug │ PrgEnv-intel/6.0.5 │ None  │ 16    │ Hello World OpenMP        │ /global/u1/s/siddiq90/gi… │
│                           │                      │                    │       │       │ scaling example with      │                           │
│                           │                      │                    │       │       │ processor count           │                           │
├───────────────────────────┼──────────────────────┼────────────────────┼───────┼───────┼───────────────────────────┼───────────────────────────┤
│ hello_world_openmp/0fe29… │ cori.slurm.knl_debug │ PrgEnv-intel/6.0.5 │ None  │ 24    │ Hello World OpenMP        │ /global/u1/s/siddiq90/gi… │
│                           │                      │                    │       │       │ scaling example with      │                           │
│                           │                      │                    │       │       │ processor count           │                           │
└───────────────────────────┴──────────────────────┴────────────────────┴───────┴───────┴───────────────────────────┴───────────────────────────┘
                                                              Batch Job Builders
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder                     ┃ executor             ┃ buildspecs                                                                             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ hello_world_openmp/a7de0abb │ cori.slurm.knl_debug │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_scale.yml │
├─────────────────────────────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ hello_world_openmp/ce755367 │ cori.slurm.knl_debug │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_scale.yml │
├─────────────────────────────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ hello_world_openmp/fa271571 │ cori.slurm.knl_debug │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_scale.yml │
├─────────────────────────────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ hello_world_openmp/0fe298ae │ cori.slurm.knl_debug │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_scale.yml │
└─────────────────────────────┴──────────────────────┴────────────────────────────────────────────────────────────────────────────────────────┘
                                                        Batch Job Builders by Processors
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder                     ┃ executor             ┃ procs ┃ buildspecs                                                                       ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ hello_world_openmp/ce755367 │ cori.slurm.knl_debug │ 8     │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_sca │
│                             │                      │       │ le.yml                                                                           │
├─────────────────────────────┼──────────────────────┼───────┼──────────────────────────────────────────────────────────────────────────────────┤
│ hello_world_openmp/fa271571 │ cori.slurm.knl_debug │ 16    │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_sca │
│                             │                      │       │ le.yml                                                                           │
├─────────────────────────────┼──────────────────────┼───────┼──────────────────────────────────────────────────────────────────────────────────┤
│ hello_world_openmp/0fe298ae │ cori.slurm.knl_debug │ 24    │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_sca │
│                             │                      │       │ le.yml                                                                           │
└─────────────────────────────┴──────────────────────┴───────┴──────────────────────────────────────────────────────────────────────────────────┘
───────────────────────────────────────────────────────────────── Building Test ─────────────────────────────────────────────────────────────────
hello_world_openmp/a7de0abb: Creating test directory:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/a7de0abb
hello_world_openmp/a7de0abb: Creating the stage directory:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/a7de0abb/stage
hello_world_openmp/a7de0abb: Writing build script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/a7de0abb/hello_world_openmp_build.sh
hello_world_openmp/ce755367: Creating test directory:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367
hello_world_openmp/ce755367: Creating the stage directory:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/stage
hello_world_openmp/ce755367: Writing build script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_openmp_build.sh
hello_world_openmp/fa271571: Creating test directory:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571
hello_world_openmp/fa271571: Creating the stage directory:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/stage
hello_world_openmp/fa271571: Writing build script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_openmp_build.sh
hello_world_openmp/0fe298ae: Creating test directory:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae
hello_world_openmp/0fe298ae: Creating the stage directory:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/stage
hello_world_openmp/0fe298ae: Writing build script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_openmp_build.sh
───────────────────────────────────────────────────────────────── Running Tests ─────────────────────────────────────────────────────────────────
Spawning 64 processes for processing builders
────────────────────────────────────────────────────────────────── Iteration 1 ──────────────────────────────────────────────────────────────────
hello_world_openmp/a7de0abb does not have any dependencies adding test to queue
hello_world_openmp/0fe298ae does not have any dependencies adding test to queue
hello_world_openmp/fa271571 does not have any dependencies adding test to queue
hello_world_openmp/ce755367 does not have any dependencies adding test to queue
In this iteration we are going to run the following tests: [hello_world_openmp/a7de0abb, hello_world_openmp/0fe298ae, hello_world_openmp/fa271571, hello_world_openmp/ce755367]
hello_world_openmp/0fe298ae: Running Test via command: bash --norc --noprofile -eo pipefail hello_world_openmp_build.sh
hello_world_openmp/ce755367: Running Test via command: bash --norc --noprofile -eo pipefail hello_world_openmp_build.sh
hello_world_openmp/a7de0abb: Running Test via command: bash --norc --noprofile -eo pipefail hello_world_openmp_build.sh
hello_world_openmp/fa271571: Running Test via command: bash --norc --noprofile -eo pipefail hello_world_openmp_build.sh
hello_world_openmp/0fe298ae: JobID 60681274 dispatched to scheduler
hello_world_openmp/a7de0abb: JobID 60681275 dispatched to scheduler
hello_world_openmp/ce755367: JobID 60681276 dispatched to scheduler
hello_world_openmp/fa271571: JobID 60681277 dispatched to scheduler
Polling Jobs in 30 seconds
hello_world_openmp/0fe298ae: Job 60681274 is complete!
hello_world_openmp/0fe298ae: Test completed in 31.868266 seconds
hello_world_openmp/0fe298ae: Test completed with returncode: 0
hello_world_openmp/0fe298ae: Writing output file -
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_openmp.out
hello_world_openmp/0fe298ae: Writing error file -
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_openmp.err
hello_world_openmp/ce755367: Job 60681276 is complete!
hello_world_openmp/ce755367: Test completed in 32.010719 seconds
hello_world_openmp/ce755367: Test completed with returncode: 0
hello_world_openmp/ce755367: Writing output file -
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_openmp.out
hello_world_openmp/ce755367: Writing error file -
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_openmp.err
                                     Running Jobs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                     ┃ executor             ┃ jobid    ┃ jobstate ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ hello_world_openmp/a7de0abb │ cori.slurm.knl_debug │ 60681275 │ RUNNING  │ 31.957  │
│ hello_world_openmp/fa271571 │ cori.slurm.knl_debug │ 60681277 │ RUNNING  │ 31.987  │
└─────────────────────────────┴──────────────────────┴──────────┴──────────┴─────────┘
                                     Completed Jobs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ builder                     ┃ executor             ┃ jobid    ┃ jobstate  ┃ runtime   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩
│ hello_world_openmp/0fe298ae │ cori.slurm.knl_debug │ 60681274 │ COMPLETED │ 31.868266 │
│ hello_world_openmp/ce755367 │ cori.slurm.knl_debug │ 60681276 │ COMPLETED │ 32.010719 │
└─────────────────────────────┴──────────────────────┴──────────┴───────────┴───────────┘
Polling Jobs in 30 seconds
hello_world_openmp/fa271571: Job 60681277 is complete!
hello_world_openmp/fa271571: Test completed in 62.153829 seconds
hello_world_openmp/fa271571: Test completed with returncode: 0
hello_world_openmp/fa271571: Writing output file -
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_openmp.out
hello_world_openmp/fa271571: Writing error file -
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_openmp.err
                                     Running Jobs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                     ┃ executor             ┃ jobid    ┃ jobstate ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ hello_world_openmp/a7de0abb │ cori.slurm.knl_debug │ 60681275 │ RUNNING  │ 62.132  │
└─────────────────────────────┴──────────────────────┴──────────┴──────────┴─────────┘
                                     Completed Jobs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ builder                     ┃ executor             ┃ jobid    ┃ jobstate  ┃ runtime   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩
│ hello_world_openmp/fa271571 │ cori.slurm.knl_debug │ 60681277 │ COMPLETED │ 62.153829 │
└─────────────────────────────┴──────────────────────┴──────────┴───────────┴───────────┘
Polling Jobs in 30 seconds
hello_world_openmp/a7de0abb: Job 60681275 is complete!
hello_world_openmp/a7de0abb: Test completed in 92.278197 seconds
hello_world_openmp/a7de0abb: Test completed with returncode: 0
hello_world_openmp/a7de0abb: Writing output file -
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/a7de0abb/hello_world_openmp.out
hello_world_openmp/a7de0abb: Writing error file -
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/a7de0abb/hello_world_openmp.err
                                     Completed Jobs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ builder                     ┃ executor             ┃ jobid    ┃ jobstate  ┃ runtime   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩
│ hello_world_openmp/a7de0abb │ cori.slurm.knl_debug │ 60681275 │ COMPLETED │ 92.278197 │
└─────────────────────────────┴──────────────────────┴──────────┴───────────┴───────────┘
                                                         Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ builder                     ┃ executor             ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returnCode ┃ runtime   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ hello_world_openmp/0fe298ae │ cori.slurm.knl_debug │ PASS   │ N/A N/A N/A                         │ 0          │ 31.868266 │
├─────────────────────────────┼──────────────────────┼────────┼─────────────────────────────────────┼────────────┼───────────┤
│ hello_world_openmp/a7de0abb │ cori.slurm.knl_debug │ PASS   │ N/A N/A N/A                         │ 0          │ 92.278197 │
├─────────────────────────────┼──────────────────────┼────────┼─────────────────────────────────────┼────────────┼───────────┤
│ hello_world_openmp/fa271571 │ cori.slurm.knl_debug │ PASS   │ N/A N/A N/A                         │ 0          │ 62.153829 │
├─────────────────────────────┼──────────────────────┼────────┼─────────────────────────────────────┼────────────┼───────────┤
│ hello_world_openmp/ce755367 │ cori.slurm.knl_debug │ PASS   │ N/A N/A N/A                         │ 0          │ 32.010719 │
└─────────────────────────────┴──────────────────────┴────────┴─────────────────────────────────────┴────────────┴───────────┘



Passed Tests: 4/4 Percentage: 100.000%
Failed Tests: 0/4 Percentage: 0.000%


Adding 4 test results to /global/u1/s/siddiq90/gitrepos/buildtest/var/report.json
Writing Logfile to: /global/u1/s/siddiq90/gitrepos/buildtest/var/logs/buildtest_ptr4xf10.log

Now let’s query the result via buildtest inspect query and examine the run. First we will need to specify the appropriate builder ids, we can specify builder name in quotes to specify a regular expression which buildtest understands when fetching record. In this test, we see that BUILDTEST_NUMPROCS is set for each test corresponding to value specified via --procs. In the build script you will notice the sbatch line for submitting the job will take into account the processor value. In the output we see each thread will print Hello World… from thread followed by name of thread where number of threads for these tests are controlled by value set by OMP_NUM_THREADS.

buildtest inspect query -t -o -b "hello_world_openmp/(fa|ce|0f)"
$ buildtest inspect query -t -o -b "hello_world_openmp/(fa|ce|0f)"
──────────────────────────────────────────── hello_world_openmp/fa271571-40e7-4a28-808c-f2ed38b47538 ────────────────────────────────────────────
Executor: cori.slurm.knl_debug
Description: Hello World OpenMP scaling example with processor count
State: PASS
Returncode: 0
Runtime: 62.153829 sec
Starttime: 2022/06/30 14:39:12
Endtime: 2022/06/30 14:40:14
Command: bash --norc --noprofile -eo pipefail hello_world_openmp_build.sh
Test Script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_openmp.sh
Build Script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_openmp_build.sh
Output File:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_openmp.out
Error File:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_openmp.err
Log File: /global/u1/s/siddiq90/gitrepos/buildtest/var/logs/buildtest_ptr4xf10.log
─ Output File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_op… ─
Hello World... from thread = 0
Hello World... from thread = 12
Hello World... from thread = 13
Hello World... from thread = 8
Hello World... from thread = 4
Hello World... from thread = 9
Hello World... from thread = 5
Hello World... from thread = 14
Hello World... from thread = 11
Hello World... from thread = 6
Hello World... from thread = 10
Hello World... from thread = 7
Hello World... from thread = 1
Hello World... from thread = 3
Hello World... from thread = 2
Hello World... from thread = 15

─ Test File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_open… ─
#!/bin/bash
#SBATCH -t 10
#SBATCH --job-name=hello_world_openmp
#SBATCH --output=hello_world_openmp.out
#SBATCH --error=hello_world_openmp.err


# name of executable
_EXEC=hello.c.exe
export OMP_NUM_THREADS="$BUILDTEST_NUMPROCS"
# Loading modules
module load PrgEnv-intel/6.0.5
# Compilation Line
cc -qopenmp -o $_EXEC /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/src/hello.c


# Run executable
./$_EXEC


─ Test File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_open… ─
#!/bin/bash
export BUILDTEST_TEST_NAME=hello_world_openmp
export BUILDTEST_TEST_ROOT=/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571
export BUILDTEST_BUILDSPEC_DIR=/global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp
export BUILDTEST_STAGE_DIR=/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/stage
export BUILDTEST_NUMPROCS=16
# source executor startup script
source /global/u1/s/siddiq90/gitrepos/buildtest/var/executor/cori.slurm.knl_debug/before_script.sh
# Run generated script
sbatch --parsable -q debug --clusters=cori -n 16 -C knl,quad,cache /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp
# Get return code
returncode=$?
# Exit with return code
exit $returncode
──────────────────────────────────────────── hello_world_openmp/0fe298ae-6704-4a3c-8253-1767e25e6edb ────────────────────────────────────────────
Executor: cori.slurm.knl_debug
Description: Hello World OpenMP scaling example with processor count
State: PASS
Returncode: 0
Runtime: 31.868266 sec
Starttime: 2022/06/30 14:39:12
Endtime: 2022/06/30 14:39:44
Command: bash --norc --noprofile -eo pipefail hello_world_openmp_build.sh
Test Script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_openmp.sh
Build Script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_openmp_build.sh
Output File:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_openmp.out
Error File:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_openmp.err
Log File: /global/u1/s/siddiq90/gitrepos/buildtest/var/logs/buildtest_ptr4xf10.log
─ Output File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_op… ─
Hello World... from thread = 0
Hello World... from thread = 8
Hello World... from thread = 12
Hello World... from thread = 16
Hello World... from thread = 13
Hello World... from thread = 9
Hello World... from thread = 20
Hello World... from thread = 17
Hello World... from thread = 21
Hello World... from thread = 15
Hello World... from thread = 10
Hello World... from thread = 19
Hello World... from thread = 14
Hello World... from thread = 18
Hello World... from thread = 4
Hello World... from thread = 5
Hello World... from thread = 11
Hello World... from thread = 1
Hello World... from thread = 23
Hello World... from thread = 22
Hello World... from thread = 3
Hello World... from thread = 7
Hello World... from thread = 6
Hello World... from thread = 2

─ Test File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_open… ─
#!/bin/bash
#SBATCH -t 10
#SBATCH --job-name=hello_world_openmp
#SBATCH --output=hello_world_openmp.out
#SBATCH --error=hello_world_openmp.err


# name of executable
_EXEC=hello.c.exe
export OMP_NUM_THREADS="$BUILDTEST_NUMPROCS"
# Loading modules
module load PrgEnv-intel/6.0.5
# Compilation Line
cc -qopenmp -o $_EXEC /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/src/hello.c


# Run executable
./$_EXEC


─ Test File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_open… ─
#!/bin/bash
export BUILDTEST_TEST_NAME=hello_world_openmp
export BUILDTEST_TEST_ROOT=/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae
export BUILDTEST_BUILDSPEC_DIR=/global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp
export BUILDTEST_STAGE_DIR=/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/stage
export BUILDTEST_NUMPROCS=24
# source executor startup script
source /global/u1/s/siddiq90/gitrepos/buildtest/var/executor/cori.slurm.knl_debug/before_script.sh
# Run generated script
sbatch --parsable -q debug --clusters=cori -n 24 -C knl,quad,cache /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp
# Get return code
returncode=$?
# Exit with return code
exit $returncode
──────────────────────────────────────────── hello_world_openmp/ce755367-4155-4721-adfd-2bd2aad36f46 ────────────────────────────────────────────
Executor: cori.slurm.knl_debug
Description: Hello World OpenMP scaling example with processor count
State: PASS
Returncode: 0
Runtime: 32.010719 sec
Starttime: 2022/06/30 14:39:12
Endtime: 2022/06/30 14:39:44
Command: bash --norc --noprofile -eo pipefail hello_world_openmp_build.sh
Test Script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_openmp.sh
Build Script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_openmp_build.sh
Output File:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_openmp.out
Error File:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_openmp.err
Log File: /global/u1/s/siddiq90/gitrepos/buildtest/var/logs/buildtest_ptr4xf10.log
─ Output File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_op… ─
Hello World... from thread = 0
Hello World... from thread = 3
Hello World... from thread = 4
Hello World... from thread = 6
Hello World... from thread = 5
Hello World... from thread = 1
Hello World... from thread = 2
Hello World... from thread = 7

─ Test File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_open… ─
#!/bin/bash
#SBATCH -t 10
#SBATCH --job-name=hello_world_openmp
#SBATCH --output=hello_world_openmp.out
#SBATCH --error=hello_world_openmp.err


# name of executable
_EXEC=hello.c.exe
export OMP_NUM_THREADS="$BUILDTEST_NUMPROCS"
# Loading modules
module load PrgEnv-intel/6.0.5
# Compilation Line
cc -qopenmp -o $_EXEC /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/src/hello.c


# Run executable
./$_EXEC


─ Test File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_open… ─
#!/bin/bash
export BUILDTEST_TEST_NAME=hello_world_openmp
export BUILDTEST_TEST_ROOT=/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367
export BUILDTEST_BUILDSPEC_DIR=/global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp
export BUILDTEST_STAGE_DIR=/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/stage
export BUILDTEST_NUMPROCS=8
# source executor startup script
source /global/u1/s/siddiq90/gitrepos/buildtest/var/executor/cori.slurm.knl_debug/before_script.sh
# Run generated script
sbatch --parsable -q debug --clusters=cori -n 8 -C knl,quad,cache /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_
# Get return code
returncode=$?
# Exit with return code
exit $returncode

Cray Burst Buffer & Data Warp

For Cray systems, you may want to stage-in or stage-out into your burst buffer to perform some computation. We can do this via #DW directive. You can see some example jobs using Burst Buffer at NERSC https://docs.nersc.gov/jobs/examples/#burst-buffer

In buildtest we support properties BB and DW which is a list of job directives that get inserted as #BW and #DW into the test script. We will create a persistent burst buffer named databuffer of size 10GB striped. We access the burst buffer using the DW directive. Finally we cd into the databuffer and write a 5GB random file.

Note

BB and DW directives are generated after scheduler directives. The #BB comes before #DW. buildtest will automatically add the directive #BB and #DW when using properties BB and DW

 1buildspecs:
 2  create_burst_buffer:
 3    type: script
 4    executor: cori.slurm.knl_debug
 5    description: Create a burst buffer
 6    tags: [jobs, cray]
 7    sbatch: ["-N 1", "-t 5", "-n 1"]
 8    BB:
 9      - create_persistent name=databuffer capacity=10GB access_mode=striped type=scratch
10    DW:
11      - persistentdw name=databuffer
12    run: |
13      cd $DW_PERSISTENT_STRIPED_databuffer
14      pwd
15      dd if=/dev/urandom of=random.txt bs=1G count=5 iflag=fullblock
16      ls -lh $DW_PERSISTENT_STRIPED_databuffer/

Next we ran this test and inspect the generated test, we see that buildtest will insert the #BB and #DW directives as specified in the buildspec. In the output we see a 5GB file called random.txt was written to the burst buffer.

$ buildtest inspect query -o -t create_burst_buffer
─────────────────────────────────────────── create_burst_buffer/2bc01707-c910-4091-ab9c-b14e5f6d56e5 ────────────────────────────────────────────
Executor: cori.slurm.knl_debug
Description: Create a burst buffer
State: PASS
Returncode: 0
Runtime: 120.848841 sec
Starttime: 2022/06/30 14:46:45
Endtime: 2022/06/30 14:48:46
Command: bash --norc --noprofile -eo pipefail create_burst_buffer_build.sh
Test Script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/create_buffer/create_burst_buffer/2bc01707/create_burst_buffer.sh
Build Script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/create_buffer/create_burst_buffer/2bc01707/create_burst_buffer_build.sh
Output File:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/create_buffer/create_burst_buffer/2bc01707/create_burst_buffer.out
Error File:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/create_buffer/create_burst_buffer/2bc01707/create_burst_buffer.err
Log File: /global/u1/s/siddiq90/gitrepos/buildtest/var/logs/buildtest_yw8xatj8.log
─ Output File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/create_buffer/create_burst_buffer/2bc01707/create_burst… ─
/var/opt/cray/dws/mounts/batch/databuffer_60681470_striped_scratch
total 5.0G
-rw-rw-r-- 1 siddiq90 siddiq90 5.0G Jun 30 14:46 random.txt

─ Test File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/create_buffer/create_burst_buffer/2bc01707/create_burst_b… ─
#!/bin/bash
#SBATCH -N 1
#SBATCH -t 5
#SBATCH -n 1
#SBATCH --job-name=create_burst_buffer
#SBATCH --output=create_burst_buffer.out
#SBATCH --error=create_burst_buffer.err
####### START OF BURST BUFFER DIRECTIVES #######
#BB create_persistent name=databuffer capacity=10GB access_mode=striped type=scratch
####### END OF BURST BUFFER DIRECTIVES   #######
####### START OF DATAWARP DIRECTIVES #######
#DW persistentdw name=databuffer
####### END OF DATAWARP DIRECTIVES   #######
# Content of run section
cd $DW_PERSISTENT_STRIPED_databuffer
pwd
dd if=/dev/urandom of=random.txt bs=1G count=5 iflag=fullblock
ls -lh $DW_PERSISTENT_STRIPED_databuffer/