Facility Test Examples
We will discuss a few test examples implemented at NERSC that are available at https://github.com/buildtesters/buildtest-nersc/.
Single Test Multiple Compilers
It’s possible to run single test across multiple compilers (gcc, intel, cray, etc…). In the
next example, we will build an OpenMP reduction test using gcc, intel and cray compilers. In this
test, we use name field to select compilers that start with gcc, intel and PrgEnv-cray
as compiler names. The default section is organized by compiler groups which inherits compiler flags
for all compilers. OpenMP flag for gcc, intel and cray differ for instance one must use -fopenmp for gcc,
--qopenmp for intel and -h omp for cray.
1 buildspecs:
2 reduction:
3 type: compiler
4 executor: cori.local.bash
5 source: src/reduction.c
6 description: OpenMP reduction example using gcc, intel and cray compiler
7 tags: [openmp]
8 compilers:
9 name: ["^(gcc|intel|PrgEnv-cray)"]
10 default:
11 all:
12 env:
13 OMP_NUM_THREADS: 4
14 gcc:
15 cflags: -fopenmp
16 intel:
17 cflags: -qopenmp
18 cray:
19 cflags: -h omp
In this example OMP_NUM_THREADS environment variable under the all section which
will be used for all compiler groups. This example was built on Cori, we expect this
test to run against every gcc, intel and PrgEnv-cray compiler module.
MPI Example
In this example we run a MPI Laplace code using 4 process on a KNL node using
the intel/19.1.2.254 compiler. This test is run on Cori through batch queue
system. We can define #SBATCH parameters using sbatch property. This program
is compiled using mpiicc wrapper that can be defined using cc parameter.
buildtest would infer icc as compiler wrapper however since this is a MPI program we must specify mpiicc as the compiler wrapper to compile this MPI C program.
It’s worth noting buildtest will detect the compiler wrapper under the compiler group defined under default such as intel, gnu, etc… since cc property is
a property of intel this will inform buildtest to use the Intel provided wrappers.
This program is run using srun job launcher, we can control
how test is executed using the run property. This test required we swap intel
modules and load impi/2020 module.
1buildspecs:
2 laplace_mpi:
3 type: compiler
4 description: Laplace MPI code in C
5 executor: slurm.knl_debug
6 tags: ["mpi"]
7 source: src/laplace_mpi.c
8 compilers:
9 name: ["^(intel/19.1.2.254)$"]
10 default:
11 all:
12 sbatch: ["-N 1", "-n 4"]
13 run: srun -n 4 $_EXEC
14 intel:
15 cc: mpiicc
16 cflags: -O3
17 config:
18 intel/19.1.2.254:
19 module:
20 load: [impi/2020.up4]
21 swap: [intel, intel/19.1.2.254]
The generated test script is show below, notice that mpiicc is used to compile the program and run via srun.
buildtest inspect query -t laplace_mpi
$ buildtest inspect query -t laplace_mpi
─────────────────────────────────────────────── laplace_mpi/1a4c7a6f-0f69-40e7-b451-f9f62843eee5 ────────────────────────────────────────────────
Executor: cori.slurm.knl_debug
Description: Laplace MPI code in C
State: PASS
Returncode: 0
Runtime: 31.496144 sec
Starttime: 2022/06/30 14:35:34
Endtime: 2022/06/30 14:36:06
Command: bash --norc --noprofile -eo pipefail laplace_mpi_build.sh
Test Script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/laplace_mpi/laplace_mpi/1a4c7a6f/laplace_mpi.sh
Build Script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/laplace_mpi/laplace_mpi/1a4c7a6f/laplace_mpi_build.sh
Output File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/laplace_mpi/laplace_mpi/1a4c7a6f/laplace_mpi.out
Error File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/laplace_mpi/laplace_mpi/1a4c7a6f/laplace_mpi.err
Log File: /global/u1/s/siddiq90/gitrepos/buildtest/var/logs/buildtest_e5cuwqhf.log
────── Test File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/laplace_mpi/laplace_mpi/1a4c7a6f/laplace_mpi.sh ───────
#!/bin/bash
#SBATCH -N 1
#SBATCH -n 4
#SBATCH --job-name=laplace_mpi
#SBATCH --output=laplace_mpi.out
#SBATCH --error=laplace_mpi.err
# name of executable
_EXEC=laplace_mpi.c.exe
# Loading modules
module swap intel intel/19.1.2.254
module load impi/2020.up4
# Compilation Line
mpiicc -O3 -o $_EXEC /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/mpi/src/laplace_mpi.c
# Run executable
srun -n 4 $_EXEC
OpenMP Hello Processor Scaling Test
In this example we have a OpenMP test that will be run across a set of processes. We define environment OMP_NUM_THREADS variable that will
take value of BUILDTEST_NUMPROCS which is set when using buildtest build --procs. Our compiler of choice is PrgEnv-intel,
the job will run via slurm scheduler. To compile OpenMP code for intel we use -qopenmp flag.
Shown below is the source code and buildspec for this test.
// OpenMP program to print Hello World
// using C language
// OpenMP header
#include <omp.h>
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char* argv[])
{
// Beginning of parallel region
#pragma omp parallel
{
printf("Hello World... from thread = %d\n",
omp_get_thread_num());
}
// Ending of parallel region
}
buildspecs:
hello_world_openmp:
type: compiler
executor: cori.slurm.knl_debug
source: src/hello.c
description: Hello World OpenMP scaling example with processor count
tags: [openmp]
compilers:
name: ["^PrgEnv-intel/6.0.5"]
default:
intel:
env:
OMP_NUM_THREADS: "$BUILDTEST_NUMPROCS"
cflags: -qopenmp
sbatch: ["-t 10"]
Next, we will run this test with 8, 16, and 24 processes. buildtest will create three builder objects for these test and each will run through slurm
scheduler. The values will be set for OMP_NUM_THREADS. Shown below is the test execution
buildtest bd -b openmp_scale.yml --procs 8 16 24
$ buildtest bd -b openmp_scale.yml --procs 8 16 24
╭───────────────────────────────────────────────── buildtest summary ──────────────────────────────────────────────────╮
│ │
│ User: siddiq90 │
│ Hostname: cori10 │
│ Platform: Linux │
│ Current Time: 2022/06/30 14:39:12 │
│ buildtest path: /global/homes/s/siddiq90/gitrepos/buildtest/bin/buildtest │
│ buildtest version: 0.14.0 │
│ python path: /global/u1/s/siddiq90/.local/share/virtualenvs/buildtest-WqshQcL1/bin/python3 │
│ python version: 3.9.7 │
│ Configuration File: /global/u1/s/siddiq90/gitrepos/buildtest-nersc/config.yml │
│ Test Directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests │
│ Report File: /global/u1/s/siddiq90/gitrepos/buildtest/var/report.json │
│ Command: /global/homes/s/siddiq90/gitrepos/buildtest/bin/buildtest bd -b openmp_scale.yml --procs 8 16 24 │
│ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
──────────────────────────────────────────────────────────── Discovering Buildspecs ────────────────────────────────────────────────────────────
Discovered buildspecs
╔════════════════════════════════════════════════════════════════════════════════════════╗
║ buildspec ║
╟────────────────────────────────────────────────────────────────────────────────────────╢
║ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_scale.yml ║
╚════════════════════════════════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
────────────────────────────────────────────────────────────── Parsing Buildspecs ───────────────────────────────────────────────────────────────
Buildtest will parse 1 buildspecs
Valid Buildspecs: 1
Invalid Buildspecs: 0
/global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_scale.yml: VALID
Total builder objects created: 4
Total compiler builder: 4
Total script builder: 0
Total spack builder: 0
Compiler Builder Details
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ description ┃ buildspecs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ hello_world_openmp/a7de0… │ cori.slurm.knl_debug │ PrgEnv-intel/6.0.5 │ None │ None │ Hello World OpenMP │ /global/u1/s/siddiq90/gi… │
│ │ │ │ │ │ scaling example with │ │
│ │ │ │ │ │ processor count │ │
├───────────────────────────┼──────────────────────┼────────────────────┼───────┼───────┼───────────────────────────┼───────────────────────────┤
│ hello_world_openmp/ce755… │ cori.slurm.knl_debug │ PrgEnv-intel/6.0.5 │ None │ 8 │ Hello World OpenMP │ /global/u1/s/siddiq90/gi… │
│ │ │ │ │ │ scaling example with │ │
│ │ │ │ │ │ processor count │ │
├───────────────────────────┼──────────────────────┼────────────────────┼───────┼───────┼───────────────────────────┼───────────────────────────┤
│ hello_world_openmp/fa271… │ cori.slurm.knl_debug │ PrgEnv-intel/6.0.5 │ None │ 16 │ Hello World OpenMP │ /global/u1/s/siddiq90/gi… │
│ │ │ │ │ │ scaling example with │ │
│ │ │ │ │ │ processor count │ │
├───────────────────────────┼──────────────────────┼────────────────────┼───────┼───────┼───────────────────────────┼───────────────────────────┤
│ hello_world_openmp/0fe29… │ cori.slurm.knl_debug │ PrgEnv-intel/6.0.5 │ None │ 24 │ Hello World OpenMP │ /global/u1/s/siddiq90/gi… │
│ │ │ │ │ │ scaling example with │ │
│ │ │ │ │ │ processor count │ │
└───────────────────────────┴──────────────────────┴────────────────────┴───────┴───────┴───────────────────────────┴───────────────────────────┘
Batch Job Builders
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder ┃ executor ┃ buildspecs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ hello_world_openmp/a7de0abb │ cori.slurm.knl_debug │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_scale.yml │
├─────────────────────────────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ hello_world_openmp/ce755367 │ cori.slurm.knl_debug │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_scale.yml │
├─────────────────────────────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ hello_world_openmp/fa271571 │ cori.slurm.knl_debug │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_scale.yml │
├─────────────────────────────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ hello_world_openmp/0fe298ae │ cori.slurm.knl_debug │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_scale.yml │
└─────────────────────────────┴──────────────────────┴────────────────────────────────────────────────────────────────────────────────────────┘
Batch Job Builders by Processors
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder ┃ executor ┃ procs ┃ buildspecs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ hello_world_openmp/ce755367 │ cori.slurm.knl_debug │ 8 │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_sca │
│ │ │ │ le.yml │
├─────────────────────────────┼──────────────────────┼───────┼──────────────────────────────────────────────────────────────────────────────────┤
│ hello_world_openmp/fa271571 │ cori.slurm.knl_debug │ 16 │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_sca │
│ │ │ │ le.yml │
├─────────────────────────────┼──────────────────────┼───────┼──────────────────────────────────────────────────────────────────────────────────┤
│ hello_world_openmp/0fe298ae │ cori.slurm.knl_debug │ 24 │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/openmp_sca │
│ │ │ │ le.yml │
└─────────────────────────────┴──────────────────────┴───────┴──────────────────────────────────────────────────────────────────────────────────┘
───────────────────────────────────────────────────────────────── Building Test ─────────────────────────────────────────────────────────────────
hello_world_openmp/a7de0abb: Creating test directory:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/a7de0abb
hello_world_openmp/a7de0abb: Creating the stage directory:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/a7de0abb/stage
hello_world_openmp/a7de0abb: Writing build script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/a7de0abb/hello_world_openmp_build.sh
hello_world_openmp/ce755367: Creating test directory:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367
hello_world_openmp/ce755367: Creating the stage directory:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/stage
hello_world_openmp/ce755367: Writing build script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_openmp_build.sh
hello_world_openmp/fa271571: Creating test directory:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571
hello_world_openmp/fa271571: Creating the stage directory:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/stage
hello_world_openmp/fa271571: Writing build script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_openmp_build.sh
hello_world_openmp/0fe298ae: Creating test directory:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae
hello_world_openmp/0fe298ae: Creating the stage directory:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/stage
hello_world_openmp/0fe298ae: Writing build script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_openmp_build.sh
───────────────────────────────────────────────────────────────── Running Tests ─────────────────────────────────────────────────────────────────
Spawning 64 processes for processing builders
────────────────────────────────────────────────────────────────── Iteration 1 ──────────────────────────────────────────────────────────────────
hello_world_openmp/a7de0abb does not have any dependencies adding test to queue
hello_world_openmp/0fe298ae does not have any dependencies adding test to queue
hello_world_openmp/fa271571 does not have any dependencies adding test to queue
hello_world_openmp/ce755367 does not have any dependencies adding test to queue
In this iteration we are going to run the following tests: [hello_world_openmp/a7de0abb, hello_world_openmp/0fe298ae, hello_world_openmp/fa271571, hello_world_openmp/ce755367]
hello_world_openmp/0fe298ae: Running Test via command: bash --norc --noprofile -eo pipefail hello_world_openmp_build.sh
hello_world_openmp/ce755367: Running Test via command: bash --norc --noprofile -eo pipefail hello_world_openmp_build.sh
hello_world_openmp/a7de0abb: Running Test via command: bash --norc --noprofile -eo pipefail hello_world_openmp_build.sh
hello_world_openmp/fa271571: Running Test via command: bash --norc --noprofile -eo pipefail hello_world_openmp_build.sh
hello_world_openmp/0fe298ae: JobID 60681274 dispatched to scheduler
hello_world_openmp/a7de0abb: JobID 60681275 dispatched to scheduler
hello_world_openmp/ce755367: JobID 60681276 dispatched to scheduler
hello_world_openmp/fa271571: JobID 60681277 dispatched to scheduler
Polling Jobs in 30 seconds
hello_world_openmp/0fe298ae: Job 60681274 is complete!
hello_world_openmp/0fe298ae: Test completed in 31.868266 seconds
hello_world_openmp/0fe298ae: Test completed with returncode: 0
hello_world_openmp/0fe298ae: Writing output file -
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_openmp.out
hello_world_openmp/0fe298ae: Writing error file -
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_openmp.err
hello_world_openmp/ce755367: Job 60681276 is complete!
hello_world_openmp/ce755367: Test completed in 32.010719 seconds
hello_world_openmp/ce755367: Test completed with returncode: 0
hello_world_openmp/ce755367: Writing output file -
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_openmp.out
hello_world_openmp/ce755367: Writing error file -
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_openmp.err
Running Jobs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ jobid ┃ jobstate ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ hello_world_openmp/a7de0abb │ cori.slurm.knl_debug │ 60681275 │ RUNNING │ 31.957 │
│ hello_world_openmp/fa271571 │ cori.slurm.knl_debug │ 60681277 │ RUNNING │ 31.987 │
└─────────────────────────────┴──────────────────────┴──────────┴──────────┴─────────┘
Completed Jobs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ builder ┃ executor ┃ jobid ┃ jobstate ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩
│ hello_world_openmp/0fe298ae │ cori.slurm.knl_debug │ 60681274 │ COMPLETED │ 31.868266 │
│ hello_world_openmp/ce755367 │ cori.slurm.knl_debug │ 60681276 │ COMPLETED │ 32.010719 │
└─────────────────────────────┴──────────────────────┴──────────┴───────────┴───────────┘
Polling Jobs in 30 seconds
hello_world_openmp/fa271571: Job 60681277 is complete!
hello_world_openmp/fa271571: Test completed in 62.153829 seconds
hello_world_openmp/fa271571: Test completed with returncode: 0
hello_world_openmp/fa271571: Writing output file -
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_openmp.out
hello_world_openmp/fa271571: Writing error file -
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_openmp.err
Running Jobs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder ┃ executor ┃ jobid ┃ jobstate ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ hello_world_openmp/a7de0abb │ cori.slurm.knl_debug │ 60681275 │ RUNNING │ 62.132 │
└─────────────────────────────┴──────────────────────┴──────────┴──────────┴─────────┘
Completed Jobs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ builder ┃ executor ┃ jobid ┃ jobstate ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩
│ hello_world_openmp/fa271571 │ cori.slurm.knl_debug │ 60681277 │ COMPLETED │ 62.153829 │
└─────────────────────────────┴──────────────────────┴──────────┴───────────┴───────────┘
Polling Jobs in 30 seconds
hello_world_openmp/a7de0abb: Job 60681275 is complete!
hello_world_openmp/a7de0abb: Test completed in 92.278197 seconds
hello_world_openmp/a7de0abb: Test completed with returncode: 0
hello_world_openmp/a7de0abb: Writing output file -
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/a7de0abb/hello_world_openmp.out
hello_world_openmp/a7de0abb: Writing error file -
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/a7de0abb/hello_world_openmp.err
Completed Jobs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ builder ┃ executor ┃ jobid ┃ jobstate ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩
│ hello_world_openmp/a7de0abb │ cori.slurm.knl_debug │ 60681275 │ COMPLETED │ 92.278197 │
└─────────────────────────────┴──────────────────────┴──────────┴───────────┴───────────┘
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returnCode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ hello_world_openmp/0fe298ae │ cori.slurm.knl_debug │ PASS │ N/A N/A N/A │ 0 │ 31.868266 │
├─────────────────────────────┼──────────────────────┼────────┼─────────────────────────────────────┼────────────┼───────────┤
│ hello_world_openmp/a7de0abb │ cori.slurm.knl_debug │ PASS │ N/A N/A N/A │ 0 │ 92.278197 │
├─────────────────────────────┼──────────────────────┼────────┼─────────────────────────────────────┼────────────┼───────────┤
│ hello_world_openmp/fa271571 │ cori.slurm.knl_debug │ PASS │ N/A N/A N/A │ 0 │ 62.153829 │
├─────────────────────────────┼──────────────────────┼────────┼─────────────────────────────────────┼────────────┼───────────┤
│ hello_world_openmp/ce755367 │ cori.slurm.knl_debug │ PASS │ N/A N/A N/A │ 0 │ 32.010719 │
└─────────────────────────────┴──────────────────────┴────────┴─────────────────────────────────────┴────────────┴───────────┘
Passed Tests: 4/4 Percentage: 100.000%
Failed Tests: 0/4 Percentage: 0.000%
Adding 4 test results to /global/u1/s/siddiq90/gitrepos/buildtest/var/report.json
Writing Logfile to: /global/u1/s/siddiq90/gitrepos/buildtest/var/logs/buildtest_ptr4xf10.log
Now let’s query the result via buildtest inspect query and examine the run. First we will need to specify the appropriate builder ids, we can specify
builder name in quotes to specify a regular expression which buildtest understands when fetching record. In this test, we see that BUILDTEST_NUMPROCS is
set for each test corresponding to value specified via --procs. In the build script you will notice the sbatch line for submitting the job will take into
account the processor value. In the output we see each thread will print Hello World… from thread followed by name of thread where number of threads for these
tests are controlled by value set by OMP_NUM_THREADS.
buildtest inspect query -t -o -b "hello_world_openmp/(fa|ce|0f)"
$ buildtest inspect query -t -o -b "hello_world_openmp/(fa|ce|0f)"
──────────────────────────────────────────── hello_world_openmp/fa271571-40e7-4a28-808c-f2ed38b47538 ────────────────────────────────────────────
Executor: cori.slurm.knl_debug
Description: Hello World OpenMP scaling example with processor count
State: PASS
Returncode: 0
Runtime: 62.153829 sec
Starttime: 2022/06/30 14:39:12
Endtime: 2022/06/30 14:40:14
Command: bash --norc --noprofile -eo pipefail hello_world_openmp_build.sh
Test Script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_openmp.sh
Build Script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_openmp_build.sh
Output File:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_openmp.out
Error File:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_openmp.err
Log File: /global/u1/s/siddiq90/gitrepos/buildtest/var/logs/buildtest_ptr4xf10.log
─ Output File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_op… ─
Hello World... from thread = 0
Hello World... from thread = 12
Hello World... from thread = 13
Hello World... from thread = 8
Hello World... from thread = 4
Hello World... from thread = 9
Hello World... from thread = 5
Hello World... from thread = 14
Hello World... from thread = 11
Hello World... from thread = 6
Hello World... from thread = 10
Hello World... from thread = 7
Hello World... from thread = 1
Hello World... from thread = 3
Hello World... from thread = 2
Hello World... from thread = 15
─ Test File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_open… ─
#!/bin/bash
#SBATCH -t 10
#SBATCH --job-name=hello_world_openmp
#SBATCH --output=hello_world_openmp.out
#SBATCH --error=hello_world_openmp.err
# name of executable
_EXEC=hello.c.exe
export OMP_NUM_THREADS="$BUILDTEST_NUMPROCS"
# Loading modules
module load PrgEnv-intel/6.0.5
# Compilation Line
cc -qopenmp -o $_EXEC /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/src/hello.c
# Run executable
./$_EXEC
─ Test File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/hello_world_open… ─
#!/bin/bash
export BUILDTEST_TEST_NAME=hello_world_openmp
export BUILDTEST_TEST_ROOT=/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571
export BUILDTEST_BUILDSPEC_DIR=/global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp
export BUILDTEST_STAGE_DIR=/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/fa271571/stage
export BUILDTEST_NUMPROCS=16
# source executor startup script
source /global/u1/s/siddiq90/gitrepos/buildtest/var/executor/cori.slurm.knl_debug/before_script.sh
# Run generated script
sbatch --parsable -q debug --clusters=cori -n 16 -C knl,quad,cache /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp
# Get return code
returncode=$?
# Exit with return code
exit $returncode
──────────────────────────────────────────── hello_world_openmp/0fe298ae-6704-4a3c-8253-1767e25e6edb ────────────────────────────────────────────
Executor: cori.slurm.knl_debug
Description: Hello World OpenMP scaling example with processor count
State: PASS
Returncode: 0
Runtime: 31.868266 sec
Starttime: 2022/06/30 14:39:12
Endtime: 2022/06/30 14:39:44
Command: bash --norc --noprofile -eo pipefail hello_world_openmp_build.sh
Test Script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_openmp.sh
Build Script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_openmp_build.sh
Output File:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_openmp.out
Error File:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_openmp.err
Log File: /global/u1/s/siddiq90/gitrepos/buildtest/var/logs/buildtest_ptr4xf10.log
─ Output File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_op… ─
Hello World... from thread = 0
Hello World... from thread = 8
Hello World... from thread = 12
Hello World... from thread = 16
Hello World... from thread = 13
Hello World... from thread = 9
Hello World... from thread = 20
Hello World... from thread = 17
Hello World... from thread = 21
Hello World... from thread = 15
Hello World... from thread = 10
Hello World... from thread = 19
Hello World... from thread = 14
Hello World... from thread = 18
Hello World... from thread = 4
Hello World... from thread = 5
Hello World... from thread = 11
Hello World... from thread = 1
Hello World... from thread = 23
Hello World... from thread = 22
Hello World... from thread = 3
Hello World... from thread = 7
Hello World... from thread = 6
Hello World... from thread = 2
─ Test File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_open… ─
#!/bin/bash
#SBATCH -t 10
#SBATCH --job-name=hello_world_openmp
#SBATCH --output=hello_world_openmp.out
#SBATCH --error=hello_world_openmp.err
# name of executable
_EXEC=hello.c.exe
export OMP_NUM_THREADS="$BUILDTEST_NUMPROCS"
# Loading modules
module load PrgEnv-intel/6.0.5
# Compilation Line
cc -qopenmp -o $_EXEC /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/src/hello.c
# Run executable
./$_EXEC
─ Test File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/hello_world_open… ─
#!/bin/bash
export BUILDTEST_TEST_NAME=hello_world_openmp
export BUILDTEST_TEST_ROOT=/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae
export BUILDTEST_BUILDSPEC_DIR=/global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp
export BUILDTEST_STAGE_DIR=/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/0fe298ae/stage
export BUILDTEST_NUMPROCS=24
# source executor startup script
source /global/u1/s/siddiq90/gitrepos/buildtest/var/executor/cori.slurm.knl_debug/before_script.sh
# Run generated script
sbatch --parsable -q debug --clusters=cori -n 24 -C knl,quad,cache /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp
# Get return code
returncode=$?
# Exit with return code
exit $returncode
──────────────────────────────────────────── hello_world_openmp/ce755367-4155-4721-adfd-2bd2aad36f46 ────────────────────────────────────────────
Executor: cori.slurm.knl_debug
Description: Hello World OpenMP scaling example with processor count
State: PASS
Returncode: 0
Runtime: 32.010719 sec
Starttime: 2022/06/30 14:39:12
Endtime: 2022/06/30 14:39:44
Command: bash --norc --noprofile -eo pipefail hello_world_openmp_build.sh
Test Script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_openmp.sh
Build Script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_openmp_build.sh
Output File:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_openmp.out
Error File:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_openmp.err
Log File: /global/u1/s/siddiq90/gitrepos/buildtest/var/logs/buildtest_ptr4xf10.log
─ Output File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_op… ─
Hello World... from thread = 0
Hello World... from thread = 3
Hello World... from thread = 4
Hello World... from thread = 6
Hello World... from thread = 5
Hello World... from thread = 1
Hello World... from thread = 2
Hello World... from thread = 7
─ Test File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_open… ─
#!/bin/bash
#SBATCH -t 10
#SBATCH --job-name=hello_world_openmp
#SBATCH --output=hello_world_openmp.out
#SBATCH --error=hello_world_openmp.err
# name of executable
_EXEC=hello.c.exe
export OMP_NUM_THREADS="$BUILDTEST_NUMPROCS"
# Loading modules
module load PrgEnv-intel/6.0.5
# Compilation Line
cc -qopenmp -o $_EXEC /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp/src/hello.c
# Run executable
./$_EXEC
─ Test File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/hello_world_open… ─
#!/bin/bash
export BUILDTEST_TEST_NAME=hello_world_openmp
export BUILDTEST_TEST_ROOT=/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367
export BUILDTEST_BUILDSPEC_DIR=/global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/apps/openmp
export BUILDTEST_STAGE_DIR=/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_scale/hello_world_openmp/ce755367/stage
export BUILDTEST_NUMPROCS=8
# source executor startup script
source /global/u1/s/siddiq90/gitrepos/buildtest/var/executor/cori.slurm.knl_debug/before_script.sh
# Run generated script
sbatch --parsable -q debug --clusters=cori -n 8 -C knl,quad,cache /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/openmp_
# Get return code
returncode=$?
# Exit with return code
exit $returncode
Cray Burst Buffer & Data Warp
For Cray systems, you may want to stage-in or stage-out into your burst buffer to perform some computation. We can do this
via #DW directive. You can see some example jobs using Burst Buffer at NERSC https://docs.nersc.gov/jobs/examples/#burst-buffer
In buildtest we support properties BB and DW which is a list of job directives
that get inserted as #BW and #DW into the test script. We will create a persistent burst buffer
named databuffer of size 10GB striped. We access the burst buffer using the DW directive. Finally we
cd into the databuffer and write a 5GB random file.
Note
BB and DW directives are generated after scheduler directives. The #BB
comes before #DW. buildtest will automatically add the directive #BB
and #DW when using properties BB and DW
1buildspecs:
2 create_burst_buffer:
3 type: script
4 executor: cori.slurm.knl_debug
5 description: Create a burst buffer
6 tags: [jobs, cray]
7 sbatch: ["-N 1", "-t 5", "-n 1"]
8 BB:
9 - create_persistent name=databuffer capacity=10GB access_mode=striped type=scratch
10 DW:
11 - persistentdw name=databuffer
12 run: |
13 cd $DW_PERSISTENT_STRIPED_databuffer
14 pwd
15 dd if=/dev/urandom of=random.txt bs=1G count=5 iflag=fullblock
16 ls -lh $DW_PERSISTENT_STRIPED_databuffer/
Next we ran this test and inspect the generated test, we see that buildtest will insert the #BB and #DW directives as specified in the buildspec. In the output we see a
5GB file called random.txt was written to the burst buffer.
$ buildtest inspect query -o -t create_burst_buffer
─────────────────────────────────────────── create_burst_buffer/2bc01707-c910-4091-ab9c-b14e5f6d56e5 ────────────────────────────────────────────
Executor: cori.slurm.knl_debug
Description: Create a burst buffer
State: PASS
Returncode: 0
Runtime: 120.848841 sec
Starttime: 2022/06/30 14:46:45
Endtime: 2022/06/30 14:48:46
Command: bash --norc --noprofile -eo pipefail create_burst_buffer_build.sh
Test Script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/create_buffer/create_burst_buffer/2bc01707/create_burst_buffer.sh
Build Script:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/create_buffer/create_burst_buffer/2bc01707/create_burst_buffer_build.sh
Output File:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/create_buffer/create_burst_buffer/2bc01707/create_burst_buffer.out
Error File:
/global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/create_buffer/create_burst_buffer/2bc01707/create_burst_buffer.err
Log File: /global/u1/s/siddiq90/gitrepos/buildtest/var/logs/buildtest_yw8xatj8.log
─ Output File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/create_buffer/create_burst_buffer/2bc01707/create_burst… ─
/var/opt/cray/dws/mounts/batch/databuffer_60681470_striped_scratch
total 5.0G
-rw-rw-r-- 1 siddiq90 siddiq90 5.0G Jun 30 14:46 random.txt
─ Test File: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/cori.slurm.knl_debug/create_buffer/create_burst_buffer/2bc01707/create_burst_b… ─
#!/bin/bash
#SBATCH -N 1
#SBATCH -t 5
#SBATCH -n 1
#SBATCH --job-name=create_burst_buffer
#SBATCH --output=create_burst_buffer.out
#SBATCH --error=create_burst_buffer.err
####### START OF BURST BUFFER DIRECTIVES #######
#BB create_persistent name=databuffer capacity=10GB access_mode=striped type=scratch
####### END OF BURST BUFFER DIRECTIVES #######
####### START OF DATAWARP DIRECTIVES #######
#DW persistentdw name=databuffer
####### END OF DATAWARP DIRECTIVES #######
# Content of run section
cd $DW_PERSISTENT_STRIPED_databuffer
pwd
dd if=/dev/urandom of=random.txt bs=1G count=5 iflag=fullblock
ls -lh $DW_PERSISTENT_STRIPED_databuffer/