Note
Please see Tutorials Setup before you proceed with this section
Buildtest E4S Testsuite Integration
The E4S Testsuite is a collection of lightweight tests assembled for over 100 software products provided by Extreme Scale Scientific Software Stack (E4S) deployments. E4S provides a spack based software stack. Not all spack packages provide internal tests and package-provided tests are static and potentially difficult to modify for use at facility systems. The E4S Testsuite provides a flexible, alternative testing framework for verifying spack-deployed software.
The E4S Testsuite operates on a hierarchy of shell scripts with each test containing its own setup.sh, clean.sh, compile.sh
and run.sh scripts. The top level driver script sets up the test environment based on a configuration file which sets
compilers, mpi run commands and other environment specific parameters. It then iterates through the selected tests
and generates output consisting of log files for individual test runs and a summary listing of each test’s success or failure. The README provides more detail on its structure and usage.
The provided tests typically include source code, build, and input files for quick-running example applications. Some import test code from the installed product’s directory or invoke spack internal tests (spack test run) directly.
Buildspecs using E4S Testsuite
E4S Testsuite provides and maintains pre-constructed tests for a number of products deployed at NERSC. It often makes sense to invoke these tests rather than develop new ones. Buildtest buildspecs that make use of the E4S Testsuite follow the pattern of loading the relevant spack modules, acquiring the testsuite and executing the test driver script on the selected test.
Shown below is an example buildspec running E4S Testsuite for testing the spack package mpich. In order to test mpich, we need to install mpich into our container environment by running spack install mpich prior to testing the package. We will clone E4S Testsuite and invoke the driver script test-all.sh for running the test.
buildspecs:
mpich_e4s_testsuite:
type: script
executor: generic.local.bash
description: Run E4S Testsuite mpich test
tags: [e4s]
run: |
spack install mpich
git clone https://github.com/E4S-Project/testsuite
cd testsuite
bash test-all.sh --color-off validation_tests/mpich --print-logs
We can see an example of the E4S Testsuite in action by running the following command.
buildtest build -b /home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml
$ buildtest build -b /home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml
╭────────────────────────────────────────────────────── buildtest summary ──────────────────────────────────────────────────────╮
│ │
│ User: spack │
│ Hostname: 0fee8530c3a4 │
│ Platform: Linux │
│ Current Time: 2023/07/14 18:46:55 │
│ buildtest path: /home/spack/buildtest/bin/buildtest │
│ buildtest version: 1.4 │
│ python path: /home/spack/pyenv/buildtest/bin/python3 │
│ python version: 3.8.6 │
│ Configuration File: /home/spack/buildtest/buildtest/settings/spack_container.yml │
│ Test Directory: /home/spack/runs │
│ Report File: /home/spack/buildtest/var/report.json │
│ Command: /home/spack/buildtest/bin/buildtest build -b /home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
───────────────────────────────────────────────────────────────────────────────────────────────────── Discovering Buildspecs ─────────────────────────────────────────────────────────────────────────────────────────────────────
Discovered buildspecs
╔══════════════════════════════════════════════════════════════╗
║ buildspec ║
╟──────────────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml ║
╚══════════════════════════════════════════════════════════════╝
Total Discovered Buildspecs: 1
Total Excluded Buildspecs: 0
Detected Buildspecs after exclusion: 1
─────────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ────────────────────────────────────────────────────────────────────────────────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml: VALID
Total builder objects created: 1
Builders by type=script
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ description ┃ buildspecs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ mpich_e4s_testsuite/0c47b479 │ script │ generic.local.bash │ None │ None │ None │ Run E4S Testsuite mpich test │ /home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml │
└──────────────────────────────┴────────┴────────────────────┴──────────┴───────┴───────┴──────────────────────────────┴──────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ──────────────────────────────────────────────────────────────────────────────────────────────────────────
mpich_e4s_testsuite/0c47b479: Creating test directory: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/0c47b479
mpich_e4s_testsuite/0c47b479: Creating the stage directory: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/0c47b479/stage
mpich_e4s_testsuite/0c47b479: Writing build script: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/0c47b479/mpich_e4s_testsuite_build.sh
────────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ──────────────────────────────────────────────────────────────────────────────────────────────────────────
Spawning 4 processes for processing builders
─────────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ───────────────────────────────────────────────────────────────────────────────────────────────────────────
mpich_e4s_testsuite/0c47b479 does not have any dependencies adding test to queue
mpich_e4s_testsuite/0c47b479: Current Working Directory : /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/0c47b479/stage
mpich_e4s_testsuite/0c47b479: Running Test via command: bash --norc --noprofile -eo pipefail mpich_e4s_testsuite_build.sh
mpich_e4s_testsuite/0c47b479: Test completed in 45.002633 seconds
mpich_e4s_testsuite/0c47b479: Test completed with returncode: 0
mpich_e4s_testsuite/0c47b479: Writing output file - /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/0c47b479/mpich_e4s_testsuite.out
mpich_e4s_testsuite/0c47b479: Writing error file - /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/0c47b479/mpich_e4s_testsuite.err
In this iteration we are going to run the following tests: [mpich_e4s_testsuite/0c47b479]
Test Summary
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ builder ┃ executor ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ mpich_e4s_testsuite/0c47b479 │ generic.local.bash │ PASS │ None None None │ 0 │ 45.002633 │
└──────────────────────────────┴────────────────────┴────────┴─────────────────────────────────────┴────────────┴───────────┘
Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%
Adding 1 test results to /home/spack/buildtest/var/report.json
Writing Logfile to: /home/spack/buildtest/var/logs/buildtest_6ytwg9ap.log
E4S Testsuite prints full test logs to the standard error instead of stdout so we must use buildtest inspect query -e to view the full test results. Let’s view the test output by running the following:
buildtest inspect query -o -e -t mpich_e4s_testsuite
$ buildtest inspect query -o -e -t mpich_e4s_testsuite
──────────────────────────────────────────────────────────────────────────────────── mpich_e4s_testsuite/0c47b479-3a67-45c4-aef5-4dfdfe61232d ─────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: Run E4S Testsuite mpich test
State: PASS
Returncode: 0
Runtime: 45.002633 sec
Starttime: 2023/07/14 18:46:55
Endtime: 2023/07/14 18:47:40
Command: bash --norc --noprofile -eo pipefail mpich_e4s_testsuite_build.sh
Test Script: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/0c47b479/mpich_e4s_testsuite.sh
Build Script: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/0c47b479/mpich_e4s_testsuite_build.sh
Output File: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/0c47b479/mpich_e4s_testsuite.out
Error File: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/0c47b479/mpich_e4s_testsuite.err
Log File: /home/spack/buildtest/var/logs/buildtest_6ytwg9ap.log
──────────────────────────────────────────────────── Output File: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/0c47b479/mpich_e4s_testsuite.out ────────────────────────────────────────────────────
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/libsigsegv-2.12-lbrx7lnfz46ukewxbhxnucmx76g23c6q
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/berkeley-db-18.1.40-4ihuiazsglf22f3pntq5hc4kyszqzexn
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/pkgconf-1.7.3-4sh6pymrm2ms4auu3ajbjjr6fiuhz5g7
==> Installing util-macros-1.19.1-gs6ag7ktdoiirb62t7bcagjw62szrrg2
==> Fetching file:///mirror/build_cache/linux-ubuntu18.04-x86_64/gcc-7.5.0/util-macros-1.19.1/linux-ubuntu18.04-x86_64-gcc-7.5.0-util-macros-1.19.1-gs6ag7ktdoiirb62t7bcagjw62szrrg2.spack
==> Extracting util-macros-1.19.1-gs6ag7ktdoiirb62t7bcagjw62szrrg2 from binary cache
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/util-macros-1.19.1-gs6ag7ktdoiirb62t7bcagjw62szrrg2
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/libiconv-1.16-jearpk4xci4zc7dkrza4fufaqfkq7rfl
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/xz-5.2.5-komekkmyciga3kl24edjmredhj3uyt7v
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/zlib-1.2.11-smoyzzo2qhzpn6mg6rd3l2p7b23enshg
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/m4-1.4.18-mkc3u4x2p2wie6jfhuku7g5rkovcrxps
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/ncurses-6.2-crhlefo3dv7lmsv5pf4icsy4gepkdorm
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/libxml2-2.9.10-yn2r3wfhiilelyulh5toteicdtxjhw7d
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/libtool-2.4.6-jdxbjftheiotj6solpomva7dowrhlerl
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/readline-8.0-t54jzdy2jj4snltjazlm3br2urcilc6v
==> Installing libpciaccess-0.16-bob4o5m3uku6vtdil5imasprgy775zg7
==> Fetching file:///mirror/build_cache/linux-ubuntu18.04-x86_64/gcc-7.5.0/libpciaccess-0.16/linux-ubuntu18.04-x86_64-gcc-7.5.0-libpciaccess-0.16-bob4o5m3uku6vtdil5imasprgy775zg7.spack
==> Extracting libpciaccess-0.16-bob4o5m3uku6vtdil5imasprgy775zg7 from binary cache
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/libpciaccess-0.16-bob4o5m3uku6vtdil5imasprgy775zg7
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/gdbm-1.18.1-4av4gywgpaspkhy3dvbb62nulqogtzbb
==> Installing hwloc-2.2.0-a4cxlu767bxxceahypqpj2gjdt7hmhvm
==> Fetching file:///mirror/build_cache/linux-ubuntu18.04-x86_64/gcc-7.5.0/hwloc-2.2.0/linux-ubuntu18.04-x86_64-gcc-7.5.0-hwloc-2.2.0-a4cxlu767bxxceahypqpj2gjdt7hmhvm.spack
==> Extracting hwloc-2.2.0-a4cxlu767bxxceahypqpj2gjdt7hmhvm from binary cache
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/hwloc-2.2.0-a4cxlu767bxxceahypqpj2gjdt7hmhvm
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/perl-5.32.0-zfdvt2jjuaees43ffrrtphqs2ky3o22t
==> Installing texinfo-6.5-p46ba5qb5wvx6a2bneh4vlurqusz35bj
==> Fetching file:///mirror/build_cache/linux-ubuntu18.04-x86_64/gcc-7.5.0/texinfo-6.5/linux-ubuntu18.04-x86_64-gcc-7.5.0-texinfo-6.5-p46ba5qb5wvx6a2bneh4vlurqusz35bj.spack
==> Extracting texinfo-6.5-p46ba5qb5wvx6a2bneh4vlurqusz35bj from binary cache
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/texinfo-6.5-p46ba5qb5wvx6a2bneh4vlurqusz35bj
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/autoconf-2.69-mm33a3ocsv3jsh2tfxc4mlab4xsurtdd
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/automake-1.16.2-d2krmb5gweivlnztcymhklzsqbrpatt6
==> Installing findutils-4.6.0-hrv4dx5txigi2la7siu3qsa7cervn7tv
==> Fetching file:///mirror/build_cache/linux-ubuntu18.04-x86_64/gcc-7.5.0/findutils-4.6.0/linux-ubuntu18.04-x86_64-gcc-7.5.0-findutils-4.6.0-hrv4dx5txigi2la7siu3qsa7cervn7tv.spack
==> Extracting findutils-4.6.0-hrv4dx5txigi2la7siu3qsa7cervn7tv from binary cache
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/findutils-4.6.0-hrv4dx5txigi2la7siu3qsa7cervn7tv
==> Installing mpich-3.3.2-vbelg5sthuytvkxbrvue2alphviggiur
==> Fetching file:///mirror/build_cache/linux-ubuntu18.04-x86_64/gcc-7.5.0/mpich-3.3.2/linux-ubuntu18.04-x86_64-gcc-7.5.0-mpich-3.3.2-vbelg5sthuytvkxbrvue2alphviggiur.spack
==> Extracting mpich-3.3.2-vbelg5sthuytvkxbrvue2alphviggiur from binary cache
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/mpich-3.3.2-vbelg5sthuytvkxbrvue2alphviggiur
===
validation_tests/mpich
---RUN LOG---
──────────────────────────────────────────────────── Error File: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/0c47b479/mpich_e4s_testsuite.err ─────────────────────────────────────────────────────
gpgconf: socketdir is '/home/spack/.gnupg'
gpgconf: no /run/user dir
gpgconf: using homedir as fallback
Cloning into 'testsuite'...
Updating files: 67% (9474/14073)
Updating files: 68% (9570/14073)
Updating files: 69% (9711/14073)
Updating files: 70% (9852/14073)
Updating files: 71% (9992/14073)
Updating files: 72% (10133/14073)
Updating files: 73% (10274/14073)
Updating files: 74% (10415/14073)
Updating files: 75% (10555/14073)
Updating files: 76% (10696/14073)
Updating files: 77% (10837/14073)
Updating files: 78% (10977/14073)
Updating files: 79% (11118/14073)
Updating files: 80% (11259/14073)
Updating files: 81% (11400/14073)
Updating files: 82% (11540/14073)
Updating files: 83% (11681/14073)
Updating files: 84% (11822/14073)
Updating files: 85% (11963/14073)
Updating files: 86% (12103/14073)
Updating files: 87% (12244/14073)
Updating files: 88% (12385/14073)
Updating files: 89% (12525/14073)
Updating files: 90% (12666/14073)
Updating files: 91% (12807/14073)
Updating files: 92% (12948/14073)
Updating files: 93% (13088/14073)
Updating files: 94% (13229/14073)
Updating files: 95% (13370/14073)
Updating files: 96% (13511/14073)
Updating files: 97% (13651/14073)
Updating files: 98% (13792/14073)
Updating files: 99% (13933/14073)
Updating files: 100% (14073/14073)
Updating files: 100% (14073/14073), done.
mpich : vbelg5s
Cleaning /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/0c47b479/stage/testsuite/validation_tests/mpich
---CLEANUP LOG---
rm -f *.o initfinalize sendrecv broadcast onesided
Compiling /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/0c47b479/stage/testsuite/validation_tests/mpich
---COMPILE LOG---
Skipping load: Environment already setup
mpicc -Wall -g -c initfinalize.c
mpicc -o initfinalize initfinalize.o
mpicc -Wall -g -c sendrecv.c
mpicc -o sendrecv sendrecv.o
mpicc -Wall -g -c broadcast.c
mpicc -o broadcast broadcast.o
mpicc -Wall -g -c onesided.c
mpicc -o onesided onesided.o
Running /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/0c47b479/stage/testsuite/validation_tests/mpich
Skipping load: Environment already setup
Running initfinalize
[PASSED]
Running sendrecv
[PASSED]
Running broadcast
[PASSED]
Running onesided
[PASSED]
Success
───────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/0c47b479/mpich_e4s_testsuite.sh ──────────────────────────────────────────────────────
#!/bin/bash
# Content of run section
spack install mpich
git clone https://github.com/E4S-Project/testsuite
cd testsuite
bash test-all.sh --color-off validation_tests/mpich --print-logs