Note

Please see Tutorials Setup before you proceed with this section

Buildtest E4S Testsuite Integration

The E4S Testsuite is a collection of lightweight tests assembled for over 100 software products provided by Extreme Scale Scientific Software Stack (E4S) deployments. E4S provides a spack based software stack. Not all spack packages provide internal tests and package-provided tests are static and potentially difficult to modify for use at facility systems. The E4S Testsuite provides a flexible, alternative testing framework for verifying spack-deployed software.

The E4S Testsuite operates on a hierarchy of shell scripts with each test containing its own setup.sh, clean.sh, compile.sh and run.sh scripts. The top level driver script sets up the test environment based on a configuration file which sets compilers, mpi run commands and other environment specific parameters. It then iterates through the selected tests and generates output consisting of log files for individual test runs and a summary listing of each test’s success or failure. The README provides more detail on its structure and usage.

The provided tests typically include source code, build, and input files for quick-running example applications. Some import test code from the installed product’s directory or invoke spack internal tests (spack test run) directly.

Buildspecs using E4S Testsuite

E4S Testsuite provides and maintains pre-constructed tests for a number of products deployed at NERSC. It often makes sense to invoke these tests rather than develop new ones. Buildtest buildspecs that make use of the E4S Testsuite follow the pattern of loading the relevant spack modules, acquiring the testsuite and executing the test driver script on the selected test.

Shown below is an example buildspec running E4S Testsuite for testing the spack package mpich. In order to test mpich, we need to install mpich into our container environment by running spack install mpich prior to testing the package. We will clone E4S Testsuite and invoke the driver script test-all.sh for running the test.

buildspecs:
  mpich_e4s_testsuite:
    type: script
    executor: generic.local.bash
    description: Run E4S Testsuite mpich test
    tags: [e4s]
    run: |
      spack install mpich
      git clone https://github.com/E4S-Project/testsuite
      cd testsuite
      bash test-all.sh --color-off validation_tests/mpich --print-logs

We can see an example of the E4S Testsuite in action by running the following command.

buildtest build -b /home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml
$ buildtest build -b /home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml 
╭────────────────────────────────────────────────────── buildtest summary ──────────────────────────────────────────────────────╮                                                                                                       
│                                                                                                                               │                                                                                                       
│ User:               spack                                                                                                     │                                                                                                       
│ Hostname:           3220cd0ac8f6                                                                                              │                                                                                                       
│ Platform:           Linux                                                                                                     │                                                                                                       
│ Current Time:       2024/02/08 19:03:40                                                                                       │                                                                                                       
│ buildtest path:     /home/spack/buildtest/bin/buildtest                                                                       │                                                                                                       
│ buildtest version:  1.7                                                                                                       │                                                                                                       
│ python path:        /home/spack/pyenv/buildtest/bin/python3                                                                   │                                                                                                       
│ python version:     3.11.6                                                                                                    │                                                                                                       
│ Configuration File: /home/spack/buildtest/buildtest/settings/spack_container.yml                                              │                                                                                                       
│ Test Directory:     /home/spack/runs                                                                                          │                                                                                                       
│ Report File:        /home/spack/buildtest/var/report.json                                                                     │                                                                                                       
│ Command:            /home/spack/buildtest/bin/buildtest build -b /home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml │                                                                                                       
│                                                                                                                               │                                                                                                       
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                                                       
───────────────────────────────────────────────────────────────────────────────────────────────────────  Discovering Buildspecs ────────────────────────────────────────────────────────────────────────────────────────────────────────
                     Discovered buildspecs                      
╔══════════════════════════════════════════════════════════════╗
║ buildspec                                                    ║
╟──────────────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml ║
╚══════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
────────────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ──────────────────────────────────────────────────────────────────────────────────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml: VALID
Total builder objects created: 1
                                                                                Builders by type=script                                                                                
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder                      ┃ type   ┃ executor           ┃ compiler ┃ nodes ┃ procs ┃ description                  ┃ buildspecs                                                   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ mpich_e4s_testsuite/65404308 │ script │ generic.local.bash │ None     │ None  │ None  │ Run E4S Testsuite mpich test │ /home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml │
└──────────────────────────────┴────────┴────────────────────┴──────────┴───────┴───────┴──────────────────────────────┴──────────────────────────────────────────────────────────────┘
──────────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
mpich_e4s_testsuite/65404308: Creating Test Directory: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/65404308
──────────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ─────────────────────────────────────────────────────────────────────────────────────────────────────────────
Spawning 8 processes for processing builders
───────────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
mpich_e4s_testsuite/65404308 does not have any dependencies adding test to queue
    Builders Eligible to Run    
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Builder                      ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ mpich_e4s_testsuite/65404308 │
└──────────────────────────────┘
mpich_e4s_testsuite/65404308: Current Working Directory : /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/65404308/stage
mpich_e4s_testsuite/65404308: Running Test via command: bash mpich_e4s_testsuite_build.sh
mpich_e4s_testsuite/65404308: failed to submit job with returncode: 1
──────────────────────────────────────────────────────────────────────────────────────────── Error Message for mpich_e4s_testsuite/65404308 ────────────────────────────────────────────────────────────────────────────────────────────
Updating files:  54% (7613/14098)
 Updating files:  55% (7754/14098)
 Updating files:  56% (7895/14098)
 Updating files:  57% (8036/14098)
 Updating files:  58% (8177/14098)
 Updating files:  59% (8318/14098)
 Updating files:  60% (8459/14098)
 Updating files:  61% (8600/14098)
 Updating files:  62% (8741/14098)
 Updating files:  63% (8882/14098)
 Updating files:  64% (9023/14098)
 Updating files:  65% (9164/14098)
 Updating files:  66% (9305/14098)
 Updating files:  67% (9446/14098)
 Updating files:  68% (9587/14098)
 Updating files:  69% (9728/14098)
 Updating files:  70% (9869/14098)
 Updating files:  71% (10010/14098)
 Updating files:  72% (10151/14098)
 Updating files:  73% (10292/14098)
 Updating files:  74% (10433/14098)
 Updating files:  75% (10574/14098)
 Updating files:  76% (10715/14098)
 Updating files:  77% (10856/14098)
 Updating files:  78% (10997/14098)
 Updating files:  79% (11138/14098)
 Updating files:  80% (11279/14098)
 Updating files:  80% (11386/14098)
 Updating files:  81% (11420/14098)
 Updating files:  82% (11561/14098)
 Updating files:  83% (11702/14098)
 Updating files:  84% (11843/14098)
 Updating files:  85% (11984/14098)
 Updating files:  86% (12125/14098)
 Updating files:  87% (12266/14098)
 Updating files:  88% (12407/14098)
 Updating files:  89% (12548/14098)
 Updating files:  90% (12689/14098)
 Updating files:  91% (12830/14098)
 Updating files:  92% (12971/14098)
 Updating files:  93% (13112/14098)
 Updating files:  94% (13253/14098)
 Updating files:  95% (13394/14098)
 Updating files:  96% (13535/14098)
 Updating files:  97% (13676/14098)
 Updating files:  98% (13817/14098)
 Updating files:  99% (13958/14098)
 Updating files: 100% (14098/14098)
 Updating files: 100% (14098/14098), done.
 mpich : cxezwh7
 Cleaning /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/65404308/stage/testsuite/validation_tests/mpich
 ---CLEANUP LOG---
 rm -f *.o initfinalize sendrecv broadcast onesided
 Compiling /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/65404308/stage/testsuite/validation_tests/mpich
 ---COMPILE LOG---
 Skipping load: Environment already setup
 mpicc  -Wall -g -c initfinalize.c
 /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/mpich-4.1.2-cxezwh7hyktdonzthd2nlfse7ekeqzwb/bin/mpicc: line 335: /usr/bin/gcc-12: No such file or directory
 make: *** [Makefile:21: initfinalize.o] Error 127
 Compile failed

mpich_e4s_testsuite/65404308: Detected failure in running test, will attempt to retry test: 1 times
mpich_e4s_testsuite/65404308: Run - 1/1
mpich_e4s_testsuite/65404308: Running Test via command: bash mpich_e4s_testsuite_build.sh
mpich_e4s_testsuite/65404308: failed to submit job with returncode: 128
mpich_e4s_testsuite/65404308: Test completed in 95.156152 seconds with returncode: 128
mpich_e4s_testsuite/65404308: Writing output file -  /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/65404308/mpich_e4s_testsuite.out
mpich_e4s_testsuite/65404308: Writing error file - /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/65404308/mpich_e4s_testsuite.err
                                    Test Summary                                     
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                      ┃ executor           ┃ status ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ mpich_e4s_testsuite/65404308 │ generic.local.bash │ FAIL   │ 128        │ 95.156  │
└──────────────────────────────┴────────────────────┴────────┴────────────┴─────────┘



Passed Tests: 0/1 Percentage: 0.000%
Failed Tests: 1/1 Percentage: 100.000%


Adding 1 test results to /home/spack/buildtest/var/report.json
Writing Logfile to /home/spack/buildtest/var/logs/buildtest_ajfmzvnn.log

E4S Testsuite prints full test logs to the standard error instead of stdout so we must use buildtest inspect query -e to view the full test results. Let’s view the test output by running the following:

buildtest inspect query -o -e -t mpich_e4s_testsuite
$ buildtest inspect query -o -e -t mpich_e4s_testsuite 
─────────────────────────────────────────────────────────────────────────────────────── mpich_e4s_testsuite/65404308-6a16-46f9-8303-7e5c342b7267 ───────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: Run E4S Testsuite mpich test
State: FAIL
Returncode: 128
Runtime: 95.156152 sec
Starttime: 2024/02/08 19:03:40
Endtime: 2024/02/08 19:05:15
Command: bash mpich_e4s_testsuite_build.sh
Test Script: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/65404308/mpich_e4s_testsuite.sh
Build Script: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/65404308/mpich_e4s_testsuite_build.sh
Output File: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/65404308/mpich_e4s_testsuite.out
Error File: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/65404308/mpich_e4s_testsuite.err
Log File: /home/spack/buildtest/var/logs/buildtest_ajfmzvnn.log
────────────────────────────────────────────────────── Output File: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/65404308/mpich_e4s_testsuite.out ───────────────────────────────────────────────────────
==> Regenerating tcl module files                                                                                                                                                                                                       
[+] /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libpciaccess-0.17-el7pkf4hkksdm7uhvtpedjweizkqoxuf                                                                                                               
[+] /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libiconv-1.17-a34xpadbggu2vigcpojbsse76dtcghep                                                                                                                   
[+] /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/xz-5.4.1-taa3gkkomd2eitsh3cesvd5b4tznpekd                                                                                                                        
[+] /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/zlib-ng-2.1.4-draqwfyptyn4fteh6kh635ivtjp3r6ly                                                                                                                   
[+] /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/ncurses-6.4-glwymeeuk3yorf4dlwyll2w3nf5po27c                                                                                                                     
[+] /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libfabric-1.19.0-46fsov7dodejcb3wdqhnugv4argyefrk                                                                                                                
[+] /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/yaksa-0.3-hmywdkqlbecfoillw7d3ezj3ffrrsmdi                                                                                                                       
[+] /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libxml2-2.10.3-damyeoscgt5gdqacxr2xv2ewplkequwl                                                                                                                  
[+] /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/hwloc-2.9.1-rvotk5aul6iynvdt2vcgx2dw7zz5mc6t                                                                                                                     
[+] /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/mpich-4.1.2-cxezwh7hyktdonzthd2nlfse7ekeqzwb                                                                                                                     
                                                                                                                                                                                                                                        
─────────────────────────────────────────────────────── Error File: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/65404308/mpich_e4s_testsuite.err ───────────────────────────────────────────────────────
fatal: destination path 'testsuite' already exists and is not an empty directory.                                                                                                                                                       
                                                                                                                                                                                                                                        
──────────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/65404308/mpich_e4s_testsuite.sh ────────────────────────────────────────────────────────
#!/usr/bin/bash                                                                                                                                                                                                                         
set -eo pipefail                                                                                                                                                                                                                        
# Content of run section                                                                                                                                                                                                                
spack install mpich                                                                                                                                                                                                                     
git clone https://github.com/E4S-Project/testsuite                                                                                                                                                                                      
cd testsuite                                                                                                                                                                                                                            
bash test-all.sh --color-off validation_tests/mpich --print-logs