Note

Please see Tutorials Setup before you proceed with this section

Buildtest E4S Testsuite Integration

The E4S Testsuite is a collection of lightweight tests assembled for over 100 software products provided by Extreme Scale Scientific Software Stack (E4S) deployments. E4S provides a spack based software stack. Not all spack packages provide internal tests and package-provided tests are static and potentially difficult to modify for use at facility systems. The E4S Testsuite provides a flexible, alternative testing framework for verifying spack-deployed software.

The E4S Testsuite operates on a hierarchy of shell scripts with each test containing its own setup.sh, clean.sh, compile.sh and run.sh scripts. The top level driver script sets up the test environment based on a configuration file which sets compilers, mpi run commands and other environment specific parameters. It then iterates through the selected tests and generates output consisting of log files for individual test runs and a summary listing of each test’s success or failure. The README provides more detail on its structure and usage.

The provided tests typically include source code, build, and input files for quick-running example applications. Some import test code from the installed product’s directory or invoke spack internal tests (spack test run) directly.

Buildspecs using E4S Testsuite

E4S Testsuite provides and maintains pre-constructed tests for a number of products deployed at NERSC. It often makes sense to invoke these tests rather than develop new ones. Buildtest buildspecs that make use of the E4S Testsuite follow the pattern of loading the relevant spack modules, acquiring the testsuite and executing the test driver script on the selected test.

Shown below is an example buildspec running E4S Testsuite for testing the spack package mpich. In order to test mpich, we need to install mpich into our container environment by running spack install mpich prior to testing the package. We will clone E4S Testsuite and invoke the driver script test-all.sh for running the test.

buildspecs:
  mpich_e4s_testsuite:
    type: script
    executor: generic.local.bash
    description: Run E4S Testsuite mpich test
    tags: [e4s]
    run: |
      spack install mpich
      git clone https://github.com/E4S-Project/testsuite
      cd testsuite
      bash test-all.sh --color-off validation_tests/mpich --print-logs

We can see an example of the E4S Testsuite in action by running the following command.

buildtest build -b /home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml
$ buildtest build -b /home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml 
╭────────────────────────────────────────────────────── buildtest summary ──────────────────────────────────────────────────────╮                                                                                                          
│                                                                                                                               │                                                                                                          
│ User:               spack                                                                                                     │                                                                                                          
│ Hostname:           f215b09d0d2f                                                                                              │                                                                                                          
│ Platform:           Linux                                                                                                     │                                                                                                          
│ Current Time:       2023/02/06 21:16:03                                                                                       │                                                                                                          
│ buildtest path:     /home/spack/buildtest/bin/buildtest                                                                       │                                                                                                          
│ buildtest version:  1.1                                                                                                       │                                                                                                          
│ python path:        /home/spack/pyenv/buildtest/bin/python3                                                                   │                                                                                                          
│ python version:     3.8.6                                                                                                     │                                                                                                          
│ Configuration File: /home/spack/buildtest/buildtest/settings/spack_container.yml                                              │                                                                                                          
│ Test Directory:     /home/spack/runs                                                                                          │                                                                                                          
│ Report File:        /home/spack/buildtest/var/report.json                                                                     │                                                                                                          
│ Command:            /home/spack/buildtest/bin/buildtest build -b /home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml │                                                                                                          
│                                                                                                                               │                                                                                                          
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                                                          
─────────────────────────────────────────────────────────────────────────────────────────────────────────  Discovering Buildspecs ─────────────────────────────────────────────────────────────────────────────────────────────────────────
                     Discovered buildspecs                      
╔══════════════════════════════════════════════════════════════╗
║ buildspec                                                    ║
╟──────────────────────────────────────────────────────────────╢
║ /home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml ║
╚══════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
─────────────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ────────────────────────────────────────────────────────────────────────────────────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml: VALID
Total builder objects created: 1
                                                                                Builders by type=script                                                                                
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder                      ┃ type   ┃ executor           ┃ compiler ┃ nodes ┃ procs ┃ description                  ┃ buildspecs                                                   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ mpich_e4s_testsuite/a14380f6 │ script │ generic.local.bash │ None     │ None  │ None  │ Run E4S Testsuite mpich test │ /home/spack/buildtest/examples/spack/e4s_testsuite_mpich.yml │
└──────────────────────────────┴────────┴────────────────────┴──────────┴───────┴───────┴──────────────────────────────┴──────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
mpich_e4s_testsuite/a14380f6: Creating test directory: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/a14380f6
mpich_e4s_testsuite/a14380f6: Creating the stage directory: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/a14380f6/stage
mpich_e4s_testsuite/a14380f6: Writing build script: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/a14380f6/mpich_e4s_testsuite_build.sh
────────────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
Spawning 4 processes for processing builders
─────────────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────
mpich_e4s_testsuite/a14380f6 does not have any dependencies adding test to queue
mpich_e4s_testsuite/a14380f6: Current Working Directory : /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/a14380f6/stage
mpich_e4s_testsuite/a14380f6: Running Test via command: bash --norc --noprofile -eo pipefail mpich_e4s_testsuite_build.sh
mpich_e4s_testsuite/a14380f6: Test completed in 10.008118 seconds
mpich_e4s_testsuite/a14380f6: Test completed with returncode: 0
mpich_e4s_testsuite/a14380f6: Writing output file -  /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/a14380f6/mpich_e4s_testsuite.out
mpich_e4s_testsuite/a14380f6: Writing error file - /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/a14380f6/mpich_e4s_testsuite.err
In this iteration we are going to run the following tests: [mpich_e4s_testsuite/a14380f6]
                                                        Test Summary                                                         
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ builder                      ┃ executor           ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returncode ┃ runtime   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ mpich_e4s_testsuite/a14380f6 │ generic.local.bash │ PASS   │ N/A N/A N/A                         │ 0          │ 10.008118 │
└──────────────────────────────┴────────────────────┴────────┴─────────────────────────────────────┴────────────┴───────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to /home/spack/buildtest/var/report.json
Writing Logfile to: /home/spack/buildtest/var/logs/buildtest_tnz33e2f.log

E4S Testsuite prints full test logs to the standard error instead of stdout so we must use buildtest inspect query -e to view the full test results. Let’s view the test output by running the following:

buildtest inspect query -o -e -t mpich_e4s_testsuite
$ buildtest inspect query -o -e -t mpich_e4s_testsuite 
──────────────────────────────────────────────────────────────────────────────────────── mpich_e4s_testsuite/a14380f6-722c-4d81-bc3d-4e051ae6f791 ─────────────────────────────────────────────────────────────────────────────────────────
Executor: generic.local.bash
Description: Run E4S Testsuite mpich test
State: PASS
Returncode: 0
Runtime: 10.008118 sec
Starttime: 2023/02/06 21:16:03
Endtime: 2023/02/06 21:16:13
Command: bash --norc --noprofile -eo pipefail mpich_e4s_testsuite_build.sh
Test Script: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/a14380f6/mpich_e4s_testsuite.sh
Build Script: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/a14380f6/mpich_e4s_testsuite_build.sh
Output File: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/a14380f6/mpich_e4s_testsuite.out
Error File: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/a14380f6/mpich_e4s_testsuite.err
Log File: /home/spack/buildtest/var/logs/buildtest_tnz33e2f.log
──────────────────────────────────────────────────────── Output File: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/a14380f6/mpich_e4s_testsuite.out ────────────────────────────────────────────────────────
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/libsigsegv-2.12-lbrx7lnfz46ukewxbhxnucmx76g23c6q                                                                                                               
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/berkeley-db-18.1.40-4ihuiazsglf22f3pntq5hc4kyszqzexn                                                                                                           
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/pkgconf-1.7.3-4sh6pymrm2ms4auu3ajbjjr6fiuhz5g7                                                                                                                 
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/util-macros-1.19.1-gs6ag7ktdoiirb62t7bcagjw62szrrg2                                                                                                            
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/libiconv-1.16-jearpk4xci4zc7dkrza4fufaqfkq7rfl                                                                                                                 
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/xz-5.2.5-komekkmyciga3kl24edjmredhj3uyt7v                                                                                                                      
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/zlib-1.2.11-smoyzzo2qhzpn6mg6rd3l2p7b23enshg                                                                                                                   
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/m4-1.4.18-mkc3u4x2p2wie6jfhuku7g5rkovcrxps                                                                                                                     
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/ncurses-6.2-crhlefo3dv7lmsv5pf4icsy4gepkdorm                                                                                                                   
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/libxml2-2.9.10-yn2r3wfhiilelyulh5toteicdtxjhw7d                                                                                                                
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/libtool-2.4.6-jdxbjftheiotj6solpomva7dowrhlerl                                                                                                                 
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/readline-8.0-t54jzdy2jj4snltjazlm3br2urcilc6v                                                                                                                  
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/libpciaccess-0.16-bob4o5m3uku6vtdil5imasprgy775zg7                                                                                                             
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/gdbm-1.18.1-4av4gywgpaspkhy3dvbb62nulqogtzbb                                                                                                                   
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/hwloc-2.2.0-a4cxlu767bxxceahypqpj2gjdt7hmhvm                                                                                                                   
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/perl-5.32.0-zfdvt2jjuaees43ffrrtphqs2ky3o22t                                                                                                                   
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/autoconf-2.69-mm33a3ocsv3jsh2tfxc4mlab4xsurtdd                                                                                                                 
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/texinfo-6.5-p46ba5qb5wvx6a2bneh4vlurqusz35bj                                                                                                                   
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/automake-1.16.2-d2krmb5gweivlnztcymhklzsqbrpatt6                                                                                                               
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/findutils-4.6.0-hrv4dx5txigi2la7siu3qsa7cervn7tv                                                                                                               
[+] /home/spack/spack-workspace/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/mpich-3.3.2-vbelg5sthuytvkxbrvue2alphviggiur                                                                                                                   
===                                                                                                                                                                                                                                        
validation_tests/mpich                                                                                                                                                                                                                     
---RUN LOG---                                                                                                                                                                                                                              
                                                                                                                                                                                                                                           
──────────────────────────────────────────────────────── Error File: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/a14380f6/mpich_e4s_testsuite.err ─────────────────────────────────────────────────────────
Cloning into 'testsuite'...                                                                                                                                                                                                                
Updating files:  95% (13345/14026)                                                                                                                                                                                                         
Updating files:  96% (13465/14026)                                                                                                                                                                                                         
Updating files:  97% (13606/14026)                                                                                                                                                                                                         
Updating files:  98% (13746/14026)                                                                                                                                                                                                         
Updating files:  99% (13886/14026)                                                                                                                                                                                                         
Updating files: 100% (14026/14026)                                                                                                                                                                                                         
Updating files: 100% (14026/14026), done.                                                                                                                                                                                                  
mpich : vbelg5s                                                                                                                                                                                                                            
Cleaning /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/a14380f6/stage/testsuite/validation_tests/mpich                                                                                                       
---CLEANUP LOG---                                                                                                                                                                                                                          
rm -f *.o initfinalize sendrecv broadcast onesided                                                                                                                                                                                         
Compiling /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/a14380f6/stage/testsuite/validation_tests/mpich                                                                                                      
---COMPILE LOG---                                                                                                                                                                                                                          
Skipping load: Environment already setup                                                                                                                                                                                                   
mpicc  -Wall -g -c initfinalize.c                                                                                                                                                                                                          
mpicc  -o initfinalize initfinalize.o                                                                                                                                                                                                      
mpicc  -Wall -g -c sendrecv.c                                                                                                                                                                                                              
mpicc  -o sendrecv sendrecv.o                                                                                                                                                                                                              
mpicc  -Wall -g -c broadcast.c                                                                                                                                                                                                             
mpicc  -o broadcast broadcast.o                                                                                                                                                                                                            
mpicc  -Wall -g -c onesided.c                                                                                                                                                                                                              
mpicc  -o onesided onesided.o                                                                                                                                                                                                              
Running /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/a14380f6/stage/testsuite/validation_tests/mpich                                                                                                        
Skipping load: Environment already setup                                                                                                                                                                                                   
Running initfinalize                                                                                                                                                                                                                       
                                 [PASSED]                                                                                                                                                                                                  
Running sendrecv                                                                                                                                                                                                                           
                                 [PASSED]                                                                                                                                                                                                  
Running broadcast                                                                                                                                                                                                                          
                                 [PASSED]                                                                                                                                                                                                  
Running onesided                                                                                                                                                                                                                           
                                 [PASSED]                                                                                                                                                                                                  
Success                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                           
───────────────────────────────────────────────────────── Test File: /home/spack/runs/generic.local.bash/e4s_testsuite_mpich/mpich_e4s_testsuite/a14380f6/mpich_e4s_testsuite.sh ──────────────────────────────────────────────────────────
#!/bin/bash                                                                                                                                                                                                                                
# Content of run section                                                                                                                                                                                                                   
spack install mpich                                                                                                                                                                                                                        
git clone https://github.com/E4S-Project/testsuite                                                                                                                                                                                         
cd testsuite                                                                                                                                                                                                                               
bash test-all.sh --color-off validation_tests/mpich --print-logs