Feature Overview

Listing Software and Modules

buildtest can report unique software and module versions found in your module system based on spider utility provided by Lmod. This feature can be useful for HPC facilities to see a count of all software packages and see what versions are installed for each package along with the filepath to module file.

buildtest collects this information via BUILDTEST_MODULEPATH which is equivalent to MODULEPATH but can be tweaked by buildtest to add & remove directory at will.

Shown below is unique software list using buildtest list --software

buildtest list --software
RHEL6-apps
cctsoft
clinpharm
deprecated
easybuild
eb
ljmedchemsoft
lmod
medsci
omics
pharmsci
ru
settarg
siterestricted
statistics
use.own.eb


Total Software Packages:  16

Similarly we can retrieve full name of module file and absolute path to module file using buildtest list --module or short option buildtest list -m

buildtest list --modules

    Full Module Name                     |      ModuleFile Path
-----------------------------------------|----------------------------- 
RHEL6-apps                               |	/nfs/grid/software/moduledomains/RHEL6-apps.lua
cctsoft                                  |	/nfs/grid/software/moduledomains/cctsoft
clinpharm                                |	/nfs/grid/software/moduledomains/clinpharm
deprecated                               |	/nfs/grid/software/moduledomains/deprecated
easybuild                                |	/nfs/grid/software/moduledomains/easybuild
eb/2017                                  |	/nfs/grid/software/moduledomains/eb/2017.lua
eb/2018                                  |	/nfs/grid/software/moduledomains/eb/2018.lua
ljmedchemsoft                            |	/nfs/grid/software/moduledomains/ljmedchemsoft
lmod/6.5.1                               |	/usr/share/lmod/lmod/modulefiles/Core/lmod/6.5.1.lua
medsci                                   |	/nfs/grid/software/moduledomains/medsci
omics                                    |	/nfs/grid/software/moduledomains/omics
pharmsci                                 |	/nfs/grid/software/moduledomains/pharmsci
ru                                       |	/nfs/grid/software/moduledomains/ru
settarg/6.5.1                            |	/usr/share/lmod/lmod/modulefiles/Core/settarg/6.5.1.lua
siterestricted                           |	/nfs/grid/software/moduledomains/siterestricted
statistics                               |	/nfs/grid/software/moduledomains/statistics
use.own.eb/append                        |	/nfs/grid/software/moduledomains/use.own.eb/append
use.own.eb/prepend                       |	/nfs/grid/software/moduledomains/use.own.eb/prepend


Total Software Modules: 18
Total LUA Modules: 5
Total non LUA Modules: 13

Module Testing

HPC sites may have hundreds if not thousand module files, it would be great to test all of them. buildtest can conduct module load testing on module files and report SUCCESS or FAIL upon module load by checking exit status.

buildtest module loadtest
module load RHEL6-apps
RUN: 1/18 STATUS: PASSED - Testing module: RHEL6-apps
________________________________________________________________________________
module load cctsoft
RUN: 2/18 STATUS: PASSED - Testing module: cctsoft
________________________________________________________________________________
module load clinpharm
RUN: 3/18 STATUS: PASSED - Testing module: clinpharm
________________________________________________________________________________
module load deprecated
RUN: 4/18 STATUS: PASSED - Testing module: deprecated
________________________________________________________________________________
module load easybuild
RUN: 5/18 STATUS: PASSED - Testing module: easybuild
________________________________________________________________________________
module load eb/2017
RUN: 6/18 STATUS: PASSED - Testing module: eb/2017
________________________________________________________________________________
module load eb/2018
RUN: 7/18 STATUS: PASSED - Testing module: eb/2018
________________________________________________________________________________
module load ljmedchemsoft
RUN: 8/18 STATUS: PASSED - Testing module: ljmedchemsoft
________________________________________________________________________________
module load lmod/6.5.1
RUN: 9/18 STATUS: PASSED - Testing module: lmod/6.5.1
________________________________________________________________________________
module load medsci
RUN: 10/18 STATUS: PASSED - Testing module: medsci
________________________________________________________________________________
module load omics
RUN: 11/18 STATUS: PASSED - Testing module: omics
________________________________________________________________________________
module load pharmsci
RUN: 12/18 STATUS: PASSED - Testing module: pharmsci
________________________________________________________________________________
module load ru
RUN: 13/18 STATUS: PASSED - Testing module: ru
________________________________________________________________________________
module load settarg/6.5.1
RUN: 14/18 STATUS: PASSED - Testing module: settarg/6.5.1
________________________________________________________________________________
module load siterestricted
RUN: 15/18 STATUS: PASSED - Testing module: siterestricted
________________________________________________________________________________
module load statistics
RUN: 16/18 STATUS: PASSED - Testing module: statistics
________________________________________________________________________________
module load use.own.eb/append
RUN: 17/18 STATUS: PASSED - Testing module: use.own.eb/append
________________________________________________________________________________
module load use.own.eb/prepend
RUN: 18/18 STATUS: PASSED - Testing module: use.own.eb/prepend
________________________________________________________________________________
Writing Results to /tmp/modules-load.out
Writing Results to /tmp/modules-load.err
________________________________________________________________________________
                     Module Load Summary
Module Trees:                            ['/nfs/grid/software/moduledomains', '/etc/modulefiles', '/usr/share/modulefiles', '/usr/share/lmod/lmod/modulefiles/Core']
PASSED:                                  18
FAILED:                                  0
________________________________________________________________________________

Building Test

To build a test, buildtest requires an input configuration file that can be specified by option -c or long option --config. This option is part of buildtest build

Shown below is an example build

buildtest build -c /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_gnu.yml -vv
________________________________________________________________________________
compiler: gnu
flags: -O3
maintainer:
- shahzeb siddiqui shahzebmsiddiqui@gmail.com
source: hello.cpp
testblock: singlesource

________________________________________________________________________________
Key Check PASSED for file /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_gnu.yml
Source File /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.cpp exists!
Programming Language Detected: c++
Compiler Check Passed
Writing Test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_gnu.yml.sh
Changing permission to 755 for test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_gnu.yml.sh
________________________________________________________________________________
#!/bin/sh
module purge
cd /tmp/siddis14/buildtest/tests/suite/compilers/helloworld
g++ -O3 -o hello.cpp.exe /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.cpp 
./hello.cpp.exe 
 rm ./hello.cpp.exe 
________________________________________________________________________________

buildtest can insert modules into test, just load the modules before you build the test and it will insert them into your test script.

ml eb/2018; ml GCC; buildtest build -c /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_gnu.yml -vv
________________________________________________________________________________
compiler: gnu
flags: -O3
maintainer:
- shahzeb siddiqui shahzebmsiddiqui@gmail.com
source: hello.cpp
testblock: singlesource

________________________________________________________________________________
Key Check PASSED for file /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_gnu.yml
Source File /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.cpp exists!
Programming Language Detected: c++
Compiler Check Passed
Writing Test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_gnu.yml.sh
Changing permission to 755 for test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_gnu.yml.sh
________________________________________________________________________________
#!/bin/sh
module purge
module load eb/2018 
module load GCCcore/6.4.0 
module load binutils/2.28-GCCcore-6.4.0 
module load GCC/6.4.0-2.28 
cd /tmp/siddis14/buildtest/tests/suite/compilers/helloworld
g++ -O3 -o hello.cpp.exe /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.cpp 
./hello.cpp.exe 
 rm ./hello.cpp.exe 
________________________________________________________________________________

System Package Test

buildtest can generate tests for system packages using the option buildtest build --package <package>. Currently, system package test only perform sanity check against binaries found in the system. The framework will automatically generate binary test by checking the system default paths i.e /usr/bin, /usr/local/bin, /usr/sbin.

For instance to build test for the system package coreutils you can do the following

buildtest build --package coreutils
Detecting System Package: coreutils
Generating  102  binary tests
Binary Tests are written in  /tmp/siddis14/buildtest/tests/system/coreutils
Writing Log file to:  /tmp/siddis14/buildtest/logs/system/coreutils/buildtest_08_29_14_05_2019.log

Running The Test

You can run the in several ways. The easiest way to run the test is via buildtest using buildtest run -S <suite>

Here is the output of the following test

buildtest run -S openmp
Running All Tests from Test Directory: /tmp/siddis14/buildtest/tests/suite/openmp
==============================================================
                         Test summary                         
Package:  openmp
Executed 14 tests
Passed Tests: 14 Percentage: 100.0%
Failed Tests: 0 Percentage: 0.0%
SUCCESS: Threshold of 100.0% was achieved
Writing results to /tmp/siddis14/buildtest/run/buildtest_08_29_14_05_2019.run

TAB Completion

buildtest use the argcomplete python module to autocomplete buildtest argument. Just press TAB key on the keyboard to fill in the arguments.

For instance if you just type buildtest followed by TAB you should see the following.

$ buildtest
benchmark     --clean-logs  -h            list          module        --scantest    --show-keys   -V            yaml
build         find          --help        --logdir      run           --show        --submitjob   --version

Note

You will need to press the TAB key few times before it shows all the arguments

Log files

All buildtest logs will be written in BUILDTEST_LOGDIR.

buildtest will store log files for buildtest build -s <app_name>/<app_ver> in BUILDTEST_LOGDIR/<app_name>/<app_ver>. If toolchain option is specified for instance buildtest build -s <app_name>/<app_ver> -t <tc_name>/<tc_ver> then buildtest will store the logs in BUILDTEST_LOGDIR/<app_name>/<app_ver>/<tc_name>/<tc_ver>.

Similarly logs for system tests like buildtest --package <package> will be stored in BUILDTEST_LOGDIR/system/<package>

You may override BUILDTEST_LOGDIR option at command line via buildtest --logdir and you may even store individual buildtest runs in separate directories such as the following

buildtest build -s OpenMPI/3.0.0-GCC-6.4.0-2.28 --logdir=/tmp