Overview

Build Options (buildtest build --help)

buildtest build --help
usage: buildtest [options] [COMMANDS] build [-h] [-s INSTALLED-SOFTWARE] [-p SYSTEM-PACKAGE] [--shell {sh,bash,csh}]
                                            [-c TEST CONFIGURATION] [-b] [-S {compilers,mpi,openmp,cuda}]
                                            [--clean-tests] [--testdir TESTDIR] [--clean-build] [-v]
                                            [-m MODULES | -co {corrupt,corrupt2,gcc540,intelmpi} | -mc COLLECTION-ID]
                                            [-pms {first,all}]

optional arguments:
  -h, --help            show this help message and exit
  -s INSTALLED-SOFTWARE, --software INSTALLED-SOFTWARE
                        Specify software package to test
  -p SYSTEM-PACKAGE, --package SYSTEM-PACKAGE
                        Build test for system packages
  --shell {sh,bash,csh}
                        Select the type of shell for testscript
  -c TEST CONFIGURATION, --config TEST CONFIGURATION
                        Specify test configuration
  -b, --binary          Conduct binary test for a package
  -S {compilers,mpi,openmp,cuda}, --suite {compilers,mpi,openmp,cuda}
                        specify test suite
  --clean-tests         delete test directory $BUILDTEST_TESTDIR
  --testdir TESTDIR     Specify alternate test directory to write test. This overrides configuration BUILDTEST_TESTDIR
  --clean-build         delete software test directory before writing test scripts
  -v, --verbose         verbosity level (default: 0)
  -m MODULES, --modules MODULES
                        Specify comma separated list of modules to build a test for every module version.
  -co {corrupt,corrupt2,gcc540,intelmpi}, --collection {corrupt,corrupt2,gcc540,intelmpi}
                        Use user Lmod module collection whenbuilding test
  -mc COLLECTION-ID, --module-collection COLLECTION-ID
                        Use internal buildtest module collection when building test.Run buildtest collection -l to
                        view list of collection ids
  -pms {first,all}, --parent-module-search {first,all}
                        control how many parent module combination to search

Test Configuration

buildtest makes use of test configuration to generate the test script. This can be done by running buildtest build -c /path/to/configuration

Shown below is an example.

$ buildtest build -c $BUILDTEST_ROOT/toolkit/buildtest/suite/compilers/helloworld/hello_gnu.yml
Writing Test: /home/siddis14/buildtest/suite/compilers/helloworld/hello_gnu.yml.sh

buildtest has two levels of verbosity that can be set by using -v option. buildtest will check the programming language, compiler and verify all the keys in configuration file before building the test.

buildtest will set the permission of test script to 755.

$ buildtest build -c $BUILDTEST_ROOT/toolkit/buildtest/suite/compilers/helloworld/hello_gnu.yml -v
Key Check PASSED for file /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_gnu.yml
Programming Language Detected: c++
Compiler Check Passed
Writing Test: /home/siddis14/buildtest/suite/compilers/helloworld/hello_gnu.yml.sh
Changing permission to 755 for test: /home/siddis14/buildtest/suite/compilers/helloworld/hello_gnu.yml.sh

You may specify additional level verbosity by -vv or specify -v -v which will give additional output including the output of configuration file and test script.

$ buildtest build -c $BUILDTEST_ROOT/toolkit/buildtest/suite/compilers/helloworld/hello_gnu.yml -v -v
________________________________________________________________________________
compiler: gnu
flags: -O3
maintainer:
- shahzeb siddiqui shahzebmsiddiqui@gmail.com
source: hello.cpp
testblock: singlesource

________________________________________________________________________________
Key Check PASSED for file /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_gnu.yml
Source File /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.cpp exists!
Programming Language Detected: c++
Compiler Check Passed
Writing Test: /home/siddis14/buildtest/suite/compilers/helloworld/hello_gnu.yml.sh
Changing permission to 755 for test: /home/siddis14/buildtest/suite/compilers/helloworld/hello_gnu.yml.sh
________________________________________________________________________________
#!/bin/sh
module purge
module load eb/2018
cd /home/siddis14/buildtest/suite/compilers/helloworld
g++ -O3 -o hello.cpp.exe /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.cpp
./hello.cpp.exe
rm ./hello.cpp.exe
________________________________________________________________________________

Test Suites

Test Suite is a collection of test configuration that is meant for organizing tests. Test suite can be found at https://github.com/HPC-buildtest/buildtest-framework/tree/master/toolkit/buildtest/suite. and each sub-directory is a separate test suite.

A test suite is capable of building all test configuration (.yml files) found in its subdirectories. To build a test suite you can execute buildtest build -S <suite>

To know more about test suite see Test Suite

Sanity Check for System Packages

buildtest can perform sanity check for all binaries defined by a system package. This may be useful when running test periodically to monitor system changes.

To build test for system package you will want to use buildtest build --package <package> and specify the name of the installed system package.

For instance, lets build the tests for coreutils package by running buildtest build --package coreutils

The output will be the following

buildtest build --package coreutils
Detecting System Package: coreutils
Generating  102  binary tests
Binary Tests are written in  /tmp/siddis14/buildtest/tests/system/coreutils
Writing Log file to:  /tmp/siddis14/buildtest/logs/system/coreutils/buildtest_08_29_14_05_2019.log

Sanity Check for Modules

buildtest can conduct sanity check for all active modules by running -b, --binary option or setting BUILDTEST_BINARY=True in your configuration file.

For instance let’s assume the following modules are active modules in your shell

$ ml

Currently Loaded Modules:
  1) eb/2018   2) GCCcore/6.4.0   3) binutils/2.28-GCCcore-6.4.0   4) GCC/6.4.0-2.28

buildtest will seek out all binary executables in each module file and run which command against the binary and load the appropriate modules

Shown below is an example.

$ buildtest build -b
Detecting Software:eb/2018
No $PATH set in your module  eb/2018   so no possible binaries can be found
There are no binaries for package: eb/2018
Detecting Software:GCCcore/6.4.0
Generating  19  binary tests
Binary Tests are written in  /home/siddis14/buildtest/software/GCCcore/6.4.0
Detecting Software:binutils/2.28-GCCcore-6.4.0
Generating  18  binary tests
Binary Tests are written in  /home/siddis14/buildtest/software/binutils/2.28-GCCcore-6.4.0
Detecting Software:GCC/6.4.0-2.28
No $PATH set in your module  GCC/6.4.0-2.28   so no possible binaries can be found
There are no binaries for package: GCC/6.4.0-2.28

modules that dont have PATH set or no binary executables are found in the directory, then buildtest will not generate any test.

Shown below is an example test script for gcc binary

#!/bin/sh


module load GCCcore/6.4.0
which gcc

Clean build (buildtest build --clean-build)

buildtest will preserve the testing directory when tests are generated. For example, if you run the following

buildtest build --package gcc --shell sh
buildtest build --package gcc --shell csh
buildtest build --package gcc --shell bash

This will write the test for shell (“sh”, “bash”, “csh”) in the same directory. If you want to remove the directory prior to running test you can do the following

buildtest build --package gcc --clean-build

Customize Test Directory (buildtest build --testdir)

If you want to customize the path to BUILDTEST_TESTDIR you may use the option --testdir or update the environment variable BUILDTEST_TESTDIR. The command line option will override environment variable and environment variable will override configuration value.

buildtest build --package gcc --testdir /home/siddis14/tmp
Detecting System Package: gcc
Generating  8  binary tests
Binary Tests are written in  /home/siddis14/tmp/system/gcc
Writing Log file to:  /tmp/siddis14/buildtest/logs/system/gcc/buildtest_08_29_14_05_2019.log

Shell Types

Currently buildtest supports sh, bash, csh shell for creating test scripts. buildtest defaults to sh but this can be tweaked

To create tests for different shell types try buildtest build --shell <shell> or set the variable BUILDTEST_SHELL in your configuration file or via environment variable

Let’s build test with csh

buildtest build -c /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_gnu.yml --shell csh
Writing Test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_gnu.yml.csh

buildtest will add the appropriate shell extension for the test script to avoid name conflicts.

Another way to build for different shell is to set BUILDTEST_SHELL as we see in example below

BUILDTEST_SHELL=bash buildtest build -c /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_gnu.yml
Writing Test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_gnu.yml.bash