Configuring buildtest

Configuration File

To configure buildtest you will need to create a YAML file at $HOME/.buildtest/settings.yml. This file is responsible for configuring buildtest to work for your test system. Shown below is the configuration file that can be found in the git repo.

# Delete Build Directory before writing tests.  Valid Options: [True, False]

# binary test support. Valid options [True, False]

# Configure shell when running test. 
# Valid options:  [bash, csh, sh]. 
# Default: [sh]

# list root of module tree where modules are installed. Specify multiple module tree as a list

# control output of Lmod spider. 
# Valid values are 
# current: retrieve modules whose abspath is a subdir of directories defined in BUILDTEST_MODULEPATH
# all: retrieve all records

# control how parent modules are retrieved.
# Valid option: first, all

# criteria for success threshold when running job. 
# Valid Option: [0.0-1.0]    

# directory where buildtest logs will be written.
BUILDTEST_LOGDIR: /tmp/$USER/buildtest/logs

# directory where test scripts will be generated
BUILDTEST_TESTDIR: /tmp/$USER/buildtest/tests

# directory where buildtest will write .run files
BUILDTEST_RUN_DIR: /tmp/$USER/buildtest/run

Variable Description





deletes test directory before writing test scripts

True, False


directory where buildtest logs will be written



directory where test scripts will be generated



colon separated list of root of module trees



shell type to use when generating tests

shell, csh, bash


Control view of Lmod spider output

current, all


directory where buildtest will write .run files



criteria for success threshold when running test



conduct sanity check for binary commands

True, False

Configuring Module Trees

BUILDTEST_MODULEPATH takes colon separated list of root of a module tree in your system that serves as module files. buildtest will read all module files and use this to figure out what modules can be tested.

Let’s assume /opt/apps and /workspace/apps are root of the module tree, so we can specify this in your configuration as follows:

- /opt/apps
- /workspace/apps

If you set an invalid directory path in BUILDTEST_MODULEPATH you will get the following message


/opt/apps directory does not exist, specified in BUILDTEST_MODULEPATH

If you don’t specify a module tree for BUILDTEST_MODULEPATH then buildtest will read the value of MODULEPATH. You may add,remove or list module tree.

To see the list of module tree you can run buildtest module -l:

$ buildtest module -l

At this time you will notice BUILDTEST_MODULEPATH is not set and it takes value of MODULEPATH:

$ cat ~/.buildtest/settings.yml  | grep -i BUILDTEST_MODULEPATH


You can add new module tree through command line using buildtest module -a which will update the configuration file:

$ buildtest module -a /usr/share/lmod/lmod/modulefiles/Core
Adding module tree: /usr/share/lmod/lmod/modulefiles/Core
Configuration File: /home/siddis14/.buildtest/settings.yml has been updated

Similarly you can remove module tree from your configuration via buildtest module -r:

(siddis14-TgVBs13r) buildtest-framework[master !?] $ buildtest module -r /etc/modulefiles
Removing module tree: /etc/modulefiles
Configuration File: /home/siddis14/.buildtest/settings.yml has been updated

Configure Spider View

Lmod spider retrieves module details in json format, buildtest is running spider -o spider-json $BUILDTEST_MODULEPATH to get all the modules. The configuration BUILDTEST_SPIDER_VIEW can control the output. When BUILDTEST_SPIDER_VIEW=all then buildtest will retrieve all records including records for modules that are not part of current MODULEPATH.

If you want to restrict the search of module retrieval to those defined in BUILDTEST_MODULEPATH then set BUILDTEST_SPIDER_VIEW=current. buildtest will only retrieve records whose modulefile absolute path is a subdirectory of BUILDTEST_MODULEPATH. When BUILDTEST_MODULEPATH is not set, it will take the value of MODULEPATH and setting BUILDTEST_SPIDER_VIEW=current can be useful in testing modules that are visible to module environment.

Configure Shell

buildtest supports test creation for sh, bash, and csh. The test are created with the appropriate extension. The default shell is sh.

To configure the shell use the variable BUILDTEST_SHELL in your configuration file:


To change the shell to bash or csh you can do either:


If you specify an invalid value you may get the following message


BUILDTEST_SHELL expects value [sh, bash, csh] current value is tcsh

Clean Build

buildtest will write test in directory specified by BUILDTEST_TESTDIR. Often times, you may want to preserve tests across subsequent builds.

For instance you may be interested in building test for different shell and preserve all tests during the previous builds, this can be done by setting export BUILDTEST_CLEAN_BUILD=False or set in configuration as follows:


Setting BUILDTEST_CLEAN_BUILD to False instructs buildtest to preserve build directory where test are written. This will allow user to keep test if they ran the following:

$ buildtest build -p gcc  --shell sh
$ buildtest build -p gcc --shell csh
$ buildtest build -p gcc --shell bash

If you want buildtest to delete the build directory before writing any tests you can set export BUILDTEST_CLEAN_BUILD=True or set the following in configuration file:


Configure Test Directory

buildtest will write test scripts in BUILDTEST_TESTDIR. This can be specified in configuration file or environment variable.

For instance you may want to set the testing directory to $HOME/tmp which can be done in configuration:


This can be done via environment variable as:


If the directory does not exist, buildtest will create it assuming you have the appropriate permissions.

You may specify this at the command line via buildtest build --testdir

See example below where we customize testdir at runtime using the command line option

buildtest build --package gcc --testdir /home/siddis14/tmp
Detecting System Package: gcc
Generating  8  binary tests
Binary Tests are written in  /home/siddis14/tmp/system/gcc
Writing Log file to:  /tmp/siddis14/buildtest/logs/system/gcc/buildtest_08_29_14_05_2019.log

Log Directory

buildtest will write the logs specified by BUILDTEST_LOGDIR this can be set at the configuration file or environment variable.

Setting this at the configuration file can be done as follows:

BUILDTEST_LOGDIR: /tmp/buildtest/logs

Once this is set you will find a log file per execution. For instance, if you build tests for GCCcore/6.4.0, you will find the logs in the following path


All log files are named as follows buildtest_HH_MM_DD_MM_YYYY.log to preserve the date and time stamp.

Here is an example for a buildtest log file from Jan 20th 2019:

$ ls -l /tmp/buildtest/logs/GCCcore/6.4.0/buildtest_21_20_20_01_2019.log
-rw-r--r-- 1 siddis14 amer 81358 Jan 20 21:20 /tmp/buildtest/logs/GCCcore/6.4.0/buildtest_21_20_20_01_2019.log

Test Threshold

buildtest provides a mechanism to set a success threshold during test execution that can be used to determine if your software passes or fails.

This can be set by using BUILDTEST_SUCCESS_THRESHOLD which is a value between [0.0-1.0] which will be used when running test.

if success_threshold >= <passed tests>/< total tests>

Here is an example test run where all test have passed and success threshold is 1.0

$ buildtest run -s GCCcore/6.4.0
Check Configuration
                         Test summary
Application:  GCCcore/6.4.0
Executed 32 tests
Passed Tests: 32    Percentage: 100.0%
Failed Tests: 0    Percentage: 0.0%
SUCCESS: Threshold of 100.0% was achieved
Writing results to /tmp/