Configuring Buildtest
We assume you are familiar with general concepts presented in getting started and your next step is to configure buildtest to run at your site. This guide will present you the necessary steps to get you started.
When you clone buildtest, we provide a default configuration that can be used to run on your laptop or workstation that supports Linux or Mac. The buildtest configuration uses a JSON schemafile settings.schema.json. for validating your configuration.
Which configuration file does buildtest read?
buildtest will read configuration files in the following order:
Command line
buildtest -c <config>.yml build
Environment variable - BUILDTEST_CONFIGFILE
User Configuration -
$HOME/.buildtest/config.yml
Default Configuration -
$BUILDTEST_ROOT/buildtest/settings/config.yml
Default Configuration
Buildtest comes with a default configuration that can be found at buildtest/settings/config.yml
relative to root of repo. At the start of buildtest execution, buildtest will load
the configuration file and validate the configuration with JSON schema settings.schema.json
.
If it’s fails to validate, buildtest will raise an error.
We recommend you copy the default configuration as a template to configure buildtest for your site.
Shown below is the default configuration provided by buildtest.
system:
generic:
# specify a list of hostnames that is a regular expression where buildtest can run.
hostnames: ['.*']
# system description
description: Generic System
# specify module system to use. Supported module systems are [lmod, environment-modules, none]
moduletool: none
# specify size of job pool (https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool). This will configure the number of processes that can run in parallel.
# If not specified then buildtest will use all available cores on the system.
poolsize: 1
# maximum number of jobs that can run in parallel. If not specified, buildtest will run all jobs in parallel.
#max_jobs: 1
# test timeout in number of seconds
# timeout: 3600
# enable pagination for buildtest
pager: false
# options for buildtest buildspec find command
buildspecs:
# determine whether to rebuild buildspec cache
rebuild: false
# determine number of records to display
count: 15
# specify format fields
#format: name,description
# display output in terse mode
terse: false
# list of paths to search for buildspecs
#directory: ["$BUILDTEST_ROOT/tutorials", "$BUILDTEST_ROOT/examples"]
# options for buildtest report command
report:
# number of records to display
count: 25
# display output in terse mode
terse: false
# specify format fields
format: name,id,state,runtime,returncode
# specify directory paths to search for binaries
#paths:
# directory path to search for slurm binaries.
#slurm: "/usr/bin"
# directory path to search for lsf binaries.
#lsf: "/usr/bin"
# directory path to search for pbs binaries.
#pbs: "/usr/bin"
# directory path to search for torque binaries.
#torque: "/usr/bin"
# path to docker directory
#docker: "/usr/local/bin"
# path to singularity directory
#singularity: "/usr/local/bin"
# path to podman directory
#podman: "/usr/local/bin"
# start of executor configuration
executors:
# local executor is used to submit jobs on local machine. In this example we have 4 executors: bash, sh, csh, zsh that will submit jobs using bash, sh, csh, zsh shell respectively.
local:
bash:
description: submit jobs on local machine using bash shell
shell: bash
sh:
description: submit jobs on local machine using sh shell
shell: sh
csh:
description: submit jobs on local machine using csh shell
shell: csh
zsh:
description: submit jobs on local machine using zsh shell
shell: zsh
# specify compiler declaration
compilers:
compiler:
# declaration of all gcc compilers
gcc:
# name of compiler
builtin_gcc:
cc: gcc
fc: gfortran
cxx: g++
# specify CDASH configuration when using 'buildtest cdash upload'
cdash:
# CDASH server
url: https://my.cdash.org/
# name of CDASH project where to push test reports
project: buildtest
# specify the site name which should generally by name of your HPC system. This is used by CDASH to group test results by site
site: generic
# specify the build name which is used to group test results by build name
buildname: tutorials
As you can see the layout of configuration starts with keyword system
which is
used to define one or more systems. Your HPC site may contain more than one cluster,
so you should define your clusters with meaningful names as this will impact when you
reference executors in buildspecs. In this example, we define one
cluster called generic
which is a dummy cluster used for running tutorial examples.
Hostnames
The hostnames
field is a list of nodes that belong to the cluster where buildtest should be run. Generally,
these hosts should be your login nodes in your cluster. buildtest will process hostnames field across
all system entry using re.fullmatch until a hostname is found, if
none is found we report an error.
In this example we defined two systems machine, machine2 with the following hostnames.
system:
machine1:
hostnames: ['loca$', '^1DOE']
machine2:
hostnames: ['BOB|JOHN']
In this example, none of the host entries match with hostname DOE-7086392.local so we get an error since buildtest needs to detect a system before proceeding.
buildtest.exceptions.BuildTestError: "Based on current system hostname: DOE-7086392.local we cannot find a matching system ['machine1', 'machine2'] based on current hostnames: {'machine1': ['loca$', '^1DOE'], 'machine2': ['BOB|JOHN']} "
Let’s assume you we have a system named mycluster
that should run on nodes login1
, login2
, and login3
.
You can specify hostnames as a list of strings
system:
mycluster:
hostnames: ["login1", "login2", "login3"]
Alternately, you can use regular expression to condense this list
system:
mycluster:
hostnames: ["login[1-3]"]
Module Tool
If your system supports environment-modules or
Lmod for managing user environment then you can
configure buildtest to use the module tool. This can be defined via moduletool
property.
# environment-modules
moduletool: environment-modules
# for lmod
moduletool: lmod
# specify N/A if you don't have modules
moduletool: N/A
The moduletool property is used for detecting compilers when you run buildtest config compilers find
.
Test directory
The default location where tests are written is $BUILDTEST_ROOT/var/tests where
$BUILDTEST_ROOT is the root of buildtest repo. You may specify testdir
in your
configuration to instruct where tests can be written. For instance, if
you want to write tests in /tmp you can set the following:
testdir: /tmp
Alternately, one can specify test directory via buildtest build --testdir <path>
which
has highest precedence and overrides configuration and default value.
Log Path
You can configure where buildtest will write logs using logdir
property. For
example, in example below buildtest will write log files $HOME/Documents/buildtest/var/logs
.
buildtest will resolve variable expansion to get real path on filesystem.
# location of log directory
logdir: $HOME/Documents/buildtest/var/logs
logdir
is not required field in configuration, if it’s not specified then buildtest will write logs
based on tempfile library which may vary
based on platform (Linux, Mac).
The buildtest logs will start with buildtest_ followed by random identifier with a .log extension.
Specify directory paths to search for binaries
The paths
property can be used to search for binaries for batch schedulers. If your scheduler binaries
are installed in a non-standard location that is not in $PATH, you can use this to specify the directory path.
In example below we will, we will specify directories for SLURM, LSF, PBS and TORQUE binaries that are not in $PATH and installed in /usr/local/slurm/bin, /usr/local/lsf/bin, /usr/local/pbs/bin, /usr/local/torque/bin respectively.
paths:
slurm: /usr/local/slurm/bin
lsf: /usr/local/lsf/bin
pbs: /usr/local/pbs/bin
torque: /usr/local/torque/bin
You can also specify paths to container runtimes, if they are installed in a non-standard location that is not in $PATH. For example, if you have docker, podman and singularity installed in /usr/local/bin you can specify the following:
paths:
docker: /usr/local/bin
podman: /usr/local/bin
singularity: /usr/local/bin
Buildtest will attempt to search for container runtime in the specified directory including what is available in $PATH.
Buildspec Cache
The buildtest buildspec find command can be configured using the configuration file to provide sensible defaults. This can be shown in the configuration file below:
buildspecs:
# whether to rebuild cache file automatically when running `buildtest buildspec find`
rebuild: False
# limit number of records to display when running `buildtest buildspec find`
count: 15
# format fields to display when running `buildtest buildspec find`, By default we will show name,description
format: "name,description"
# enable terse mode
terse: False
# specify list of directories to search for buildspecs when building cache
#root: [ $BUILDTEST_ROOT/examples, /tmp/buildspecs ]
The rebuild: False
means buildtest won’t rebuild the buildcache every time you run buildtest buildspec find
. If the
cache file is not present, it will automatically rebuild the cache, otherwise it will build the cache if one specifies
--rebuild
option or rebuild: True
is set in the configuration file.
The buildspec cache is built by reading the contents of the buildspec file on the filesystem; therefore if you make changes
to the buildspecs then you will need to rebuild the buildspec cache by running buildtest buildspec find --rebuild
.
If you want buildtest to always rebuild cache you can set the following in your configuration file
buildspecs:
rebuild: True
The configuration options such as count
, format
, terse
can be tweaked to your preference. These configuration values
can be overridden by command line option.
Searching for buildspecs when building Buildspec Cache
When building the buildspec cache, buildtest will search for buildspecs in a list of directories specified in the configuration file.
Buildtest will search for buildspecs by recursively searching for files with .yml extension. The directory
property in configuration file
is a list of directories to search for buildspecs. The directory
property is not required in configuration file, but it can be a good
idea to set this value if you have a pre-determined location where buildspecs are stored.
You can specify the directory path via command line buildtest buildspec find --directory <dir1> --directory <dir2>
which will override the configuration value. In a
practical situation, you will want to write your buildspecs in a separate repository which you can clone in your filesystem. Let’s say they are cloned in
your $HOME directory named $HOME/buildtest-examples. You have one of two options, one you can specify the path in configuration file as shown below:
buildspecs:
directory: [ $HOME/buildtest-examples ]
This configuration will instruct buildtest to search for buildspecs in $HOME/buildtest-examples
directory, and you won’t
have to specify the --directory
option when running buildtest buildspec find
. The second option would be to specify the --directory
everytime
you need to build the cache. If neither is specified, buildtest will load the default buildspecs which are $BUILDTEST_ROOT/tutorials and
$BUILDTEST_ROOT/general_tests.
Configuring buildtest report
The report
section in configuration file allows you to configure behavior of buildtest report
command. The
report
section is shown below:
report:
count: 25
#enable terse mode for report
terse: False
format: "name,id,state,runtime,returncode"
The count
property limits the number of records to display when running buildtest report
command. The format
property
controls the fields to display when running buildtest report
. The terse
property enables terse mode for buildtest report
.
CDASH Configuration
buildtest can be configured to push test to CDASH. The default configuration file provides a CDASH configuration for buildtest project is the following.
cdash:
url: https://my.cdash.org/
project: buildtest
site: generic
buildname: tutorials
The cdash section can be summarized as follows:
url
: URL to CDASH server
project
: Project Name in CDASH server
site
: Site name that shows up in CDASH entry. This should be name of your system name
buildname
: Build Name that shows up in CDASH, this can be any name you want.
The cdash settings can be used with buildtest cdash
command. For more details
see CDASH Integration (buildtest cdash).
Test Timeout
The timeout
property is number of seconds a test can run before it is called. The timeout property must be a positive integer.
For instance if you want all test to timeout within 60 sec you can do the following
timeout: 60
The timeout
field is not set by default, it can be configured in the configuration file but can be overridden via command line
option buildtest build --timeout
. For more details see Test Timeout (buildtest build --timeout)
Pool Size
buildtest makes use of multiprocessing.Pool which is used
to control pool size for worker processes used for processing builders during run phase. We can use the poolsize
property
to control the size of pool. The pool size must be 1 or higher, if value exceeds maximum CPU count (i.e. os.cpu_count())
then value is set to maximum CPU count.
Shown below we set poolsize
to 1.
system:
generic:
# specify list of hostnames where buildtest can run for given system record
hostnames: [".*"]
# system description
description: Generic System
# specify module system used at your site (environment-modules, lmod)
moduletool: N/A
# specify test timeout duration in number of seconds
# timeout: 60
poolsize: 1
Maximum Jobs
The max_jobs
property is used to limit number of jobs that can run concurrently. This is useful if you want to limit,
the workload on your system. Buildtest will run all jobs in parallel by default, if max_jobs
is not specified.
If you want to run all tests in serial, you can set max_jobs: 1
as shown below.
max_jobs: 1
This value can be overridden via buildtest build --max-jobs
option. For more details see Limit Maximum Jobs that can run concurrently (buildtest build --max-jobs)
Managing Profiles
The profile
section allows you to define build profiles that can be used to encapsulate buildtest build
options.
This section is auto-generated when using buildtest build --save-profile
option, see Using Profiles (buildtest build --profile) for more details.
Shown below is an example profile, the python-tests
is the name of the profile. The tags
property is a list of tags to use
which are used by buildtest build --tags
option. The testdir
option is the path where tests are written that is used by buildtest build --testdir
.
profiles:
python-tests:
tags:
- python
testdir: /Users/siddiq90/Documents/github/buildtest/var/tests
The profile can be configured with many other options supported by buildtest build
, shown below are additional examples.
Configuration properties like rebuild
, limit
, timeout
are integer and must be positive numbers.
profiles:
profile-2:
buildspecs:
- /Users/siddiq90/Documents/github/buildtest/tutorials/job_dependency
exclude-buildspecs:
- tutorials/job_dependency/ex1.yml
tags:
- python
exclude-tags:
- network
executors:
- generic.local.csh
rebuild: 2
limit: 10
timeout: 10
account: dev
procs:
- 2
- 4
nodes:
- 1
- 2
testdir: /Users/siddiq90/Documents/github/buildtest/var/tests
executor-type: local
Shown below is a generated profile using
buildtest build -b tutorials --filter "tags=pass;maintainers=@shahzebsiddiqui;type=script" --save-profile=filter_profile
. The filter
is an object and attributes tags
, maintainers
, type
correspond to the filter fields.
profiles:
filter_profile:
buildspecs:
- /Users/siddiq90/Documents/github/buildtest/tutorials
testdir: /Users/siddiq90/Documents/github/buildtest/var/tests
filter:
tags:
- pass
maintainers:
- '@shahzebsiddiqui'
type:
- script
We have added additional checks in the JSON schema for valid values for each type, for instance if you specify an invalid value for type
field
which is used to filter buildspecs by the type
field, then you will get an invalid configuration file.
Listing Profiles
This section in the profile permits you to enumerate the profiles available for encapsulating buildtest build options.
Lets create a profile by running the following buildtest command.
$ buildtest build -t python --save-profile=python
Saved profile python to configuration file /tmp/tmpr88132k2/config.yml
The –save-profile is used to specify name of profile that will be written in configuration file.
In order to see all profiles you can run buildtest config profiles list
as shown below
$ buildtest config profiles list
python
$ buildtest config profiles list --yaml
python:
dry-run: false
remove-stagedir: false
strict: false
tags:
- python
testdir: /tmp/tmpr88132k2/var/tests
validate: false
Removing Profiles
You can remove a profile by running buildtest config profiles remove <profile>
, where <profile> is the name of profile.
This command will update your configuration file and remove the profile name from configuration. You can remove multiple profiles at once, buildtest will check if profile name exist and attempt to remove it. If its not found, it will simply skip it.
First, lets create two profile using buildtest build --save-profile
Creating profiles
$ buildtest build -t python --save-profile=prof1
Saved profile prof1 to configuration file /tmp/tmpr88132k2/config.yml
$ buildtest build -b tutorials/shell_examples.yml --save-profile=prof2
Saved profile prof2 to configuration file /tmp/tmpr88132k2/config.yml
Now we will list the profiles to confirm they are created and remove them. Next we will rerun buildtest config profiles list
to confirm
profiles are removed
Example on how to remove profiles
$ buildtest config profiles list
python-tests
python
prof1
prof2
$ buildtest config profiles remove prof1 prof2
Removing profile: prof1
Removing profile: prof2
Updating configuration file: /Users/siddiq90/Documents/github/buildtest/buildtest/settings/config.yml
$ buildtest config profiles list
python-tests
python