Summary of buildtest

Background

HPC System and Software Stack are tightly integrated with underlying architecture which makes them highly sensitive to changes in system such as OS, kernel, driver, or vendor updates. We need a testing framework to automate acceptance testing of an HPC system so that HPC Support Teams can increase confidence of their HPC system throughout the system lifecycle.

Motivation

There are many build automations tools for compiling source code into binary code, the most used tool is the make utility found in most Linux systems. Build scripts like configure, cmake and autoconf can generate files used by make for installing the software. Makefile is a file used by make program that shows how to compile and link a program which is the basis for building a software package. One can invoke make test which will run the target named test in Makefile that dictates how tests are compiled and run. Makefile is hard to interpret and requires in-depth experience with shell-scripting and strong understanding of how package is built and tested. Note that package maintainers must provide the source files, headers, and additional libraries to test the software and make test simply the test compilation and execution. Tools like configure, cmake and autoconf are insufficient for testing because HPC software stack consist of applications packaged in many formats and some are make-incompatible.

We wanted a framework that hides the complexity for compiling source code and provide an easy markup language to define test configuration to create the test. This leads to buildtest, which is a testing framework that generates test-scripts using YAML that is validated with JSON Schemas. YAML was picked given its ease-of-use and it lowers the barrier for writing tests.

Inception of buildtest

buildtest was founded by Shahzeb Siddiqui in 2017 when he was at Pfizer tasked for testing software stack for a data center migration.

Shahzeb was tasked with testing the software ecosystem by focusing on the most important application due to time constraints. During this period, several dozen test scripts were developed in shell-script that targeted core HPC tools such as compilers, MPI, R, Python, etc. A single master script was used to run all the tests which led to buildtest.

Preview of buildtest

Building test

buildtest will process YAML files called buildspecs (build specification) and generate test scripts. To build and run tests you use buildtest build command. There are several ways to build buildspecs, shown below are few examples:

# process a single file
$ buildtest build -b /path/to/file.yml

# recursively find all .yml files in a directory
$ buildtest build -b /path/to/directory

# append -b multiple times
$ buildtest -b /path/to/directory -b /path/to/file.yml

# process entire directory but exclude file system/kernel.yml
$ buildtest -b system/ -x system/kernel.yml

# process entire directory but exclude sub-directory.
$ buildtest build -b system -x system/lustre

# discover buildspecs based on tags
$ buildtest build --tags network

# filter tests by tag name network
$ buildtest build --tags network --filter-tags network

# discover buildspecs based on executors
$ buildtest build --executor generic.local.sh

# combine all options together
$ buildtest build -b example.yml -b system --tags network -x system/kernel.yml

For more details see Building Test via buildtest.

Querying Test Report

buildtest will retrieve test results using buildtest report command which will display all test records. We can filter and format output of test records using the --filter and --format option. Shown below is example usage:

# report all tests
$ buildtest report

# filter and format test report
$ buildtest report --filter KEY1=VALUE1,KEY2=VALUE2 --format <field1>,<field2>

# show filter fields
$ buildtest report --helpfilter

# show format fields
$ buildtest report --helpformat

# get oldest test record
$ buildtest report --oldest

# get latest test record
$ buildtest report --latest

To learn more about buildtest report see querying test reports.

Inspect Tests

If you want to see more detailed results for tests you should use buildtest inspect which will extract the JSON record from the report file. Shown below are few command usage:

# list all test names and id
$ buildtest inspect list

# query record for test name 'python_hello'
$ buildtest inspect name python_hello

# query record for test name 'hello_world' 'shell_opts'
$ buildtest inspect name hello_world shell_opts

# query record based on test id
$ buildtest inspect id <identifier-1> <identifier-2>

To learn more about buildtest inspect see Inspect Tests Records.

Query Buildspecs

buildtest can discover and validate buildspecs with corresponding JSON schema. This feature is handy when you want to see all tests in your acceptance test. To see all buildspecs you need to use buildtest buildspec find:

# build your buildspec cache and report all validated buildspecs
$ buildtest buildspec find

# rebuild buildspec cache and discover new buildspecs
$ buildtest buildspec find --rebuild

# view all tags
$ buildtest buildspec find --tags

# view all executors
$ buildtest buildspec find --executors

# view all maintainers
$ buildtest buildspec find --maintainers

# filter and format buildspec cache
$ buildtest buildspec find --filter KEY1=VALUE1,KEY2=VALUE2 --format <field1>,<field2>

To learn more about buildtest buildspec find see Buildspecs Interface.

Configuration

buildtest provides command line interface via buildtest config to query buildtest configuration file. As you start configuring buildtest at your site, you will want to run:

$ buildtest config validate

If you want to view your configuration file you can do:

$ buildtest config view

buildtest can define compilers in your configuration file which can be used to compile source code in a test. The buildtest config compilers command is responsible for querying and finding compilers:

# list compilers in flat listing
$ buildtest config compilers

# list compilers in YAML format
$ buildtest config compilers --yaml

# list compilers in JSON format
$ buildtest config compilers --json

# find compilers and update configuration file
$ buildtest config compilers find

Schema Interface

buildtest has a command line interface to buildtest schemas via buildtest schema command. We provide a list of available schemas, including schema content and schema examples validated for each schema. This can be queried as follows:

# show available schemas
$ buildtest schema

# show content of schema global.schema.json
$ buildtest schema -n global.schema.json --json

# show schema examples of schema global.schema.json
$ buildtest schema -n global.schema.json --example

We encourage you go over the Getting Started guide as you learn buildtest and try the examples on your machine.

Target Audience & Use Case

buildtest target audience is HPC Staff that wants to perform acceptance & regression testing of their HPC system.

buildtest is not

  • replacement for make, cmake, autoconf, ctest

  • a software build framework (easybuild, spack, nix , guix)

  • a replacement for benchmark tools or test suite from upstream package

  • a replacement for writing tests, you will need to write your tests defined by buildtest schemas, however you can copy/paste & adapt tests from other sites that are applicable to you.

Typical use-case:

  • Run your test suite during system maintenance

  • Perform daily tests for testing various system components. These tests should be short

  • Run weekly/biweekly test on medium/large workload including micro-benchmark

  • Run tests for newly installed software package typically requested by user.

If you are interested trying out buildtest check out Getting Started and Join Slack Channel.

Timeline

Date

Description

Mar 31th 2021

v0.9.5 add support for PBS scheduler and reimplement buildtest inspect command

Mar 14th 2021

v0.9.4 introduced major change in buildtest configuration file (settings.schema.json) to define multiple HPC systems in configuration file. This lead to change in how executors are referenced in buildspec file.

Feb 22nd 2021

v0.9.3 change Copyright details for project to include LBNL. We added dependabot for managing dependencies, added OLCF facility pipelines for running regression test.

Jan 12th 2021

v0.9.2 contains major refactor to compiler-v1.0-schema.json for writing compiler test using regular expression to search for compilers that are defined in configuration file.

Nov 24st 2020

v0.9.1 added support for Cobalt Scheduler.

Sep 3rd 2020

v0.8.0 introduced JSON Schema for validating buildspec. Add support for Slurm and LSF scheduler for job submission. Add support for building buildspecs by file, directory and tagname and command line interface to schema.

Mar 3rd 2020

A spin-off project called lmodule was formed based on buildtest module features.

Sep 11th 2018

In v0.4.0 buildtest was ported from Python 2 to 3.

Aug 20th 2017

In v0.1.5 buildtest was converted from bash to Python and project was moved into github https://github.com/HPC-buildtest/buildtest.

Feb 18th 2017

Start of project