Summary of buildtest

Background

HPC System and Software Stack are tightly integrated with underlying architecture which makes them highly sensitive to changes in system such as OS, kernel, driver, or vendor updates. We need a testing framework to automate acceptance testing of an HPC system so that HPC Support Teams can increase confidence of their HPC system throughout the system lifecycle.

Motivation

There are many build automations tools for compiling source code into binary code, the most used tool is the make utility found in most Linux systems. Build scripts like configure, cmake and autoconf can generate files used by make for installing the software. Makefile is a file used by make program that shows how to compile and link a program which is the basis for building a software package. One can invoke make test which will run the target named test in Makefile that dictates how tests are compiled and run. Makefile is hard to interpret and requires in-depth experience with shell-scripting and strong understanding of how package is built and tested. Note that package maintainers must provide the source files, headers, and additional libraries to test the software and make test simply the test compilation and execution. Tools like configure, cmake and autoconf are insufficient for testing because HPC software stack consist of applications packaged in many formats and some are make-incompatible.

We wanted a framework that hides the complexity for compiling source code and provide an easy markup language to define test configuration to create the test. This leads to buildtest, which is a testing framework that generates test-scripts using YAML that is validated with JSON Schemas. YAML was picked given its ease-of-use and it lowers the barrier for writing tests.

Inception of buildtest

buildtest was founded by Shahzeb Siddiqui in 2017 when he was at Pfizer tasked for testing software stack for a data center migration.

Shahzeb was tasked with testing the software ecosystem by focusing on the most important application due to time constraints. During this period, several dozen test scripts were developed in shell-script that targeted core HPC tools such as compilers, MPI, R, Python, etc. A single master script was used to run all the tests which led to buildtest.

Preview of buildtest

You can run buildtest help followed by name of command and it will provide an overview of the buildtest.

Building Test

$ buildtest help build
                                                  Building buildspecs                                                   
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command                                                 ┃ Description                                                ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest build -b <file>                               │ Build a single buildspec file                              │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest build -b <dir>                                │ Build all buildspecs recursively in a given directory      │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest build -b <file> -b <dir>                      │ Build buildspecs by file and directory                     │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest build -b <file> -b <dir> -x <file> -x <dir>   │ Exclude files and directory when building buildspecs       │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest build -t pass -t python                       │ Build buildspecs by tagname 'pass' and 'python'            │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest build -e <executor1> -e <executor2>           │ Building buildspecs by executor                            │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest build -b <file> -t <tagname1> -e <executor1>  │ Building buildspecs with file, directory, tags, and        │
│                                                         │ executors                                                  │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest build -b tutorials  --filter type=script      │ Build all tests in directory 'tutorials' and filter tests  │
│                                                         │ by type='script'                                           │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest build -b tutorials  --filter tags=pass        │ Build all tests in directory 'tutorials' and filter tests  │
│                                                         │ by tags='pass'                                             │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest build -b tutorials  --filter maintainers=@bob │ Build all tests in directory 'tutorials' and filter tests  │
│                                                         │ by maintainers='@bob'                                      │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest build --helpfilter                            │ Show list of filter fields used with --filter option       │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest -c config.yml build -b <file>                 │ Use buildtest configuration file 'config.yml'              │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest build -b <file> --rebuild 5                   │ Rebuild a test 5 times                                     │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest build -b <file> --testdir /tmp                │ Write tests in /tmp                                        │
└─────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────┘

Buildspec Interface

$ buildtest help buildspec
                                                   Finding Buildspecs                                                   
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command                                                 ┃ Description                                                ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest buildspec find                                │ Discover and validate all buildspecs and load all          │
│                                                         │ validated buildspecs in cache                              │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find --rebuild                      │ Rebuild cache file                                         │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find --root /tmp --rebuild          │ Discover buildspecs in /tmp and rebuild buildspec cache    │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find --paths                        │ Print all root directories for buildspecs                  │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find --buildspec                    │ List all available buildspecs from cache                   │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find --executors                    │ List all unique executors from cache                       │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find --maintainers                  │ List all maintainers from cache                            │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find --maintainers-by-buildspecs    │ Show breakdown of all buildspecs by maintainer names       │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find --filter type=script,tags=pass │ Filter buildspec cache based on type=script and            │
│                                                         │ tags='pass'                                                │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find --filter buildspec=<path>      │ Filter cache by buildspec file                             │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find --format name,description      │ Format table columns by field: 'name', and 'description'   │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find --group-by-tags                │ Group tests by tag name                                    │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find --group-by-executor            │ Group tests by executor name                               │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find --helpfilter                   │ Show all filter fields                                     │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find --helpformat                   │ Show all format fields                                     │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find --terse                        │ Display output in terse format                             │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find invalid                        │ Show invalid buildspecs                                    │
├─────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ buildtest buildspec find invalid --error                │ Show invalid buildspecs with error messages                │
└─────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────┘
                                                 Validating Buildspecs                                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command                                               ┃ Description                                                  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest buildspec validate -b <file>                │ Validate a buildspec with JSON Schema                        │
├───────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ buildtest buildspec validate -b /tmp/ -x /tmp/network │ Validate all buildspecs in directory /tmp but exclude        │
│                                                       │ /tmp/network                                                 │
├───────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ buildtest buildspec validate -t python -t mac         │ Validate all buildspecs for tagname 'python' and 'mac'       │
├───────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ buildtest buildspec validate -e generic.local.bash    │ Validate all buildspecs for executor 'generic.local.bash'    │
└───────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────┘
                                   Additional Features of Buildspecs                                   
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command                               ┃ Description                                                 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest buildspec summary           │ Show summary of buildspec cache file                        │
├───────────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ buildtest buildspec show python_hello │ Show content of buildspec based on test name 'python_hello' │
└───────────────────────────────────────┴─────────────────────────────────────────────────────────────┘

Query Report

$ buildtest help report
                                                 Viewing Test Report                                                  
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command                                                ┃ Description                                               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest report                                       │ Display all test results                                  │
├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ buildtest report --filter returncode=0                 │ Filter test results by returncode=0                       │
├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ buildtest report --filter state=PASS,tags=python       │ Filter test by filter fields 'state', 'tags'.             │
├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ buildtest report --filter buildspec=tutorials/vars.yml │ Filter report by buildspec file 'tutorials/vars.yml       │
├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ buildtest report --format name,state,buildspec         │ Format report table by field 'name', 'state', 'buildspec' │
├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ buildtest report --helpfilter                          │ List all filter fields                                    │
├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ buildtest report --helpformat                          │ List all format fields                                    │
├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ buildtest report --latest                              │ Retrieve latest record for all tests                      │
├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ buildtest report -r <report-file>                      │ Specify alternate report file to display test results     │
├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ buildtest report --terse                               │ Print report in terse format                              │
├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ buildtest report list                                  │ List all report files                                     │
├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ buildtest report clear                                 │ Remove content of default report file                     │
├────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ buildtest report summary                               │ Show summary of test report                               │
└────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────┘

Inspect Tests

$ buildtest help inspect
                                                   Inspecting a test                                                    
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command                                        ┃ Description                                                         ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest inspect list                         │ Display all test names, ids, and corresponding buildspec file       │
├────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ buildtest inspect list -t                      │ Show output in terse format                                         │
├────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ buildtest inspect name hello                   │ Display last run for test name 'hello'                              │
├────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ buildtest inspect name hello/9ac bar/ac9       │ Display record for test 'hello/9ac' and 'bar/ac9'. Will find first  │
│                                                │ match for each test ID                                              │
├────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ buildtest inspect buildspec tutorials/vars.yml │ Fetch latest runs for all tests in buildspec file                   │
│                                                │ 'tutorials/vars.yml'                                                │
├────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ buildtest inspect query -o hello               │ Display content of output file for test name 'hello'                │
├────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ buildtest inspect query -e hello               │ Display content of error file for test name 'hello'                 │
├────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ buildtest inspect query -d first -o -e foo bar │ Display first record of tests 'foo', 'bar', and show output and     │
│                                                │ error file                                                          │
├────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ buildtest inspect query -o hello               │ Display all runs for tests 'foo'                                    │
└────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────┘

Target Audience & Use Case

buildtest target audience is HPC Staff that wants to perform acceptance & regression testing of their HPC system.

buildtest is not

  • replacement for make, cmake, autoconf, ctest

  • a software build framework (easybuild, spack, nix , guix)

  • a replacement for benchmark tools or test suite from upstream package

  • a replacement for writing tests, you will need to write your tests defined by buildtest schemas, however you can copy/paste & adapt tests from other sites that are applicable to you.

Typical use-case:

  • Run your test suite during system maintenance

  • Perform daily tests for testing various system components. These tests should be short

  • Run weekly/biweekly test on medium/large workload including micro-benchmark

  • Run tests for newly installed software package typically requested by user.

If you are interested trying out buildtest check out Buildtest Tutorial and Join Slack Channel.

Timeline

Date

Version

Description

Sep 9th 2021

v0.11.0

Reimplement core implementation of running and polling jobs using asynchronous job submission. In addition we added several new commands including: buildtest cd, buildtest path and buildtest path and we enable alias for sub-commands.

Aug 16th 2021

v0.10.2

Add support for multi executor support in buildspec via executors property. Add new commands buildtest report summary for summary of report file. The buildtest buildspec show command shows content of buildspec file given a test name. The buildtest edit command can be used to edit buildspec and validate with JSON schema upon closing file. In this release, we added buildtest inspect buildspec command to view records based on buildspec file.

Jul 30th 2021

v0.10.1

Add new commands buildtest buildspec summary, buildtest buildspec invalid to show summary of buildspec cache and invalid buildspecs. Add buildtest build --filter to filter buildspecs during build. Add --terse option for several commands including buildtest history list, buildtest report, buildtest buildspec find. Add new command buildtest inspect query for querying test records. Added support for metrics property for defining arbitrary metrics in buildspec based on environment variable, variables or regular expression on stdout/stderr

Jul 13th 2021

v0.10.0

In this release we added spack support in buildtest by creating a new schema to write buildspecs that will generate spack commands. For more details see spack schema. We added bash completion for buildtest commands which is enabled when installing buildtest. We added a new command buildtest buildspec validate that can be used for validating buildspecs with JSON Schema.

Jun 11th 2021

v0.9.6

Added buildtest CDASH integration using buildtest cdash to upload test results. In this release we added buildtest history command to retrieve build history and query logfiles. Add global option -c in buildtest to specify alternate configuration file.

Mar 31th 2021

v0.9.5

Add support for PBS scheduler and reimplement buildtest inspect command

Mar 14th 2021

v0.9.4

Introduced major change in buildtest configuration file (settings.schema.json) to define multiple HPC systems in configuration file. This lead to change in how executors are referenced in buildspec file.

Feb 22nd 2021

v0.9.3

Change Copyright details for project to include LBNL. We added dependabot for managing dependencies, added OLCF facility pipelines for running regression test.

Jan 12th 2021

v0.9.2

Contains major refactor to compiler-v1.0-schema.json for writing compiler test using regular expression to search for compilers that are defined in configuration file.

Nov 24st 2020

v0.9.1

Added support for Cobalt Scheduler.

Sep 3rd 2020

v0.8.0

Introduced JSON Schema for validating buildspec. Add support for Slurm and LSF scheduler for job submission. Add support for building buildspecs by file, directory and tagname and command line interface to schema.

Sep 11th 2018

v0.4.0

buildtest was ported from Python 2 to 3.

Aug 20th 2017

v0.1.5

buildtest was converted from bash to Python and project was moved into github https://github.com/HPC-buildtest/buildtest.

Feb 18th 2017

N/A

Start of project