Summary of buildtest¶
Contents
Background¶
HPC System and Software Stack are tightly integrated with underlying architecture which makes them highly sensitive to changes in system such as OS, kernel, driver, or vendor updates. We need a testing framework to automate acceptance testing of an HPC system so that HPC Support Teams can increase confidence of their HPC system throughout the system lifecycle.
Motivation¶
There are many build automations tools for compiling source code into binary code, the most used tool is the make utility found in most Linux systems. Build scripts like configure, cmake and autoconf can generate files used by make for installing the software. Makefile is a file used by make program that shows how to compile and link a program which is the basis for building a software package. One can invoke make test which will run the target named test in Makefile that dictates how tests are compiled and run. Makefile is hard to interpret and requires in-depth experience with shell-scripting and strong understanding of how package is built and tested. Note that package maintainers must provide the source files, headers, and additional libraries to test the software and make test simply the test compilation and execution. Tools like configure, cmake and autoconf are insufficient for testing because HPC software stack consist of applications packaged in many formats and some are make-incompatible.
We wanted a framework that hides the complexity for compiling source code and provide an easy markup language to define test configuration to create the test. This leads to buildtest, which is a testing framework that generates test-scripts using YAML that is validated with JSON Schemas. YAML was picked given its ease-of-use and it lowers the barrier for writing tests.
Inception of buildtest¶
buildtest was founded by Shahzeb Siddiqui in 2017 when he was at Pfizer tasked for testing software stack for a data center migration.
Shahzeb was tasked with testing the software ecosystem by focusing on the most important application due to time constraints. During this period, several dozen test scripts were developed in shell-script that targeted core HPC tools such as compilers, MPI, R, Python, etc. A single master script was used to run all the tests which led to buildtest.
Preview of buildtest¶
You can run buildtest help
which will provide an overview of buildtest commands.
$ buildtest help
Building Buildspecs
--------------------
Build a single buildspec file
$ buildtest build -b <file>
Search all buildspecs recursively in a given directory for all '.yml' files and build all buildspecs
$ buildtest build -b <dir>
Build buildspecs by file and directory
$ buildtest build -b <file> -b <dir>
Exclude files when building buildspecs
$ buildtest build -b <file> -b <dir> -x <file> -x <dir>
Building buildspecs by tag
$ buildtest build -t <tagname1> -t <tagname2>
Building buildspecs by executor
$ buildtest build -e <executor1> -e <executor2>
building buildspecs with file, directory, tags, and executors
$ buildtest build -b <file> -b <dir> -t <tagname1> -e <executor1>
Filter tests by tagname for all buildspecs discovered
$ buildtest build -b <file> -t <tagname1> -ft <filter-tagname1> -ft <filter-tagname2>
Specify alternate buildtest configuration file when building test.
$ buildtest -c <config> build -b <file>
Specify alternate location of test directory (/tmp)
$ buildtest build -b <file> --testdir /tmp
Set max-pend-time to 30s which overrides 'max_pend_time' field in configuration file. buildtest will cancel job
job if its pending in queue beyond max-pend-time value
$ buildtest build -b <file> --max-pend-time 30
Set pollinterval to 60s which overrides 'pollinterval' field in configuration file. buildtest will poll jobs based on 'pollinterval' value.
$ buildtest build -b <file> --poll-interval 60
View report
------------
Display all tests results
$ buildtest report
Filter test results by returncode=0
$ buildtest report --filter returncode=0
Filter test by multiple filter fields. In this example, buildtest will report all tests that 'PASS' for tagname 'python'
$ buildtest report --filter state=PASS,tags=python
Format report table by --format fields, in this example we only report columns 'name', 'state', 'buildspec'
$ buildtest report --format name,state,buildspec
List all filter fields
$ buildtest report --helpfilter
List all format fields
$ buildtest report --helpformat
Retrieve oldest record for all tests based on timestamp
$ buildtest report --oldest
Retrieve latest record for all tests based on timestamp
$ buildtest report --latest
Specify alternate report file to display test results
$ buildtest report -r <report-file>
Inspect Tests
-------------
Display record of all runs for test name 'hello'.
$ buildtest inspect name hello
Display record of test name 'foo' and 'bar'
$ buildtest inspect name foo bar
Display all test names and ids
$ buildtest inspect list
Display record of test by unique identifer
$ buildtest inspect id <ID>
Finding Buildspecs
----------------------
Discover and validate all buildspecs and load all validated buildspecs in cache. Finally buildtest will report all tests for all validated buildspecs
$ buildtest buildspec find
Rebuild cache
$ buildtest buildspec find --rebuild
List all available buildspecs from cache
$ buildtest buildspec find --buildspec
List all unique tags from cache
$ buildtest buildspec find --tags
List all unique executors from cache
$ buildtest buildspec find --executors
List all maintainers from cache
$ buildtest buildspec find --maintainers
Show breakdown of all buildspecs by maintainer names.
$ buildtest buildspec find --maintainers-by-buildspecs
Validate buildspecs
---------------------
Validate buildspecs by file and directory
$ buildtest buildspec validate -b <file> -b <dir>
Validate buildspecs by tagname 'python' and 'mac'
$ buildtest buildspec validate -t python -t mac
Validate buildspecs by executor name
$ buildtest buildspec validate -e <executor>
Build History
---------------
The 'buildtest history' command can report history of all builds when you run 'buildtest build' command
along with log files.
To display a list of all builds
$ buildtest history list
Query information for a given build by unique identifier.
$ buildtest history query <ID>
Get logfile for given build
$ buildtest history query <ID> --log
Buildtest Configuration
------------------------
View content of configuration file
$ buildtest config view
Validate configuration file with JSON schema
$ buildtest config validate
List all executors from configuration file
$ buildtest config executors
List all available system entries in configuration file
$ buildtest config systems
Specify alternate configuration file
$ buildtest -c <config> config validate
Upload Tests to CDASH
---------------------
Upload all tests to cdash with build name 'DEMO'
$ buildtest cdash upload DEMO
Upload all tests from report file /tmp/result.json with build name 'DAILY_CHECK'
$ buildtest cdash upload 'DAILY_CHECK' --report /tmp/result.json
Upload tests to CDASH with site named called 'laptop'
$ buildtest cdash upload --site laptop DEMO
Open CDASH project in web-browser
$ buildtest cdash view
Open CDASH project with url name
$ buildtest cdash view --url https://my.cdash.org/index.php?project=buildtest
Target Audience & Use Case¶
buildtest target audience is HPC Staff that wants to perform acceptance & regression testing of their HPC system.
buildtest is not
replacement for make, cmake, autoconf, ctest
a replacement for benchmark tools or test suite from upstream package
a replacement for writing tests, you will need to write your tests defined by buildtest schemas, however you can copy/paste & adapt tests from other sites that are applicable to you.
Typical use-case:
Run your test suite during system maintenance
Perform daily tests for testing various system components. These tests should be short
Run weekly/biweekly test on medium/large workload including micro-benchmark
Run tests for newly installed software package typically requested by user.
If you are interested trying out buildtest check out Getting Started and Join Slack Channel.
Timeline¶
Date |
Version |
Description |
---|---|---|
Jul 13th 2021 |
In this release we added spack support in buildtest by creating a new schema to write buildspecs that will generate spack commands. For more details see Spack Schema. We added bash completion for buildtest commands which is enabled when installing buildtest. We added a new command |
|
Jun 11th 2021 |
Added buildtest CDASH integration using |
|
Mar 31th 2021 |
Add support for PBS scheduler and reimplement |
|
Mar 14th 2021 |
Introduced major change in buildtest configuration file ( |
|
Feb 22nd 2021 |
Change Copyright details for project to include LBNL. We added dependabot for managing dependencies, added OLCF facility pipelines for running regression test. |
|
Jan 12th 2021 |
Contains major refactor to |
|
Nov 24st 2020 |
Added support for Cobalt Scheduler. |
|
Sep 3rd 2020 |
Introduced JSON Schema for validating buildspec. Add support for Slurm and LSF scheduler for job submission. Add support for building buildspecs by file, directory and tagname and command line interface to schema. |
|
Sep 11th 2018 |
buildtest was ported from Python 2 to 3. |
|
Aug 20th 2017 |
buildtest was converted from bash to Python and project was moved into github https://github.com/HPC-buildtest/buildtest. |
|
Feb 18th 2017 |
N/A |
Start of project |