Summary of buildtest

Background

HPC System and Software Stack are tightly integrated with underlying architecture which makes them highly sensitive to changes in system such as OS, kernel, driver, or vendor updates. We need a testing framework to automate acceptance testing of an HPC system so that HPC Support Teams can increase confidence of their HPC system throughout the system lifecycle.

Motivation

There are many build automations tools for compiling source code into binary code, the most used tool is the make utility found in most Linux systems. Build scripts like configure, cmake and autoconf can generate files used by make for installing the software. Makefile is a file used by make program that shows how to compile and link a program which is the basis for building a software package. One can invoke make test which will run the target named test in Makefile that dictates how tests are compiled and run. Makefile is hard to interpret and requires in-depth experience with shell-scripting and strong understanding of how package is built and tested. Note that package maintainers must provide the source files, headers, and additional libraries to test the software and make test simply the test compilation and execution. Tools like configure, cmake and autoconf are insufficient for testing because HPC software stack consist of applications packaged in many formats and some are make-incompatible.

We wanted a framework that hides the complexity for compiling source code and provide an easy markup language to define test configuration to create the test. This leads to buildtest, which is a testing framework that generates test-scripts using YAML that is validated with JSON Schemas. YAML was picked given its ease-of-use and it lowers the barrier for writing tests.

Inception of buildtest

buildtest was founded by Shahzeb Siddiqui in 2017 when he was at Pfizer tasked for testing software stack for a data center migration.

Shahzeb was tasked with testing the software ecosystem by focusing on the most important application due to time constraints. During this period, several dozen test scripts were developed in shell-script that targeted core HPC tools such as compilers, MPI, R, Python, etc. A single master script was used to run all the tests which led to buildtest.

Preview of buildtest

You can run buildtest show followed by name of command and it will provide an overview of the buildtest.

buildtest show build
$ buildtest show build
                              Building buildspecs                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command                               ┃ Description                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest build -b <file>             │ Build a single buildspec file        │
│ buildtest build -b <dir>              │ Build all buildspecs recursively in  │
│                                       │ a given directory                    │
│ buildtest build -b <file> -b <dir>    │ Build buildspecs by file and         │
│                                       │ directory                            │
│ buildtest build -b <file> -b <dir> -x │ Exclude files and directory when     │
│ <file> -x <dir>                       │ building buildspecs                  │
│ buildtest build -t pass -t python     │ Build buildspecs by tagname 'pass'   │
│                                       │ and 'python'                         │
│ buildtest build -t pass,fail          │ Build buildspecs by tagnames that    │
│                                       │ are specified as comma separated     │
│                                       │ list                                 │
│ buildtest build -b <file> -xt python  │ Build buildspecs by file and exclude │
│                                       │ tests with tagname 'python'          │
│ buildtest build -b <file> -xt         │ Build buildspecs by file and exclude │
│ pass,fail                             │ tests with tagname 'pass' and        │
│                                       │ 'fail'. Tag names can be specified   │
│                                       │ as comma separated list              │
│ buildtest build --name hello_world    │ Discover buildspecs by test name     │
│ --name add_numbers                    │ 'hello_world' and 'add_numbers'      │
│ buildtest build -e <executor1> -e     │ Building buildspecs by executor      │
│ <executor2>                           │                                      │
│ buildtest build -b <file> -t          │ Building buildspecs with file,       │
│ <tagname1> -e <executor1>             │ directory, tags, and executors       │
│ buildtest build -b tutorials          │ Build all tests in directory         │
│ --filter type=script                  │ 'tutorials' and filter tests by      │
│                                       │ type='script'                        │
│ buildtest build -b tutorials          │ Build all tests in directory         │
│ --filter tags=pass                    │ 'tutorials' and filter tests by      │
│                                       │ tags='pass'                          │
│ buildtest build -b tutorials          │ Build all tests in directory         │
│ --filter maintainers=@bob             │ 'tutorials' and filter tests by      │
│                                       │ maintainers='@bob'                   │
│ buildtest build --helpfilter          │ Show list of filter fields used with │
│                                       │ --filter option                      │
│ buildtest -c config.yml build -b      │ Use buildtest configuration file     │
│ <file>                                │ 'config.yml'                         │
│ buildtest build -b <file> --rebuild 5 │ Rebuild a test 5 times               │
│ buildtest build -b <file> --testdir   │ Write tests in /tmp                  │
│ /tmp                                  │                                      │
│ buildtest build --rerun               │ Run last successful 'buildtest       │
│                                       │ build' command                       │
│ buildtest -r $HOME/python.json build  │ Write test to report file            │
│ -t python                             │ $HOME/python.json for all test run   │
│                                       │ via 'python' tag                     │
│ buildtest build -b <file>             │ For every test run 'module purge'    │
│ --module-purge --modules gcc,python   │ and then load 'gcc' and 'python'     │
│                                       │ module                               │
│ buildtest build -b <file>             │ For every test run 'module unload    │
│ --unload-modules gcc/9.3.0 --modules  │ gcc/9.3.0' and then load             │
│ gcc/10.3.0                            │ 'gcc/10.3.0'                         │
│ buildtest build -b /tmp/hostname.yml  │ Poll jobs every 10 seconds and       │
│ --maxpendtime 120 --pollinterval 10   │ maximum pending time for jobs to 120 │
│                                       │ sec when submitting batch job. Job   │
│                                       │ will be cancelled after 120sec if    │
│                                       │ job is pending                       │
│ buildtest build -b <file> --account   │ Use project 'dev' when submitting    │
│ dev                                   │ batch jobs                           │
│ buildtest build -b <file> --timeout   │ Test will run till it reaches        │
│ 60                                    │ timeout of 60sec and then it will be │
│                                       │ cancelled if it exceeds the limit.   │
│ buildtest build -t python --limit=5   │ Limit number of test runs to 5       │
│ buildtest build -t python             │ Save buildtest options to profile    │
│ --save-profile=python-profile         │ name 'python-profile'                │
│ buildtest build                       │ Run buildtest from profile name      │
│ --profile=python-profile              │ 'python-profile'                     │
│ buildtest build --validate -b <file>  │ Test will validate buildspecs and    │
│                                       │ stop after parsing stage             │
└───────────────────────────────────────┴──────────────────────────────────────┘
buildtest show buildspec
$ buildtest show buildspec
                               Finding Buildspecs                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command                               ┃ Description                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest buildspec find              │ Discover and validate all buildspecs │
│                                       │ and load all validated buildspecs in │
│                                       │ cache                                │
│ buildtest buildspec find --rebuild    │ Rebuild cache file                   │
│ buildtest buildspec find --pager      │ Paginate output of buildspec cache   │
│ buildtest buildspec find --directory  │ Discover buildspecs in /tmp and      │
│ /tmp --rebuild                        │ rebuild buildspec cache              │
│ buildtest buildspec find --quiet      │ Rebuild cache file but don't display │
│ --rebuild                             │ output of cache                      │
│ buildtest buildspec find --paths      │ Print all root directories for       │
│                                       │ buildspecs                           │
│ buildtest buildspec find --buildspec  │ List all available buildspecs from   │
│                                       │ cache                                │
│ buildtest buildspec find --executors  │ List all unique executors from cache │
│ buildtest buildspec find --filter     │ Filter buildspec cache based on      │
│ type=script,tags=pass                 │ type=script and  tags='pass'         │
│ buildtest buildspec find --filter     │ Filter cache by buildspec file       │
│ buildspec=<path>                      │                                      │
│ buildtest buildspec find --format     │ Format table columns by field:       │
│ name,description                      │ 'name', and 'description'            │
│ buildtest buildspec find              │ Group tests by tag name              │
│ --group-by-tags                       │                                      │
│ buildtest buildspec find              │ Group tests by executor name         │
│ --group-by-executor                   │                                      │
│ buildtest buildspec find --helpfilter │ Show all filter fields               │
│ buildtest buildspec find --helpformat │ Show all format fields               │
│ buildtest buildspec find --terse      │ Display output in terse format       │
│ buildtest buildspec find --row-count  │ Print total count of records from    │
│                                       │ the table                            │
│ buildtest buildspec find --count=5    │ Limit output of buildspec cache to 5 │
│                                       │ elements                             │
│ buildtest buildspec find invalid      │ Show invalid buildspecs              │
│ buildtest buildspec find invalid      │ Show invalid buildspecs with error   │
│ --error                               │ messages                             │
│ buildtest buildspec find invalid      │ Print total count of records from    │
│ --row-count                           │ the table                            │
└───────────────────────────────────────┴──────────────────────────────────────┘
                             Validating Buildspecs                              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command                               ┃ Description                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest buildspec validate -b       │ Validate a buildspec with JSON       │
│ <file>                                │ Schema                               │
│ buildtest buildspec validate -b /tmp/ │ Validate all buildspecs in directory │
│ -x /tmp/network                       │ /tmp but exclude /tmp/network        │
│ buildtest buildspec validate -t       │ Validate all buildspecs for tagname  │
│ python -t mac                         │ 'python' and 'mac'                   │
│ buildtest buildspec validate -e       │ Validate all buildspecs for executor │
│ generic.local.bash                    │ 'generic.local.bash'                 │
└───────────────────────────────────────┴──────────────────────────────────────┘
                       Additional Features of Buildspecs                        
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command                               ┃ Description                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest buildspec summary           │ Show summary of buildspec cache file │
│ buildtest buildspec summary --pager   │ Paginate the output of summary for   │
│                                       │ buildspec cache                      │
│ buildtest buildspec show python_hello │ Show content of buildspec based on   │
│                                       │ test name 'python_hello'             │
│ buildtest buildspec show python_hello │ Use color theme 'emacs' for showing  │
│ --theme emacs                         │ content of test                      │
│ buildtest buildspec show-fail         │ Show content of buildspec on all     │
│                                       │ failed tests                         │
│ buildtest buildspec show-fail         │ Show content of test 'exit1_fail'    │
│ exit1_fail                            │                                      │
│ buildtest buildspec edit-test         │ Open test 'python_hello' in editor   │
│ python_hello                          │ and validate file upon closing       │
│ buildtest buildspec edit-file         │ Open file                            │
│ $BUILDTEST_ROOT/tutorials/sleep.yml   │ $BUILDTEST_ROOT/tutorials/sleep.yml  │
│                                       │ in editor and validate file upon     │
│                                       │ closing                              │
│ buildtest buildspec maintainers find  │ Find buildspec with maintainer name  │
│ johndoe                               │ 'johndoe'                            │
│ buildtest buildspec maintainers       │ List all maintainers from buildspec  │
│                                       │ cache                                │
│ buildtest buildspec maintainers       │ Print total count of records from    │
│ --row-count                           │ the table                            │
│ buildtest buildspec maintainers       │ List all maintainers in machine      │
│ --terse --no-header                   │ readable format without header       │
│ buildtest buildspec maintainers       │ Show breakdown of maintainers by     │
│ --breakdown                           │ buildspecs                           │
└───────────────────────────────────────┴──────────────────────────────────────┘
buildtest show report
$ buildtest show report
                              Viewing Test Report                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command                              ┃ Description                           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest report                     │ Display all test results              │
│ buildtest report --pager             │ Paginate output of test results       │
│ buildtest report --filter            │ Filter test results by returncode=0   │
│ returncode=0                         │                                       │
│ buildtest report --filter            │ Filter test by filter fields 'state', │
│ state=PASS,tags=python               │ 'tags'.                               │
│ buildtest report --filter            │ Filter report by buildspec file       │
│ buildspec=tutorials/vars.yml         │ 'tutorials/vars.yml                   │
│ buildtest report --format            │ Format report table by field 'name',  │
│ name,state,buildspec                 │ 'state', 'buildspec'                  │
│ buildtest report --helpfilter        │ List all filter fields                │
│ buildtest report --helpformat        │ List all format fields                │
│ buildtest report --latest            │ Retrieve latest record for all tests  │
│ buildtest report --count=5           │ Retrieve 5 records from report file   │
│ buildtest -r /tmp/result.json report │ Read report file /tmp/result.json and │
│                                      │ display result                        │
│ buildtest report --fail              │ Show all test failures                │
│ buildtest report --pass              │ Show all test passed                  │
│ buildtest report --start 2022-01-01  │ Show all test records in the date     │
│ --end 2022-01-05                     │ range from [2022-01-01, 2022-01-05]   │
│ buildtest report --terse             │ Print report in terse format          │
│ buildtest report --row-count         │ Print total count of records from the │
│                                      │ table                                 │
│ buildtest report list                │ List all report files                 │
│ buildtest report clear               │ Remove content of default report file │
│ buildtest report path                │ Print full path to the report file    │
│                                      │ being used                            │
│ buildtest report summary             │ Show summary of test report           │
│ buildtest report summary --detailed  │ Show detailed summary of test report  │
│ buildtest report summary --pager     │ Paginate output of report summary     │
└──────────────────────────────────────┴───────────────────────────────────────┘
buildtest show inspect
$ buildtest show inspect
                               Inspecting a test                                
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command                              ┃ Description                           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest inspect list               │ Display all test names, ids, and      │
│                                      │ corresponding buildspec file          │
│ buildtest inspect list -t            │ Show output in terse format           │
│ buildtest inspect list --pager       │ Paginate output of inspect list       │
│ buildtest inspect list --row-count   │ Print total count of records from the │
│                                      │ table                                 │
│ buildtest inspect name hello         │ Display last run for test name        │
│                                      │ 'hello'                               │
│ buildtest inspect name hello/9ac     │ Display record for test 'hello/9ac'   │
│ bar/ac9                              │ and 'bar/ac9'. Will find first match  │
│                                      │ for each test ID                      │
│ buildtest inspect buildspec          │ Fetch latest runs for all tests in    │
│ tutorials/vars.yml                   │ buildspec file 'tutorials/vars.yml'   │
│ buildtest inspect query -o           │ Display content of output file for    │
│ exit1_fail                           │ latest run for test name 'exit1_fail' │
│ buildtest inspect query -e hello     │ Display content of error file for     │
│                                      │ test name 'hello'                     │
│ buildtest inspect query exit1_fail/  │ Display all runs for tests            │
│                                      │ 'exit1_fail'                          │
│ buildtest inspect query              │ Use regular expression when searching │
│ 'exit1_fail/(24|52)'                 │ for test via 'buildtest inspect       │
│                                      │ query'                                │
└──────────────────────────────────────┴───────────────────────────────────────┘
buildtest show config
$ buildtest show config
                             Configuring Buildtest                              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Command                               ┃ Description                          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildtest config view                 │ View content of configuration file   │
│ buildtest config view --pager         │ Paginate output of configuration     │
│                                       │ file                                 │
│ buildtest config validate             │ Validate configuration file with     │
│                                       │ JSON schema                          │
│ buildtest config edit                 │ Edit configuration file in your      │
│                                       │ preferred editor                     │
│ buildtest config executors list       │ List all executors in flat listing   │
│                                       │ from configuration file              │
│ buildtest config executors list       │ Show executor configuration in YAML  │
│ --yaml                                │ format                               │
│ buildtest config executors list       │ Show executor configuration in JSON  │
│ --json                                │ format                               │
│ buildtest config executors list       │ List all disabled executors          │
│ --disabled                            │                                      │
│ buildtest config executors list       │ List all invalid executors           │
│ --json                                │                                      │
│ buildtest config executors list --all │ List all available executors         │
│ buildtest config executors remove     │ Remove executor names                │
│ generic.local.bash generic.local.sh   │ 'generic.local.bash' and             │
│                                       │ 'generic.local.sh' from              │
│                                       │ configuration file                   │
│ buildtest config path                 │ Show path to configuration file      │
│ buildtest config systems              │ List all available system entries in │
│                                       │ configuration file                   │
│ buildtest -c /tmp/config.yml config   │ Validate configuration file          │
│ validate                              │ /tmp/config.yml                      │
│ buildtest config compilers list       │ List all compilers from              │
│                                       │ configuration file in flat listing   │
│ buildtest config compilers find       │ Detect compilers and update          │
│                                       │ configuration file                   │
│ buildtest config compilers find       │ Show detailed output when finding    │
│ --detailed --update                   │ compiler and update configuration    │
│                                       │ file with new compilers              │
│ buildtest config compilers test       │ Test each compiler instance by       │
│                                       │ performing module load test          │
│ buildtest config compilers remove     │ Remove compiler instance gcc/7.5.0   │
│ gcc/7.5.0                             │                                      │
│ buildtest config profiles list        │ Listing all profiles from            │
│                                       │ configuration file                   │
│ buildtest config profiles remove      │ Remove a profile 'python' from       │
│ python                                │ configuration file                   │
└───────────────────────────────────────┴──────────────────────────────────────┘

Target Audience & Use Case

buildtest target audience is HPC Staff, Developers, and Vendors that wants to perform acceptance & regression testing of their HPC system.

buildtest is not

  • replacement for make, cmake, autoconf, ctest

  • a software build framework (easybuild, spack, nix , guix)

  • a replacement for benchmark tools or test suite from upstream package

  • a replacement for writing tests, you will need to write your tests defined by buildtest schemas, however you can copy/paste & adapt tests from other sites that are applicable to you.

Typical use-case:

  • Run your test suite during system maintenance

  • Perform daily tests for testing various system components. These tests should be short

  • Run weekly/biweekly test on medium/large workload including micro-benchmark

  • Run tests for newly installed software package typically requested by user.

If you are interested trying out buildtest check out Buildtest Command Line Reference.