buildtest

This documentation was rebuilt on May 26, 2023 and is intended for version 1.4.

If you are working off the latest release please see https://buildtest.readthedocs.io/en/latest/ for documentation. If you are working off the devel branch then please refer to https://buildtest.readthedocs.io/en/devel/ which references the devel branch.

Useful Links

• Source Code: https://github.com/buildtesters/buildtest

• Documentation: http://buildtest.rtfd.io/

• Schema Docs: https://buildtesters.github.io/buildtest/

• ReadTheDocs: https://readthedocs.org/projects/buildtest/

• CodeCov: https://codecov.io/gh/buildtesters/buildtest

• Slack Channel: http://hpcbuildtest.slack.com

• Slack Invite: https://communityinviter.com/apps/hpcbuildtest/buildtest-slack-invitation

• CodeFactor: https://www.codefactor.io/repository/github/buildtesters/buildtest

• Snyk: https://app.snyk.io/org/buildtesters/

• NERSC Test Repository: https://github.com/buildtesters/buildtest-nersc

Description

buildtest is a testing framework to help HPC sites write test for their system as part of their routine acceptance and regression testing. Buildtest provides a YAML interface to write tests which buildtest can validate and then use to generate shell scripts that run on your HPC system. The test template (YAML) is called a buildspec which can define one or more test instances and is validated by a json schema. Buildtest supports the following batch schedulers: IBM Spectrum LSF, Slurm, PBS and Cobalt. We publish the schema documentation, json schemas, and schema examples at https://buildtesters.github.io/buildtest/ which is useful when you are writing buildspecs.

To get started with buildtest, please see Installing buildtest and Getting Started Guide.

A spin-off project called lmodule is a Python API for Lmod. The buildtest module features were deprecated and moved to lmodule with the main objective is to automate module load testing. For more details on lmodule see https://github.com/buildtesters/lmodule

Background

• Summary of buildtest
  • Background
  • Motivation
  • Inception of buildtest
  • Preview of buildtest
  • Target Audience & Use Case
  • Related Projects and community efforts

Getting Started

• Installing buildtest
  • Requirements
  • Cloning buildtest
  • Installing buildtest
  • Specify Python Wrapper via BUILDTEST_PYTHON
  • Development Dependencies (Optional)
  • Usage (buildtest --help)
• Quick Start Guide
  • Building Test
  • Inspecting a test
  • Querying Test Report
  • Developing a test
  • Buildtest Configuration

Tutorial

• Buildtest Command Line Tutorial
  • Building Test
  • Querying Test Report
  • Inspecting Test
  • Interacting with Buildspecs
  • Buildtest Configuration
• Buildspec Tutorial
  • Buildspec Overview
  • Using Compiler Schema
  • Buildtest Spack Integration
  • Buildtest E4S Testsuite Integration
  • Tutorials Setup
• Buildtest Tutorial on Perlmutter
  • Setup
  • Exercise 1: Running a Batch Job
  • Exercise 2: Performing Status Check
  • Exercise 3: Querying Buildspec Cache
  • Exercise 4: Querying Test Reports
  • Exercise 5: Specifying Performance Checks

How-to-guides

• How to configure buildtest
  • Overview
  • Defining Compilers
  • Site Examples
• Batch Scheduler Support
  • Slurm
  • LSF
  • PBS
  • Cobalt
  • Jobs exceeds max_pend_time
  • Access PBS Container
• Troubleshooting Buildtest
  • How to enable debugging?
  • How to troubleshoot buildtest configuration?
  • How to troubleshoot a buildspec?
  • How to view past builds?

Explanation

• Build and Test Process
  • Discover Buildspecs
  • Parse Buildspecs
  • Building Buildspecs
  • Running Buildspecs
  • Sanity Check
  • Update Report
• Using buildtest at HPC sites
  • Picking a version of buildtest
  • Configuring buildtest for your site
  • Writing Test

Reference

• Buildtest Command Line Reference
  • Building Test (buildtest build)
  • Buildspecs Interface
  • Query Test Report
• Writing Buildspecs
  • Global Schema
  • Performance Checks
  • Test Dependency
  • Running Test Across Multiple Executors
  • Customize Shell
• Additional Features
  • Accessing build history (buildtest history)
  • Buildtest Info (buildtest info)
  • Accessing buildtest documentation
  • Enabling colored output for table entries
  • Listing available color options (buildtest --helpcolor)
  • Disabling Colored Output
  • CDASH Integration (buildtest cdash)
  • Cleaning buildtest files (buildtest clean)
  • Changing Directories (buildtest cd)
  • Get Path for Test (buildtest path)
  • Test Statistics (buildtest stats)
  • Buildtest Debug Report (buildtest debugreport)
  • Command Line Interface to buildtest configuration (buildtest config)
• Buildtest Schemas
  • Schema Naming Convention
  • CLI for buildtest schemas (buildtest schema)
  • Schema Files
  • Schema Examples
• Facility Test Examples
  • Single Test Multiple Compilers
  • MPI Example
  • OpenMP Hello Processor Scaling Test
  • Cray Burst Buffer & Data Warp
• buildtest API Reference
  • buildtest
• Buildtest Command Reference
  • Named Arguments
  • COMMANDS
  • Sub-commands
• Conference and Publications
  • Talks
  • Publications
  • Article

Development Guide

• Contributing Guide
  • Overview
  • Contributing Topics

Useful Links

• Buildtest Github repo
• Buildtest Slack
• buildtest at NERSC

License

buildtest is released under the MIT license

Indices and tables

• Index

• Module Index

• Search Page