.. _summary_of_buildtest: Summary of buildtest ====================== Background ------------ HPC System and Software Stack are tightly integrated with underlying architecture which makes them highly sensitive to changes in system such as OS, kernel, driver, or vendor updates. We need a testing framework to automate acceptance testing of an HPC system so that HPC Support Teams can increase **confidence** of their HPC system throughout the system lifecycle. Motivation ----------- There are many build automations tools for compiling source code into binary code, the most used tool is the **make** utility found in most Linux systems. Build scripts like **configure**, **cmake** and **autoconf** can generate files used by make for installing the software. Makefile is a file used by make program that shows how to compile and link a program which is the basis for building a software package. One can invoke **make test** which will run the target named **test** in Makefile that dictates how tests are compiled and run. Makefile is hard to interpret and requires in-depth experience with shell-scripting and strong understanding of how package is built and tested. Note that package maintainers must provide the source files, headers, and additional libraries to test the software and make test simply the test compilation and execution. Tools like `configure`, `cmake` and `autoconf` are insufficient for testing because HPC software stack consist of applications packaged in many formats and some are make-incompatible. We wanted a framework that hides the complexity for compiling source code and provide an easy markup language to define test configuration to create the test. This leads to buildtest, which is a testing framework that generates test-scripts using YAML that is validated with JSON Schemas. YAML was picked given its ease-of-use and it lowers the barrier for writing tests. Inception of buildtest --------------------------- buildtest was founded by `Shahzeb Siddiqui `_ in 2017 when he was at `Pfizer `_ tasked for testing software stack for a data center migration. Shahzeb was tasked with testing the software ecosystem by focusing on the most important application due to time constraints. During this period, several dozen test scripts were developed in shell-script that targeted core HPC tools such as compilers, **MPI**, **R**, **Python**, etc. A single master script was used to run all the tests which led to `buildtest`. Preview of buildtest ---------------------- You can run ``buildtest show`` followed by name of command and it will provide an overview of the buildtest. .. dropdown:: ``buildtest show build`` .. command-output:: buildtest show build .. dropdown:: ``buildtest show buildspec`` .. command-output:: buildtest show buildspec .. dropdown:: ``buildtest show report`` .. command-output:: buildtest show report .. dropdown:: ``buildtest show inspect`` .. command-output:: buildtest show inspect .. dropdown:: ``buildtest show config`` .. command-output:: buildtest show config Target Audience & Use Case --------------------------- buildtest target audience is **HPC Staff**, **Developers**, and **Vendors** that wants to perform acceptance & regression testing of their HPC system. buildtest is not - replacement for `make`, `cmake`, `autoconf`, `ctest` - a software build framework (`easybuild `_, `spack `__, `nix `_ , `guix `_) - a replacement for benchmark tools or test suite from upstream package - a replacement for writing tests, you will need to write your tests defined by buildtest schemas, however you can copy/paste & adapt tests from other sites that are applicable to you. Typical use-case: - Run your test suite during system maintenance - Perform daily tests for testing various system components. These tests should be short - Run weekly/biweekly test on medium/large workload including micro-benchmark - Run tests for newly installed software package typically requested by user. If you are interested trying out buildtest check out :ref:`getting_started`. Related Projects and community efforts --------------------------------------- +--------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | Project | Description | State | +======================================================================================+=====================================================================================================================================================================================================================================================================================================================+==========+ | `ReFrame `_ | is a high level regression framework for writing regression test for HPC systems. Tests are written in Python class andit has support for cray programming environment, job scheduler, module integration, parameter tests, test dependency,and sanity check. The project is led by `CSCS `_. | Active | +--------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | `Pavilion2 `_ | is a framework for running and analyzing tests targeting HPC systems. Tests are written in YAML and majority of pavilion commands are implemented through python plugins using yapsy. Pavilion2 is developed by `LANL `_. | Active | +--------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | `Automatic Testing of Installed Software (ATIS) `_ | This project was presented by Xavier Besseron in `FOSDEM14 `_ that targets MPI testing using ctest and cdash. This project is no longer in development. | Obsolete | +--------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | `hpcswtest `_ | is a HPC Software Stack Testing Framework developed by `Idaho National Lab `_. The framework is built using C++11 and JSON file to define test configuration. | Obsolete | +--------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | `PVCS `_ | is a validation engine to run large tests for HPC systems, the framework is written in Perl and recipe known as **Test Expression (TE)** are written in YAML. This project is developed by `CEA `_. | Obsolete | +--------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ The `System Test Working Group `_ hosted a BOF `HPC System Testing: Procedures, Acceptance, Regression Testing, and Automation `_ in SuperComputing '19. This working group is aimed at discussing acceptance and regression testing procedure and lessons learned from other HPC centers.