Build and Test Process¶
The buildtest build command is responsible for building and running tests. Every buildspec goes through a pipeline that discovers buildspecs, validates the buildspec and builds and runs the test. The buildspec must go through each stage of the pipeline, if it fails in one of the stage, the buildspec will be ignored.
buildtest will discover buildspecs based on command line arguments since you can build by file, directory, executor, or tags. In discover stage, buildtest will detect buildspecs which is discussed in Discover Buildspecs .
For every discovered buildspecs, buildtest will validate the buildspecs in the parse stage which is performed using jsonschema.validate library. The parser will validate every buildspec with the global schema named global.schema.json and one of the sub-schemas, check parsing buildspecs section for more details.
A buildspec file may contain one or more test sections specified via
field. Each test is validated by a sub-schema specified by
buildtest will validate the buildspec with global schema first followed by sub-schema
by using the
version field to look up the schema version for sub-schema. buildtest
will look up the schema from its schema library and validate the test section
Buildspecs will be ignored if it fails validation process for instance you may have an Invalid Buildspecs. Invalid buildspecs won’t be sent to build stage since we can’t reliably build a test-script.
buildtest will send all valid buildspecs to build phase which is responsible for building
a shell-script from the buildspec file. In this stage, we create a Builder object
that is an instance of BuilderBase class that is a base
class for building a buildspec. There is a sub-class for BuilderBase class such as ScriptBuilder
and CompilerBuilder that implements
how to build a test-script based on the sub-schema selection (
During build phase, there are additional checks on buildspecs to ensure we can generate a test-script. In the event of failure, buildtest will raise an exception and buildspec will be ignored. The ignored buildspecs are not sent to run stage
In this stage, we run the test based on executors defined in configuration file. buildtest will
select the executor defined by
executor property in buildspec which is responsible for running the test. There is a BaseExecutor
that is a base-class for all executors. We have sub-class for each executor type (Local, Slurm, Cobalt, PBS, Cobalt). In this stage,
we run the test and get output, error, returncode and detect status of test (
FAIL). If test is run via scheduler,
we submit job to scheduler and poll jobID until it is finished.
Upon completion of test, we update the Builder object with the test results which is written to report file.