Build and Test Process
The buildtest build command is responsible for building and running tests on your system given one or more buildspecs.
Every buildspec goes through a pipeline that includes the following stages:
The buildspec must go through each stage of the pipeline, if it fails in one of the stage, the buildspec will be ignored.
buildtest will discover buildspecs based on command line arguments since you can build by file, directory, executor, or tags.
The buildspec search resolution is described as follows:
If file or directory specified by
--buildspecoption doesn’t exist we exit immediately.
If buildspec path is a directory, traverse directory recursively to find all
If buildspec path is a file, check if file extension is not
.yml, exit immediately
If user specifies
--executorwe search in buildspec cache to discover buildspecs.
Shown below is a diagram on how buildtest discovers buildspecs. The user can build buildspecs
--buildspec, –tags, or –executor
which will discover the buildspecs. You can exclude buildspecs
--exclude option which is processed after discovering buildspecs. The
excluded buildspecs are removed from list if found and final list of buildspecs
For every discovered buildspec, buildtest will validate the content in the parse stage to ensure buildspec complies to the JSON schema.
A buildspec is composed of one or more test sections specified via
field. Each test is validated by a sub-schema specified by
type field. In this diagram below,
we have a test hello_world that uses
type: script which informs buildtest to validate
this test with schema
buildtest will ignore any buildspecs that fail validation process for instance you may have an invalid buildspec. buildtest will send valid buildspecs to the build stage which is responsible for building a shell-script.
In this stage, buildtest is responsible for building a shell-script by parsing the content of buildspec and writing test to disk.
Buildtest will create a Builder object that is an instance of BuilderBase class
which embodies the test that will be run. Every Builder object has a unique hash that is used to identify the test. The hash is used to create
unique directory on disk where buildtest will write contents of test and corresponding files. Assuming a test was run successfully, you can
buildtest path to retrieve directory path. In example below we show root directory of a test and content of the directory.
For each test (always_fail), buildtest will write .out and .err files which correspond to output and error file. The generated script is
a wrapper script
always_fail_build.sh is the script ran by buildtest which will invoke the generated script.
buildtest path always_fail/b758eb5a-854e-4ef4-909c-b4376d970117 /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.sh/explicit_state/always_fail/b758eb5a ls -l $(buildtest path always_fail/b758eb5a-854e-4ef4-909c-b4376d970117) total 24 -rw-r--r-- 1 siddiq90 92503 0 Jan 23 13:29 always_fail.err -rw-r--r-- 1 siddiq90 92503 0 Jan 23 13:29 always_fail.out -rwxr-xr-x 1 siddiq90 92503 42 Jan 23 13:29 always_fail.sh -rwxr-xr-x 1 siddiq90 92503 771 Jan 23 13:29 always_fail_build.sh -rw-r--r-- 1 siddiq90 92503 3220 Jan 23 13:29 build-env.txt drwxr-xr-x 15 siddiq90 92503 480 Jan 23 13:29 stage
In the event of failure, buildtest will raise an exception and buildspec will be ignored.
In this stage, we run the test based on executors defined in configuration file. buildtest will
select the executor defined by
executor property in buildspec which is responsible for running the test. There is a
BaseExecutor that is a base-class for
all executors. buildtest will run tests in parallel and wait for completion.
buildtest can submit jobs to batch scheduler depending on content of test, in this event buildtest will
dispatch job, retrieve jobID and poll job until completion.
Once test is complete, buildtest will write output and error to disk.
Once test is complete, buildtest will run a series of sanity check to determine state of test which can be
The default behavior is returncode, if its 0 test is
PASS otherwise its a
FAIL. buildtest supports several status checks
in addition to performance checks.
Buildtest will write all tests and corresponding metadata for each test to report file. The report file is a JSON file that is
updated upon every buildtest build command which allows buildtest to keep track of previous builds. The report file can be queried
buildtest report or
buildtest will write report to default report file unless
buildtest build -r /path/to/report.json is specified.