Quick Start Guide
Building Test
The buildtest build
command is used for running a test. There are several ways to run the test,
the most common way to run arbitrary test is to specify file path:
buildtest build -b <file>
To specify multiple files you can append -b
option:
buildtest build -b <file1> -b <file2>
To run all tests within a directory, specify path to directory, let’s assume we want to run all tests in examples directory you can do the following:
buildtest build -b examples
buildtest can exclude buildspecs via -x
option which can operate on file or directory similar to -b
. This can be useful
if you want to run tests in a directory but exclude one file. For example you can do the following to exclude test examples/hello.yml
:
buildtest build -b examples -x examples/hello.yml
To build tests by tagname you can use the -t
option where buildtest will discover all tests that belong to the corresponding tag.
For instance to run all tests for tagname network you can do the following:
buildtest build -t network
Inspecting a test
Upon execution of test you can inspect test results for a given test, where name is the name of test:
buildtest inspect query <name>
To see content of output and error file for a given test, you can run:
buildtest inspect query -o -e <name>
You can inspect multiple tests simultaneously since test names are positional arguments, let’s assume we have two test names jobA and jobB, we can do the following:
buildtest inspect query jobA jobB
Every test has a unique identifier, if you want to select a specific test run you specify the test identifier such as following:
buildtest inspect query always_fail/c0e886c6-8a1f-4063-846f-d16bc4733b04
Note buildtest has tab completion so test IDs will be auto-completed as you type.
You can use regular expression with buildtest inspect query
command which can be useful to select multiple
tests. You must specify test names in quotes in order for buildtest to process regular expression:
buildtest inspect query "always_fail/(c0|8e)"
To see a list of available tests you can run:
buildtest inspect query list
You can retrieve path attributes for a given test via buildtest path
command which expects a test name. To
retrieve path for a test you can run:
buildtest path hello_world
Note you can only specify one test name. You can retrieve path to output or error file by running:
buildtest path -o hello_world
buildtest path -e hello_world
Querying Test Report
buildtest will keep track of all tests in a report file, to see all tests you can run:
buildtest report
To see all pass or failed tests you can run:
buildtest report --pass
buildtest report --fail
If you want to filter tests by returncode you can use --filter
field which expects a key=value pair. You can see
all available filter fields via buildtest report --helpfilter
. To filter by returncode 1 you can do:
buildtest report --filter returncode=1
If you want to see a summary of the report file you can run:
buildtest report summary
Developing a test
When you are creating a test known as buildspec you will want to make sure buildspec is valid, this can
be done by running buildtest buildspec validate. This command works similar to buildtest build
where one
can specify file or directory to search for buildspecs. To validate multiple buildspecs you can do the following:
buildtest buildspec validate -b <file1> -b <file2>
buildtest buildspec validate -b <dir1> -b <file1>
buildtest can load all buildspecs in the cache so you can query cache to find all buildspecs, query all tags, search for test maintainers and much more. To see all available buildspecs in the cache you can run:
buildtest buildspec find
If you want to see a summary of the buildspec cache you can run:
buildtest buildspec summary
To see all invalid buildspecs in your cache you can run:
buildtest buildspec find invalid
The buildtest buildspec edit-test
and buildtest buildspec edit-file
are builtin commands to open
buildspec in editor similar to how one would open in your preferred editor. The one benefit of using these commands,
is buildtest will validate the test after closing file. The difference between the two commands is one operates on test
names while the later operates on filepaths. Shown below is example usage for both commands:
buildtest buildspec edit-test hello_world
buildtest buildspec edit-file examples/foo.yml
To view content of buildspec, you can use buildtest buildspec show
where argument is name of test. You can
specify multiple test name to see content of all tests such as shown below:
buildtest buildspec show hello_world foo_bar
The buildtest buildspec show-fail
command will show content of all buildspecs that failed tests during execution which
are all tests reported by buildtest report --fail
. Note this is not to be confused with all invalid buildspecs.
Buildtest Configuration
The buildtest configuration file is used to How to configure buildtest for your system. In order to use buildtest, your configuration file must be valid. You can validate the configuration file by running:
buildtest config validate
To view content of your configuration file you can run:
buildtest config view
You can retrieve path to configuration file by running:
buildtest config path
When building tests, you can specify configuration file to use via:
buildtest -c <config> build -b <file>
To see a list of available executors defined in your configuration file you can run:
buildtest config executors list