Buildspecs Interface
Now that we learned how to build tests, in this section we will discuss how one can query a buildspec cache. In buildtest, one can load all buildspecs which is equivalent to validating all buildspecs with the appropriate schema. Buildtest will ignore all invalid buildspecs and store them in a separate file.
Note
buildtest bc is an alias for buildtest buildspec command.
Finding Buildspecs - buildtest buildspec find
The buildtest buildspec find command is used for finding buildspecs from buildspec
cache. This command is also used for generating the buildspec cache. Shown below is a list of options for
provided for this command.
buildtest buildspec find --help
$ buildtest buildspec find --help
usage: buildtest [options] [COMMANDS] buildspec find [-h] [-c COUNT] [-n]
[--pager] [--row-count]
[--terse] [-b] [-e]
[--group-by-tags]
[--group-by-executor]
[-p] [-t]
[--filter FILTER]
[--format FORMAT]
[--filterfields]
[--formatfields]
[--helpfilter]
[--helpformat] [-q] [-r]
[-d DIRECTORY] [-f FILE]
...
positional arguments:
invalid Show invalid buildspecs
optional arguments:
-h, --help show this help message and exit
-c COUNT, --count COUNT
Retrieve limited number of rows that get printed
-n, --no-header Do not print header columns in terse output (--terse)
--pager Enable PAGING when viewing result
--row-count Display number of rows from query shown in table
--terse Print output in machine readable format
query:
query options to retrieve from buildspec cache
-b, --buildspec Get all buildspec files from cache
-e, --executors Get all unique executors from buildspecs
--group-by-tags Group tests by tag name
--group-by-executor Group tests by executor name
-p, --paths Print all root buildspec paths
-t, --tags List all available tags
filter:
filter and format options
--filter FILTER Filter buildspec cache with filter fields in format
--filter key1=val1,key2=val2
--format FORMAT Format buildspec cache with format fields in format
--format field1,field2,...
--filterfields Print raw filter fields for --filter option for
filtering buildspec cache output
--formatfields Print raw format fields for --format option for
formatting buildspec cache output
--helpfilter Show filter fields for --filter option for filtering
buildspec cache output
--helpformat Show format fields for --format option for formatting
buildspec cache output
extra:
All extra options
-q, --quiet Don't print output of buildspec cache when rebuilding
cache
-r, --rebuild Rebuild buildspec cache and find all buildspecs again
-d DIRECTORY, --directory DIRECTORY
Specify root buildspecs (directory) path to load
buildspecs into buildspec cache.
-f FILE, --file FILE Specify buildspec file to load into buildspec cache.
The buildtest buildspec find command will discover all buildspecs by recursively searching all .yml extensions.
buildtest will validate each buildspec file with the json schema and buildtest will display all valid buildspecs in the output,
all invalid buildspecs will be stored in a file for post-processing.
buildtest buildspec find
$ buildtest buildspec find
Buildspec Cache: /tmp/tmpr88132k2/var/buildspecs/cache.json
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ name ┃ type ┃ executor ┃ tags ┃ description ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ bash_login_sheba │ script │ generic.local.bas │ tutorials │ customize │
│ ng │ │ h │ │ shebang line │
│ │ │ │ │ with bash login │
│ │ │ │ │ shell │
│ bash_nonlogin_sh │ script │ generic.local.bas │ tutorials │ customize │
│ ebang │ │ h │ │ shebang line │
│ │ │ │ │ with default │
│ │ │ │ │ bash (nonlogin) │
│ │ │ │ │ shell │
│ post_run_example │ script │ generic.local.bas │ │ post run example │
│ │ │ h │ │ that will remove │
│ │ │ │ │ symbolic link │
│ create_burst_buf │ script │ generic.local.(sh │ jobs │ Create a burst │
│ fer_executors │ │ |bash) │ │ buffer for │
│ │ │ │ │ multiple │
│ │ │ │ │ executors │
│ linux_strict_tes │ script │ generic.local.bas │ │ This example │
│ t │ │ h │ │ test will show │
│ │ │ │ │ how returncode │
│ │ │ │ │ will change when │
│ │ │ │ │ using --strict │
│ │ │ │ │ flag │
│ _bin_sh_shell │ script │ generic.local.sh │ tutorials │ /bin/sh shell │
│ │ │ │ │ example │
│ _bin_bash_shell │ script │ generic.local.bas │ tutorials │ /bin/bash shell │
│ │ │ h │ │ example │
│ bash_shell │ script │ generic.local.bas │ tutorials │ bash shell │
│ │ │ h │ │ example │
│ sh_shell │ script │ generic.local.sh │ tutorials │ sh shell example │
│ shell_options │ script │ generic.local.sh │ tutorials │ shell options │
│ skip_all_tests │ script │ generic.local.bas │ tutorials │ All test in this │
│ │ │ h │ │ buildspec are │
│ │ │ │ │ skipped │
│ this_test_is_als │ script │ generic.local.bas │ tutorials │ This test is │
│ o_skipped │ │ h │ │ also skipped │
│ │ │ │ │ even if skip is │
│ │ │ │ │ defined in test │
│ python_hello │ script │ generic.local.bas │ python │ Hello World │
│ │ │ h │ │ python │
│ foo_bar │ script │ generic.local.sh │ tutorials │ prints variable │
│ │ │ │ │ $FOO │
│ csh_shell │ script │ generic.local.csh │ tutorials │ csh shell │
│ │ │ │ │ example │
└──────────────────┴────────┴───────────────────┴───────────┴──────────────────┘
buildtest will load all discovered buildspecs in a cache file (JSON) which is created upon
running buildtest buildspec find. Any subsequent runs will read from cache and update
if any new buildspecs are added. If you make changes to buildspec you should rebuild the
buildspec cache by running:
$ buildtest buildspec find --rebuild
The --quiet option can be used to suppress output when using buildtest buildspec find this can be useful
if you want to rebuild the cache without seeing output of cache. Take for instance the following command
buildtest buildspec find --quiet --rebuild
$ buildtest buildspec find --quiet --rebuild
Clearing cache file: /tmp/tmpr88132k2/var/buildspecs/cache.json
Buildspec Paths: ['/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials', '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/general_tests']
Updating buildspec cache file: /tmp/tmpr88132k2/var/buildspecs/cache.json
If you want to limit the number of entries to display in output, you can use --count option which expects a positive number. For instance
let’s limit output to 5 entries, we can run the following
buildtest buildspec find --count=5
$ buildtest buildspec find --count=5
Buildspec Cache: /tmp/tmpr88132k2/var/buildspecs/cache.json
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ name ┃ type ┃ executor ┃ tags ┃ description ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ bash_login_sheba │ script │ generic.local.bas │ tutorials │ customize │
│ ng │ │ h │ │ shebang line │
│ │ │ │ │ with bash login │
│ │ │ │ │ shell │
│ bash_nonlogin_sh │ script │ generic.local.bas │ tutorials │ customize │
│ ebang │ │ h │ │ shebang line │
│ │ │ │ │ with default │
│ │ │ │ │ bash (nonlogin) │
│ │ │ │ │ shell │
│ post_run_example │ script │ generic.local.bas │ │ post run example │
│ │ │ h │ │ that will remove │
│ │ │ │ │ symbolic link │
│ create_burst_buf │ script │ generic.local.(sh │ jobs │ Create a burst │
│ fer_executors │ │ |bash) │ │ buffer for │
│ │ │ │ │ multiple │
│ │ │ │ │ executors │
│ linux_strict_tes │ script │ generic.local.bas │ │ This example │
│ t │ │ h │ │ test will show │
│ │ │ │ │ how returncode │
│ │ │ │ │ will change when │
│ │ │ │ │ using --strict │
│ │ │ │ │ flag │
└──────────────────┴────────┴───────────────────┴───────────┴──────────────────┘
You can use the --row-count option to retrieve total number of records from the buildtest buildspec find query. In the next example, we show retrieved number of
tests available in the buildspec cache by running the following.
buildtest buildspec find --row-count
$ buildtest buildspec find --row-count
15
You may find it useful to use the --row-count feature while filtering the buildspec cache. For instance, you can get the number of buildspecs in the cache with
the script schema type, this can be done by running buildtest bc find --filter type=script --row-count as shown below.
buildtest buildspec find --filter type=script --row-count
$ buildtest buildspec find --filter type=script --row-count
15
Finding buildspec files
If you want to find all buildspec files in cache you can run buildtest buildspec find --buildspec.
Shown below is an example output.
buildtest buildspec find --buildspec
$ buildtest buildspec find --buildspec
List of Buildspecs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Buildspecs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ tutorials/shebang.yml │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ tutorials/post_run.yml │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ tutorials/burstbuffer_datawarp_executors.yml │
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
...
Adding buildspecs to cache
The buildtest buildspec find --paths will display a list of root directories buildtest will search for
buildspecs when running buildtest buildspec find. One can define these directories in the configuration file
or pass them via command line.
buildtest buildspec find --paths
$ buildtest buildspec find --paths
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/general_tests
buildtest will search buildspecs when building cache that can be configured via
configuration or command line. If you want to load buildspecs from a directory, you can use the --directory option.
buildtest will rebuild cache when –directory option is specified. Note that to rebuild cache you typically
need to pass –rebuild option but that is not required when using –directory option because we want
buildtest to load buildspecs into cache.
The –directory option must be path to a directory, if you specify a file path then buildtest will report an error message similar to one below
buildtest buildspec find --directory $BUILDTEST_ROOT/README.rst
$ buildtest buildspec find --directory $BUILDTEST_ROOT/README.rst Path: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/README.rst must be a directory not a file Clearing cache file: /tmp/tmpr88132k2/var/buildspecs/cache.json ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ │ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │ │ bin/buildtest:38 in <module> │ │ │ │ 35 │ │ 36 if __name__ == "__main__": │ │ 37 │ │ ❱ 38 sys.exit(main()) │ │ 39 │ │ │ │ ╭───────────────────────────────── locals ─────────────────────────────────╮ │ │ │ buildtest_file = '/home/docs/checkouts/readthedocs.org/user_builds/buil… │ │ │ │ main = <function main at 0x7f262820c280> │ │ │ │ os = <module 'os' from │ │ │ │ '/home/docs/.asdf/installs/python/3.9.19/lib/python3.9… │ │ │ │ prefix = '/home/docs/checkouts/readthedocs.org/user_builds/buil… │ │ │ │ sys = <module 'sys' (built-in)> │ │ │ ╰──────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │ │ buildtest/main.py:104 in main │ │ │ │ 101 │ │ │ 102 │ # implementation for 'buildtest buildspec find' │ │ 103 │ elif args.subcommands in ["buildspec", "bc"]: │ │ ❱ 104 │ │ handle_buildspec_command(args, configuration, report_file, bui │ │ 105 │ │ │ 106 │ # running buildtest inspect │ │ 107 │ elif args.subcommands in ["inspect", "it"]: │ │ │ │ ╭───────────────────────────────── locals ─────────────────────────────────╮ │ │ │ args = Namespace(subcommands='buildspec', configfile=None, │ │ │ │ debug=False, loglevel='DEBUG', editor=None, │ │ │ │ view_log=False, logpath=False, print_log=False, │ │ │ │ color=None, no_color=False, helpcolor=False, │ │ │ │ report=None, listopts=False, verbose=False, │ │ │ │ buildspecs_subcommand='find', count=None, │ │ │ │ no_header=False, pager=False, row_count=False, │ │ │ │ terse=False, buildspec=False, executors=False, │ │ │ │ group_by_tags=False, group_by_executor=False, │ │ │ │ paths=False, tags=False, filter=None, format=None, │ │ │ │ filterfields=False, formatfields=False, │ │ │ │ helpfilter=False, helpformat=False, quiet=False, │ │ │ │ rebuild=False, │ │ │ │ directory=['$BUILDTEST_ROOT/README.rst'], file=None, │ │ │ │ buildspec_find_subcommand=None) │ │ │ │ buildtest_editor = '/usr/bin/vim' │ │ │ │ configuration = <buildtest.config.SiteConfiguration object at │ │ │ │ 0x7f26262f2c70> │ │ │ │ parser = <buildtest.cli.BuildTestParser object at │ │ │ │ 0x7f262838aa90> │ │ │ │ report_file = None │ │ │ │ system = <buildtest.system.BuildTestSystem object at │ │ │ │ 0x7f26262f2d00> │ │ │ ╰──────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │ │ buildtest/main.py:354 in handle_buildspec_command │ │ │ │ 351 │ │ 352 def handle_buildspec_command(args, configuration, report_file, buildte │ │ 353 │ if args.buildspecs_subcommand in ["find", "f"]: │ │ ❱ 354 │ │ buildspec_find(args=args, configuration=configuration) │ │ 355 │ elif args.buildspecs_subcommand in ["summary", "sm"]: │ │ 356 │ │ summarize_buildspec_cache( │ │ 357 │ │ │ pager=args.pager, configuration=configuration, color=args. │ │ │ │ ╭───────────────────────────────── locals ─────────────────────────────────╮ │ │ │ args = Namespace(subcommands='buildspec', configfile=None, │ │ │ │ debug=False, loglevel='DEBUG', editor=None, │ │ │ │ view_log=False, logpath=False, print_log=False, │ │ │ │ color=None, no_color=False, helpcolor=False, │ │ │ │ report=None, listopts=False, verbose=False, │ │ │ │ buildspecs_subcommand='find', count=None, │ │ │ │ no_header=False, pager=False, row_count=False, │ │ │ │ terse=False, buildspec=False, executors=False, │ │ │ │ group_by_tags=False, group_by_executor=False, │ │ │ │ paths=False, tags=False, filter=None, format=None, │ │ │ │ filterfields=False, formatfields=False, │ │ │ │ helpfilter=False, helpformat=False, quiet=False, │ │ │ │ rebuild=False, │ │ │ │ directory=['$BUILDTEST_ROOT/README.rst'], file=None, │ │ │ │ buildspec_find_subcommand=None) │ │ │ │ buildtest_editor = '/usr/bin/vim' │ │ │ │ configuration = <buildtest.config.SiteConfiguration object at │ │ │ │ 0x7f26262f2c70> │ │ │ │ report_file = None │ │ │ ╰──────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │ │ buildtest/cli/buildspec.py:1435 in buildspec_find │ │ │ │ 1432 │ │ configuration (buildtest.config.SiteConfiguration): instance │ │ 1433 │ """ │ │ 1434 │ │ │ ❱ 1435 │ cache = BuildspecCache( │ │ 1436 │ │ rebuild=args.rebuild, │ │ 1437 │ │ filterfields=args.filter, │ │ 1438 │ │ formatfields=args.format, │ │ │ │ ╭───────────────────────────────── locals ─────────────────────────────────╮ │ │ │ args = Namespace(subcommands='buildspec', configfile=None, │ │ │ │ debug=False, loglevel='DEBUG', editor=None, │ │ │ │ view_log=False, logpath=False, print_log=False, │ │ │ │ color=None, no_color=False, helpcolor=False, │ │ │ │ report=None, listopts=False, verbose=False, │ │ │ │ buildspecs_subcommand='find', count=None, │ │ │ │ no_header=False, pager=False, row_count=False, │ │ │ │ terse=False, buildspec=False, executors=False, │ │ │ │ group_by_tags=False, group_by_executor=False, │ │ │ │ paths=False, tags=False, filter=None, format=None, │ │ │ │ filterfields=False, formatfields=False, │ │ │ │ helpfilter=False, helpformat=False, quiet=False, │ │ │ │ rebuild=False, directory=['$BUILDTEST_ROOT/README.rst'], │ │ │ │ file=None, buildspec_find_subcommand=None) │ │ │ │ configuration = <buildtest.config.SiteConfiguration object at │ │ │ │ 0x7f26262f2c70> │ │ │ ╰──────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │ │ buildtest/cli/buildspec.py:132 in __init__ │ │ │ │ 129 │ │ self.cache = {} │ │ 130 │ │ │ │ 131 │ │ self.load_paths() │ │ ❱ 132 │ │ self.build() │ │ 133 │ │ │ │ 134 │ │ self._check_filter_fields() │ │ 135 │ │ self._check_format_fields() │ │ │ │ ╭───────────────────────────────── locals ─────────────────────────────────╮ │ │ │ buildspec_files = None │ │ │ │ color = None │ │ │ │ configuration = <buildtest.config.SiteConfiguration object at │ │ │ │ 0x7f26262f2c70> │ │ │ │ count = None │ │ │ │ directory = ['$BUILDTEST_ROOT/README.rst'] │ │ │ │ filterfields = None │ │ │ │ formatfields = None │ │ │ │ header = False │ │ │ │ pager = False │ │ │ │ rebuild = False │ │ │ │ row_count = False │ │ │ │ self = <buildtest.cli.buildspec.BuildspecCache object at │ │ │ │ 0x7f26262e7dc0> │ │ │ │ terse = False │ │ │ ╰──────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │ │ buildtest/cli/buildspec.py:218 in build │ │ │ │ 215 │ │ # all buildspecs paths and traverse directory to find all .ym │ │ 216 │ │ │ │ 217 │ │ if not is_file(BUILDSPEC_CACHE_FILE): │ │ ❱ 218 │ │ │ self.build_cache() │ │ 219 │ │ │ │ 220 │ │ self.cache = load_json(BUILDSPEC_CACHE_FILE) │ │ 221 │ │ │ │ ╭───────────────────────────────── locals ─────────────────────────────────╮ │ │ │ buildspecs = [] │ │ │ │ self = <buildtest.cli.buildspec.BuildspecCache object at │ │ │ │ 0x7f26262e7dc0> │ │ │ ╰──────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │ │ buildtest/cli/buildspec.py:350 in build_cache │ │ │ │ 347 │ │ # for path in self.paths: │ │ 348 │ │ # self.update_cache[path] = {} │ │ 349 │ │ │ │ ❱ 350 │ │ buildspecs = self._discover_buildspecs() │ │ 351 │ │ │ │ 352 │ │ self.update_cache["invalids"] = {} │ │ 353 │ │ │ │ ╭───────────────────────────────── locals ─────────────────────────────────╮ │ │ │ self = <buildtest.cli.buildspec.BuildspecCache object at 0x7f26262e7dc0> │ │ │ ╰──────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │ │ buildtest/cli/buildspec.py:233 in _discover_buildspecs │ │ │ │ 230 │ │ # recursively and add them to list │ │ 231 │ │ │ │ 232 │ │ if not self.paths: │ │ ❱ 233 │ │ │ raise BuildTestError( │ │ 234 │ │ │ │ "Unable to search any buildspecs, please specify a di │ │ 235 │ │ │ ) │ │ 236 │ │ │ │ ╭───────────────────────────────── locals ─────────────────────────────────╮ │ │ │ buildspecs = [] │ │ │ │ self = <buildtest.cli.buildspec.BuildspecCache object at │ │ │ │ 0x7f26262e7dc0> │ │ │ ╰──────────────────────────────────────────────────────────────────────────╯ │ ╰──────────────────────────────────────────────────────────────────────────────╯ BuildTestError: 'Unable to search any buildspecs, please specify a directory'
If you want to specify multiple root paths you can specify the –directory options multiple times.
Let’s rebuild the cache again by running buildtest buildspec find which will load the default buildspecs into the cache
$ buildtest buildspec find --rebuild --quiet
Buildspec Paths: ['/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials', '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/general_tests']
Updating buildspec cache file: /tmp/tmpr88132k2/var/buildspecs/cache.json
Clearing cache file: /tmp/tmpr88132k2/var/buildspecs/cache.json
Buildspec Paths: ['/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials', '/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/general_tests']
Updating buildspec cache file: /tmp/tmpr88132k2/var/buildspecs/cache.json
In addition to --directory option, one can specify a list of files to load into cache using the --file option. This can be useful
if you want to load specific buildspecs into cache without having to specify --directory. You can use --file option with --directory
and buildtest will recursively search directories and load files specified in --file option.
If you specify an invalid file path, a directory or file without .yml extension, buildtest will report a message and skip to next file.
Shown below, we specify a list of files to load into cache using --file option.
buildtest buildspec find --file $BUILDTEST_ROOT/tutorials/vars.yml
$ buildtest buildspec find --file $BUILDTEST_ROOT/tutorials/vars.yml
Buildspec Cache: /tmp/tmpr88132k2/var/buildspecs/cache.json
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ name ┃ type ┃ executor ┃ tags ┃ description ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ bash_login_sheba │ script │ generic.local.bas │ tutorials │ customize │
│ ng │ │ h │ │ shebang line │
│ │ │ │ │ with bash login │
│ │ │ │ │ shell │
│ bash_nonlogin_sh │ script │ generic.local.bas │ tutorials │ customize │
│ ebang │ │ h │ │ shebang line │
│ │ │ │ │ with default │
│ │ │ │ │ bash (nonlogin) │
│ │ │ │ │ shell │
│ post_run_example │ script │ generic.local.bas │ │ post run example │
│ │ │ h │ │ that will remove │
│ │ │ │ │ symbolic link │
│ create_burst_buf │ script │ generic.local.(sh │ jobs │ Create a burst │
│ fer_executors │ │ |bash) │ │ buffer for │
│ │ │ │ │ multiple │
│ │ │ │ │ executors │
│ linux_strict_tes │ script │ generic.local.bas │ │ This example │
│ t │ │ h │ │ test will show │
│ │ │ │ │ how returncode │
│ │ │ │ │ will change when │
│ │ │ │ │ using --strict │
│ │ │ │ │ flag │
│ _bin_sh_shell │ script │ generic.local.sh │ tutorials │ /bin/sh shell │
│ │ │ │ │ example │
│ _bin_bash_shell │ script │ generic.local.bas │ tutorials │ /bin/bash shell │
│ │ │ h │ │ example │
│ bash_shell │ script │ generic.local.bas │ tutorials │ bash shell │
│ │ │ h │ │ example │
│ sh_shell │ script │ generic.local.sh │ tutorials │ sh shell example │
│ shell_options │ script │ generic.local.sh │ tutorials │ shell options │
│ skip_all_tests │ script │ generic.local.bas │ tutorials │ All test in this │
│ │ │ h │ │ buildspec are │
│ │ │ │ │ skipped │
│ this_test_is_als │ script │ generic.local.bas │ tutorials │ This test is │
│ o_skipped │ │ h │ │ also skipped │
│ │ │ │ │ even if skip is │
│ │ │ │ │ defined in test │
│ python_hello │ script │ generic.local.bas │ python │ Hello World │
│ │ │ h │ │ python │
│ foo_bar │ script │ generic.local.sh │ tutorials │ prints variable │
│ │ │ │ │ $FOO │
│ csh_shell │ script │ generic.local.csh │ tutorials │ csh shell │
│ │ │ │ │ example │
└──────────────────┴────────┴───────────────────┴───────────┴──────────────────┘
We can confirm the file is loaded into cache using the -b option which list all buildspecs in cache and pipe via grep to search for vars.yml. Note that
we specify --count=-1 to show all buildspecs in cache.
$ buildtest buildspec find -b --terse --count=-1 | grep vars.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/vars.yml
Filtering buildspec
Once you have a buildspec cache, we can query the buildspec cache for certain attributes.
When you run buildtest buildspec find it will report all buildspecs from cache which can
be difficult to process. Therefore, we have a filter option (--filter) to restrict our search.
Let’s take a look at the available filter fields that are acceptable with filter option.
buildtest buildspec find --helpfilter
$ buildtest buildspec find --helpfilter
Filter Field Description
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Field ┃ Type ┃ Description ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ buildspecs │ Filter tests by buildspec │ FILE │
├────────────┼───────────────────────────┼─────────────┤
│ executor │ Filter by executor name │ STRING │
├────────────┼───────────────────────────┼─────────────┤
│ tags │ Filter by tag name │ STRING │
├────────────┼───────────────────────────┼─────────────┤
│ type │ Filter by schema type │ STRING │
└────────────┴───────────────────────────┴─────────────┘
The --filter option expects an arguments in key=value format as follows:
buildtest buildspec find --filter key1=value1,key2=value2,key3=value3
We can filter buildspec cache by tags=fail which will query all tests with
associated tag field in test.
buildtest buildspec find --filter tags=fail
$ buildtest buildspec find --filter tags=fail
Buildspec Cache: /tmp/tmpr88132k2/var/buildspecs/cache.json
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ name ┃ type ┃ executor ┃ tags ┃ description ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ exit1_fail │ script │ generic.local. │ tutorials fail │ exit 1 by │
│ │ │ bash │ │ default is FAIL │
│ returncode_list │ script │ generic.local. │ tutorials fail │ exit 2 failed │
│ _mismatch │ │ bash │ │ since it failed │
│ │ │ │ │ to match │
│ │ │ │ │ returncode 1 │
└─────────────────┴────────┴────────────────┴────────────────┴─────────────────┘
In addition, we can query buildspecs by schema type using type property. In this
example we query all tests by type property
buildtest buildspec find --filter type=script
$ buildtest buildspec find --filter type=script
Buildspec Cache: /tmp/tmpr88132k2/var/buildspecs/cache.json
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ name ┃ type ┃ executor ┃ tags ┃ description ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ bash_login_sheba │ script │ generic.local.bas │ tutorials │ customize │
│ ng │ │ h │ │ shebang line │
│ │ │ │ │ with bash login │
│ │ │ │ │ shell │
│ bash_nonlogin_sh │ script │ generic.local.bas │ tutorials │ customize │
│ ebang │ │ h │ │ shebang line │
│ │ │ │ │ with default │
│ │ │ │ │ bash (nonlogin) │
│ │ │ │ │ shell │
│ post_run_example │ script │ generic.local.bas │ │ post run example │
│ │ │ h │ │ that will remove │
│ │ │ │ │ symbolic link │
│ create_burst_buf │ script │ generic.local.(sh │ jobs │ Create a burst │
│ fer_executors │ │ |bash) │ │ buffer for │
│ │ │ │ │ multiple │
│ │ │ │ │ executors │
│ linux_strict_tes │ script │ generic.local.bas │ │ This example │
...
Finally, we can combine multiple filter fields separated by comma, in the next example
we can query all buildspecs with tags=tutorials, executor=generic.local.sh, and type=script
buildtest buildspec find --format name,tags,executor,type --filter tags=tutorials,executor=generic.local.sh,type=script
$ buildtest buildspec find --format name,tags,executor,type --filter tags=tutorials,executor=generic.local.sh,type=script
Buildspec Cache: /tmp/tmpr88132k2/var/buildspecs/cache.json
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ name ┃ tags ┃ executor ┃ type ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ _bin_sh_shell │ tutorials │ generic.local.sh │ script │
│ sh_shell │ tutorials │ generic.local.sh │ script │
│ shell_options │ tutorials │ generic.local.sh │ script │
│ foo_bar │ tutorials │ generic.local.sh │ script │
│ timelimit_min_max │ tutorials │ generic.local.sh │ script │
│ timelimit_min │ tutorials │ generic.local.sh │ script │
│ timelimit_max │ tutorials │ generic.local.sh │ script │
│ timelimit_min_fail │ tutorials │ generic.local.sh │ script │
│ timelimit_max_fail │ tutorials │ generic.local.sh │ script │
└────────────────────┴───────────┴──────────────────┴────────┘
We can filter output of buildspec cache by buildspec using --filter buildspec=<path> which
expects a path to buildspec file. The buildspec must be in the cache and file path must exist in order to
fetch the result. The path can be absolute or relative path.
In this next example, we will filter cache by file tutorials/test_status/pass_returncode.yml and use --format name,buildspec
to format columns. The --format buildspec will show full path to buildspec and name refers to name of test.
For more details on –format see Format buildspec cache.
buildtest buildspec find --filter buildspec=tutorials/test_status/pass_returncode.yml --format name,buildspec
$ buildtest buildspec find --filter buildspec=tutorials/test_status/pass_returncode.yml --format name,buildspec
Buildspec Cache: /tmp/tmpr88132k2/var/buildspecs/cache.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name ┃ buildspec ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ exit1_fail │ /home/docs/checkouts/readthedocs.org/user_builds/ │
│ │ buildtest/checkouts/stable/tutorials/test_status/ │
│ │ pass_returncode.yml │
│ exit1_pass │ /home/docs/checkouts/readthedocs.org/user_builds/ │
│ │ buildtest/checkouts/stable/tutorials/test_status/ │
│ │ pass_returncode.yml │
│ returncode_list_mismatch │ /home/docs/checkouts/readthedocs.org/user_builds/ │
│ │ buildtest/checkouts/stable/tutorials/test_status/ │
│ │ pass_returncode.yml │
│ returncode_int_match │ /home/docs/checkouts/readthedocs.org/user_builds/ │
│ │ buildtest/checkouts/stable/tutorials/test_status/ │
│ │ pass_returncode.yml │
└──────────────────────────┴───────────────────────────────────────────────────┘
Format buildspec cache
We have seen how one can filter buildspecs, but we can also configure which columns to display in the output of buildtest buildspec find. By default, we show a pre-selected format fields in the output, however there are more format fields available that can be configured at the command line.
The format fields are specified in comma separated format such as buildtest buildspec find --format <field1>,<field2>,....
You can see a list of all format fields by --helpformat option as shown below
buildtest buildspec find --helpformat
$ buildtest buildspec find --helpformat
Format Field Description
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Field ┃ Description ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ buildspec │ Display name of buildspec file │
├─────────────┼─────────────────────────────────────┤
│ description │ Show description of test │
├─────────────┼─────────────────────────────────────┤
│ executor │ Display 'executor' property in test │
├─────────────┼─────────────────────────────────────┤
│ name │ Display name of test │
├─────────────┼─────────────────────────────────────┤
│ tags │ Display 'tag' property in test │
├─────────────┼─────────────────────────────────────┤
│ type │ Display 'type' property in test │
└─────────────┴─────────────────────────────────────┘
In the next example, we utilize --format field to show how format fields affect table columns.
buildtest will display the table in order of format fields specified in command line.
buildtest buildspec find --format name,description,buildspec
$ buildtest buildspec find --format name,description,buildspec
Buildspec Cache: /tmp/tmpr88132k2/var/buildspecs/cache.json
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name ┃ description ┃ buildspec ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ bash_login_shebang │ customize shebang line │ /home/docs/checkouts/rea │
│ │ with bash login shell │ dthedocs.org/user_builds │
│ │ │ /buildtest/checkouts/sta │
│ │ │ ble/tutorials/shebang.ym │
│ │ │ l │
│ bash_nonlogin_shebang │ customize shebang line │ /home/docs/checkouts/rea │
│ │ with default bash │ dthedocs.org/user_builds │
│ │ (nonlogin) shell │ /buildtest/checkouts/sta │
│ │ │ ble/tutorials/shebang.ym │
│ │ │ l │
│ post_run_example │ post run example that │ /home/docs/checkouts/rea │
│ │ will remove symbolic │ dthedocs.org/user_builds │
│ │ link │ /buildtest/checkouts/sta │
│ │ │ ble/tutorials/post_run.y │
│ │ │ ml │
│ create_burst_buffer_exe │ Create a burst buffer │ /home/docs/checkouts/rea │
│ cutors │ for multiple executors │ dthedocs.org/user_builds │
│ │ │ /buildtest/checkouts/sta │
│ │ │ ble/tutorials/burstbuffe │
│ │ │ r_datawarp_executors.yml │
│ linux_strict_test │ This example test will │ /home/docs/checkouts/rea │
│ │ show how returncode │ dthedocs.org/user_builds │
│ │ will change when using │ /buildtest/checkouts/sta │
│ │ --strict flag │ ble/tutorials/strict_exa │
│ │ │ mple.yml │
│ _bin_sh_shell │ /bin/sh shell example │ /home/docs/checkouts/rea │
│ │ │ dthedocs.org/user_builds │
│ │ │ /buildtest/checkouts/sta │
│ │ │ ble/tutorials/shell_exam │
│ │ │ ples.yml │
│ _bin_bash_shell │ /bin/bash shell example │ /home/docs/checkouts/rea │
│ │ │ dthedocs.org/user_builds │
│ │ │ /buildtest/checkouts/sta │
│ │ │ ble/tutorials/shell_exam │
│ │ │ ples.yml │
│ bash_shell │ bash shell example │ /home/docs/checkouts/rea │
│ │ │ dthedocs.org/user_builds │
│ │ │ /buildtest/checkouts/sta │
│ │ │ ble/tutorials/shell_exam │
│ │ │ ples.yml │
│ sh_shell │ sh shell example │ /home/docs/checkouts/rea │
│ │ │ dthedocs.org/user_builds │
│ │ │ /buildtest/checkouts/sta │
│ │ │ ble/tutorials/shell_exam │
│ │ │ ples.yml │
│ shell_options │ shell options │ /home/docs/checkouts/rea │
│ │ │ dthedocs.org/user_builds │
│ │ │ /buildtest/checkouts/sta │
│ │ │ ble/tutorials/shell_exam │
│ │ │ ples.yml │
│ skip_all_tests │ All test in this │ /home/docs/checkouts/rea │
│ │ buildspec are skipped │ dthedocs.org/user_builds │
│ │ │ /buildtest/checkouts/sta │
│ │ │ ble/tutorials/skip_build │
│ │ │ spec.yml │
│ this_test_is_also_skipp │ This test is also │ /home/docs/checkouts/rea │
│ ed │ skipped even if skip is │ dthedocs.org/user_builds │
│ │ defined in test │ /buildtest/checkouts/sta │
│ │ │ ble/tutorials/skip_build │
│ │ │ spec.yml │
│ python_hello │ Hello World python │ /home/docs/checkouts/rea │
│ │ │ dthedocs.org/user_builds │
│ │ │ /buildtest/checkouts/sta │
│ │ │ ble/tutorials/python-hel │
│ │ │ lo.yml │
│ foo_bar │ prints variable $FOO │ /home/docs/checkouts/rea │
│ │ │ dthedocs.org/user_builds │
│ │ │ /buildtest/checkouts/sta │
│ │ │ ble/tutorials/maintainer │
│ │ │ s_example.yml │
│ csh_shell │ csh shell example │ /home/docs/checkouts/rea │
│ │ │ dthedocs.org/user_builds │
│ │ │ /buildtest/checkouts/sta │
│ │ │ ble/tutorials/csh_shell_ │
│ │ │ examples.yml │
└─────────────────────────┴─────────────────────────┴──────────────────────────┘
Querying buildspec executor
If you want to know all executors in your buildspec cache use the
buildtest buildspec find --executors command. This can be useful when
you want to build by executors (buildtest build --executor).
buildtest buildspec find --executors
$ buildtest buildspec find --executors
List of Executors
┏━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Executors ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ generic.local.csh │
│ generic.local.(sh|bash) │
│ generic.local.(bash|sh) │
│ generic.local.sh │
│ generic.local.bash │
└─────────────────────────┘
Similar to --group-by-tags, buildtest has an option to group tests by executor
using --group-by-executor option. This will show tests grouped by executor,
name of test and test description. Shown below is an example output.
buildtest buildspec find --group-by-executor
$ buildtest buildspec find --group-by-executor
Tests by Executors
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Executors ┃ Name ┃ Description ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ generic.local.bash │ bash_login_shebang │ customize shebang line with │
│ │ │ bash login shell │
│ generic.local.bash │ bash_nonlogin_shebang │ customize shebang line with │
│ │ │ default bash (nonlogin) │
│ │ │ shell │
│ generic.local.bash │ post_run_example │ post run example that will │
│ │ │ remove symbolic link │
│ generic.local.bash │ linux_strict_test │ This example test will show │
│ │ │ how returncode will change │
│ │ │ when using --strict flag │
│ generic.local.bash │ _bin_bash_shell │ /bin/bash shell example │
│ generic.local.bash │ bash_shell │ bash shell example │
│ generic.local.bash │ skip_all_tests │ All test in this buildspec │
│ │ │ are skipped │
│ generic.local.bash │ this_test_is_also_skipped │ This test is also skipped │
│ │ │ even if skip is defined in │
│ │ │ test │
│ generic.local.bash │ python_hello │ Hello World python │
│ generic.local.bash │ skip │ This test is skipped │
│ generic.local.bash │ unskipped │ This test is not skipped │
│ generic.local.bash │ sleep │ sleep 2 seconds │
│ generic.local.bash │ add_numbers │ Add X+Y │
│ generic.local.bash │ circle_area │ Calculate circle of area │
│ │ │ given a radius │
│ generic.local.bash │ bash_env_variables │ Declare environment │
│ │ │ variables in default shell │
│ │ │ (bash) │
...
Terse Output
You can use the --terse option to print output of buildtest buildspec find in terse format that can
be useful if you want to parse content of file. In example below, we will print output of tags in terse format, the
first entry tags is the header followed by list of unique tags. The --no-header option
can be used to disable printing of header title.
buildtest buildspec find -t --terse
$ buildtest buildspec find -t --terse
Tags
filesystem
benchmark
pass
containers
singularity
storage
python
system
fail
slurm
network
jobs
configuration
lsf
tutorials
You can also use --count with terse option, note that heading is not counted as an element, the –count will only limit number
of entries reported from the buildspec cache. Shown below we retrieve 5 test results in terse mode and disable heading via -n option.
buildtest buildspec find --terse -n --count=5
$ buildtest buildspec find --terse -n --count=5
bash_login_shebang|script|generic.local.bash|tutorials|customize shebang line with bash login shell
bash_nonlogin_shebang|script|generic.local.bash|tutorials|customize shebang line with default bash (nonlogin) shell
post_run_example|script|generic.local.bash||post run example that will remove symbolic link
create_burst_buffer_executors|script|generic.local.(sh|bash)|jobs|Create a burst buffer for multiple executors
linux_strict_test|script|generic.local.bash||This example test will show how returncode will change when using --strict flag
Invalid Buildspecs - buildtest buildspec find invalid
buildtest will store invalid buildspecs in the cache file which can be retrieved using buildtest buildspec find invalid. buildtest
will attempt to parse each buildspec and store error message for every buildspec. If you run without any options it will
report a list of invalid buildspecs as shown below
buildtest buildspec find invalid
$ buildtest buildspec find invalid
Invalid Buildspecs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Buildspecs ┃ Exception ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /home/docs/checkouts/readthedocs.org/ │ <class │
│ user_builds/buildtest/checkouts/stabl │ 'buildtest.exceptions.InvalidBuildsp │
│ e/tutorials/invalid_buildspec_section │ ecSchemaType'> │
│ .yml │ │
│ /home/docs/checkouts/readthedocs.org/ │ <class │
│ user_builds/buildtest/checkouts/stabl │ 'jsonschema.exceptions.ValidationErr │
│ e/tutorials/invalid_tags.yml │ or'> │
│ /home/docs/checkouts/readthedocs.org/ │ <class │
│ user_builds/buildtest/checkouts/stabl │ 'buildtest.exceptions.InvalidBuildsp │
│ e/tutorials/invalid_executor.yml │ ecExecutor'> │
│ /home/docs/checkouts/readthedocs.org/ │ <class │
│ user_builds/buildtest/checkouts/stabl │ 'buildtest.exceptions.InvalidBuildsp │
│ e/tutorials/containers/container_exec │ ecExecutor'> │
│ utor/python_container.yml │ │
│ /home/docs/checkouts/readthedocs.org/ │ <class │
│ user_builds/buildtest/checkouts/stabl │ 'buildtest.exceptions.InvalidBuildsp │
│ e/tutorials/containers/container_exec │ ecExecutor'> │
│ utor/ubuntu.yml │ │
│ /home/docs/checkouts/readthedocs.org/ │ <class │
│ user_builds/buildtest/checkouts/stabl │ 'jsonschema.exceptions.ValidationErr │
│ e/tutorials/test_status/file_linecoun │ or'> │
│ t_invalid.yml │ │
│ /home/docs/checkouts/readthedocs.org/ │ <class │
│ user_builds/buildtest/checkouts/stabl │ 'jsonschema.exceptions.ValidationErr │
│ e/tutorials/test_status/file_exists_e │ or'> │
│ xception.yml │ │
│ /home/docs/checkouts/readthedocs.org/ │ <class │
│ user_builds/buildtest/checkouts/stabl │ 'jsonschema.exceptions.ValidationErr │
│ e/tutorials/metrics/invalid_metrics.y │ or'> │
│ ml │ │
│ /home/docs/checkouts/readthedocs.org/ │ <class │
│ user_builds/buildtest/checkouts/stabl │ 'jsonschema.exceptions.ValidationErr │
│ e/tutorials/metrics/missing_required_ │ or'> │
│ in_metrics.yml │ │
│ /home/docs/checkouts/readthedocs.org/ │ <class │
│ user_builds/buildtest/checkouts/stabl │ 'jsonschema.exceptions.ValidationErr │
│ e/tutorials/metrics/invalid_metric_na │ or'> │
│ me.yml │ │
│ /home/docs/checkouts/readthedocs.org/ │ <class │
│ user_builds/buildtest/checkouts/stabl │ 'buildtest.exceptions.InvalidBuildsp │
│ e/general_tests/sched/pbs/hostname.ym │ ecExecutor'> │
│ l │ │
└───────────────────────────────────────┴──────────────────────────────────────┘
If you want to see error messages for each buildspec you can pass the -e or --error option which will display output of
each buildspec followed by error message.
buildtest buildspec find invalid --error
$ buildtest buildspec find invalid --error ─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─ { │ 'msg': '"[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/invalid_buildspec_section.yml]: Schema type must be one of the following: [\'script\', \'spack\']. "', │ 'exception': "<class 'buildtest.exceptions.InvalidBuildspecSchemaType'>" } ─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─ { │ 'msg': '<ValidationError: "[\'network\', \'network\'] is not valid under any of the given schemas">', │ 'exception': "<class 'jsonschema.exceptions.ValidationError'>" } ─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─ { │ 'msg': '"[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/invalid_executor.yml]: Unable to find executor: badexecutor in [\'generic.local.bash\', \'generic.local.sh\', \'generic.local.csh\']"', │ 'exception': "<class 'buildtest.exceptions.InvalidBuildspecExecutor'>" } ─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─ { │ 'msg': '"[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/containers/container_executor/python_container.yml]: Unable to find executor: generic.container.python in [\'generic.local.bash\', \'generic.local.sh\', \'generic.local.csh\']"', │ 'exception': "<class 'buildtest.exceptions.InvalidBuildspecExecutor'>" } ─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─ { │ 'msg': '"[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/containers/container_executor/ubuntu.yml]: Unable to find executor: generic.container.ubuntu in [\'generic.local.bash\', \'generic.local.sh\', \'generic.local.csh\']"', │ 'exception': "<class 'buildtest.exceptions.InvalidBuildspecExecutor'>" } ─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─ { │ 'msg': "<ValidationError: '-1 is less than the minimum of 0'>", │ 'exception': "<class 'jsonschema.exceptions.ValidationError'>" } ─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─ { │ 'msg': '<ValidationError: "1 is not of type \'string\'">', │ 'exception': "<class 'jsonschema.exceptions.ValidationError'>" } ─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─ { │ 'msg': '<ValidationError: "{\'type\': \'str\', \'file_regex\': {\'file\': \'file.txt\', \'exp\': \'BAR\'}, \'regex\': {\'stream\': \'stdout\', \'exp\': \'BAR\'}} is valid under each of {\'required\': [\'file_regex\']}, {\'required\': [\'regex\']}">', │ 'exception': "<class 'jsonschema.exceptions.ValidationError'>" } ─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─ { │ 'msg': '<ValidationError: "{\'type\': \'str\'} is not valid under any of the given schemas">', │ 'exception': "<class 'jsonschema.exceptions.ValidationError'>" } ─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─ { │ 'msg': '<ValidationError: "\'(foo-bar\' does not match \'^[A-Za-z0-9_.-]+$\'">', │ 'exception': "<class 'jsonschema.exceptions.ValidationError'>" } ─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─ { │ 'msg': '"[/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/general_tests/sched/pbs/hostname.yml]: Unable to find executor: generic.pbs.workq in [\'generic.local.bash\', \'generic.local.sh\', \'generic.local.csh\']"', │ 'exception': "<class 'buildtest.exceptions.InvalidBuildspecExecutor'>" }
Query Maintainers (buildtest buildspec maintainers)
buildtest keeps track of maintainers (i.e authors) for a given buildspec provided that you
specify the maintainers property. This is stored in the buildspec cache which can be used
to query some interesting details.
Shown below is the help for buildtest buildspec maintainers --help
buildtest buildspec maintainers --help
$ buildtest buildspec maintainers --help
usage: buildtest [options] [COMMANDS] buildspec maintainers
[-h] [-c COUNT] [-n] [--pager] [--row-count] [--terse] [-b] {find} ...
positional arguments:
{find}
find Find buildspecs based on maintainer name
optional arguments:
-h, --help show this help message and exit
-c COUNT, --count COUNT
Retrieve limited number of rows that get printed
-n, --no-header Do not print header columns in terse output (--terse)
--pager Enable PAGING when viewing result
--row-count Display number of rows from query shown in table
--terse Print output in machine readable format
-b, --breakdown Breakdown of buildspecs by maintainers
If you want to see a listing of all maintainers you can run the following
buildtest buildspec maintainers
$ buildtest buildspec maintainers
List of Maintainers
┏━━━━━━━━━━━━━━━━━━┓
┃ Maintainers ┃
┡━━━━━━━━━━━━━━━━━━┩
│ @johndoe │
│ @bobsmith │
│ @shahzebsiddiqui │
└──────────────────┘
If you prefer a machine readable format, then you can use --terse and --no-header.
buildtest buildspec maintainers --terse --no-header
$ buildtest buildspec maintainers --terse --no-header
@johndoe
@bobsmith
@shahzebsiddiqui
If you want to see a breakdown of all buildspecs by maintainers you can use --breakdown which will
display the following information
buildtest buildspec maintainers --breakdown
$ buildtest buildspec maintainers --breakdown
False
List of Maintainers
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Maintainers ┃ Buildspecs ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ @johndoe │ /home/docs/checkouts/readthedocs.org/user_builds/buildtes │
│ │ t/checkouts/stable/tutorials/maintainers_example.yml │
│ @bobsmith │ /home/docs/checkouts/readthedocs.org/user_builds/buildtes │
│ │ t/checkouts/stable/tutorials/maintainers_example.yml │
│ @shahzebsiddiqui │ /home/docs/checkouts/readthedocs.org/user_builds/buildtes │
│ │ t/checkouts/stable/tutorials/hello_world.yml:/home/docs/c │
│ │ heckouts/readthedocs.org/user_builds/buildtest/checkouts/ │
│ │ stable/general_tests/configuration/ulimits.yml │
└──────────────────┴───────────────────────────────────────────────────────────┘
The buildtest buildspec maintainers find command can be used to report buildspec given a maintainer
name which works similar to --breakdown but doesn’t report information for all maintainers. Shown
below, we query all buildspecs by maintainer @shahzebsiddiqui
buildtest buildspec maintainers find @shahzebsiddiqui
$ buildtest buildspec maintainers find @shahzebsiddiqui
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/hello_world.yml
/home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/general_tests/configuration/ulimits.yml
Cache Summary - buildtest buildspec summary
The buildtest buildspec summary command can be used to provide a summary of the buildspec cache. This command
can be used assuming your cache is built via buildtest buildspec find. Shown below is a summary of the cache file.
buildtest buildspec summary
$ buildtest buildspec summary
╭──────────────────────────────────────────────────────────────────────────────╮
│ │
│ Reading Buildspec Cache File: /tmp/tmpr88132k2/var/buildspecs/cache.jso │
│ Total Valid Buildspecs: 86 │
│ Total Invalid Buildspecs: 11 │
│ Total Unique Tags: 16 │
│ Total Maintainers: 3 │
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
Tag Breakdown Executor Breakdown Maintainers Breakdown
┏━━━━━━━━━━━━━━━┳━━━━━━━┓ ┏━━━━━━━━━━━━━━━━━┳━━━━━━━┓┏━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃ Tag ┃ Total ┃ ┃ Executor ┃ Total ┃┃ Maintainers ┃ Total ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━┩ ┡━━━━━━━━━━━━━━━━━╇━━━━━━━┩┡━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│ tutorials │ 43 │ │ generic.local.b │ 141 ││ @johndoe │ 1 │
│ jobs │ 1 │ │ ash │ ││ @bobsmith │ 1 │
│ python │ 2 │ │ generic.local.( │ 1 ││ @shahzebsiddiqu │ 2 │
│ network │ 2 │ │ sh|bash) │ ││ i │ │
│ ping │ 1 │ │ generic.local.s │ 13 │└─────────────────┴───────┘
│ benchmark │ 1 │ │ h │ │
│ system │ 11 │ │ generic.local.c │ 3 │
│ fail │ 2 │ │ sh │ │
│ pass │ 2 │ │ generic.local.( │ 4 │
│ containers │ 8 │ │ bash|sh) │ │
│ singularity │ 8 │ └─────────────────┴───────┘
│ filesystem │ 1 │
Invalid Buildspecs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Buildspecs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ tutorials/invalid_buildspec_section.yml │
├──────────────────────────────────────────────────────────────────────────────┤
│ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/ │
│ tutorials/invalid_tags.yml │
Validate Buildspecs - buildtest buildspec validate
buildtest can validate buildspecs through the buildtest buildspec validate command which provides
analogous options for buildtest build for selecting buildspecs such as -b, -e, -n, -t and -x.
This command can be used to validate buildspecs with the JSON Schema which can be useful if you are writing a buildspec
and want to validate the buildspec without running the test.
Shown below are the available command options.
buildtest buildspec validate --help
$ buildtest buildspec validate --help
usage: buildtest [options] [COMMANDS] buildspec validate [-h] [-b BUILDSPEC]
[-x EXCLUDE]
[-e EXECUTOR]
[-t TAG] [-n NAME]
optional arguments:
-h, --help show this help message and exit
-b BUILDSPEC, --buildspec BUILDSPEC
Specify path to buildspec (file, or directory) to
validate
-x EXCLUDE, --exclude EXCLUDE
Specify path to buildspec to exclude (file or
directory) during validation
-e EXECUTOR, --executor EXECUTOR
Specify buildspecs by executor name to validate
-t TAG, --tag TAG Specify buildspecs by tag name to validate
-n NAME, --name NAME Specify buildspecs by name to validate
The -b option can be used to specify path to buildspec file or directory to validate buildspecs. If its a directory, buildtest will traverse all directories recursively and find any .yml file extensions and attempt to validate each buildspec. Shown below is an example output of what it may look like
buildtest buildspec validate -b tutorials/vars.yml
$ buildtest buildspec validate -b tutorials/vars.yml
buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/vars.yml is valid
All buildspecs passed validation!!!
If buildtest detects an error during validation, the error message will be displayed to screen with a non-zero returncode.
buildtest buildspec validate -b tutorials/invalid_tags.yml
$ buildtest buildspec validate -b tutorials/invalid_tags.yml
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─
['network', 'network'] is not valid under any of the given schemas
Failed validating 'oneOf' in schema['properties']['tags']:
{'oneOf': [{'type': 'string'},
{'$ref': '#/definitions/list_of_strings'}]}
On instance['tags']:
['network', 'network']
1 buildspecs failed to validate
Similarly we can search buildspecs based on tags if you want to validate a group of buildspecs using the -t option. We can
append -t option multiple times to search by multiple tag names. In this next example, we
will validate all buildspecs for python and pass tags.
buildtest buildspec validate -t python -t pass
$ buildtest buildspec validate -t python -t pass
buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/python-hello.yml is valid
buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/test_status/pass_returncode.yml is valid
buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/python-shell.yml is valid
All buildspecs passed validation!!!
You can mix and match different options for searching buildspecs to validate. For example, we can search by buildspec, tags, and name in the following example
buildtest buildspec validate -t python -n hello_world -b tutorials/vars.yml
$ buildtest buildspec validate -t python -n hello_world -b tutorials/vars.yml
buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/python-shell.yml is valid
buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/python-hello.yml is valid
buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/hello_world.yml is valid
buildspec: /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable/tutorials/vars.yml is valid
All buildspecs passed validation!!!
Show buildspec buildtest buildspec show
buildtest can display content of buildspec file given a test name via buildtest buildspec show command which expects a
positional argument that is the name of test. This can be quick way to see content of buildspec without remembering the full path
to the buildspec.
In this next example, we will instruct buildtest to show content of buildspec for test name python_hello.
buildtest buildspec show python_hello
$ buildtest buildspec show python_hello
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─
buildspecs:
python_hello:
type: script
description: Hello World python
executor: generic.local.bash
tags: python
run: python hello.py
────────────────────────────────────────────────────────────────────────────────
You can pass multiple arguments to buildtest buildspec show to show content of each test
buildtest buildspec show python_hello circle_area
$ buildtest buildspec show python_hello circle_area
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─
buildspecs:
python_hello:
type: script
description: Hello World python
executor: generic.local.bash
tags: python
run: python hello.py
────────────────────────────────────────────────────────────────────────────────
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─
buildspecs:
circle_area:
executor: generic.local.bash
type: script
shell: python
description: "Calculate circle of area given a radius"
tags: [tutorials, python]
run: |
import math
radius = 2
area = math.pi * radius * radius
print("Circle Radius ", radius)
print("Area of circle ", area)
────────────────────────────────────────────────────────────────────────────────
There is bash completion for this command which will show list of test names available in the cache assuming you have run
buildtest buildspec find. If you specify an invalid test name you will get an error followed by list of tests that are available
in the cache
buildtest buildspec show python_hello
$ buildtest buildspec show XYZ123! Invalid test name: XYZ123!
You can use --theme option to define the color scheme used for printing content of buildspecs. The available comlor schemes can be found at
https://pygments.org/docs/styles/#getting-a-list-of-available-styles. buildtest supports tab completion on the available themes which you can see below
$ buildtest bc show --theme
abap autumn default friendly_grayscale igor manni native pastie sas stata-dark vim
algol borland dracula fruity inkpot material one-dark perldoc solarized-dark stata-light vs
algol_nu bw emacs gruvbox-dark lilypond monokai paraiso-dark rainbow_dash solarized-light tango xcode
arduino colorful friendly gruvbox-light lovelace murphy paraiso-light rrt stata trac zenburn
Show fail buildspec buildtest buildspec show-fail
buildtest can display content of buildspec file of all failed tests via buildtest buildspec show-fail command.
This can be quick way to see content of buildspec file given a failed test name such as buildtest buildspec show-fail exit1_fail.
buildtest buildspec show-fail exit1_fail
$ buildtest buildspec show-fail exit1_fail
─ /home/docs/checkouts/readthedocs.org/user_builds/buildtest/checkouts/stable… ─
buildspecs:
exit1_fail:
executor: generic.local.bash
type: script
description: exit 1 by default is FAIL
tags: [tutorials, fail]
run: exit 1
exit1_pass:
executor: generic.local.bash
type: script
description: report exit 1 as PASS
run: exit 1
tags: [tutorials, pass]
status:
returncode: [1]
returncode_list_mismatch:
executor: generic.local.bash
type: script
description: exit 2 failed since it failed to match returncode 1
run: exit 2
tags: [tutorials, fail]
status:
returncode: [1, 3]
returncode_int_match:
executor: generic.local.bash
type: script
description: exit 128 matches returncode 128
run: exit 128
tags: [tutorials, pass]
status:
returncode: 128
────────────────────────────────────────────────────────────────────────────────
If you run buildtest buildspec show-fail without any argument, then buildtest will show content of all failed tests with
corresponding buildspec. buildtest will automatically filter out duplicate buildspec entries where multiple test correspond to
same buildspec to avoid printing content of buildspec multiple times.
Editing buildspecs in your preferred editor
buildtest provides an interface to automatically open your buildspecs in editor and validate them after closing file.
You are welcome to open your buildspec in your editor (vim, emacs, nano) but you won’t be able to validate the buildspec
unless you explicitly run the test or use buildtest buildspec validate to see if your buildspec is valid. buildtest comes
with two commands to edit your buildspecs buildtest buildspec edit-test and buildtest buildspec edit-file which we will
discuss below.
Editing by Test buildtest buildspec edit-test
The buildtest buildspec edit-test allows one to specify a list of test as positional
arguments to edit-test in your preferred editor. buildtest will provide tab completion for this
command to show all test available in cache which works similar to buildtest buildspec show command.
For instance, we can see the following test are available as part of command completion
$ buildtest buildspec edit-test
_bin_bash_shell download_stream nodes_state_down show_host_groups string_tag
_bin_sh_shell executor_regex_script_schema nodes_state_idle show_jobs systemd_default_target
add_numbers executors_sbatch_declaration nodes_state_reboot show_lsf_configuration tcsh_env_declaration
always_fail executors_vars_env_declaration pullImage_dockerhub show_lsf_models test_fail_returncode_match
always_pass exit1_fail pullImage_shub show_lsf_queues test_pass_returncode_mismatch
bash_env_variables exit1_pass pullImage_sylabscloud show_lsf_queues_current_user timelimit_max
bash_login_shebang foo_bar python_hello show_lsf_queues_formatted timelimit_max_fail
bash_nonlogin_shebang gcc_version qdel_version show_lsf_resources timelimit_min
bash_shell get_partitions qmove_version show_lsf_user_groups timelimit_min_fail
bhosts_version hello_world qselect_version show_partition timelimit_min_max
build_remoteimages inspect_image qsub_version show_qos ulimit_cputime_unlimited
build_sandbox_image jobA returncode_int_match show_queues ulimit_filedescriptor_4096
build_sif_from_dockerimage jobB returncode_list_mismatch show_tres ulimit_filelock_unlimited
circle_area jobC root_disk_usage show_users ulimit_max_user_process_2048
cqsub_version kernel_swapusage runImage sinfo_version ulimit_stacksize_unlimited
csh_env_declaration list_of_strings_tags run_stream skip ulimit_vmsize_unlimited
csh_shell lsf_version selinux_disable sleep unskipped
current_user_queue metric_regex_example sh_shell slurm_config variables_bash
dead_nodes node_down_fail_list_reason shell_options status_regex_fail
display_hosts_format nodes_state_allocated show_accounts status_regex_pass
display_lsf_hosts nodes_state_completing show_all_jobs status_returncode_by_executors
Let’s take for instance we want to edit the following test, buildtest will search the buildspec cache and find the buildspec file, open in editor and once changes are written to disk, the next file will be processed until all files are written to disk.
$ buildtest buildspec edit-test sleep _bin_bash_shell add_numbers
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml is valid
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/shell_examples.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/shell_examples.yml is valid
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/add_numbers.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/add_numbers.yml is valid
If you specify an invalid test, then buildtest will ignore the test and report a message and skip to next test as shown below
$ buildtest buildspec edit-test invalid_test sleep
Unable to find test invalid_test in cache
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml is valid
Edit buildspecs buildtest buildspec edit-file
The buildtest buildspec edit-file command can be used to edit buildspec based on filename as pose to testname.
This command works similar to buildtest buildspec edit-test where each file is open in editor and validated upon completion.
You can use this command to create new buildspec whereas buildtest buildspec edit-test only works on existing buildspecs loaded
in cache. You can pass multiple filenames as arguments if you want to edit several files.
$ buildtest buildspec edit-file $BUILDTEST_ROOT/tutorials/sleep.yml
Writing file: /Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml
/Users/siddiq90/Documents/GitHubDesktop/buildtest/tutorials/sleep.yml is valid