Global Schema¶
The global schema is validated with for all schema types and is the top-level schema when defining a Buildspec.
For more details see Global Schema Documentation.
Schema Files¶
Global Keys in buildspec¶
The global keys required for buildspec are version and buildspecs. The
version key is required to validate with sub-schema when used with type field.
The buildspecs is the start of test section. The maintainers is an optional
field that is an array which can be used to identity maintainer of test. To understand
how buildtest validates the buildspec see Parse Stage.
Shown below is an example buildspec:
version: "1.0"
buildspecs:
hello_world:
executor: local.bash
type: script
description: "hello world example"
run: echo "hello world!"
maintainers:
- "@shahzebsiddiqui"
In this example, the global schema validates the following section:
version: "1.0"
buildspecs:
hello_world:
maintainers:
- "@shahzebsiddiqui"
The field version buildspecs and maintainers are validated with global.schema.json
using jsonschema.validate
method. The sub-schema is the following section which is validated with the type schema:
hello_world:
executor: local.bash
type: script
description: "hello world example"
run: echo "hello world!"
Every sub-schema requires type field in this case, type: script directs
buildtest to validate with the script schema. All type schemas have a version,
currently buildtest supports 1.0 version for all type schemas. The
version: "1.0" is used to select the version of the type schema,
in this example we validate with the schema script-v1.0.schema.json.
Test Names¶
The buildspecs is an object that defines one or more test, the
test names take the following pattern "^[A-Za-z_][A-Za-z0-9_]*$".
In the previous example the test name is hello_world. You must have unique test
names in your buildspecs section, otherwise you will have an invalid buildspec
file.
Note
We refer to the entire YAML content as buildspec file, this is not to be confused with the buildspecs field.
You may define multiple tests in a single buildspec file, shown below is an example using both script and compiler schema:
version: "1.0"
buildspecs:
hello_f:
type: compiler
description: "Hello World Fortran Compilation"
executor: local.bash
module:
- "module purge && module load gcc"
build:
source: "src/hello.f90"
name: gnu
fflags: -Wall
environment_variables:
executor: local.bash
type: script
env:
FIRST_NAME: avocado
LAST_NAME: dinosaur
run: |
hostname
whoami
echo $USER
printf "${FIRST_NAME} ${LAST_NAME}\n"
In this example we have two tests hello_f and environment_variables. The
test hello_f is using the compiler-v1.0.schema.json
for validation because type: compiler is set in sub-schema while environment_variables test
is using script-v1.0.schema.json
for validation because type: script is set.
Schema Naming Convention¶
All schema files use the file extension .schema.json to distinguish itself
as a json schema definition from an ordinary json file. All sub-schemas
must be versioned, with the exception of global.schema.json.
If you have got this far you may proceed with Writing buildspecs
Examples¶
You can see below a list of global schema examples that can be accessible
via buildtest schema -n global.schema.json -e
$ buildtest schema -n global.schema.json -e
File: /Users/siddiq90/Documents/buildtest/buildtest/schemas/examples/global.schema.json/valid/examples.yml
Valid State: True
________________________________________________________________________________
version: "1.0"
buildspecs:
# testing all caps
ABCDEFGHIJKLMNOPQRSTUVWXYZ:
type: script
executor: local.bash
run: "hostname"
# testing all lowercase letters
abcdefghijklmnopqrstuvwxyz:
type: script
executor: local.bash
run: "hostname"
# testing both caps and lowercase and numbers
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789:
type: script
executor: local.bash
run: "hostname"
# testing '_' followed by all caps, lowercase and numbers
_ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789:
type: script
executor: local.bash
run: "hostname"
# testing '_' in middle and end of word
ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz_0123456789_:
type: script
executor: local.bash
run: "hostname"
File: /Users/siddiq90/Documents/buildtest/buildtest/schemas/examples/global.schema.json/invalid/maintainers_type_mismatch.yml
Valid State: FAIL
________________________________________________________________________________
version: "1.0"
# wrong type for maintainers key, expects a string
maintainers: 1
buildspecs:
hostname:
type: script
run: "hostname"
________________________________________ Validation Error ________________________________________
1 is not of type 'array'
Failed validating 'type' in schema['properties']['maintainers']:
{'description': 'One or more maintainers or aliases',
'items': {'type': 'string'},
'minItems': 1,
'type': 'array'}
On instance['maintainers']:
1
File: /Users/siddiq90/Documents/buildtest/buildtest/schemas/examples/global.schema.json/invalid/invalid_pattern.yml
Valid State: FAIL
________________________________________________________________________________
version: "1.0"
buildspecs:
# invalid pattern for test. Must be matching regex "^[A-Za-z_.][A-Za-z0-9_]*$" when declaring a dict
(badname:
type: script
run: "ping login 1"
________________________________________ Validation Error ________________________________________
'(badname' does not match '^[A-Za-z_][A-Za-z0-9_]*$'
Failed validating 'pattern' in schema['properties']['buildspecs']['propertyNames']:
{'pattern': '^[A-Za-z_][A-Za-z0-9_]*$'}
On instance['buildspecs']:
'(badname'
File: /Users/siddiq90/Documents/buildtest/buildtest/schemas/examples/global.schema.json/invalid/missing-version.yml
Valid State: FAIL
________________________________________________________________________________
buildspecs:
# Shell would be accepted to indicate a single line shell command (or similar)
login_node_check:
type: script
run: "ping login 1"
________________________________________ Validation Error ________________________________________
'version' is a required property
Failed validating 'required' in schema:
{'$id': 'https://buildtesters.github.io/schemas/schemas/global.schema.json',
'$schema': 'http://json-schema.org/draft-07/schema#',
'additionalProperties': False,
'definitions': {'bsub': {'description': 'This field is used for '
'specifying #BSUB options in '
'test script. buildtest will '
'insert #BSUB in front of '
'each value',
'items': {'type': 'string'},
'type': 'array'},
'env': {'description': 'One or more key value pairs '
'for an environment '
'(key=value)',
'items': {'minItems': 1,
'propertyNames': {'pattern': '^[A-Za-z_][A-Za-z0-9_]*$'},
'type': 'object'},
'minItems': 1,
'type': 'object'},
'executor': {'description': 'Select one of the '
'executor name defined in '
'your configuration file '
'(``config.yml``). Every '
'buildspec must have an '
'executor which is '
'responsible for running '
'job. ',
'type': 'string'},
'sbatch': {'description': 'This field is used for '
'specifying #SBATCH options '
'in test script. buildtest '
'will insert #SBATCH in '
'front of each value',
'items': {'type': 'string'},
'type': 'array'},
'skip': {'description': 'The ``skip`` is a boolean '
'field that can be used to '
'skip tests during builds. By '
'default buildtest will build '
'and run all tests in your '
'buildspec file, if ``skip: '
'True`` is set it will skip '
'the buildspec.',
'type': 'boolean'},
'status': {'additionalProperties': False,
'description': 'The status section '
'describes how buildtest '
'detects PASS/FAIL on test. '
'By default returncode 0 is '
'a PASS and anything else '
'is a FAIL, however '
'buildtest can support '
'other types of PASS/FAIL '
'conditions.',
'properties': {'regex': {'description': 'Perform '
'regular '
'expression '
'search '
'using '
'``re.search`` '
'python '
'module '
'on '
'stdout/stderr '
'stream '
'for '
'reporting '
'if '
'test '
'``PASS``. ',
'properties': {'exp': {'description': 'Specify '
'a '
'regular '
'expression '
'to '
'run '
'with '
'input '
'stream '
'specified '
'by '
'``stream`` '
'field. '
'buildtest '
'uses '
're.search '
'when '
'performing '
'regex',
'type': 'string'},
'stream': {'description': 'The '
'stream '
'field '
'can '
'be '
'stdout '
'or '
'stderr. '
'buildtest '
'will '
'read '
'the '
'output '
'or '
'error '
'stream '
'after '
'completion '
'of '
'test '
'and '
'check '
'if '
'regex '
'matches '
'in '
'stream',
'enum': ['stdout',
'stderr'],
'type': 'string'}},
'required': ['stream',
'exp'],
'type': 'object'},
'returncode': {'description': 'By '
'default, '
'returncode '
'0 '
'is '
'PASS, '
'if '
'you '
'want '
'to '
'emulate '
'a '
'non-zero '
'returncode '
'to '
'pass '
'then '
'specify '
'an '
'expected '
'return '
'code. '
'buildtest '
'will '
'match '
'actual '
'returncode '
'with '
'one '
'defined '
'in '
'this '
'field, '
'if '
'there '
'is '
'a '
'match '
'buildtest '
'will '
'report '
'as '
'``PASS``',
'type': 'integer'},
'slurm_job_state_codes': {'description': 'This '
'field '
'can '
'be '
'used '
'for '
'checking '
'Slurm '
'Job '
'State, '
'if '
'there '
'is '
'a '
'match '
'buildtest '
'will '
'report '
'as '
'``PASS`` ',
'enum': ['COMPLETED',
'FAILED',
'OUT_OF_MEMORY',
'TIMEOUT'],
'type': 'string'}},
'type': 'object'},
'tags': {'description': 'Classify tests using a tag '
'name, this can be used for '
'categorizing test and '
'building tests using '
'``--tags`` option',
'items': {'type': 'string'},
'minItems': 1,
'type': 'array'}},
'description': 'buildtest global schema is validated for all '
'buildspecs. The global schema defines top-level '
'structure of buildspec and defintions that are '
'inherited for sub-schemas',
'properties': {'buildspecs': {'description': 'This section is used to '
'define one or more '
'tests (buildspecs). '
'Each test must be '
'unique name',
'propertyNames': {'pattern': '^[A-Za-z_][A-Za-z0-9_]*$'},
'type': 'object'},
'maintainers': {'description': 'One or more '
'maintainers or aliases',
'items': {'type': 'string'},
'minItems': 1,
'type': 'array'},
'version': {'description': 'The semver version of the '
'schema to use (x.x).',
'type': 'string'}},
'required': ['version', 'buildspecs'],
'title': 'global schema',
'type': 'object'}
On instance:
{'buildspecs': {'login_node_check': {'run': 'ping login 1',
'type': 'script'}}}
global.schema.json¶
$ buildtest schema -n global.schema.json -j
{
"$id": "https://buildtesters.github.io/schemas/schemas/global.schema.json",
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "global schema",
"description": "buildtest global schema is validated for all buildspecs. The global schema defines top-level structure of buildspec and defintions that are inherited for sub-schemas",
"type": "object",
"required": [
"version",
"buildspecs"
],
"additionalProperties": false,
"properties": {
"version": {
"type": "string",
"description": "The semver version of the schema to use (x.x)."
},
"maintainers": {
"type": "array",
"description": "One or more maintainers or aliases",
"minItems": 1,
"items": {
"type": "string"
}
},
"buildspecs": {
"type": "object",
"description": "This section is used to define one or more tests (buildspecs). Each test must be unique name",
"propertyNames": {
"pattern": "^[A-Za-z_][A-Za-z0-9_]*$"
}
}
},
"definitions": {
"env": {
"type": "object",
"description": "One or more key value pairs for an environment (key=value)",
"minItems": 1,
"items": {
"type": "object",
"minItems": 1,
"propertyNames": {
"pattern": "^[A-Za-z_][A-Za-z0-9_]*$"
}
}
},
"tags": {
"type": "array",
"description": "Classify tests using a tag name, this can be used for categorizing test and building tests using ``--tags`` option",
"minItems": 1,
"items": {
"type": "string"
}
},
"skip": {
"type": "boolean",
"description": "The ``skip`` is a boolean field that can be used to skip tests during builds. By default buildtest will build and run all tests in your buildspec file, if ``skip: True`` is set it will skip the buildspec."
},
"executor": {
"type": "string",
"description": "Select one of the executor name defined in your configuration file (``config.yml``). Every buildspec must have an executor which is responsible for running job. "
},
"sbatch": {
"type": "array",
"description": "This field is used for specifying #SBATCH options in test script. buildtest will insert #SBATCH in front of each value",
"items": {
"type": "string"
}
},
"bsub": {
"type": "array",
"description": "This field is used for specifying #BSUB options in test script. buildtest will insert #BSUB in front of each value",
"items": {
"type": "string"
}
},
"status": {
"type": "object",
"description": "The status section describes how buildtest detects PASS/FAIL on test. By default returncode 0 is a PASS and anything else is a FAIL, however buildtest can support other types of PASS/FAIL conditions.",
"additionalProperties": false,
"properties": {
"slurm_job_state_codes": {
"type": "string",
"enum": [
"COMPLETED",
"FAILED",
"OUT_OF_MEMORY",
"TIMEOUT"
],
"description": "This field can be used for checking Slurm Job State, if there is a match buildtest will report as ``PASS`` "
},
"returncode": {
"type": "integer",
"description": "By default, returncode 0 is PASS, if you want to emulate a non-zero returncode to pass then specify an expected return code. buildtest will match actual returncode with one defined in this field, if there is a match buildtest will report as ``PASS``"
},
"regex": {
"type": "object",
"description": "Perform regular expression search using ``re.search`` python module on stdout/stderr stream for reporting if test ``PASS``. ",
"properties": {
"stream": {
"type": "string",
"enum": [
"stdout",
"stderr"
],
"description": "The stream field can be stdout or stderr. buildtest will read the output or error stream after completion of test and check if regex matches in stream"
},
"exp": {
"type": "string",
"description": "Specify a regular expression to run with input stream specified by ``stream`` field. buildtest uses re.search when performing regex"
}
},
"required": [
"stream",
"exp"
]
}
}
}
}
}