buildtest.system

This module detects System changes defined in class BuildTestSystem.

Module Contents

Classes

BuildTestSystem	BuildTestSystem is a class that detects system configuration
Scheduler	This is a base Scheduler class used for implementing common methods for
Slurm	The Slurm class implements common functions to query Slurm cluster
LSF	The LSF class checks for LSF binaries and returns a list of LSF queues
Cobalt	The Cobalt class checks for Cobalt binaries and gets a list of Cobalt queues
PBS	The PBS class checks for PBS binaries and gets a list of available queues

class buildtest.system.BuildTestSystem

BuildTestSystem is a class that detects system configuration

Constructor method for BuildTestSystem(). Defines all system configuration using class variable system which is a dictionary.

system

supported_platforms = ['Linux', 'Darwin']

get()

check()

Based on the module “distro” get system details like linux distro, processor, hostname, etc…

check_scheduler()

Check existence of batch scheduler and if so determine which scheduler it is. Currently we support Slurm, LSF, and Cobalt we invoke each class and see if its valid state. The checks determine if scheduler binaries exist in $PATH.

detect_module_tool()

Check if module tool exists, we check for Lmod or environment-modules by checking if environment variable LMOD_VERSION, MODULE_VERSION or MODULES_CMD exist. We check this with input specification in buildtest configuration. If user specifies lmod as the module tool but detected environment-modules, buildtest should pick this up and report this as part of configuration check

class buildtest.system.Scheduler

This is a base Scheduler class used for implementing common methods for detecting Scheduler details. The subclass implement specific queries that are scheduler specific. The Slurm, LSF, PBS and Cobalt class inherit from Base Class Scheduler.

logger

check_binaries(binaries)

Check if binaries exist binary exist in $PATH

active()

Returns True if buildtest is able to retrieve queues from Scheduler otherwises returns False

class buildtest.system.Slurm

Bases: Scheduler

The Slurm class implements common functions to query Slurm cluster including partitions, qos, cluster. We check existence of slurm binaries in $PATH and return if slurm cluster is in valid state.

binaries = ['sbatch', 'sacct', 'sacctmgr', 'sinfo', 'scancel']

active()

Slurm scheduler is active if we are able to retrieve partitions or qos from scheduler. This method will return a boolean type where True indicates that slurm executors can be validated.

_get_partitions()

Get list of all partitions slurm partitions using sinfo -a -h -O partitionname. The output is a list of queue names

$ sinfo -a -h -O partitionname
system
system_shared
debug_hsw
debug_knl
jupyter

_get_clusters()

Get list of slurm clusters by running sacctmgr list cluster -P -n format=Cluster. The output is a list of slurm clusters something as follows

$ sacctmgr list cluster -P -n format=Cluster
cori
escori

_get_qos()

Retrieve a list of all slurm qos by running sacctmgr list qos -P -n  format=Name. The output is a list of qos. Shown below is an example output

$ sacctmgr list qos -P -n  format=Name
normal
premium
low
serialize
scavenger

class buildtest.system.LSF

Bases: Scheduler

The LSF class checks for LSF binaries and returns a list of LSF queues

binaries = ['bsub', 'bqueues', 'bkill', 'bjobs']

_get_queues()

Return json dictionary of available LSF Queues and their queue states. The command we run is the following: bqueues -o 'queue_name status' -json which returns a JSON record of all queue details.

$ bqueues -o 'queue_name status' -json
    {
      "COMMAND":"bqueues",
      "QUEUES":2,
      "RECORDS":[
        {
          "QUEUE_NAME":"batch",
          "STATUS":"Open:Active"
        },
        {
          "QUEUE_NAME":"test",
          "STATUS":"Open:Active"
        }
      ]
    }

class buildtest.system.Cobalt

Bases: Scheduler

The Cobalt class checks for Cobalt binaries and gets a list of Cobalt queues

binaries = ['qsub', 'qstat', 'qdel', 'nodelist', 'showres', 'partlist']

_get_queues()

Get all Cobalt queues by running qstat -Ql and parsing output

class buildtest.system.PBS

Bases: Scheduler

The PBS class checks for PBS binaries and gets a list of available queues

binaries = ['qsub', 'qstat', 'qdel', 'qstart', 'qhold', 'qmgr']

check(binaries)

_get_queues()

Get queue configuration using qstat -Q -f -F json and retrieve a list of queues.