Module Operation

Module Options (buildtest module --help)

buildtest module --help
usage: buildtest [options] [COMMANDS] module [-h] [--diff-trees DIFF_TREES] [-eb] [--spack]
                                             {loadtest,tree,collection} ...

optional arguments:
  -h, --help            show this help message and exit
  --diff-trees DIFF_TREES
                        Show difference between two module trees
  -eb, --easybuild      reports modules that are built by easybuild
  --spack               reports modules that are built by spack

subcommands:
  valid subcommands

  {loadtest,tree,collection}
    loadtest            module load test
    tree                module tree operation
    collection          module collection operation

Difference Between Module Trees (buildtest module --diff-trees)

buildtest can report differences between two module trees that can be useful if you deploy your software in a stage/prod module tree and you want to keep these trees in sync.

If your HPC site builds software stack for each architecture and your environment is heterogeneous then --diff-trees option will be helpful.

buildtest takes two trees as argument in the form of buildtest --diff-tree tree1,tree2 where trees are separated by a comma. The tree must point to the root of the module tree in your system and buildtest will walk through the entire tree. We expect this operation to be quick given that the module tree is on the order of few thousand module files which is a reasonable count of module files in a large HPC facility.

buildtest module --diff-trees /clust/app/easybuild/2018/commons/modules/all,/usr/share/lmod/lmod/modulefiles/Core
			 Comparing Module Trees for differences in module files
			 -------------------------------------------------------
Module Tree 1:  /clust/app/easybuild/2018/commons/modules/all
Module Tree 2:  /usr/share/lmod/lmod/modulefiles/Core
ID       |     Module                                                   |   Module Tree 1    |   Module Tree 2
---------|--------------------------------------------------------------|--------------------|----------------------
1        | lmod/6.5.1                                                   | NOT FOUND          | FOUND
2        | CUDA/9.1.85                                                  | FOUND              | NOT FOUND
3        | CUDA/7.5.18                                                  | FOUND              | NOT FOUND
4        | EasyBuild/3.6.0                                              | FOUND              | NOT FOUND
5        | EasyBuild/3.5.3                                              | FOUND              | NOT FOUND
6        | git-lfs/2.4.0                                                | FOUND              | NOT FOUND
7        | Anaconda2/5.1.0                                              | FOUND              | NOT FOUND
8        | IGV/2.3.98-Java-1.8.0_152                                    | FOUND              | NOT FOUND
9        | Anaconda3/5.1.0                                              | FOUND              | NOT FOUND
10       | CUDA/8.0.61                                                  | FOUND              | NOT FOUND
11       | settarg/6.5.1                                                | NOT FOUND          | FOUND
12       | cuDNN/7.1-CUDA-9.1.85                                        | FOUND              | NOT FOUND
13       | Java/1.8.0_152                                               | FOUND              | NOT FOUND

If your site supports multiple architecture and you want to find difference between the stacks then you will find --diff-trees to be handy. If the stacks are same you will see the following message

buildtest module --diff-trees /clust/app/easybuild/2018/Broadwell/redhat/7.3/modules/all,/clust/app/easybuild/2018/IvyBridge/redhat/7.3/modules/all
No difference found between module tree:  /clust/app/easybuild/2018/Broadwell/redhat/7.3/modules/all  and module tree:  /clust/app/easybuild/2018/IvyBridge/redhat/7.3/modules/all

Module Load Testing (buildtest module loadtest)

buildtest provides feature to test module load functionality on all module files in a module tree. This assumes you have the module tree in MODULEPATH in order for module command to work properly.

To use this feature specify the appropriate module tree for parameter BUILDTEST_MODULEPATH in settings.yml or via environment variable. To use this feature you need to use buildtest module loadtest

To demonstrate let’s start off with an example where we test module load for a single module tree.

[siddis14@amrndhl1228 buildtest-framework]$ buildtest --show | grep BUILDTEST_MODULEPATH
BUILDTEST_MODULEPATH                              (C) = /nfs/grid/software/RHEL7/non-easybuild/modules/all

Let’s start the test

buildtest module loadtest
module load RHEL6-apps
RUN: 1/18 STATUS: PASSED - Testing module: RHEL6-apps
________________________________________________________________________________
module load cctsoft
RUN: 2/18 STATUS: PASSED - Testing module: cctsoft
________________________________________________________________________________
module load clinpharm
RUN: 3/18 STATUS: PASSED - Testing module: clinpharm
________________________________________________________________________________
module load deprecated
RUN: 4/18 STATUS: PASSED - Testing module: deprecated
________________________________________________________________________________
module load easybuild
RUN: 5/18 STATUS: PASSED - Testing module: easybuild
________________________________________________________________________________
module load eb/2017
RUN: 6/18 STATUS: PASSED - Testing module: eb/2017
________________________________________________________________________________
module load eb/2018
RUN: 7/18 STATUS: PASSED - Testing module: eb/2018
________________________________________________________________________________
module load ljmedchemsoft
RUN: 8/18 STATUS: PASSED - Testing module: ljmedchemsoft
________________________________________________________________________________
module load lmod/6.5.1
RUN: 9/18 STATUS: PASSED - Testing module: lmod/6.5.1
________________________________________________________________________________
module load medsci
RUN: 10/18 STATUS: PASSED - Testing module: medsci
________________________________________________________________________________
module load omics
RUN: 11/18 STATUS: PASSED - Testing module: omics
________________________________________________________________________________
module load pharmsci
RUN: 12/18 STATUS: PASSED - Testing module: pharmsci
________________________________________________________________________________
module load ru
RUN: 13/18 STATUS: PASSED - Testing module: ru
________________________________________________________________________________
module load settarg/6.5.1
RUN: 14/18 STATUS: PASSED - Testing module: settarg/6.5.1
________________________________________________________________________________
module load siterestricted
RUN: 15/18 STATUS: PASSED - Testing module: siterestricted
________________________________________________________________________________
module load statistics
RUN: 16/18 STATUS: PASSED - Testing module: statistics
________________________________________________________________________________
module load use.own.eb/append
RUN: 17/18 STATUS: PASSED - Testing module: use.own.eb/append
________________________________________________________________________________
module load use.own.eb/prepend
RUN: 18/18 STATUS: PASSED - Testing module: use.own.eb/prepend
________________________________________________________________________________
Writing Results to /tmp/modules-load.out
Writing Results to /tmp/modules-load.err
________________________________________________________________________________
                     Module Load Summary
Module Trees:                            ['/nfs/grid/software/moduledomains', '/etc/modulefiles', '/usr/share/modulefiles', '/usr/share/lmod/lmod/modulefiles/Core']
PASSED:                                  18
FAILED:                                  0
________________________________________________________________________________

buildtest will attempt to run module load against each module to verify modules are working properly.

You may specify additional module trees using BUILDTEST_MODULEPATH for module testing.

If you want to test all modules that were detected by spider utility, you can set BUILDTEST_SPIDER_VIEW=all in your configuration or environment variable or just run as follows:

BUILDTEST_SPIDER_VIEW=all buildtest module loadtest

This will test all modules retrieved by spider utility.

Module Trees Operation

buildtest supports adding, removing and listing module trees. Internally, buildtest is modifying BUILDTEST_MODULEPATH which is synonymous to MODULEPATH though, buildtest makes use of BUILDTEST_MODULEPATH when querying modules using spider command.

At your site, you may be interested in testing software by each stack.

By default, BUILDTEST_MODULEPATH is set to an empty list [] in configuration file $HOME/.buildtest/settings.yml. In this case, BUILDTEST_MODULEPATH will read from MODULEPATH.

Listing Module Tree (buildtest module tree -l)

To list the module trees in buildtest you can run buildtest module tree -l which shows one module tree per line

buildtest module tree -l
/nfs/grid/software/moduledomains
/etc/modulefiles
/usr/share/modulefiles
/usr/share/lmod/lmod/modulefiles/Core

For this run, BUILDTEST_MODULEPATH is not set in configuration file so it is reading from MODULEPATH

$ cat ~/.buildtest/settings.yml  | grep -i BUILDTEST_MODULEPATH
BUILDTEST_MODULEPATH: []

Adding Module Tree (buildtest module tree -a)

You can add new module tree through command line using buildtest module tree -a /path/to/tree which will update the configuration file

buildtest module tree -a /usr/share/lmod/lmod/modulefiles/Core
Adding module tree: ['/usr/share/lmod/lmod/modulefiles/Core']
Configuration File: /home/siddis14/.buildtest/settings.yml has been updated

Removing Module Tree (buildtest module tree -r)

Similarly you can remove module tree from your configuration via buildtest module tree -r /path/to/tree

buildtest module tree -r /usr/share/lmod/lmod/modulefiles/Core
Removing module tree: ['/usr/share/lmod/lmod/modulefiles/Core']
Configuration File: /home/siddis14/.buildtest/settings.yml has been updated

Report Easybuild Modules (buildtest module --easybuild)

buildtest can detect modules that are built by Easybuild. An easybuild module will contain a string in module file as follows:

Built with EasyBuild version 3.7.1

buildtest will check all module trees defined by BUILDTEST_MODULEPATH and search for string without the version number. To enable this feature use buildtest module --easybuild or short option buildtest module -eb.

Shown below is the output of easybuild retrieval.

buildtest module --easybuild


Total Easybuild Modules: 0
Total Modules Searched: 18

If you want buildtest to retrieve all records from spider to seek out all easybuild modules consider setting BUILDTEST_SPIDER_VIEW=all in configuration or environment variable. Shown below is an output when running BUILDTEST_SPIDER_VIEW=all buildtest module --easybuild

Module: /nfs/grid/software/RHEL7/easybuild/modules/all/Compiler/GCCcore/5.4.0/zlib/.1.2.8.lua is built with Easybuild
Module: /nfs/grid/software/RHEL7/easybuild/modules/all/MPI/GCC/5.4.0-2.27/OpenMPI/2.0.0/zlib/.1.2.8.lua is built with Easybuild
Module: /nfs/grid/software/RHEL7/easybuild/modules/all/Compiler/GCCcore/6.2.0/zlib/.1.2.8.lua is built with Easybuild
Module: /nfs/grid/software/RHEL7/easybuild/modules/all/MPI/intel/2017.1.132-GCC-5.4.0-2.27/impi/2017.1.132/zlib/.1.2.8.lua is built with Easybuild
Module: /nfs/grid/software/RHEL7/easybuild/modules/all/MPI/intel-CUDA/2017.1.132-GCC-5.4.0-2.27-8.0.44/impi/2017.1.132/zlib/.1.2.8.lua is built with Easybuild
Module: /clust/app/easybuild/2018/Broadwell/redhat/7.3/modules/all/zlib/1.2.11-GCCcore-6.4.0.lua is built with Easybuild


Total Easybuild Modules: 404
Total Modules Searched: 824

Report Spack Modules (buildtest module --spack)

buildtest can detect Spack modules. A spack module has a string to denote this module was created by spack with timestamp of module creation. Shown below is an example:

Module file created by spack (https://github.com/spack/spack) on 2019-04-11 11:38:31.191604

buildtest will search for string Module file created by spack in modulefile. buildtest will run this for all modules in module trees defined by BUILDTEST_MODULEPATH.

buildtest module --spack


Total Spack Modules: 0
Total Modules Searched: 18

To retrieve all records spider to find all spack modules in your system consider running BUILDTEST_SPIDER_VIEW=all buildtest module --spack.

BUILDTEST_SPIDER_VIEW=all buildtest module --spack
Module: /nfs/grid/software/RHEL7/medsci/modules/all/ffmpeg/3.2.4-n6ulc43.lua is built with Spack
Module: /nfs/grid/software/RHEL7/medsci/software/spack/dev/share/spack/lmod/linux-rhel7-x86_64/openmpi/3.1.3-bs5h3cj/Core/fftw/3.3.8-67xoq34.lua is built with Spack
Module: /nfs/grid/software/RHEL7/medsci/software/spack/dev/share/spack/lmod/linux-rhel7-x86_64/openmpi/3.1.3-bs5h3cj/Core/gromacs/2018.1-orqoszc.lua is built with Spack
Module: /nfs/grid/software/RHEL7/medsci/modules/all/gromacs/2018.1-orqoszc.lua is built with Spack
Module: /nfs/grid/software/RHEL7/medsci/modules/all/gromacs/2018.4-45ev4jn.lua is built with Spack
Module: /nfs/grid/software/RHEL7/medsci/software/spack/dev/share/spack/lmod/linux-rhel7-x86_64/openmpi/3.1.3-bs5h3cj/Core/gromacs/2018.4-unxhn3r.lua is built with Spack
Module: /nfs/grid/software/RHEL7/medsci/modules/all/openmpi/.3.1.3-bs5h3cj.lua is built with Spack
Module: /nfs/grid/software/RHEL7/medsci/software/spack/dev/share/spack/lmod/linux-rhel7-x86_64/openmpi/3.1.3-bs5h3cj/Core/plumed/2.4.2-kgdtnfu.lua is built with Spack


Total Spack Modules: 8
Total Modules Searched: 824

Parent Modules (buildtest module --module-deps)

Parent modules are modules that set MODULEPATH in the modulefile. This technique is used in Hierarchical Module Naming Scheme where modules like compilers, mpi, numlibs expose new module trees. These modules are called parent modules.

buildtest can report list of modules depended on a parent module. First, buildtest will seek out all parent module from file BUILDTEST_ROOT/vars/modules.json.

To seek out modules that depend on parent modules use the option buildtest module --module-deps or short option buildtest module -d.

Shown below are the list of parent modules that can be used with buildtest module -d upon tab completion.

(siddis14-TgVBs13r) buildtest-framework[master ?] $ buildtest module -d
cctsoft                           eb/2018                           GCCcore/.6.2.0                    impi/2017.1.132                   OpenMPI/2.0.1                     RHEL6-apps
CUDA/8.0.44                       GCC/5.4.0-2.27                    hpc/eb-2017-core                  medsci                            OpenMPI/2.0.2
deprecated                        GCC/6.2.0-2.27                    icc/.2017.1.132-GCC-5.4.0-2.27    omics                             openmpi/.3.1.3-bs5h3cj
eb/2017                           GCCcore/.5.4.0                    ifort/.2017.1.132-GCC-5.4.0-2.27  OpenMPI/2.0.0                     pharmsci

Shown below is a sample run for parent module OpenMPI/2.0.1. buildtest will report the content of the module file and list of modules that are depended upon the module.

buildtest module -d OpenMPI/2.0.1
Module File: /nfs/grid/software/RHEL7/easybuild/modules/all/Compiler/GCC/6.2.0-2.27/OpenMPI/2.0.1.lua
________________________________________________________________________________
help([[The Open MPI Project is an open source MPI-2 implementation. - Homepage: http://www.open-mpi.org/]])

whatis([[Description: The Open MPI Project is an open source MPI-2 implementation. - Homepage: http://www.open-mpi.org/]])

local root = "/nfs/grid/software/RHEL7/easybuild/software/Compiler/GCC/6.2.0-2.27/OpenMPI/2.0.1"

conflict("OpenMPI")

load("hwloc/1.11.3")
prepend_path("MODULEPATH", "/nfs/grid/software/RHEL7/easybuild/modules/all/MPI/GCC/6.2.0-2.27/OpenMPI/2.0.1")

prepend_path("CPATH", pathJoin(root, "include"))
prepend_path("LD_LIBRARY_PATH", pathJoin(root, "lib"))
prepend_path("LIBRARY_PATH", pathJoin(root, "lib"))
prepend_path("MANPATH", pathJoin(root, "share/man"))
prepend_path("PATH", pathJoin(root, "bin"))
prepend_path("PKG_CONFIG_PATH", pathJoin(root, "lib/pkgconfig"))
setenv("EBROOTOPENMPI", root)
setenv("EBVERSIONOPENMPI", "2.0.1")
setenv("EBDEVELOPENMPI", pathJoin(root, "easybuild/Compiler-GCC-6.2.0-2.27-OpenMPI-2.0.1-easybuild-devel"))


family("MPI") 
local lsf_libdir = os.getenv("LSF_LIBDIR") 
prepend_path("LD_LIBRARY_PATH",lsf_libdir)

-- Built with EasyBuild version 3.1.2

________________________________________________________________________________
Modules that depend on OpenMPI/2.0.1
/nfs/grid/software/RHEL7/easybuild/modules/all/MPI/GCC/6.2.0-2.27/OpenMPI/2.0.1/ScaLAPACK/2.0.2-OpenBLAS-0.2.19-LAPACK-3.6.0.lua
/nfs/grid/software/RHEL7/easybuild/modules/all/MPI/GCC/6.2.0-2.27/OpenMPI/2.0.1/GLPK/4.60.lua
/nfs/grid/software/RHEL7/easybuild/modules/all/MPI/GCC/6.2.0-2.27/OpenMPI/2.0.1/OpenBLAS/0.2.19-LAPACK-3.6.0.lua
/nfs/grid/software/RHEL7/easybuild/modules/all/MPI/GCC/6.2.0-2.27/OpenMPI/2.0.1/FFTW/3.3.5.lua
/nfs/grid/software/RHEL7/easybuild/modules/all/MPI/GCC/6.2.0-2.27/OpenMPI/2.0.1/Boost/1.60.0.lua


Total Modules Found: 5