Module Collections

Lmod Module Collection

buildtest supports building test with Lmod user collections module collection comes in handy when testing software with different module sets.

You can run module -t savelist to see a list of collection. Shown below is an example:

$ module -t savelist
corrupt
corrupt2
default
intelmpi

To restore a module collection you can run:

module restore <collection-name>

To build a test with a module collection use the --collection option which is a choice field that is the name of the module collection. To demonstrate, lets build a test with a module collection intelmpi.

buildtest build -c /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_intel_fortran.yml -co intelmpi -vv
________________________________________________________________________________
compiler: intel
flags: -O3
maintainer:
- shahzeb siddiqui shahzebmsiddiqui@gmail.com
source: hello.f90
testblock: singlesource

________________________________________________________________________________
Key Check PASSED for file /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_intel_fortran.yml
Source File /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90 exists!
Programming Language Detected: fortran
Compiler Check Passed
Writing Test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml.sh
Changing permission to 755 for test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml.sh
________________________________________________________________________________
#!/bin/sh
module restore intelmpi
cd /tmp/siddis14/buildtest/tests/suite/compilers/helloworld
ifort -O3 -o hello.f90.exe /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90 
./hello.f90.exe 
 rm ./hello.f90.exe 
________________________________________________________________________________

Note the module restore command will load the modules from the collection. To use this feature, you will need to have a module collection which are typically stored in $HOME/.lmod.d/

buildtest module collection

buildtest keeps track of its own module collection which is stored in BUILDTEST_ROOT/vars/default.json. This file is automatically generated at start of buildtest.

buildtest module collection works as follows.

First you load a set of modules:

$ ml

Currently Loaded Modules:
  1) eb/2018   2) CUDA/9.1.85

Next add the modules to your collection by running:

$ buildtest module collection -a
{
    "collection": [
        [
            "eb/2018",
            "CUDA/9.1.85"
        ]
    ]
}

Updating collection file: /home/siddis14/buildtest-framework/var/default.json

Next, view the module collection by running:

$ buildtest module collection -l
0: ['eb/2018', 'CUDA/9.1.85']

buildtest will display module collection with index starting 0 followed by name of modules. Let’s add a couple more collections by repeating same step by adding different modules.

Shown below, we have 4 module collection that shows the index followed by a list of modules

$ buildtest module collection -l
0: ['eb/2018', 'CUDA/9.1.85']
1: ['eb/2018', 'GCCcore/6.4.0', 'binutils/2.28-GCCcore-6.4.0', 'GCC/6.4.0-2.28']
2: ['eb/2017', 'icc/.2017.1.132-GCC-5.4.0-2.27', 'GCCcore/.5.4.0', 'binutils/.2.27', 'ifort/.2017.1.132-GCC-5.4.0-2.27', 'impi/2017.1.132', 'imkl/2017.1.132', 'intel/2017.01']
3: ['eb/2017', 'GCCcore/.6.2.0', 'binutils/.2.27', 'GCC/6.2.0-2.27']

Let’s build a test with the 2nd module collection by using --module-collection option or short option -mc.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
(siddis14-TgVBs13r) buildtest-framework[master !?+] $ buildtest build -c  $(pwd)/toolkit/buildtest/suite/compilers/helloworld/hello_intel_fortran.yml -mc 2 -vv
________________________________________________________________________________
compiler: intel
flags: -O3
maintainer:
- shahzeb siddiqui shahzebmsiddiqui@gmail.com
source: hello.f90
testblock: singlesource

________________________________________________________________________________
Key Check PASSED for file /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_intel_fortran.yml
Source File /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90 exists!
Programming Language Detected: fortran
Compiler Check Passed
Writing Test: /tmp/siddis14/buildtest/suite/compilers/helloworld/hello_intel_fortran.yml.sh
Changing permission to 755 for test: /tmp/siddis14/buildtest/suite/compilers/helloworld/hello_intel_fortran.yml.sh
________________________________________________________________________________
#!/bin/sh
module load eb/2017 icc/.2017.1.132-GCC-5.4.0-2.27 GCCcore/.5.4.0 binutils/.2.27 ifort/.2017.1.132-GCC-5.4.0-2.27 impi/2017.1.132 imkl/2017.1.132 intel/2017.01
cd /tmp/siddis14/buildtest/suite/compilers/helloworld
ifort -O3 -o hello.f90.exe /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90
./hello.f90.exe
 rm ./hello.f90.exe
________________________________________________________________________________

The option --module-collection takes an integer argument that is a choice field prepopulated by finding the total index in the collection key in file default.json

If you pass an invalid index, buildtest will report an error as you can see

Error

buildtest build: error: argument -mc/–module-collection: invalid choice: -1 (choose from 0, 1, 2, 3)

Module Permutation

buildtest can build a single test configuration with all version of a module . The spider utility by Lmod keeps track of metadata for all modules in your system as a json object. buildtest will formulate a modified json object that is written in $BUILDTEST_ROOT/var/modules.json.

Here is an example json object for intel.

"intel": {
    "/nfs/grid/software/RHEL6/general/intel/16.0.lua": {
        "fullName": "intel/16.0",
        "parent": [
            [
                "RHEL6-apps"
            ]
        ]
    },
    "/nfs/grid/software/RHEL6/general/intel/14.0.lua": {
        "fullName": "intel/14.0",
        "parent": [
            [
                "RHEL6-apps"
            ]
        ]
    },
    "/nfs/grid/software/RHEL7/easybuild/modules/all/Core/intel/2017.01.lua": {
        "fullName": "intel/2017.01",
        "parent": [
            [
                "eb/2017"
            ],
            [
                "medsci",
                "hpc/eb-2017-core"
            ]
        ]
    },

Shown below is a list of intel modules available in this system

$ module -t spider intel
intel/14.0
intel/15.0
intel/16.0
intel/16.0.1
intel/16.0.3
intel/2017.01
intel/2018a

To demonstrate an example, let’s build a test using the module permutation option --modules on all intel modules.

 buildtest build -c  /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_intel_fortran.yml --modules intel -vv
Module Permutation Detected.
Each test will be built with 7 module permutations
Module Permutation List
__________________________________________________
module load RHEL6-apps intel/16.0
module load RHEL6-apps intel/14.0
module load eb/2017 intel/2017.01
module load RHEL6-apps intel/16.0.3
module load RHEL6-apps intel/15.0
module load RHEL6-apps intel/16.0.1
module load eb/2018 intel/2018a
________________________________________________________________________________
compiler: intel
flags: -O3
maintainer:
- shahzeb siddiqui shahzebmsiddiqui@gmail.com
source: hello.f90
testblock: singlesource

________________________________________________________________________________
Key Check PASSED for file /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_intel_fortran.yml
Source File /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90 exists!
Programming Language Detected: fortran
Compiler Check Passed
Writing Test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0xbb23d23609a7a848b9ebe7ef002a9323.sh
Changing permission to 755 for test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0xbb23d23609a7a848b9ebe7ef002a9323.sh
________________________________________________________________________________
#!/bin/sh
module load RHEL6-apps intel/16.0
cd /tmp/siddis14/buildtest/tests/suite/compilers/helloworld
ifort -O3 -o hello.f90.exe /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90 
./hello.f90.exe 
 rm ./hello.f90.exe 
________________________________________________________________________________
Writing Test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0x710d2b7cfdfb9a935b735f48330554ed.sh
Changing permission to 755 for test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0x710d2b7cfdfb9a935b735f48330554ed.sh
________________________________________________________________________________
#!/bin/sh
module load RHEL6-apps intel/14.0
cd /tmp/siddis14/buildtest/tests/suite/compilers/helloworld
ifort -O3 -o hello.f90.exe /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90 
./hello.f90.exe 
 rm ./hello.f90.exe 
________________________________________________________________________________
Writing Test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0x88ce320d3c7ed9da689eb4017b9bb8f5.sh
Changing permission to 755 for test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0x88ce320d3c7ed9da689eb4017b9bb8f5.sh
________________________________________________________________________________
#!/bin/sh
module load eb/2017 intel/2017.01
cd /tmp/siddis14/buildtest/tests/suite/compilers/helloworld
ifort -O3 -o hello.f90.exe /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90 
./hello.f90.exe 
 rm ./hello.f90.exe 
________________________________________________________________________________
Writing Test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0x361f018c8eee60f878f988be7aaee8cd.sh
Changing permission to 755 for test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0x361f018c8eee60f878f988be7aaee8cd.sh
________________________________________________________________________________
#!/bin/sh
module load RHEL6-apps intel/16.0.3
cd /tmp/siddis14/buildtest/tests/suite/compilers/helloworld
ifort -O3 -o hello.f90.exe /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90 
./hello.f90.exe 
 rm ./hello.f90.exe 
________________________________________________________________________________
Writing Test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0xbe42206e12a730e3d397c6ee4dee83f8.sh
Changing permission to 755 for test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0xbe42206e12a730e3d397c6ee4dee83f8.sh
________________________________________________________________________________
#!/bin/sh
module load RHEL6-apps intel/15.0
cd /tmp/siddis14/buildtest/tests/suite/compilers/helloworld
ifort -O3 -o hello.f90.exe /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90 
./hello.f90.exe 
 rm ./hello.f90.exe 
________________________________________________________________________________
Writing Test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0xc7eb5591c162974bb90c62b08a26f7d1.sh
Changing permission to 755 for test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0xc7eb5591c162974bb90c62b08a26f7d1.sh
________________________________________________________________________________
#!/bin/sh
module load RHEL6-apps intel/16.0.1
cd /tmp/siddis14/buildtest/tests/suite/compilers/helloworld
ifort -O3 -o hello.f90.exe /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90 
./hello.f90.exe 
 rm ./hello.f90.exe 
________________________________________________________________________________
Writing Test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0x737e4ea256880ff1f5380bca5f2788ca.sh
Changing permission to 755 for test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0x737e4ea256880ff1f5380bca5f2788ca.sh
________________________________________________________________________________
#!/bin/sh
module load eb/2018 intel/2018a
cd /tmp/siddis14/buildtest/tests/suite/compilers/helloworld
ifort -O3 -o hello.f90.exe /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90 
./hello.f90.exe 
 rm ./hello.f90.exe 
________________________________________________________________________________
Writing 7 tests for /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_intel_fortran.yml

Each test will be uniquely identified with a 128 random number in the test script to avoid name conflicts.

In this example, buildtest is building the test for every intel modules found in the system.

buildtest will select the first parent combination should there be multiple parent combination to load the module. This is controlled by variable BUILDTEST_PARENT_MODULE_SEARCH that is defined in configuration file.

The default configuration for BUILDTEST_PARENT_MODULE_SEARCH is first which will select the first parent combination. The other option is all which will select all parent combination when building test.

Shown below is a snapshot of VMD record from modules.json

"VMD": {
    "/nfs/grid/software/RHEL7/easybuild/modules/all/MPI/intel/2017.1.132-GCC-5.4.0-2.27/impi/2017.1.132/VMD/1.9.3-Python-2.7.12.lua": {
        "fullName": "VMD/1.9.3-Python-2.7.12",
        "parent": [
            [
                "eb/2017",
                "icc/.2017.1.132-GCC-5.4.0-2.27",
                "impi/2017.1.132"
            ],
            [
                "eb/2017",
                "ifort/.2017.1.132-GCC-5.4.0-2.27",
                "impi/2017.1.132"
            ],
            [
                "icc/.2017.1.132-GCC-5.4.0-2.27",
                "impi/2017.1.132"
            ],
            [
                "ifort/.2017.1.132-GCC-5.4.0-2.27",
                "impi/2017.1.132"
            ],
            [
                "medsci",
                "hpc/eb-2017-core",
                "icc/.2017.1.132-GCC-5.4.0-2.27",
                "impi/2017.1.132"
            ],
            [
                "medsci",
                "hpc/eb-2017-core",
                "ifort/.2017.1.132-GCC-5.4.0-2.27",
                "impi/2017.1.132"
            ]
        ]
    }
},

The fullName and parent key define how to load a module with all the parent combinations which you are required in order to load the desired module.

To demonstrate let’s build a test with all parent combination for VMD module.

BUILDTEST_PARENT_MODULE_SEARCH=all buildtest build -c  /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_intel_fortran.yml --modules VMD -vv
Module Permutation Detected.
Each test will be built with 4 module permutations
Module Permutation List
__________________________________________________
module load eb/2017 icc/.2017.1.132-GCC-5.4.0-2.27 impi/2017.1.132 VMD/1.9.3-Python-2.7.12
module load eb/2017 ifort/.2017.1.132-GCC-5.4.0-2.27 impi/2017.1.132 VMD/1.9.3-Python-2.7.12
module load medsci hpc/eb-2017-core icc/.2017.1.132-GCC-5.4.0-2.27 impi/2017.1.132 VMD/1.9.3-Python-2.7.12
module load medsci hpc/eb-2017-core ifort/.2017.1.132-GCC-5.4.0-2.27 impi/2017.1.132 VMD/1.9.3-Python-2.7.12
________________________________________________________________________________
compiler: intel
flags: -O3
maintainer:
- shahzeb siddiqui shahzebmsiddiqui@gmail.com
source: hello.f90
testblock: singlesource

________________________________________________________________________________
Key Check PASSED for file /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_intel_fortran.yml
Source File /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90 exists!
Programming Language Detected: fortran
Compiler Check Passed
Writing Test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0xc80e042db904de9a3fa899d7e58ed78e.sh
Changing permission to 755 for test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0xc80e042db904de9a3fa899d7e58ed78e.sh
________________________________________________________________________________
#!/bin/sh
module load eb/2017 icc/.2017.1.132-GCC-5.4.0-2.27 impi/2017.1.132 VMD/1.9.3-Python-2.7.12
cd /tmp/siddis14/buildtest/tests/suite/compilers/helloworld
ifort -O3 -o hello.f90.exe /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90 
./hello.f90.exe 
 rm ./hello.f90.exe 
________________________________________________________________________________
Writing Test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0x33ee0f531d623d52cde69680dd973ee3.sh
Changing permission to 755 for test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0x33ee0f531d623d52cde69680dd973ee3.sh
________________________________________________________________________________
#!/bin/sh
module load eb/2017 ifort/.2017.1.132-GCC-5.4.0-2.27 impi/2017.1.132 VMD/1.9.3-Python-2.7.12
cd /tmp/siddis14/buildtest/tests/suite/compilers/helloworld
ifort -O3 -o hello.f90.exe /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90 
./hello.f90.exe 
 rm ./hello.f90.exe 
________________________________________________________________________________
Writing Test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0x973a918edc31a04c62638eb390bba18d.sh
Changing permission to 755 for test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0x973a918edc31a04c62638eb390bba18d.sh
________________________________________________________________________________
#!/bin/sh
module load medsci hpc/eb-2017-core icc/.2017.1.132-GCC-5.4.0-2.27 impi/2017.1.132 VMD/1.9.3-Python-2.7.12
cd /tmp/siddis14/buildtest/tests/suite/compilers/helloworld
ifort -O3 -o hello.f90.exe /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90 
./hello.f90.exe 
 rm ./hello.f90.exe 
________________________________________________________________________________
Writing Test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0x412dfac1afd2b8092e45330602c4f9d4.sh
Changing permission to 755 for test: /tmp/siddis14/buildtest/tests/suite/compilers/helloworld/hello_intel_fortran.yml_0x412dfac1afd2b8092e45330602c4f9d4.sh
________________________________________________________________________________
#!/bin/sh
module load medsci hpc/eb-2017-core ifort/.2017.1.132-GCC-5.4.0-2.27 impi/2017.1.132 VMD/1.9.3-Python-2.7.12
cd /tmp/siddis14/buildtest/tests/suite/compilers/helloworld
ifort -O3 -o hello.f90.exe /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/src/hello.f90 
./hello.f90.exe 
 rm ./hello.f90.exe 
________________________________________________________________________________
Writing 4 tests for /home/siddis14/buildtest-framework/toolkit/buildtest/suite/compilers/helloworld/hello_intel_fortran.yml

Note all 6 parent combination for module VMD/1.9.3-Python-2.7.12 were used when writing the test. It is worth noting, that any parent combination is sufficient when loading the desired module.