Handling Compiler and other Package Dependencies

When creating a collection of software (applications and libraries) for users to use, there is the problem of ensuring that the user is using the correct builds of everything. Generally, if an user is attempting to compile code making use of the system software collection, you want to ensure that the user is compiling his code with the same compiler that was used to compile the library. This tends to be particularly true of C++ and Fortran code using modules, and parallel codes using MPI libraries.

As a result, in environments supporting multiple compilers, software libraries often end up with multiple installs of libraries and applications of the same version, depending on the compiler and other libraries used to build them. Sometimes there are even additional installs for variants with different threading models, number formats, level of vectorization support, etc. This cookbook describes various strategies for handling the modulefiles to support all these different builds for each package.

For each strategy, we will provide an overview of how it works, and then show how an user might interact with it, usually a similar sequence for each case. We wish to explore how, and how well, each strategy succeeds in handling the multiple builds of the same version of a package, including

  1. Basic dependency handling: seeing how well the strategy supports the loading of the correct build of a package depending on the previously loaded dependencies. And if no appropriate build is available, they should error accordingly.

  2. The module switch command and more advanced dependency handling: how well the strategy supports more advanced cases. E.g. a case wherein several modules are loaded and the user replaces a module upon which other modules currently loaded depend. In general, how well the strategy prevents the user's set of loaded modules from being incompatible.

  3. Visibility into what packages are available. This includes being able to readily see all of the packages installed, seeing what versions of packages are available for a given compiler/MPI/etc combination, and seeing for which compiler/MPI/etc combinations a specific version of a package is available.

  4. How easily the user can navigate the modules for the builds. This includes how well partial modulenames (e.g. omitting version, etc) are handled by the different strategies.

We then try to summarize the strengths, weaknesses, and other attributes of each strategy. We also try to discuss differences in using on older (3.x) and newer (4.x) Environment Modules versions.

In addition to displaying examples for each strategy in this document, we have set up a the test environment as a playground in which you can explore.

Contents

  1. Overview of Examples

  2. Flavours Strategy

  3. Homebrewed Flavors Strategy

  4. Modulerc-based Strategy

  5. Modulepath-based Strategy

  6. Comparison of Strategies

Overview of Examples

The examples are a bit more elaborate than in some other cookbooks, so the directory structure under doc/example/compiler-etc-dependencies is similarly more complicated.

Example Software Library

For the purpose of the examples and the playground, we have created a fake example software library, rooted at the subdirectory doc/example/compiler-etc-dependencies/dummy-sw-root beneath where you placed the modules source files. This software tree is intended to represent some of the features you might see in a real software tree, which supports various compiler and MPI libraries, and that has been added to over time, and not always in the most systematic way.

The example software library does not contain any real code; there are dummy scripts for e.g. gcc, mpirun, etc. which just echo then name of the code and what version, compiler, etc. it was supposed to be built for, which is handy to show in the examples that the modulefiles are working as expected. It also shows how such a directory tree might be laid out --- the details of the layout will affect some of the code in the modulefiles, etc. The directory structure can be altered to fit your standards, but would require some minor modification to the modulefiles, etc. Note that there are also a bunch of subdirectories named 1 containing symlinks, these are for the strategy using the Flavours add-on and are discussed in that section.

The software in the example software library consists of:

  • GNU compiler versions 8.2.0 and 9.1.0

  • Intel Parallel Studio suite versions 2018 and 2019 (includes compilers, MPI and MKL)

  • PGI compiler suite versions 18.4 and 19.4

  • OpenMPI version 4.0, built for:

    • gcc/9.1.0

    • intel/2019

    • pgi/19.4

  • OpenMPI version 3.1, built for:

    • gcc versions 8.2.0 and 9.1.0

    • intel versions 2018 and 2019

    • pgi versions 18.4 and 19.4

  • mvapich version 2.3.1, built for:

    • gcc/9.1.0

    • intel/2019

    • pgi/19.4

  • mvapich version 2.1, built for:

    • gcc versions 8.2.0 and 9.1.0

    • intel versions 2018 and 2019

    • pgi versions 18.4 and 19.4

  • foo version 2.4, built for:

    • gcc/9.1.0 and openmpi/4.0

    • gcc/9.1.0 and mvapich/2.3.1

    • gcc/9.1.0 and no MPI

    • intel/2019 and openmpi/4.0

    • intel/2019 and mvapich/2.3.1

    • intel/2019 and intelmpi

    • intel/2019 and no MPI

    • pgi/19.4 and openmpi/3.1

    • pgi/19.4 and no MPI

  • foo version 1.1, built for:

    • gcc/8.2.0 and openmpi/3.1

    • gcc/8.2.0 and mvapich/2.1

    • gcc/8.2.0 and no MPI

    • intel/2018 and openmpi/3.1

    • intel/2018 and mvapich/2.1

    • intel/2018 and intelmpi

    • intel/2018 and no MPI

    • pgi/18.4 and openmpi/3.1

    • pgi/18.4 and mvapich/2.1

    • pgi/18.4 and no MPI

  • bar version 5.4, built with:

    • gcc/9.1.0 and supporting avx2

    • gcc/9.1.0 and supporting avx

  • bar version 4.7, built for:

    • gcc/8.2.0 and supporting avx

    • gcc/8.2.0 and supporting sse4.1

I.e., we have 3 families of compiler suites with 2 different versions each. And two MPI families (openmpi and mvapich) with two versions each, with the most recent version only built with the latest compiler version of each family, and the older version built with both versions of each compiler family. In addition, it is assumed that the intel compiler suites include Intel's MPI library built for that compiler. The application foo depends on the compiler and optionally on MPI libraries and has two versions; the newer version mostly has builds for the latest compiler and MPI (for pgi it only supports the latest compiler and older openmpi), and the older version mostly has builds for the older compiler and MPI. The bar application depends on compiler and has variants depending on size of integers used in the API.

We also assume that the gcc/8.2.0 compiler is the system default; i.e. it is the compiler provided by default by the Linux distro used by the system, and therefore might potentially be available to users without loading any modules.

More directories under doc/example/compiler-etc-dependencies

The modulefiles for the different strategies do not play well with each other, in part because we use the same names for many of the modules between strategies. So in addition to the dummy-sw-root subdirectory, each strategy has its own modulepath tree subdirectory underneath doc/example/compiler-etc-dependencies. There are some minor differences between the modulefiles for the Modulerc-based Strategy depending on whether Environment Modules 3.x or 4.x is being used, so we actually have two trees for that case (modulerc3 and modulerc4).

As there are a fair number of modulefiles, we make use of various tricks in the cookbook Tips for Code Reuse in Modulefiles to minimize the amount of repeated code. In general, the actual modulefiles are small "stubfiles", setting one or a few Tcl variables, and then sourcing a common tcl file which does all the real work. Symlinks are used where possible to avoid duplicating files. The Modulerc-based Strategy also uses some complicated .modulerc files; these are fairly generic and to avoid redundancy are symlinked into the appropriate places in modulepath tree from the modrc_common directory.

We also in some cases use Tcl procedures; for the sake of the examples these our sourced in the files as needed, but if one were to use the strategies needing such in production it would be better to follow the suggestions in Expose procedures and variables to modulefiles and place the required procedures in site config script. The various tcl procedures are placed in the tcllib sub-directory, outside of the modulepaths. These are actually broken up into multiple files for the purpose of this cookbook (so that smaller chunks of code can be looked at in this document).

The example-sessions subdirectory contains various shell scripts used for the usage examples for each strategy (shell scripts are used because there are some slight variations required between the strategies), as well as the outputs of running such scripts. Subdirectories exist for each strategy, and beneath them for each of the two Environment Modules versions (3.2.10 and 4.3.1) used; for brevity not all of them are shown in this document, especially as the 3.x and 4.x differences are often small. In the example outputs, the Environment Modules version and the strategy being employed is indicated in the shell prompt (e.g. mod3-flavours or mod4 (modulerc)).

Using the playground environments

Although we strive to provide a decent discussion in this cookbook, you are encouraged to try things out in the playground in order to get a better feel for things.

Because we use some modulefile names (e.g. gcc, intel, pgi, openmpi, etc) that likely are present on your system as well, it is recommended that if you wish to explore the playground environment that you spawn a new shell, do a "module purge", and then set your MODULEPATH environment variable appropriately for the specific strategy.

The Flavours_strategy, as will be discussed, requires some modifications to your Environment Modules installation. It is recommended that you you make a copy or new installation (for Flavours, a 3.x install works best), and then spawn a new shell and initialize the new Flavours install in that first. Flavours code is not provided with this cookbook.

Some of the modulefiles, etc. require knowledge of where they were installed. To avoid requiring you to update lines in numerous files, we require you to set the environment variable MOD_GIT_ROOTDIR to location where the modules git working directory was cloned. E.g., if you issued the command git clone https://github.com/cea-hpc/modules.git ~/modules.test you should set MOD_GIT_ROOTDIR to ~/modules.test. Please ensure it is exported (use setenv in csh and related shells, or export in Bourne derived shells like bash). This is just a hack to make the examples work better; if you opt to use one of these strategies in production, you will want to hard code some relevant paths; the comments in the modulefiles will describe what needs to be done.

Some more detail on setting up the playground is given at the start of the Examples section for each strategy.

Flavours Strategy

The Flavours strategy uses the Flavours extension to Environment Modules. Unlike the other strategies discussed, this requires the separate download and installation of an extension to Environment Modules.

Installation and Implementation

More details can be found at the website for this extension, but to install this you basically just need to:

  1. Clone the git repo somewhere (git clone https://git.code.sf.net/p/flavours/code flavours-code)

  2. Rename the standard Environment Modules modulecmd file (in the bin subdirectory under the installation root) to modulecmd.wrapped. (It is recommended that you do this in a copy of your production installation, or better yet, in a new install of the 3.x Environment Modules (as Flavours has been developed for Modules 3.x))

  3. Copy the modulecmd.wrapper file from Flavours to the bin subdirectory above. Make sure the modulecmd.wrapper file is executable.

  4. Symlink modulecmd.wrapper to modulecmd

  5. Edit modulecmd.wrapper where indicated to give fully qualified path to modulecmd.wrapped

  6. Copy the flavours.tcl and pkgIndex.tcl files to some (possibly new) directory under the modules installation root, and set TCLLIBPATH to that directory (you probably will want to add that to the various modules init scripts)

The module command invokes modulecmd, which in this case is results in the Flavours wrapper bash script modulecmd.wrapper being invoked. This calls the renamed standard modulecmd.wrapped command. This wrapper command catches and processes certain output from the modulefile evaluation intended for its consumption.

The modulefiles themselves make use of various commands in the Tcl module flavours. Many of these are just flavours variants of standard modulefile commands, e.g. flavours prepend-path versus prepend-path. Some important flavours commands:

  • package require flavours: This loads the Tcl package flavours, and should occur near the top of your modulefile

  • flavours init: This initializes the flavours package, and should be the first of the flavours commands issued. Typically call right after the package load.

  • flavours prereq: Like the standard prereq command, this declares a prerequisite. But it also does quite a bit more, as is discussed further below.

  • flavours root: This is used to set the root for where the package is actually installed. This is used when generating the flavours path

  • flavours revision: seem intended to allow for changes in the path format in future versions of flavours. It is used in constructing the final path to the package.

  • flavours conflict: This is similar to the standard conflict command, but enhanced to recognize the flavours prereqs above.

  • flavours commit: This should be called after the root, revision, and prereq sub-commands of flavours are called, and before any of the path sub-commands. It seems to be responsible for taking all those values to above and constructing the path to the package.

  • flavours path: This returns a string with the path to the specific build of the package.

  • flavours prepend-path, flavour append-path: These work much like the standard prepend-path and append-path, except that the value being prepended/appended to the environment variable has the path (as returned by flavours path) prepended to it with the appropriate directory separator. E.g., to add to the PATH variable the bin subdirectory of the root directory where the specific build was installed, use flavours prepend-path PATH bin

  • flavours cleanup: This should be called after all flavours sub-commands are finished and before exiting the script to ensure proper cleanup. Among other things, it ensures that any packages that depend on this package will get reloaded if this package is switched out.

The flavours prereq command accepts the new -class parameter, allowing it to require a class of packages; e.g. one could use -class compiler to indicate that it has a prereq on a compiler (any of the modules gnu, intel, or pgi). The allowable classes, and the package basenames that are in each class, is defined in flavours.tcl in the Tcl associative array _class. The ones shipped by default are

  • compiler: consisting of gnu, intel, and pgi

  • mpi: consisting of openmpi, mvapich2, mvapich, intelmpi

  • linalg: consisting of mkl, atlas, acml, netlib

You will likely want to adjust these if you go with flavours in production.

The flavours prereq command also accepts the parameter -optional, which declares optional prerequisites. Although it sounds a little oxymoronic, this comes into play with the secondary purpose of the command in declaring the components of the path, as discussed below. If a prereq is not optional, the modulefile will complain if nothing satisfying the prereq has been module loaded previously. If the prereq is optional, the modulefile will not complain if it was not loaded, but will use the prereq in constructing the path to the build of the package if it was loaded.

The flavours prereq command also defines the components which will comprise the final path to the directory containing the specific build of the package. The order of the prereq commands controls the order of the components in the path.

The modulefile will check that all non-optional flavours prereq commands are satisfied, and then construct a path to the installation root for this build of the package using the packages satisfying the prereqs. The resultant path is composed of:

  • the value from flavours root

  • directory separator (/)

  • the value from flavours revision

  • directory separator (/)

  • a prefix created by concatenating the package names satisfying the prereqs, in order. The package name and version will be separated by a hyphen (-), as will the different components.

So if flavours root was set to /local/software/foo/1.7, revision to 1, and the package had prereqs compiler and mpi, and gnu/9.1.0 and openmpi/4.0 were loaded, the resulting path would be /local/software/foo/1.7/1/gnu-9.1.0-openmpi-4.0.

The modulefile actually will test for the existence of that directory, and if not found will return an error to the that the package was not built for that combination of prereqs. You either need to install your packages using the above directory schema, or create symlinks linking that scheme to where you actually install the packages.

Examples

We now look at the example modulefiles for flavours. To use the examples, you must

  1. Have Flavours extension installed. NOTE these examples will NOT work without the Flavours installed.

  2. Set (and export) MOD_GIT_ROOTDIR to where you git-cloned the modules source

  3. Do a module purge, and then set your MODULEPATH to $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/flavours

We start with the module avail command:

[mod4-flavours]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/flavours -
bar/4.7  foo/2.4    intel/2018        mvapich/2.1    openmpi/4.0  simd/avx
bar/5.4  gnu/8.2.0  intel/2019        mvapich/2.3.1  pgi/18.4     simd/avx2
foo/1.1  gnu/9.1.0  intelmpi/default  openmpi/3.1    pgi/19.4     simd/sse4.1

We note that we only see the package names and versions; e.g. foo/2.4, without any mention of the compilers and MPI libraries for which it is built. This terser type was an intentional design goal of the authors. Also of note are the intelmpi and simd packages. The Flavours approach relies on seeing what modules have been loaded previously in order to determine what 'flavor' of the requested package should be loaded. To support the different builds of bar which depend on the CPU vectorization commands supported, we need to add a "dummy" package simd. The module definition is quite trivial; a simple stub file like

#%Module
# Modulefile for CPU vectorization support

set simd avx
set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/common

and the main content in the common file:

# Common stuff for "simd" modulefiles
# Using "flavours" strategy
#
# This file expects the following Tcl variable to have been previously defined:
#	simd: The level of simd support, eg. avx, avx2, sse4.1

# Initialise "flavours"
package require flavours
flavours init

proc ModulesHelp { } {
   global simd
   puts stderr "
This is a dummy modulefile to indicate that the CPU vectorization
should be set to '$simd' where possible.

"
}

module-whatis "CPU Vectorization support level: $simd"

# Even in production, this modulefile would not do anything
conflict simd

# Reload any modules with this as a prerequisite
flavours cleanup

Basically it just declares a help procedure and whatis text. This way, an user can load the appropriate simd module to control which variant of bar they will get. The only interesting aspect is that near the beginning of the file we do a package require flavours and flavours init, and add a flavours cleanup near the bottom. The lines at the beginning instruct Tcl command to load the Flavours package, and then initialize the package. The flavours cleanup is required so that if the simd module is switched out, any modulefiles that depend on it get reloaded.

In our example, we assumed that the Intel MPI libraries are automatically set up properly if one were to load the intel module, and we assumed the Intel MPI libraries were not supported for either the GNU or PGI compilers. However, we also wished to allow for foo to be used without any MPI support. So we need a way to distinguish if someone wants to use an Intel compiler build of foo without MPI or with the Intel MPI libraries. Our choice for this example was to require one to explicitly module load intelmpi if one wished to use the Intel MPI variant --- we do not bother with a real version number because assuming the version is determined by the version of intel (the Intel Parallel Studio version). So the intelmpi modulefile is similar to the simd modulefiles, a dummy modulefile. Again, it includes the flavours init and flavours cleanup wrapping to ensure proper reloading of dependent modules should it be switched out.

If you were to support Intel MPI for non-intel compilers, you could create your intelmpi modulefiles as usual, and then add a default or intel "dummy" version to use the version that is part of the intel Parallel Studio. Or you could separate the intelmpi bits from the intel modulefile so both non-intel and intel compilers need to explicitly module load intelmpi.

The modulefiles for the various compilers are all pretty much standard, except for the same three flavours lines as the simd modulefile: package require flavours, flavours init, and flavours cleanup. These are required to ensure dependent modulefiles get reloaded if the compiler is switched out. We also note that the modulefile for the GNU Compiler Collection is referred to as gnu, not gcc (this is due to how the compiler class is defined in flavours.tcl, and we did not bother to change that for the purposes of this cookbook).

With the openmpi and mvapich MPI libraries, things start to get interesting. These all should setup the environment for a different build depending on the compiler loaded. The real work is done in the common tcl file, as shown below:

# Common stuff for "openmpi" modulefiles
# Using "flavours" strategy
#
# Expects the following Tcl variables to have been previously set:
#	version: version of openmpi

# Common parts of modulefile for openmpi

# Initialise "flavours"
package require flavours
flavours init

proc ModulesHelp { } {
   global version
   puts stderr "
openmpi: Test dummy version of OpenMPI $version

For testing packages depending on compilers/MPI

"
}

module-whatis "Dummy openmpi $version"

# Construct flavour name
flavours prereq -class compiler
flavours conflict openmpi

# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root

flavours root     $swroot/openmpi/$version
flavours revision 1
flavours commit

# Set environment variables
setenv MPI_DIR [flavours path]

# Prepend to environment variables (paths relative to
# the directory containing the flavour)
flavours prepend-path PATH            bin
flavours prepend-path LIBRARY_PATH    lib
flavours prepend-path LD_LIBRARY_PATH lib
flavours prepend-path CPATH           include

# Reload any modules with this as a prerequisite
flavours cleanup

Like the previous cases, the file starts with the Tcl command to load the package, followed by the flavours init command.

The flavours prereq command states that this package requires a compiler to have been previously loaded, and that the path to the specific build to use will depend on that. We note the use of the -class parameter; the exact definition of the compiler class is in the compiler field of the Tcl associative hash _class defined in flavours.tcl.

The flavours root sets the root directory of where the builds for this package is installed. We use the MOD_GIT_ROOTDIR environment variable for convenience in this example, but in production you would generally hardcode a path. The result of all the directives is that the build will be found in a path named after the compiler (since in this case there is only one flavour prereq); e.g. for gcc version 9.1.0, we expect to find the build in $swroot/openmpi/4.0/1/gnu-9.1.0. If you do not use that naming convention for your installation directories, you can use symlinks to fake it.

The flavours path command in the setenv MPI_DIR statement sets MPI_DIR to the aforementioned build path. The flavours prepend-path commands prepend to the environment variable specified by the first argument the result of prepending the flavours path to their second argument. E.g., the first such, assuming openmpi version 4.0 was requested and gnu/9.1.0 loaded, would be basically the same as a standard Modules command:

prepend-path PATH $swroot/openmpi/4.0/1/gnu-9.1.0/bin

The following shows how this would appear to the user:

[mod4-flavours]$ module purge
[mod4-flavours]$ module load pgi/19.4
[mod4-flavours]$ module load openmpi/4.0
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/4.0
[mod4-flavours]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4-flavours]$ module unload openmpi
[mod4-flavours]$ module switch pgi intel/2019
[mod4-flavours]$ module load openmpi/4.0
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0
[mod4-flavours]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4-flavours]$ module unload openmpi
[mod4-flavours]$ module switch intel gnu/9.1.0
[mod4-flavours]$ module load openmpi/4.0
[mod4-flavours]$ mpirun
mpirun (openmpi/4.0, gcc/9.1.0)
[mod4-flavours]$ module unload openmpi
[mod4-flavours]$ module switch gnu gnu/8.2.0
[mod4-flavours]$ module load openmpi/4.0
openmpi/4.0 - no flavour compatible with modules 'gnu/8.2.0'

Loading openmpi/4.0
  ERROR: Module evaluation aborted
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) gnu/8.2.0

Here we note that once a compiler is loaded, the PATH and the other environment variables are set appropriately to point to the bin dir for the particular build of openmpi/4.0, as evidenced by the output of our dummy mpirun command. At the end, we attempt to load openmpi/4.0 for gnu/8.2.0, and receive an error because our dummy SW library does not contain a matching build. This is determined from the flavours path; if the path does not exist (in this example $swroot/openmpi/4.0/1/gnu-8.2.0) it will abort in this fashion.

In the above, we have explicitly unloaded openmpi, switched the compilers, and then reloaded openmpi. A nice feature of Flavours is that it can handle the switching out of compilers or other modulefiles which other modulefiles depend on, as:

[mod3-flavours]$ module purge
[mod3-flavours]$ module load pgi/19.4
[mod3-flavours]$ module load openmpi
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
  1) pgi/19.4      2) openmpi/4.0
[mod3-flavours]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod3-flavours]$ module switch pgi intel/2019
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
  1) intel/2019    2) openmpi/4.0
[mod3-flavours]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod3-flavours]$ module switch intel intel/2018
openmpi/4.0 - no flavour compatible with modules 'intel/2018'
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
  1) intel/2019    2) openmpi/4.0
[mod3-flavours]$ mpirun
mpirun (openmpi/4.0, intel/2019)

Note that when we switched between the pgi and intel compilers above, Flavours automatically "unloaded" and "reloaded" the openmpi module. This happens in the flavours cleanup portion of the compiler modulefiles, and is due to openmpi declaring a flavours prereq on the compiler class.

Note

The above behavior with switch was done with version 3.2.10 of Environment Modules; it does not appear to work with 4.3.1.

Note that when we further tried to replace version 2019 of the intel compiler with the 2018 version, the module switch of the compilers failed because openmpi/4.0 was not built with intel/2018. Since the user never explicitly requested version 4.0 of openmpi (it was defaulted in the initial load as the latest version of openmpi available for pgi/19.4), it would have been nicer had the attempted reload of openmpi allowed it t:o default to the 3.1 version (as the latest version available for intel/2018). Nevertheless, it behaved well in this situation; the module switch failed with a reasonable error message and the resulting set of modules was still consistent.

We also note that if we attempt to load openmpi without having previously loading a compiler, we will get an error:

[mod4-flavours]$ module purge
[mod4-flavours]$ module load openmpi/3.1
openmpi/3.1 depends on one of the module(s) 'gnu intel pgi'

Loading openmpi/3.1
  ERROR: Module evaluation aborted
[mod4-flavours]$ module list
No Modulefiles Currently Loaded.
[mod4-flavours]$ module purge
[mod4-flavours]$ module load gnu/8.2.0
[mod4-flavours]$ module load openmpi
openmpi/4.0 - no flavour compatible with modules 'gnu/8.2.0'

Loading openmpi/4.0
  ERROR: Module evaluation aborted
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) gnu/8.2.0

In particular, there is no support for a "default" compiler; if e.g. you wished to make the distribution supplied gcc the default compiler, you will need to have the initializations scripts automatically do a module load of that compiler (possibly a dummy modulefile like simd/intelmpi if the compiler is already in the user's path) in your user's start up dot files or similar. We also note that there is no additional intelligence in the version defaulting --- in the last example, we have gnu/8.2.0 loaded and if we try to load openmpi without specifying a version, it defaults to version 4.0 as that is the latest version of openmpi without regard for the fact that there is no build of openmpi version 4.0 for gnu/8.2.0 (but there is such for openmpi/3.1).

The situation for foo is more complicated, as it depends both on the compiler and optionally on the MPI library. But with Flavours, the modulefile is only slightly more complicated, e.g. for the common file is:

# Common stuff for "foo" modulefiles
# Using "flavours" strategy
#
# Expects the following Tcl variables to have been previously set:
#	version: version of foo

# Initialise "flavours"
package require flavours
flavours init

proc ModulesHelp { } {
   global version
   puts stderr "
foo: Test dummy version of foo $version

For testing packages depending on compilers/MPI

"
}

module-whatis "Dummy foo $version"

# Construct flavour name
flavours prereq -class compiler
flavours prereq -optional -class mpi
flavours conflict foo

# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root

flavours root     $swroot/foo/$version
flavours revision 1
flavours commit

# Set environment variables
setenv FOO_DIR [flavours path]

# Prepend to environment variables (paths relative to
# the directory containing the flavour)
flavours prepend-path PATH            bin
flavours prepend-path LIBRARY_PATH    lib
flavours prepend-path LD_LIBRARY_PATH lib
flavours prepend-path CPATH           include

# Reload any modules with this as a prerequisite
flavours cleanup

Basically, the main difference is the addition of the line flavours prereq -optional -class mpi. This instructs Flavours that there is an additional, optional prereq. The order of the prereq lines matter, as that controls the resultant flavors path. With the current configuration, assuming gnu/9.1.0 and openmpi/4.0 were loaded, the path would become $swroot/foo/2.4/1/gnu-9.1.0-openmpi-4.0. If the order were reversed, the openmpi-4.0 would precede the gnu-9.1.0. Because the MPI requirement is optional, if gnu/9.1.0 was loaded and no MPI library loaded, the path would evaluate to $swroot/foo/2.4/1/gnu-9.1.0.

We show how it works below:

[mod4-flavours]$ module purge
[mod4-flavours]$ module load pgi/19.4
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (pgi/19.4, nompi)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module load openmpi/3.1
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/3.1   3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (pgi/19.4, openmpi/3.1)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module unload openmpi
[mod4-flavours]$ module switch pgi intel/2019
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (intel/2019, nompi)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module load intelmpi
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) intelmpi/default   3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (intel/2019, intelmpi)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module switch intelmpi mvapich/2.3.1
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) mvapich/2.3.1   3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (intel/2019, mvapich/2.3.1)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module switch mvapich openmpi/4.0
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0   3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (intel/2019, openmpi/4.0)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module unload openmpi
[mod4-flavours]$ module switch intel/2019 gnu/9.1.0
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) gnu/9.1.0   2) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module load mvapich/2.3.1
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) gnu/9.1.0   2) mvapich/2.3.1   3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (gcc/9.1.0, mvapich/2.3.1)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module switch mvapich openmpi/4.0
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) gnu/9.1.0   2) openmpi/4.0   3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (gcc/9.1.0, openmpi/4.0)

So basically, if the user loads a compiler, the the environment variables (PATH, etc) are set up for the correct build of foo. If no MPI library was loaded, a version of foo built without MPI will be loaded, otherwise, a version of foo built with the loaded MPI library will be loaded. This is shown by the output of the foo command. Note also how we use the dummy intelmpi package to indicate a desire for the intelmpi enabled version.

The 3.x version of Environment Modules supports using the switch command on either the compiler or MPI library, and will result in reloading of foo and the MPI library.

[mod3-flavours]$ module purge
[mod3-flavours]$ module load pgi/18.4
[mod3-flavours]$ module load openmpi/3.1
[mod3-flavours]$ module load foo/1.1
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
  1) pgi/18.4      2) openmpi/3.1   3) foo/1.1
[mod3-flavours]$ foo
foo 1.1 (pgi/18.4, openmpi/3.1)
[mod3-flavours]$ module switch pgi intel/2018
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
  1) intel/2018    2) openmpi/3.1   3) foo/1.1
[mod3-flavours]$ foo
foo 1.1 (intel/2018, openmpi/3.1)
[mod3-flavours]$ mpirun
mpirun (openmpi/3.1, intel/2018)
[mod3-flavours]$ module purge
[mod3-flavours]$ module load intel/2019
[mod3-flavours]$ module load foo
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
  1) intel/2019   2) foo/2.4
[mod3-flavours]$ foo
foo 2.4 (intel/2019, nompi)
[mod3-flavours]$ module load openmpi
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
  1) intel/2019    2) foo/2.4       3) openmpi/4.0
[mod3-flavours]$ foo
foo 2.4 (intel/2019, openmpi/4.0)

In particular note the final case, wherein we load intel/2019 then foo, and get the version of foo built without MPI. When we subsequently load openmpi, foo is reloaded to be the openmpi version (this is because the hooks to reload foo are in the flavours cleanup part of the openmpi modulefile, and foo declared its optional dependency on MPI). Also, we don't bother showing it, but if you were to attempt to load foo without at least a compiler loaded, it would display an error.

Our final example for flavours is the bar command. Here in addition to the compiler dependency, we have versions for different SIMD vectorization supported. Again, the difference in the modulefile is small, e.g.

# Common stuff for "bar" modulefiles
# Using "flavours" strategy
#
# Expects the following Tcl variables to have been previously set:
#	version: version of bar

# Initialise "flavours"
package require flavours
flavours init

proc ModulesHelp { } {
   global version
   puts stderr "
bar: Test dummy version of bar $version

For testing packages depending on compilers/MPI

"
}

module-whatis "Dummy bar $version"

# Construct flavour name
flavours prereq -class compiler
flavours prereq simd
flavours conflict bar

# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root

flavours root     $swroot/bar/$version
flavours revision 1
flavours commit

# Set environment variables
setenv BAR_DIR [flavours path]

# Prepend to environment variables (paths relative to
# the directory containing the flavour)
flavours prepend-path PATH            bin
flavours prepend-path LIBRARY_PATH    lib
flavours prepend-path LD_LIBRARY_PATH lib
flavours prepend-path CPATH           include

# Reload any modules with this as a prerequisite
flavours cleanup

Basically, the optional flavours prereq on the mpi class from the foo package is replaced by a (mandatory) flavours prereq on the simd dummy package. We note that Flavours package knows nothing about our simd dummy package until we add it as a prereq for bar. (This is in contrast to the compiler and mpi classes). Usage would be like:

[mod4-flavours]$ module purge
[mod4-flavours]$ module load gnu/9.1.0
[mod4-flavours]$ module load simd/avx2
[mod4-flavours]$ module load bar/5.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) gnu/9.1.0   2) simd/avx2   3) bar/5.4
[mod4-flavours]$ bar
bar 5.4 (gcc/9.1.0, avx2)
[mod4-flavours]$ module unload bar
[mod4-flavours]$ module switch simd simd/avx
[mod4-flavours]$ module load bar/5.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) gnu/9.1.0   2) simd/avx   3) bar/5.4
[mod4-flavours]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4-flavours]$ module unload bar
[mod4-flavours]$ module switch simd simd/sse4.1
[mod4-flavours]$ module load bar/5.4
bar/5.4 - no flavour compatible with modules 'gnu/9.1.0 simd/sse4.1'

Loading bar/5.4
  ERROR: Module evaluation aborted
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) gnu/9.1.0   2) simd/sse4.1

Here we note that as both the compiler and simd prereqs are non-optional, it complains unless both have been previously loaded. When both have been loaded, the PATH and other environment variables are set appropriately for the requested build; and if it does not exist and error is produced.

Summary of Flavours

  • It is an external extension to Environment Modules, requiring additional installation steps.

  • The git repository appears to have been last updated in 2013; although which means that it has not been updated for Environment Modules 4.x, a simple experimentation indicates that it still works, with the exception of automatic reloading of a module if any of the modules it depends on are switched. However, the Flavours extension does not appear to be actively supported.

  • The Flavours package (using Environment Modules 3.x) fully supports the module switch syntax, with the switching out of a dependency (e.g. a compiler) causing the reload of all modulefiles depending on it. (however the test of this feature failed when using Environment Modules 4.x.)

  • The syntax for modulefiles is elegant, and one can easily extend the basic compiler dependency modulefile to add additional dependencies. Even for packages/dummy packages that the Flavours extension knows nothing about (e.g. simd in the above example).

  • The Flavours package will provide a shorter module avail output, only e.g. giving package name and version and not listing a separate modulefile for each combination of package, version, compiler+version, MPI library+version, etc.

  • The Flavours package will fail with an error message if user tries to load a package which was not built for the values of the dependency packages loaded.

  • The Flavours package will fail to load with an error message if any dependent package is not already loaded. In particular, it will not attempt to default these. So if such defaults are desired, you will need to have initialization scripts automatically load the appropriate modules.

  • The Flavours package does not include any mechanism for more intelligent defaulting. I.e., if an user requests to load a package without specifying the version desired, the version will be defaulted to the latest version (or whatever the .modulerc file specifies) without regard for which versions support the versions of the compiler and other prereq-ed packages the user has loaded. While one could write custom .modulerc files for such, Flavours does not provide any tools for simplifying such.

Homebrewed Flavors Strategy

Although the Flavours extension described above has an elegance about it, one can achieve much of the same functionality in modulefiles using standard Environment Modules and Tcl commands. This can be facilitated by the definition of some useful Tcl procedures. For lack of a better name, we will refer to this strategy as Homebrewed flavors.

Implementation

This strategy just makes use of standard Environment Modules and Tcl procedures to query what modules of a given type are loaded and to construct the path to the software package accordingly. To avoid needless (and error prone) repetition of code, we collect these into several Tcl procedures of our own. Ideally, these should be placed in a site configuration Tcl file and exposed to modulefiles as explained in the cookbook Expose procedures and variables to modulefiles. However, to avoid the need for that in these examples, we instead have placed them into a file and use the MOD_GIT_ROOTDIR to locate and source that file in the relevant modulefiles. (Actually, we have a single tcl file that is sourced both for this and some other strategies, and it sources several files so that we can break up the discussion of the the Tcl procedures. All of that is just for the purposes of this cookbook; normally you just put the procedures you need in the one site config file).

We discuss the various Tcl procedures here, as they are what provide most of the functionality. We start with the routines for generic loaded modules:

#--------------------------------------------------------------------
# GetLoadedModules
#
# Returns a tcl list of all modules loaded.  From $ENV{LOADEDMODULES}
proc GetLoadedModules { } {
   global env
   #Handle case if no modules loaded
   if {![info exists env(LOADEDMODULES)]} {
      #No modules loaded, return empty list
      return [ list ]
   }
   set loadedenv $env(LOADEDMODULES)
   set loaded [ split $loadedenv : ]
   return $loaded
}

#--------------------------------------------------------------------
# GetTagOfModuleLoaded(pkg)
#
# Looks for a loaded module matching ^$pkg/, and returns the tag matched.
# Returns {} if no tags matched (i.e. module pkg not loaded)
proc GetTagOfModuleLoaded { mymodule } {
   set loadedlist [ GetLoadedModules ]
   set regex "^$mymodule/"
   set fndidx [ lsearch -regex $loadedlist $regex ]
   if {  $fndidx == -1 }  { return {} }
   set found [ lindex $loadedlist $fndidx ]
   return $found
}

This defines the two Tcl procedures:

  • GetLoadedModules : this returns the list of loaded modules, from the LOADEDMODULES

  • GetTagOfModuleLoaded : this takes as argument the base name of a package, and returns the first full spec for the matching package, or an empty string if no matching package found.

The Tcl procedure GetTagOfModuleLoaded can be used to find out what version of a given package is loaded, and is enough for many packages. However, for compilers, and similar, a bit more is needed. For compilers:

#--------------------------------------------------------------------
# GetDefaultCompiler:
#
# Returns the default compiler, gcc/8.2.0
proc GetDefaultCompiler { } {
   return "gcc/8.2.0"
}

#--------------------------------------------------------------------
# RequireCompiler:
#
# Does a module load of specified compiler $mycomp.
# Includes special handling if $mycomp is the default compiler
proc RequireCompiler { mycomp } {
   # If your module tree is set up so that there is no module for the
   # default compiler (because e.g. it is available w/out loading a module
   # anyway), you can uncomment the following block which will cause
   # RequireCompiler to do nothing if mycomp is the default compiler
   #set defComp [GetDefaultCompiler]
   #if { $mycomp eq $defComp } {
      #return
   #}
   module load $mycomp
}

#--------------------------------------------------------------------
# GetKnownCompilerFamilies:
#
# Returns a list of recognized compiler family names
#E.g. gcc, intel, pgi
proc GetKnownCompilerFamilies { } {
   set cfamilies {gcc intel pgi}
   return $cfamilies
}

#--------------------------------------------------------------------
# GetLoadedCompiler:
#
# Returns the string for the compiler we are using (i.e. was previously
# module loaded).  E.g., gcc/8.2.0
# If no compiler was previously loaded, then if the optional parameter
# $pathDefault is set, it will look for a compiler family and
# version in the last components to the path to the current modulefile or
# .modulerc, and if found, uses that.
# If still no path found, it will return the value of [GetDefaultCompiler]
# if $useDefault is set.
# Otherwise, returns empty string.
#
# Takes the following arguments:
#	pathDefault: boolean, default false.  If set, attempt to determine
#		the compiler from the full path to the modulefile if
#		no compiler was loaded.
#	useDefault: boolean, default false.  If set, return the value of
#		GetDefaultCompiler if no compiler is loaded or found from
#		path (if $pathDefault).
#	loadIt: boolean, default false.  If set and a compiler
#		was defaulted from path of GetDefaultCompiler, we will
#		module load that compiler.
#		Ignored unless either pathDefault or useDefault is set
#	requireIt: boolean, default false.  If set, we will prereq the
#		compiler before returning.
proc GetLoadedCompiler {{pathDefault 0} { useDefault 0}
   {loadIt 0 } { requireIt 0 } } {
   global ModulesCurrentModulefile
   set ctag {}
   set cfams [ GetKnownCompilerFamilies ]
   foreach cfam $cfams {
      if { [ is-loaded $cfam ] } {
         set ctag [ GetTagOfModuleLoaded $cfam  ]
         if { $requireIt } { prereq $ctag }
         return $ctag
      }
   }

   # No loaded compiler found, try to default from path to modulefile?
   if { $pathDefault} {
      set moduledir [file dirname $ModulesCurrentModulefile ]
      set cversion [file tail $moduledir]
      set tmppath [file dirname $moduledir]
      set cfamily [file tail $tmppath]

      if { [lsearch $cfams $cfamily] > -1 } {
         # We matched a known compiler family in our path
         set ctag "$cfamily/$cversion"
         if { $loadIt } { RequireCompiler $ctag }
         if { $requireIt } { prereq $ctag }
         return $ctag
      }
   }

   # Still no compiler, default to GetDefaultCompiler>
   if { $useDefault } {
      set ctag [ GetDefaultCompiler ]
      if { $loadIt } { RequireCompiler $ctag }
      if { $requireIt } { prereq $ctag }
      return $ctag
   }

   #Nothing found, and not defaulting
   return $ctag
}

We defined four procedures above:

  • GetDefaultCompiler : this simply returns the name of our default compiler, which for this example is gcc/8.2.0

  • RequireCompiler : this simply does a module load on the specified compiler. It is kept as a separate procedure just in case you wish to intercept and prevent the loading of the default compiler (e.g. because no modulefile exists for it). In our example, there is a modulefile for it and so it is just a wrapper for module load.

  • GetKnownCompilerFamilies : this simply returns a Tcl list of known compiler families.

  • GetLoadedCompiler: this is the procedure that does the main work, and is described in detail below.

The GetLoadedCompiler procedure basically checks if any packages matching the names in GetKnownCompilerFamilies have been previously loaded. If so, it returns the modulefile specification for the first one found, and returns. If not, if pathDefault is set and there is a recognized compiler name and version in the last two components of the module specification, it will return that compiler. Otherwise, if the optional flag useDefault is set, it will return the value from GetDefaultCompiler. If all else fails, returns the empty string.

If the optional parameter loadIt is set, if a compiler was defaulted (i.e. not returned because it was already loaded), the procedure will call RequireCompiler to module load it.

If the optional parameter requireIt is set, we invoke prereq on the compiler found before returning.

A similar set of procedures exist for the MPI libraries, namely:

#--------------------------------------------------------------------
# RequireMPI:
#
# Does a module load of specified MPI library $mympi
# Includes special handling if $mympi is nompi or intelmpi (or one of its aliases)
# If $mympi is nompi, nothing is loaded.
# If the optional parameter noLoadIntel is set (default false), and if
# $mympi is intelmpi (or intel or impi), then we do not load module if
# the loaded compiler is intel (as we assume that provides intel MPI as well)
proc RequireMPI { mympi {noLoadIntel 0} } {
   # We do not do anything if mympi is nompi
   if { $mympi eq {nompi} } { return }
   if { $noLoadIntel } {
      # Get the basename of requested MPI library
      set mympiSplit [ split $mympi / ]
      set mympiBase [ lindex $mympiSplit 0 ]
      # Check if we requested an intel MPI
      set intelList "intelmpi impi intel intelmpi-mt impi-mt intel-mt"
      if { [lsearch $intelList $mympiBase ] > -1 } {
         # We requested intelmpi in some form
         # Check if an intel compiler was loaded
         set curComp [ GetLoadedCompiler 1]
         if { $curComp ne {} } {
            set curCompSplit [ split $curComp / ]
            set curCompBase [ lindex $curCompSplit 0 ]
            if { $curCompBase eq {intel} } {
               # $noLoadIntel is set, requested MPI is intel MPI, and intel compiler loaded
               return
            }
         }
      }
   }
   module load $mympi
}

#--------------------------------------------------------------------
# GetKnownMpiFamilies:
#
# Returns a list of recognized MPI library family names
#E.g. gcc, intel, pgi
proc GetKnownMpiFamilies { } {
   set mfamilies {openmpi mpavich intelmpi}
   return $mfamilies
}

#--------------------------------------------------------------------
# GetLoadedMPI:
#
# Returns the string for the MPI library we are using (i.e. was previously
# module loaded), or empty string if nothing loaded (from is-loaded command)
# E.g., intel/2013.1.117
# Takes an optional argument,
#	useIntel: boolean, default false.  If set, returns 'intelmpi' if
#		no MPI library is loaded but intel compiler is loaded
#	forceIt: boolean, default false.  If set, prereq MPI lib before returning.
#	requireIt: boolean, default false.  If set, prereq the MPI library
proc GetLoadedMPI { { useIntel 0} {forceIt 0} {requireIt 0} } {
   set mtag {}
   foreach mfam [ GetKnownMpiFamilies ] {
      if { [ is-loaded $mfam ] } {
         set mtag [ GetTagOfModuleLoaded $mfam  ]
         if { $requireIt } { prereq $ctag }
         return $mtag
      }
   }
   # No loaded compiler found, should we check for Intel compiler and return intelmpi?
   if { $useIntel } {
      #Yes
      set ctag [ GetCompilerLoaded ]
      set cSplit [ split $ctag / ]
      set cBase [ lindex $cSplit 0 ]
      if { $cBase eq intel } { return intelmpi }
   }
   return $mtag
}

The three procedures here are analogues of the compiler versions:

  • RequireMPI : this basically does a module load of the specified MPI library. It has some added logic so that it will not do a module load if the MPI library is nompi. Also, if the optional parameter noLoadIntel is set, if the MPI library is intelmpi (or a variant of that name) and the loaded compiler in intel, we assume that no additional module needs to be loaded. For this strategy, we want to load intelmpi modules, because, just like in the Flavours Strategy, we need to provide dummy intelmpi modules to allow one to request the use of the Intel MPI library.

  • GetKnownMpiFamilies : this returns a list of known MPI library family names. Used in GetLoadedMPI

  • GetLoadedMPI : This is the analogue of GetLoadedCompiler. If an MPI library is loaded, it will return the name of that module. If the optional requireIt flag is set, it will do a prereq on the MPI library before returning. The first optional argument, useIntel, indicates whether this module should return intelmpi if no MPI library is loaded but an Intel compiler is loaded.

The modulefiles for the compilers are basically standard; unlike the Flavours Strategy there is nothing special needed in these. Likewise for the dummy simd and intelmpi modules (the latter is only this basic because we assume intelmpi is only available if an Intel compiler is loaded. If one allowed for intelmpi with other compilers, it would more closely resemble the other MPI libraries).

We also define some Tcl procedures for generating warning and error messages, namely

#--------------------------------------------------------------------
# PrintIfLoading:
#
# Prints supplied text to stderr but only if in "load" mode
proc PrintIfLoading { args } {
   if [ module-info mode load ] {
      set tmp [ join $args ]
      puts stderr "$tmp"
   }
}

#--------------------------------------------------------------------
# PrintLoadInfo:
#
# Prints supplied text to stderr as informational message, but only
# if actually trying to load the module.
proc PrintLoadInfo { args } {
   set tmp [ join $args ]
   PrintIfLoading "
\[INFO\] $tmp
"
}

#--------------------------------------------------------------------
# PrintLoadWarning:
#
# Prints supplied text to stderr as warning message, but only
# if actually trying to load the module.
proc PrintLoadWarning { args } {
   set tmp [ join $args ]
   PrintIfLoading "
WARNING:
$tmp

"
}

#--------------------------------------------------------------------
# PrintLoadError:
#
# Like PrintLoadWarning, but as error message and does a "break"
proc PrintLoadError { args } {
   set tmp [ join $args ]
   PrintIfLoading "
**** ERROR *****:
$tmp

"
   if [ module-info mode load ] {
      break
   }
}

These procedures:

  • PrintIfLoading: will print supplied text to stderr only when in load mode

  • PrintLoadInfo: will print supplied text as informational text, but only when trying to load a module

  • PrintLoadWarning: will print supplied text as warning text, but only when trying to load a module

  • PrintLoadError: will print supplied text as error text and abort, but only when trying to load a module

The gist of this is that we might wish to print errors if an user tries to load an incompatible modulefile, but do not wish to print errors if they are merely doing a help, display, or whatis command.

The interesting bit begins with the openmpi and mvapich modulefiles. These both depend on the compiler, we show the main part of the openmpi modulefile below:

# Common modulefile for openmpi
# Using "homebrewed flavors" strategy
# Expects the following variables to have been
# previously defined:
#	version: version of openmpi

# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root

# Also get location of and load common procedures
# This is a hack for the cookbook examples, in production
# one should either
# 1) declare the procedures in a site config file (preferred)
# 2) hardcode the path to $tcllibdir and common_utilities.tcl
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

proc ModulesHelp { } {
   global version
   puts stderr "
openmpi: Test dummy version of OpenMPI $version

For testing packages depending on compilers/MPI

"
}

module-whatis "Dummy openmpi $version"

# Figure out what compiler we have loaded
# Optional args ensure will default to and load/prereq default compiler
# if no compiler currently loaded
set ctag [ GetLoadedCompiler 1 1 1 ]

# Compute the installation prefix
set pkgroot $swroot/openmpi
set vroot $pkgroot/$version
set prefix $vroot/$ctag

# Make sure there is a build for this openmpi version/compiler
# I.e. that the prefix exists
if ![ file exists $prefix ] {
   # Not built for this compiler, alert user and abort
   PrintLoadError "
openmpi/$version does not appear to be built for compiler $ctag

Please select a different openmpi version or different compiler.
"
}

# We need to prereq the compiler to allow autohandling to work
prereq $ctag

# Set environment variables
setenv MPI_DIR $prefix

set bindir $prefix/bin
set libdir $prefix/lib
set incdir $prefix/include

prepend-path PATH            $bindir
prepend-path LIBRARY_PATH    $libdir
prepend-path LD_LIBRARY_PATH $libdir
prepend-path CPATH           $incdir

We begin by sourcing the common_utilities file which defined the previously described Tcl procedures. Normally it is recommended that you put those procedures in a site config Tcl script and expose them to the modulefiles using the techniques described in the cookbook Expose procedures and variables to modulefiles. Even if you opt against that and decide to source a Tcl file, it is recommended to hard code the path.

The next interesting bit comes when we set the local Tcl variable ctag by calling the GetLoadedCompiler procedure. We allow the procedure to default the compiler, and because we have a default compiler defined we should always get a value. (If no default compiler was defined, one would have to handle the error if no compiler was loaded/defaulted.) We then use the value of ctag to set the path to the build of the package. To ensure that the package is built for this compiler, we do a quick check that the package installation path exists.

The modulefile for foo is a bit more complex:

# Common stuff for "foo" modulefiles
# Using "homebrewed flavors" strategy
#
# This file expects the following Tcl variables to have been
# previously set:
#	version: version of foo

# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root

# Also get location of and load common procedures
# This is a hack for the cookbook examples, in production
# one should either
# 1) declare the procedures in a site config file (preferred)
# 2) hardcode the path to $tcllibdir and common_utilities.tcl
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

proc ModulesHelp { } {
   global version
   puts stderr "
foo: Test dummy version of foo $version

For testing packages depending on compilers/MPI

"
}

module-whatis "Dummy foo $version"

# Figure out what compiler we have loaded
# Will default to and load default compiler if none loaded
set ctag [ GetLoadedCompiler 1 1 1 ]

# Figure out what MPI we have loaded
# We do NOT want to default to intelmpi if intel compiler loaded
set mtag [ GetLoadedMPI ]
# Set mtag it nompi if no MPI library found loaded
if { $mtag eq {} } { set mtag nompi }
# Set mtag to intelmpi if intelmpi/default found loaded
if { $mtag eq {intelmpi/default} } { set mtag intelmpi }

# Compute the installation prefix
set pkgroot $swroot/foo
set vroot $pkgroot/$version
set prefix $vroot/$ctag/$mtag

# Make sure there is a build for this foo version/compiler/MPI library
# I.e. that the prefix exists
if ![ file exists $prefix ] {
   # Not built for this compiler/MPI, alert user and abort
   PrintLoadError "
foo/$version does not appear to be built for compiler $ctag and MPI $mtag

Please select a different openmpi version or different compiler/MPI library combination.
"
}

# We need to prereq the compiler to allow autohandling to work
prereq $ctag
# and the MPI library if used
if { $mtag ne {nompi} } {
   # We currently require one to load intelmpi to use Intel MPI with
   # intel compilers. So we prereq on intelmpi as well
   # But if not, we could place the prereq below in an if-then block
   # so is only executed if mtag != intelmpi
   prereq $mtag
}

# Set environment variables
setenv FOO_DIR $prefix

set bindir $prefix/bin
set libdir $prefix/lib
set incdir $prefix/include

prepend-path PATH            $bindir
prepend-path LIBRARY_PATH    $libdir
prepend-path LD_LIBRARY_PATH $libdir
prepend-path CPATH           $incdir

conflict foo

The main difference between this modulefile, depending on both compiler and optionally MPI, and the openmpi modulefile above, is that in addition to detecting which compiler is loaded, we call GetLoadedMPI to determine the MPI library which was loaded, and use both of them in constructing the prefix to the installed foo.

Examples

We now look at the example modulefiles for the Homebrewed flavors strategy. To use the examples, you must #. Set (and export) MOD_GIT_ROOTDIR to where you git-cloned the modules source #. Do a module purge, and then set your MODULEPATH to:

$MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/homebrewed

The Homebrewed flavors strategy behaves much like the Flavours Strategy in practice. The module avail command,

[mod4 (homebrewed)]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/homebrewed -
bar/4.7  foo/2.4    intel/2018        mvapich/2.1    openmpi/4.0  simd/avx
bar/5.4  gcc/8.2.0  intel/2019        mvapich/2.3.1  pgi/18.4     simd/avx2
foo/1.1  gcc/9.1.0  intelmpi/default  openmpi/3.1    pgi/19.4     simd/sse4.1

looks basically the same, showing the a concise listing of packages and versions without information on the compilers and MPI libraries they were built with.

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load pgi/19.4
[mod4 (homebrewed)]$ module load openmpi/4.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/4.0
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (homebrewed)]$ module unload openmpi
[mod4 (homebrewed)]$ module switch --auto pgi intel/2019
[mod4 (homebrewed)]$ module load openmpi/4.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4 (homebrewed)]$ module unload openmpi
[mod4 (homebrewed)]$ module switch --auto intel gcc/9.1.0
[mod4 (homebrewed)]$ module load openmpi/4.0
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, gcc/9.1.0)
[mod4 (homebrewed)]$ module unload openmpi
[mod4 (homebrewed)]$ module switch --auto gcc gcc/8.2.0
[mod4 (homebrewed)]$ module load openmpi/4.0

**** ERROR *****:
openmpi/4.0 does not appear to be built for compiler gcc/8.2.0
Please select a different openmpi version or different compiler.


Loading openmpi/4.0
  ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0

Again, once a compiler is loaded, loading openmpi will set the PATH, etc. for the correct build of openmpi, as evidenced by the output of the dummy mpirun command. Notice that the module list command only shows the version of openmpi loaded, and contains no information about what compiler it was built with (you just have to assume it matches the loaded compiler). And again we note that if one attempts to load a version of openmpi (e.g. 4.0) that was not built for the specified compiler (e.g. gcc/8.2.0), an error is generated.

Unlike in Flavours Strategy, we did not put any code in the modulefiles to cause dependent modulefiles to be reloaded if a module they depend on gets switched out. However, starting with Environment Modules 4.2, a feature called Automated module handling was added. Without this feature, attempting to switch out a module upon which other modules depended could be problematic, as evidenced in this sequence below (using Environment Modules 3.2.10 and so without automated module handling mode):

[mod3 (homebrewed)]$ module purge
[mod3 (homebrewed)]$ module load pgi/19.4
[mod3 (homebrewed)]$ module load openmpi
[mod3 (homebrewed)]$ module list
Currently Loaded Modulefiles:
  1) pgi/19.4      2) openmpi/4.0
[mod3 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod3 (homebrewed)]$ module switch pgi intel/2019
[mod3 (homebrewed)]$ module list
Currently Loaded Modulefiles:
  1) intel/2019    2) openmpi/4.0
[mod3 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)

Here we note that we were able to switch out the pgi compiler for the intel compiler, but the openmpi module was not reloaded and the environment is still set for openmpi compiled with the pgi compiler, and that this inconsistency is not readily determined from the module list command.

Environment Modules 4.x, even with automated module handling disabled, is better --- in a command sequence as above the module switch from pgi to intel would fail due to the prereq module. However, with the Automated module handling enabled (this feature is disabled by default but could be enabled on a per-command basis with the --auto flag), things work much better, as evidenced below:

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load pgi/19.4
[mod4 (homebrewed)]$ module load openmpi
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/4.0
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (homebrewed)]$ module switch --auto pgi intel/2019
Switching from pgi/19.4 to intel/2019
  Unloading dependent: openmpi/4.0
  Reloading dependent: openmpi/4.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4 (homebrewed)]$ module switch --auto intel intel/2018

**** ERROR *****:
openmpi/4.0 does not appear to be built for compiler intel/2018
Please select a different openmpi version or different compiler.


Loading openmpi/4.0
  ERROR: Module evaluation aborted

Switching from intel/2019 to intel/2018
  WARNING: Reload of dependent openmpi/4.0 failed
  Unloading dependent: openmpi/4.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2018
[mod4 (homebrewed)]$ mpirun
mpirun: command not found

When we switch out the pgi compiler for the intel/2019 compiler, the openmpi module is automatically unloaded before the compiler switch and reload afterwards, so we end up with the correct build of openmpi for the newly loaded intel/2019 compiler. If one then switches out intel/2019 replacing it with intel/2018, the openmpi module is first unloaded, the compilers are switched, and as there is no openmpi/4.0 build for intel/2018, a warning is given and the openmpi module is left unloaded. Since the user never specifically requested version 4.0 of openmpi (it was defaulted in the original module load of openmpi as that was the latest version available for pgi/19.4), it would have been nicer if on the switch of intel compiler versions the reload only attempted a module load openmpi instead of module load openmpi/4.0, but nevertheless this well behaved. The openmpi module is dropped with a warning and the user has a consistent set of modules loaded.

We note that the modulefile is able to default the compiler, so when we attempt to load openmpi without having previously loaded a compiler, as in

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load openmpi/3.1
Loading openmpi/3.1
  Loading requirement: gcc/8.2.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) openmpi/3.1
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/3.1, gcc/8.2.0)
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load gcc/8.2.0
[mod4 (homebrewed)]$ module load openmpi

**** ERROR *****:
openmpi/4.0 does not appear to be built for compiler gcc/8.2.0
Please select a different openmpi version or different compiler.


Loading openmpi/4.0
  ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0

it will default to the default compiler, gcc/8.2.0. Note however, that if one does not specify version 3.1 of openmpi, it will still default to 4.0 and fail to load as there is no build of openmpi/4.0 for gcc/8.2.0.

The situation is similar for foo:

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load pgi/19.4
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (pgi/19.4, nompi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module load openmpi/3.1
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/3.1   3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (pgi/19.4, openmpi/3.1)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module unload openmpi
[mod4 (homebrewed)]$ module switch --auto pgi intel/2019
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module load intelmpi
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) intelmpi/default   3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, intelmpi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module switch --auto intelmpi mvapich/2.3.1
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) mvapich/2.3.1   3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module switch --auto mvapich openmpi/4.0
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0   3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, openmpi/4.0)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module unload openmpi
[mod4 (homebrewed)]$ module switch --auto intel/2019 gcc/9.1.0
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module load mvapich/2.3.1
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) mvapich/2.3.1   3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module switch --auto mvapich openmpi/4.0
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) openmpi/4.0   3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (gcc/9.1.0, openmpi/4.0)

Again, one can load a compiler without an MPI library to get the non-MPI version of foo, or a compiler and MPI library to get the MPI version. The dummy intelmpi modulefile is used to allow one to indicate that the Intel MPI library is desired. The automated module handling mode can again allow the switch functionality work properly, as in

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load pgi/18.4
[mod4 (homebrewed)]$ module load openmpi/3.1
[mod4 (homebrewed)]$ module load foo/1.1
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) pgi/18.4   2) openmpi/3.1   3) foo/1.1
[mod4 (homebrewed)]$ foo
foo 1.1 (pgi/18.4, openmpi/3.1)
[mod4 (homebrewed)]$ module switch --auto pgi intel/2018
Switching from pgi/18.4 to intel/2018
  Unloading dependent: foo/1.1 openmpi/3.1
  Reloading dependent: openmpi/3.1 foo/1.1
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2018   2) openmpi/3.1   3) foo/1.1
[mod4 (homebrewed)]$ foo
foo 1.1 (intel/2018, openmpi/3.1)
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/3.1, intel/2018)
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load intel/2019
[mod4 (homebrewed)]$ module load foo
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (homebrewed)]$ module load openmpi
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) foo/2.4   3) openmpi/4.0
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, nompi)

Here we note a deficiency in the switch support as compared to Flavours Strategy. In the last example after loading intel/2019 and foo, we have the non-MPI build of foo as expected. However, upon subsequently loading the openmpi module, we still have the non-MPI version of foo loaded, as evidenced by the output of the dummy foo command. I.e., the foo package was not automatically reloaded, as there was no prereq in the foo modulefile on an MPI library (as in the non-MPI build there is no MPI library to prereq). Also note that module list does not really inform one of this fact.

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load foo

**** ERROR *****:
foo/2.4 does not appear to be built for compiler gcc/8.2.0 and MPI nompi
Please select a different openmpi version or different compiler/MPI library combination.


Loading foo/2.4
  ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
No Modulefiles Currently Loaded.
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load foo/1.1
Loading foo/1.1
  Loading requirement: gcc/8.2.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) foo/1.1
[mod4 (homebrewed)]$ foo
foo 1.1 (gcc/8.2.0, nompi)
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load pgi/18.4
[mod4 (homebrewed)]$ module load foo

**** ERROR *****:
foo/2.4 does not appear to be built for compiler pgi/18.4 and MPI nompi
Please select a different openmpi version or different compiler/MPI library combination.


Loading foo/2.4
  ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) pgi/18.4

Above, we see once more that the compiler can be defaulted, but that the defaulting mechanism is not smart enough to default the version of foo based on the compiler loaded (or defaulted to).

The situation with bar is basically the same; with a compiler and simd module loaded, the environment for the appropriate build of bar is loaded when you module load bar.

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load gcc/9.1.0
[mod4 (homebrewed)]$ module load simd/avx2
[mod4 (homebrewed)]$ module load bar/5.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) simd/avx2   3) bar/5.4
[mod4 (homebrewed)]$ bar
bar 5.4 (gcc/9.1.0, avx2)
[mod4 (homebrewed)]$ module unload bar
[mod4 (homebrewed)]$ module switch --auto simd simd/avx
[mod4 (homebrewed)]$ module load bar/5.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) simd/avx   3) bar/5.4
[mod4 (homebrewed)]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4 (homebrewed)]$ module unload bar
[mod4 (homebrewed)]$ module switch --auto simd simd/sse4.1
[mod4 (homebrewed)]$ module load bar/5.4

**** ERROR *****:
foo/5.4 does not appear to be built for compiler gcc/9.1.0 and simd/sse4.1
Please select a different openmpi version or different compiler/simd combination.


Loading bar/5.4
  ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) simd/sse4.1

And an error is generated if there is no build for that combination of compiler and simd. The automatic handling of modules again allows the switch command to work as expected:

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load gcc/9.1.0
[mod4 (homebrewed)]$ module load simd/avx
[mod4 (homebrewed)]$ module load bar/5.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) simd/avx   3) bar/5.4
[mod4 (homebrewed)]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4 (homebrewed)]$ module switch --auto simd simd/avx2
Switching from simd/avx to simd/avx2
  Unloading dependent: bar/5.4
  Reloading dependent: bar/5.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) simd/avx2   3) bar/5.4
[mod4 (homebrewed)]$ bar
bar 5.4 (gcc/9.1.0, avx2)

and both the simd level and compiler can be defaulted, but one still has to choose a version of bar which supports the defaults.

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load bar

[INFO] Setting simd to simd/sse4.1

**** ERROR *****:
foo/5.4 does not appear to be built for compiler gcc/8.2.0 and simd/sse4.1
Please select a different openmpi version or different compiler/simd combination.


Loading bar/5.4
  ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
No Modulefiles Currently Loaded.
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load bar/4.7

[INFO] Setting simd to simd/sse4.1

Loading bar/4.7
  Loading requirement: gcc/8.2.0 simd/sse4.1
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) simd/sse4.1   3) bar/4.7
[mod4 (homebrewed)]$ bar
bar 4.7 (gcc/8.2.0, sse4.1)

Summary of Homebrewed flavors strategy

  • The Automated module handling feature (introduced in Environment Modules 4.2) allows for the switching out of a dependency (e.g. a compiler) to cause the reload of all modulefiles depending on it. Without the automated module handling (as is default for 4.x, and the only option for 3.x), the switching of a compiler only changes the compiler and leaves the modulefiles that depend on the compiler unchanged.

  • The various Tcl procedures make it somewhat easy to determine which compiler, MPI, etc. modules have been loaded and set the paths appropriately. Not as elegant or easy to use as Flavours Strategy, but not difficult

  • Like the Flavours package, the module avail output is concise, only e.g. giving package name and version rather than listing a separate modulefile for each combination of package, version, compiler+version, MPI library+version, etc.

  • Modules will fail with an error message if user tries to load a package which was not built for the values of the dependency packages loaded.

  • With Automated module handling mode, the dependencies of a module asked for load could automatically be loaded. But it requires that these dependencies are clearly specified with mention of the versions for which a build is available.

  • It does not include a mechanism for more intelligent defaulting. I.e., if an user requests to load a package without specifying the version desired, the version will be defaulted to the latest version (or whatever the .modulerc file specifies) without regard for which versions support the versions of the compiler and other prereq-ed packages the user has loaded.

  • Note that future releases of Environment Modules will introduce additional mechanisms to the Automated module handling mode, which will improve the user experience on such Homebrewed flavors setup.

Modulerc-based Strategy

The previous two strategies used additional code in the modulefile to determine which compiler, etc. was loaded and adjust the values for PATH, etc. accordingly. The Modulerc-based strategy instead uses .modulerc files to direct the module command to the proper modulefile depending on what compiler, etc. was previously loaded. Because of this, there are a number of differences in behavior and what is seen by the user, most notably many, many more modulefiles. Whether this is good or bad is a matter of taste.

Implementation

Whereas the Homebrewed Flavors Strategy had the modulefile invoke a Tcl procedure to determine which, if any, version of a module like a compiler was loaded and then adjust paths, the Modulerc-based strategy instead uses the same Tcl procedures to default the modulefile which will be loaded. This implies that there is a distinct modulefile for every build of the package, and an immediate consequence is that this strategy has many more modulefiles than the others. We make use of the techniques in the cookbook Tips for Code Reuse in Modulefiles to reduce the total amount of code; the actual modulefiles for each build are typically small stubfiles defining a couple of Tcl variables and then sourcing a common script (unique to each package) which does all the real work.

The modules will be named with components for the different dependencies, so the one for openmpi version 4.0 built with gcc version 9.1.0 would be openmpi/4.0/gcc/9.1.0; similarly the module for foo version 1.1 built for pgi version 18.4 and mvapich 2.1 would be foo/1.1/pgi/18.4/mvapich/2.1.

The .modulerc files themselves are not trivial, but these can generally be written in a generic fashion, usable by multiple packages, and can just be symlinked to the appropriate locations. We define five such files which can be linked as .modulerc:

Two of these files can be linked in the module tree at various places as .modulerc for defaulting the compiler. One to default to the family portion of the compiler (e.g. gcc, intel, or pgi), and one for the version. For the family portion of the compiler, we have the file modulerc.select_compiler_family as below:

#%Module
# Choose compiler family
#
# This check if any compiler was previously "module loaded", or if no compiler
# previously loaded, will use the default compiler.  Either way, it will check
# if there is a subdirectory matching the compiler family name, and if so
# will default to that.
#
# If no subdirectory matching compiler family name is found, we just return
# without defaulting.  The modulecmd will default based on its internal rules,
# and it is up to the resulting module file to either load an appropriate
# compiler if no compiler is loaded or to abort with appropriate error
# messages if it wants an incompatible compiler.
#
# Usage:
# In most cases, can simply symlink .modulerc to this file
#
# In more complicated cases, .modulerc can source this file, and can
# then test the variable _did_default, which will be true if we set
# a default for modules for the next level, or false otherwise (in which
# case your .modulerc can set one)

# Source some required Tcl procedures here.  Hack for cookbook
# making use of environment variable for location.
# In production, this should ideally be in a site config file.
# At minimum, hardcode the path
set rootdir $::env(MOD_GIT_ROOTDIR)
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

set moduledir [file dirname $ModulesCurrentModulefile]
set baseComponent [ file tail $moduledir ]

# Get the currently loaded compiler, or default comp.  Don't module load/prereq it
set fullCompilerTag [ GetLoadedCompiler 1 1 ]
set tmpCompTag [ GetPackageFamilyVersion $fullCompilerTag ]
set compilerFamily [ lindex $tmpCompTag 0 ]
set compFamList "$compilerFamily"

# Allow either gnu or gcc for GCC compilers
if { $compilerFamily eq {gcc} || $compilerFamily  eq {gnu} } {
   set compFamList "gnu gcc"
}

# _did_default will be true if we actually default something
# Useful in case a .modulerc sources us, and wants to set a default if we
# did not
set _did_default false

if { $compilerFamily ne {} } {
   # Default to the family of currently loaded compiler
   set firstChild [ FirstChildModuleInList $compFamList ]
   if { $firstChild ne {} } {
      # Compiler family found, so default to it
      module-version $baseComponent/$firstChild default
      set _did_default true
   }
}

The file starts by sourcing a set of useful Tcl procedures. For the purpose of the example for this cookbook this is done based on the MOD_GIT_ROOTDIR environment variable. If you were to use this in production, it is recommended that the Tcl procedures be placed in a site configuration script and exposed to modulefiles via the techniques described in the cookbook Expose procedures and variables to modulefiles. At the minimum, it is recommended to hardcode the path to the common_utilities.tcl file.

The modulerc script then determines the directory it is in using the Tcl variable ModulesCurrentModulefile. It then uses the GetLoadedCompiler Tcl procedure (which was discussed in the section on the Homebrewed Flavors Strategy. We then parse the resulting module name into family and version pieces (we will discuss the procedure GetPackageFamilyVersion later; for now suffice to say it takes a module name and returns a Tcl list with family and version).

We do some trickery to support the use of either gcc or gnu as family names for the GNU compiler suite, and then see if there is a directory or modulefile underneath the directory containing the .modulerc file, and if so defaults to it. For this purpose, we use the Tcl procedure FirstChildModuleInList (defined in common_utilities.tcl). This and a related procedure are defined as:

#--------------------------------------------------------------------
# FirstChildModuleInList
#
# Given a list of names of modulefile components, returns the
# first one found in the directory of the current modulefile
# (typically a .modulerc file).  Returns empty string if none
# of them were found.
# NOTE: we do NOT check for module magic signature or even that symlink
# targets exist.
proc FirstChildModuleInList { modlist } {
   global ModulesCurrentModulefile
   # Get directory for current modulefile
   set moduledir [ file dirname $ModulesCurrentModulefile ]
   # See if any names in $modlist in moduledir
   foreach mcomp $modlist {
      if [ file exists $moduledir/$mcomp ] { return $mcomp }
   }
   # Nothing found
   return
}

#--------------------------------------------------------------------
# ChildModuleExists
#
# Takes a module path component, and returns true if that path component
# exists beneath the current level, i.e. in the directory from which
# this .modulerc file was called.
# NOTE: we do NOT check for module magic signature or even that symlink
# targets exist.
proc ChildModuleExists { pathcomponent } {
   global ModulesCurrentModulefile
   # Get directory for current modulefile
   set moduledir [ file dirname $ModulesCurrentModulefile ]
   # See if file exists in moduledir
   return [ file exists $moduledir/$pathcomponent ]
}

and basically make use of the Tcl variable ModulesCurrentModulefile. They are limited, however, in that they will not work properly if the module is split across multiple modulepaths. That is, ChildModuleExists and FirstChildModuleInList will only detect children in the same directory as the modulerc file they are called from; e.g., if two paths in the MODULEPATH both have directories for openmpi, one with an intel subdirectory and one with a pgi subdirectory and the modulerc invoking ChildModuleExists, the ChildModuleExists procedure will not see the intel directory.

The modulerc.select_compiler_version file is similar,

#%Module
# Choose compiler version
#
# This check if any compiler was previously "module loaded", or if no compiler
# previously loaded, will use the default compiler.  Either way, it will check
# if there is a subdirectory matching the compiler family name, and if so
# will default to that.
#
# If no subdirectory matching compiler family name is found, we just return
# without defaulting.  The modulecmd will default based on its internal rules,
# and it is up to the resulting module file to either load an appropriate
# compiler if no compiler is loaded or to abort with appropriate error
# messages if it wants an incompatible compiler.
#
# Usage:
# In most cases, can simply symlink .modulerc to this file
#
# In more complicated cases, .modulerc can source this file, and can
# then test the variable _did_default, which will be true if we set
# a default for modules for the next level, or false otherwise (in which
# case your .modulerc can set one)

# Source some required Tcl procedures here.  Hack for cookbook
# making use of environment variable for location.
# In production, this should ideally be in a site config file.
# At minimum, hardcode the path
set rootdir $::env(MOD_GIT_ROOTDIR)
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

set moduledir [file dirname $ModulesCurrentModulefile]
set parentdir [ file tail $moduledir ]

# _did_default will be true if we actually default something
# Useful in case a .modulerc sources us, and wants to set a default if we
# did not
set _did_default false

# Get the currently loaded compiler or default, but do not load/prereq it
set compilerTag [ GetLoadedCompiler 1 1 ]

if { $compilerTag ne {} } {
   # Compiler loaded or defaulted
   # Make sure it matches our parent dir
   set tmpCompTag [ GetPackageFamilyVersion $compilerTag ]
   set compilerFamily [ lindex $tmpCompTag 0 ]
   set compilerVersion [ lindex $tmpCompTag 1 ]

   # Convert any gnus to gccs
   set tmp1 $compilerFamily
   if { $tmp1 eq {gnu} } { set tmp1 gcc }
   set tmp2 $parentdir
   if { $tmp2 eq {gnu} } { set tmp2 gcc }

   if { $tmp1 eq $tmp2 } {
      # If reach here, our parent dir agrees with
      # the family of our loaded/defaulted compiler
      if { $compilerVersion ne {} } {
         # We have a version, does a modulefile/dir exist for it
         if [ ChildModuleExists $compilerVersion ] {
            # There is a modulefile/dir for this compiler version, default to it
            module-version $compilerFamily/$compilerVersion default
            set _did_default true
         }
      }
   }
}

Again, we source the common_utitilies.tcl file and use ModulesCurrentModulefile to get the directory in which the .modulerc script resides. It then uses GetLoadedCompiler to determine what if any compiler was loaded. If one was, we split into family and version, and ensure that the directory containing the .modulerc file matches the family name. If so, it checks if there is a directory or modulefile in that directory matching the version name, and if so defaults to it.

For both of the .modulerc scripts we examined, we note that in case of errors, etc., the script just exits without defaulting. One might be tempted to have the .modulerc script actually return an error in such cases (perhaps using module-info mode to only have it output during load operations). This, however, runs into issues:

  • For Environment Modules 3.x: for some reason, when modulecmd processes .modulerc scripts, module-info mode load always returns true. Because of this, users would get spurious errors when doing a module avail if the .modulerc scripts error-ed, because they all get processed regardless of what compiler is loaded.

  • For versions 4.x: the modulecmd process tends to process all .modulerc files underneath the package root during a load, which will similarly generate spurious errors if the .modulerc scripts throw errors.

Because of this, if an user explicitly requests a modulefile that conflicts with the loaded compiler (e.g. the user does a module load pgi/19.4 followed by a module load openmpi/4.0/intel/2019), the modulefile needs to detect this and error appropriately. A similar situation can happen if one of the .modulerc scripts fail to default, presumably because there is no appropriate build for the requested package; the modulecmd will default to something, but likely not what is wanted. To facilitate this, we define the Tcl procedure LoadedCompilerMatches:

#--------------------------------------------------------------------
# LoadedCompilerMatches:
#
# Takes the tag of the desired compiler $wanted (in form of FAMILY/VERSION)
# Checks if any compiler is loaded, and will throw error if one is
# loaded and it does not match $wanted.
# If compilers match, it will prereq the compiler if $requireIt set
# If no compiler is loaded, will load if optional boolean parameter
# $loadIt is set (default unset), and prereq if $requireIt is set, and
# then return.
# Option parameters:
# requireIt: boolean, default false. If set, require $wanted
# loadIt: boolean, default false.  If set, load $wanted if no compiler
#	already loaded
# modTag: string, defaults go [module-info specified].  Used in error messages
proc LoadedCompilerMatches { wanted {requireIt 0} { loadIt 0 } {modTag {} } } {
   # Default modTag
   if { $modTag eq {} } { set modTag [ module-info specified ] }
   # If no compiler given in $wanted, just return
   if { $wanted eq {} } { return }

   # Get loaded compiler (w/out defaults)
   set loaded [ GetLoadedCompiler 0  0 ]
   # If no compiler is loaded, then require it if asked, and return
   if { $loaded eq {} } {
      if { $loadIt } {
         RequireCompiler $wanted
         if { $requireIt } { prereq $wanted }
      }
      return
   }

   # Have a loaded compiler, split into family and version
   set tmpLoaded [ GetPackageFamilyVersion $loaded ]
   set loadedFam [ lindex $tmpLoaded 0 ]
   set loadedVer [ lindex $tmpLoaded 1 ]
   # Always use 'gcc' not 'gnu'
   if { $loadedFam eq {gnu} } { set loadedFam gcc }

   # Split wanted into family and version
   set tmpWanted [ split $wanted / ]
   set wantedLen [ llength $tmpWanted ]
   #Nothing to do if no components to wanted
   if { $wantedLen < 1 } { return }

   set wantedFam [ lindex $tmpWanted 0 ]
   # Always use 'gcc' not 'gnu'
   if { $wantedFam eq {gnu} } { set wantedFam gcc }

   # Ensure families match
   if { $loadedFam ne $wantedFam } {
      PrintLoadError "Compiler Mismatch
Package $modTag does not appear to be built for currently
loaded compiler $loaded."
   }

   # OK, families match
   # If no version specified in $wanted, we are basically done
   if { $wantedLen < 2 } {
      if { $requireIt } { prereq $wanted }
      return
   }

   # Ensure versions match
   set wantedVer [ lindex $tmpWanted 1 ]
   if { $loadedVer eq $wantedVer } {
      if { $requireIt } { prereq $wanted }
      return
   }
   # Versions don't match
   PrintLoadError "Compiler Mismatch
Package $modTag does not appear to be built for currently
loaded compiler $loaded."
}

The procedure takes a string for the family and version of the compiler that the modulefile expects, and ensures that if a compiler is loaded, they match. If they match, the procedure just returns, and if not, it spits out an error indicating a compiler mismatch. It also takes optional boolean flags: requireIt to determine if the procedure should prereq the compiler before returning, and loadIt to determine if, when no compiler was previously loaded, whether the procedure should load it (using RequireCompiler so it can properly handle the default compiler specially if needed). The requireIt flag should generally be set for Environment Modules 4.x (as that will allow the automatic handling of modules to properly recognize dependencies, even though as we will see that does not work too well for this strategy as yet), and should not be set for versions 3.x (as the loading of modules with loadIt does not occur until after the prereq, causing module loads to fail).

The resulting modulefile for something depending only on the compiler, using mvapich as an example, then would look like:

# Common modulefile for mvapich
# Using "modulerc based" strategy
# Expects the following variables to have been
# previously defined:
#	version: version of mvapich
#	compilerTag: the compiler to use

# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root

# Also get location of and load common procedures
# This is a hack for the cookbook examples, in production
# one should either
# 1) declare the procedures in a site config file (preferred)
# 2) hardcode the path to $tcllibdir and common_utilities.tcl
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

proc ModulesHelp { } {
   global version compilerTag
   puts stderr "
mvapich: Test dummy version of mvapich $version

mvapich version: $version
Compiler: $compilerTag

For testing packages depending on compilers/MPI

"
}

module-whatis "Dummy mvapich $version (built for $compilerTag)"

# Make sure loadedCompiler matches what we want.  And
# load compiler if no compiler loaded
LoadedCompilerMatches $compilerTag 1 1
# For Env Modules 3.x, set requireIt to 0
#LoadedCompilerMatches $compilerTag 0 1

# Compute the installation prefix
set pkgroot $swroot/mvapich
set vroot $pkgroot/$version
set prefix $vroot/$compilerTag

# Set environment variables
setenv MPI_DIR $prefix

set bindir $prefix/bin
set libdir $prefix/lib
set incdir $prefix/include

prepend-path PATH            $bindir
prepend-path LIBRARY_PATH    $libdir
prepend-path LD_LIBRARY_PATH $libdir
prepend-path CPATH           $incdir

Basically, the modulefile knows what compiler it wants (in the above example, that is set by the stubfile and passed as the Tcl variable compilerTag into the common script above), and then calls the LoadedCompilerMatches procedure above to ensure the loaded compiler matches what the modulefile wants (and to load it if no compiler is loaded).

So the mvapich directory in the MODULEPATH consists of subdirectories for each version of mvapich supported (e.g. 2.1 and 2.3.1). In each of the version subdirectories, we symlink modulerc.select_compiler_family as .modulerc, and create an additional layer of subdirectories, one for each compiler family for which the version of mvapich is built (e.g. gcc, intel, pgi). In each compiler family directory under each mvapich version, we symlink modulerc.select_compiler_version as .modulerc, and create a stub modulefile named for the compiler version. The stub modulefile then defines some Tcl variables for the version of mvapich and the compiler family/version, and sources the common file above. E.g., for mvapich/2.3.1/intel/2019, the stubfile would look like

#%Module
# Dummy modulefile for mvapich

set version 2.3.1
set compilerTag intel/2019
set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/../../common

So the mvapich directory in MODULEPATH would have a structure like

mvapich
├── 2.1
│   ├── gcc
│   │   ├── 8.2.0
│   │   ├── 9.1.0
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── intel
│   │   ├── 2018
│   │   ├── 2019
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── .modulerc -> symlink to modulerc.select_compiler_family
│   └── pgi
│       ├── 18.4
│       ├── 19.4
│       └── .modulerc -> symlink to modulerc.select_compiler_version
├── 2.3.1
│   ├── gcc
│   │   ├── 9.1.0
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── intel
│   │   ├── 2019
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── .modulerc -> symlink to modulerc.select_compiler_family
│   └── pgi
│       ├── 19.4
│       └── .modulerc -> symlink to modulerc.select_compiler_version
└── common

With this directory structure, an user can load a compiler and then module load a specific version of mvapich, and if a build of that version of mvapich exists for that compiler, the environment will be set up properly, and otherwise an error reported. However, if the user module loads mvapich without specifying a version, it will simply default the version to the latest version of mvapich available, regardless of whether there is a build of that version of mvapich for the loaded compiler. Ideally, we would like it so that if one issues a module load of a package without specifying a version, it should load the latest version available that is compatible with any loaded compilers or other modules.

A simple change can enable that. We add a symlink to modulerc.select_compiler_family as .modulerc directly under the mvapich directory, and create an additional subdirectory under the mvapich directory for each compiler family. We symlink modulerc.select_compiler_version as .modulerc under each compiler family subdirectory, along with a subdirectory for each version of that compiler family that a version of mvapich is built for. We then symlink the various stub modulefiles under the mvapich/VERSION into the corresponding mvapich/COMPILER_FAMILY/COMPILER_VERSION directory, with the symlink's name being the mvapich version number. So the mvapich directory tree will now look like:

mvapich
├── 2.1
│   ├── gcc
│   │   ├── 8.2.0
│   │   ├── 9.1.0
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── intel
│   │   ├── 2018
│   │   ├── 2019
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── .modulerc -> symlink to modulerc.select_compiler_family
│   └── pgi
│       ├── 18.4
│       ├── 19.4
│       └── .modulerc -> symlink to modulerc.select_compiler_version
├── 2.3.1
│   ├── gcc
│   │   ├── 9.1.0
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── intel
│   │   ├── 2019
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── .modulerc -> symlink to modulerc.select_compiler_family
│   └── pgi
│       ├── 19.4
│       └── .modulerc -> symlink to modulerc.select_compiler_version
├── common
├── gcc
│   ├── 8.2.0
│   │   └── 2.1 -> symlink to ../../2.1/gcc/8.2.0
│   ├── 9.1.0
│   │   ├── 2.1 -> symlink to ../../2.1/gcc/9.1.0
│   │   └── 2.3.1 -> symlink to ../../2.3.1/gcc/9.1.0
│   └── .modulerc -> symlink to modulerc.select_compiler_version
├── intel
│   ├── 2018
│   │   └── 2.1 -> symlink to ../../2.1/intel/2018
│   ├── 2019
│   │   ├── 2.1 -> symlink to ../../2.1/intel/2019
│   │   └── 2.3.1 -> symlink to ../../2.3.1/intel/2019
│   └── .modulerc -> symlink to modulerc.select_compiler_version
├── .modulerc -> symlink to modulerc.select_compiler_family
└── pgi
    ├── 18.4
    │   └── 2.1 -> symlink to ../../2.1/pgi/18.4
    ├── 19.4
    │   ├── 2.1 -> symlink to ../../2.1/pgi/19.4
    │   └── 2.3.1 -> symlink to ../../2.3.1/pgi/19.4
    └── .modulerc -> symlink to modulerc.select_compiler_version

When one attempts to module load mvapich without specifying a version, the .modulerc file will default to the family of the loaded compiler (or the default compiler if none loaded). It then descends into the corresponding compiler family directory, where the .modulerc there will default to the version of the loaded or defaulted compiler. After descending into the compiler version directory, there is no .modulerc, so the standard modulecmd defaulting mechanism will select the latest version.

We now have two ways to refer to the same build of mvapich, namely one using the traditional form

PACKAGE/PKGVERSION/COMPILER_FAMILY/COMPILER_VERSION
(e.g. mvapich/2.3.1/pgi/19.4)

and a new one giving the compiler family and version immediately after package name, with the version of the package coming last

PACKAGE/COMPILER_FAMILY/COMPILER_VERSION/PKGVERSION
(e.g. mvapich/pgi/19.4/2.3.1)

These both refer to the same build of the package (e.g. version 2.3.1 of mvapich, built for version 19.4 of the PGI compiler suite). Indeed, through the judicious use of symlinks, these actually refer to the modulefile on disk, but with two distinct names depending on the path used to get to it. In general, we allow for modules to be named with multiple dependency packages and/or flags if doing so would make it easier for a user to default to the correct package.

This is the primary reason for the procedure GetPackageFamilyVersion --- it splits a module name into components and takes the first component as the family name. It then examines the second component, and if it matches the name of a known package (like a compiler family), it uses the last component as the version. Otherwise, the second component is assumed to be the version. The code is as follows:

#--------------------------------------------------------------------
# GetPackageFamilyVersion
#
# Given the fully qualified spec for a modulefile, returns a list
# with the package family name and version.
#
# E.g., for foo/1.1/gcc/8.2.0/openmpi/3.1 would return {foo 1.1}
# But also handles stuff like foo/gcc/8.2.0/openmpi/3.1/1.1 (returning
# again {foo 1.1})
proc GetPackageFamilyVersion { tag } {
   # Return empty list if given empty tag
   if { $tag eq {} } { return {} }

   # Split tag into components
   set components [ split $tag / ]
   set compLen [ llength $components ]
   if { $compLen < 1 } { return {} }

   # Family should always be first
   set family [ lindex $components 0 ]
   if { $compLen < 2 } { return $family }

   # First guess, version immediately follows family
   set version [ lindex $components 1 ]
   # Check if it matches known non-version stuff following family
   # Compilers, MPIs, other things branch on
   set nonVers { gnu gcc pgi intel openmpi ompi intelmpi impi mvapich
      mvapich2 avx avx2 sse4.1 sse3 python perl hdf hdf4 hdf5 }
   set tmp [ lsearch $nonVers $version ]
   if { $tmp > -1 } {
      # The component immediately following family was NOT a version
      # Use last component as version
      set version [ lindex $components end ]
   }
   return  "$family $version"
}

Similarly, there are two corresponding .modulerc scripts for defaulting the MPI library: one to default the family (e.g. openmpi, mvapich, intelmpi) and one to default the version. The family version, shown below:

#%Module
# Choose MPI family
#
# This check if any MPI lib was previously "module loaded"
#
# If no MPI library was previously module loaded (and intel compiler suite
# was not loaded, see below) and nompiFlag is set (default is set), then
# if a "nompi" subdirectory exists, we will default to that.
# If nompiFlag is not set, then we just return w/out setting any default
# (allowing modulecmd to default on its own.  It is the responsibility
# and the resulting modulefile to lado/require the MPI lib it needs or err
# as appropriate).
#
# If an MPI library was loaded, then we check for a subdirectory named after
# the family of the loaded MPI library, and if such exists, default to that.
# If it does not exist, we return w/out defaulting, allowing modulecmd to
# default according to its own rules.  The resulting modulefile is responsible
# for aborting/complaining about the MPI library mismatch.
#
# Special handling is needed for intel MPI, as it does not get explicitly
# loaded if the Intel compiler suite is being used.  So if no MPI library
# is loaded, we check to see if an Intel compiler was loaded.  If so, we
# check for an "intelmpi" or similar subdirectory, and if found default to
# that.  If not found, we proceed as above for the no MPI library case
# (defaulting to "nompi" if found and nompiFlag set, or just returning w/out
# defaulting otherwise).
#
# Usage:
# In most cases, can simply symlink .modulerc to this file
#
# In more complicated cases, .modulerc can source this file, and can
# then test the variable _did_default, which will be true if we set
# a default for modules for the next level, or false otherwise (in which
# case your .modulerc can set one)

# Source some required Tcl procedures here.  Hack for cookbook
# making use of environment variable for location.
# In production, this should ideally be in a site config file.
# At minimum, hardcode the path
set rootdir $::env(MOD_GIT_ROOTDIR)
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

#Default nompiFlag to set
if ![info exists nompiFlag] { set nompiFlag true }

# _did_default will be true if we actually default something
# Useful in case a .modulerc sources us, and wants to set a default if we
# did not
set _did_default false

set moduledir [file dirname $ModulesCurrentModulefile]
set baseComponent [ file tail $moduledir ]

# Get the currently loaded MPI library
set fullMpiTag [ GetLoadedMPI 0 ]
set tmpMpiTag [ GetPackageFamilyVersion $fullMpiTag ]
set mpiFamily [ lindex $tmpMpiTag 0 ]

if { $mpiFamily eq {} } {
   # No MPI module loaded
   # What compiler is loaded (default from path if needed, but not GetDefaultCompiler)
   set fullComp [ GetLoadedCompiler 1 0 ]
   set tmpComp [ GetPackageFamilyVersion $fullComp ]
   set compFam [ lindex $tmpComp 0 ]
   if { $compFam eq {intel} } {
      # OK, no MPI explicitly loaded, but using Intel compiler
      # So set mpiFamily to first found in list below
      set mpiList {intelmpi impi intel intelmpi-mt impi-mt intel-mp}
      set mpiFamily [ FirstChildModuleInList $mpiList ]
   }
}

if { $mpiFamily eq {} } {
   # No MPI lib was explicitly loaded, and if intel compiler was loaded,
   # did not find a subdir matching intelmpi or one of its variants
   # Looks for a "nompi" dir if $nompiFlag is set
   if { $nompiFlag } {
      if [ ChildModuleExists nompi ] {
         # nompi subdir exists, default to it
         module-version $baseComponent/nompi default
         set _did_default true
      }
   }
} else {
   # Either MPI lib was explicitly loaded, or implied intelmpi
   # Default to the subdir named after MPI family, if it exists
   if [ ChildModuleExists $mpiFamily ] {
      module-version $baseComponent/$mpiFamily default
      set _did_default true
   }
}

is similar to the corresponding compiler version, but contains additional logic to handle the intelmpi case. We treat intelmpi specially, because in our example here we assume the environment for Intel MPI is setup properly when the user loads the intel module. If no MPI library is explicitly loaded, but the intel module is loaded, than we look for a subdirectory named intelmpi (or one of its aliases) and default to it if it exists. Otherwise, if no MPI module is loaded, it looks for and will default to a directory or modulefile named nompi if it exists.

The modulerc.select_mpi_version script is also similar to its compiler counterpart,

#%Module
# Choose MPI version
#
# This check if any MPI lib was previously "module loaded"
#
# If an MPI library was loaded, and the family of loaded MPI lib matches
# the name of the parent dir for this .modulerc, and there is a subdir
# of that directory matching the MPI version, then we default to that
# subdir.
#
# If no MPI library was previously module loaded, or the loaded MPI family
# does not match the name of the parent directory, or no subdir matching
# the MPI version is found, then we just return w/out defaulting.
# The modulecmd will then default according to its own rules, and it is
# up to the resulting modulefile to either load the appropriate MPI library
# or abort/complain about an MPI mismatch as appropriate.
#
# NOTE: no special handling is needed for intelmpi, as we assume that if
# intelmpi is selected because the intel compiler was loaded, then we
# are to use the intelmpi that shipped with the compiler suite, and so
# there is no needed for additional versioning of the intelmpi beneath
# the intelmpi subdir.  If intelmpi is being used with a non-intel compiler,
# then it is assumed intelmpi was explicitly loaded.
#
# Usage:
# In most cases, can simply symlink .modulerc to this file
#
# In more complicated cases, .modulerc can source this file, and can
# then test the variable _did_default, which will be true if we set
# a default for modules for the next level, or false otherwise (in which
# case your .modulerc can set one)

# Source some required Tcl procedures here.  Hack for cookbook
# making use of environment variable for location.
# In production, this should ideally be in a site config file.
# At minimum, hardcode the path
set rootdir $::env(MOD_GIT_ROOTDIR)
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

# _did_default will be true if we actually default something
set _did_default false

set moduledir [file dirname $ModulesCurrentModulefile]
set parentDir [ file tail $moduledir ]

# Get the currently loaded compiler, w/out defaulting
set fullMpiTag [ GetLoadedMPI 0 ]

if { $fullMpiTag ne {} } {
   # We have an MPI module loaded
   set tmpMpiTag [ GetPackageFamilyVersion $fullMpiTag ]
   set mpiFamily [ lindex $tmpMpiTag 0 ]
   set mpiVersion [ lindex $tmpMpiTag 1 ]

   # Canonicalize intelmpi variants in parentDir and mpiFamily for comparison
   set intelList "intelmpi impi intel intelmpi-mt impi-mt intel-mt"
   set tmpMpiFamily $mpiFamily
   if { [lsearch $intelList $tmpMpiFamily] > -1 } {
	set tmpMpiFamily intelmpi
   }
   set tmpParentDir $parentDir
   if { [lsearch $intelList $tmpParentDir] > -1 } {
	set tmpParentDir intelmpi
   }

   if { $tmpParentDir eq $tmpMpiFamily } {
      # The loaded MPI lib family name matches parentDir
      # So see if have subdir matching version, and if so, default it
      if { $mpiVersion ne {} } {
         if [ ChildModuleExists $mpiVersion ] {
            module-version $mpiFamily/$mpiVersion default
            set _did_default true
         }
      }
   }
}

It checks if any MPI library was explicitly loaded, and if so it checks if the family of the loaded MPI module matches the name of the parent directory of this .modulerc file. If so, it checks for the existence of a subdirectory matching the version, and if found it defaults to it. There is no special handling needed for the nompi case, as since there is no version attached to the MPI library in the nompi case, this .modulerc is not present there. Similarly, for the case of Intel MPI when the Intel compiler suite is loaded, we expect the Intel MPI version to be that from the Intel compiler suite, so no version or this .modulerc is needed.

As the MPI modules depend themselves on the compiler, it is assumed that any package depending on the MPI libraries also depend on the compiler, and so will have a structure like the one described below for foo

foo
├── 1.1
│   ├── gcc
│   │   ├── 8.2.0
│   │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│   │   │   ├── mvapich
│   │   │   │   ├── 2.1
│   │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   │   ├── nompi
│   │   │   └── openmpi
│   │   │       ├── 3.1
│   │   │       └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── intel
│   │   ├── 2018
│   │   │   ├── intelmpi
│   │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│   │   │   ├── mvapich
│   │   │   │   ├── 2.1
│   │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   │   ├── nompi
│   │   │   └── openmpi
│   │   │       ├── 3.1
│   │   │       └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── .modulerc -> symlink to modulerc.select_compiler_family
│   └── pgi
│       ├── 18.4
│       │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│       │   ├── mvapich
│       │   │   ├── 2.1
│       │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│       │   ├── nompi
│       │   └── openmpi
│       │       ├── 3.1
│       │       └── .modulerc -> symlink to modulerc.select_mpi_version
│       └── .modulerc -> symlink to modulerc.select_compiler_version
├── 2.4
│   ├── gcc
│   │   ├── 9.1.0
│   │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│   │   │   ├── mvapich
│   │   │   │   ├── 2.3.1
│   │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   │   ├── nompi
│   │   │   └── openmpi
│   │   │       ├── 4.0
│   │   │       └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── intel
│   │   ├── 2019
│   │   │   ├── intelmpi
│   │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│   │   │   ├── mvapich
│   │   │   │   ├── 2.3.1
│   │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   │   ├── nompi
│   │   │   └── openmpi
│   │   │       ├── 4.0
│   │   │       └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── .modulerc -> symlink to modulerc.select_compiler_family
│   └── pgi
│       ├── 19.4
│       │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│       │   ├── nompi
│       │   └── openmpi
│       │       ├── 3.1
│       │       └── .modulerc -> symlink to modulerc.select_mpi_version
│       └── .modulerc -> symlink to modulerc.select_compiler_version
├── common
├── gcc
│   ├── 8.2.0
│   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│   │   ├── mvapich
│   │   │   ├── 2.1
│   │   │   │   └── 1.1 -> ../../../../1.1/gcc/8.2.0/mvapich/2.1
│   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   ├── nompi
│   │   │   └── 1.1 -> ../../../1.1/gcc/8.2.0/nompi
│   │   └── openmpi
│   │       ├── 3.1
│   │       │   └── 1.1 -> ../../../../1.1/gcc/8.2.0/openmpi/3.1
│   │       └── .modulerc -> symlink to modulerc.select_mpi_version
│   ├── 9.1.0
│   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│   │   ├── mvapich
│   │   │   ├── 2.3.1
│   │   │   │   └── 2.4 -> ../../../../2.4/gcc/9.1.0/mvapich/2.3.1
│   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   ├── nompi
│   │   │   └── 2.4 -> ../../../2.4/gcc/9.1.0/nompi
│   │   └── openmpi
│   │       ├── 4.0
│   │       │   └── 2.4 -> ../../../../2.4/gcc/9.1.0/openmpi/4.0
│   │       └── .modulerc -> symlink to modulerc.select_mpi_version
│   └── .modulerc -> symlink to modulerc.select_compiler_version
├── intel
│   ├── 2018
│   │   ├── intelmpi
│   │   │   └── 1.1 -> ../../../1.1/intel/2018/intelmpi
│   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│   │   ├── mvapich
│   │   │   ├── 2.1
│   │   │   │   └── 1.1 -> ../../../../1.1/intel/2018/mvapich/2.1
│   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   ├── nompi
│   │   │   └── 1.1 -> ../../../1.1/intel/2018/nompi
│   │   └── openmpi
│   │       ├── 3.1
│   │       │   └── 1.1 -> ../../../../1.1/intel/2018/openmpi/3.1
│   │       └── .modulerc -> symlink to modulerc.select_mpi_version
│   ├── 2019
│   │   ├── intelmpi
│   │   │   └── 2.4 -> ../../../2.4/intel/2019/intelmpi
│   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│   │   ├── mvapich
│   │   │   ├── 2.3.1
│   │   │   │   └── 2.4 -> ../../../../2.4/intel/2019/mvapich/2.3.1
│   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   ├── nompi
│   │   │   └── 2.4 -> ../../../2.4/intel/2019/nompi
│   │   └── openmpi
│   │       ├── 4.0
│   │       │   └── 2.4 -> ../../../../2.4/intel/2019/openmpi/4.0
│   │       └── .modulerc -> symlink to modulerc.select_mpi_version
│   └── .modulerc -> symlink to modulerc.select_compiler_version
├── .modulerc -> symlink to modulerc.select_compiler_family
└── pgi
    ├── 18.4
    │   ├── .modulerc -> symlink to modulerc.select_mpi_family
    │   ├── mvapich
    │   │   ├── 2.1
    │   │   │   └── 1.1 -> ../../../../1.1/pgi/18.4/mvapich/2.1
    │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
    │   ├── nompi
    │   │   └── 1.1 -> ../../../1.1/pgi/18.4/nompi
    │   └── openmpi
    │       ├── 3.1
    │       │   └── 1.1 -> ../../../../1.1/pgi/18.4/openmpi/3.1
    │       └── .modulerc -> symlink to modulerc.select_mpi_version
    ├── 19.4
    │   ├── .modulerc -> symlink to modulerc.select_mpi_family
    │   ├── nompi
    │   │   └── 2.4 -> ../../../2.4/pgi/19.4/nompi
    │   └── openmpi
    │       ├── 3.1
    │       │   └── 2.4 -> ../../../../2.4/pgi/19.4/openmpi/3.1
    │       └── .modulerc -> symlink to modulerc.select_mpi_version
    └── .modulerc -> symlink to modulerc.select_compiler_version

Although the directory tree above is somewhat lengthy, it is similar to the case for a module depending only on the compiler, but expanded to handle MPI dependencies as well. There is a subdirectory under foo for each version of foo. Each of these subdirectories have further subdirectories for the compiler families for which that version of foo was built, and under the subdirectories for each compiler family are subdirectories for each version of that compiler family for which foo was built. Underneath these are stub modulefiles for nompi (and for intel compilers, intelmpi) builds of the package, and subdirectories for each MPI family, containing stub modulefiles for each version of the MPI family the package was built for.

Additionally, there are subdirectories directly under foo for each compiler family the package was built for, with each having subdirectories for the version of the compiler. Beneath each of those are one or two more layers of subdirectories for the MPI family and version (the version layer is omitted in the nompi or intelmpi cases), underneath which are symlinks to the corresponding stub modulefile from the foo/FOOVERSION path.

By placing the appropriate .modulerc files (actually, symlinks to the correct modulerc file), when the user enters a partial modulename the modulecmd will descend this directory tree based on the previously loaded compiler and MPI library to get to correct build of the package, if it exists. If no MPI library was previously loaded, it will search for an intelmpi build if an intel compiler was loaded, or a nompi build if no intelmpi build found or a non-intel compiler was used. If only MPI builds were made, it will default (using standard module defaulting rules) to one of the MPI builds, and the appropriate MPI library will be loaded when the foo module gets loaded (via the LoadedMPIMatches Tcl procedure described below). If a compiler and MPI library were loaded but no build for that combination exists, again a modulefile will be defaulted to (using standard module defaulting rules), but an error will be generated by the one of the LoadedCompilerMatches or LoadedMPIMatches calls in the foo modulefile (see below).

The common code of the modulefile is fairly standard, as shown below

# Common stuff for "foo" modulefiles
# Using "modulerc based" strategy
#
# This file expects the following Tcl variables to have been
# previously set:
#	version: version of foo
#	compilerTag: compiler was built for
#	mpiTag: mpi was built for

# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root

# Also get location of and load common procedures
# This is a hack for the cookbook examples, in production
# one should either
# 1) declare the procedures in a site config file (preferred)
# 2) hardcode the path to $tcllibdir and common_utilities.tcl
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

proc ModulesHelp { } {
   global version compilerTag mpiTag
   puts stderr "
foo: Test dummy version of foo $version

For testing packages depending on compilers/MPI

Foo: $version
Compiler: $compilerTag
MPI: $mpiTag

"
}

module-whatis "Dummy foo $version ($compilerTag, $mpiTag)"

# Make sure loaded compiler and MPI match what we want.
# And load them if not already loaded
LoadedCompilerMatches $compilerTag 1 1
# But don't load intelmpi if compiler is intel
LoadedMpiMatches $mpiTag 1 1 1
# For Env Modules 3.x, set requireIt to false
#LoadedCompilerMatches $compilerTag 0 1
#LoadedMpiMatches $mpiTag 0 1 1

# Compute the installation prefix
set pkgroot $swroot/foo
set vroot $pkgroot/$version
set prefix $vroot/$compilerTag/$mpiTag

# Set environment variables
setenv FOO_DIR $prefix

set bindir $prefix/bin
set libdir $prefix/lib
set incdir $prefix/include

prepend-path PATH            $bindir
prepend-path LIBRARY_PATH    $libdir
prepend-path LD_LIBRARY_PATH $libdir
prepend-path CPATH           $incdir

conflict foo

The main difference from a standard modulefile is the inclusion of the invocations of LoadedCompilerMatches and LoadedMpiMatches. This is needed to ensure that the compiler and MPI for the modulefile being processed are compatible with any which are loaded. The LoadedMpiMatches procedure is defined by:

#--------------------------------------------------------------------
# LoadedMpiMatches:
#
# Takes the tag of the desired MPI library $wanted (in form of FAMILY/VERSION)
# Checks if any MPI is loaded, and will throw error if one is
# loaded and it does not match $wanted.
# If MPIs match, it will prereq the MPI lib if $requireIt set
# If no MPI is loaded, will load if optional boolean parameter
# $loadIt is set (default unset), and prereq if $requireIt is set, and
# then return.
# Option parameters:
# requireIt: boolean, default false. If set, will prereq $wanted
# loadIt: boolean, default false. If set, load $wanted
#	if no MPI is already loaded
# noLoadIntel: boolean, passed to RequireMPI.  If set,
#	will not attempt to load intelmpi variants if compiler is intel
# forceNoMpi: boolean, default false.  If set and $wanted is 'nompi',
#  	insist that no MPI module is loaded
# modTag: string, defaults go [module-info specified].  Used in error messages
proc LoadedMpiMatches { wanted {requireIt 0} { loadIt 0 } { noLoadIntel 0 }
   { forceNoMpi 0 } {modTag {} } } {
   # Default modTag
   if { $modTag eq {} } { set modTag [ module-info specified ] }
   # If no MPI given in $wanted, just return
   if { $wanted eq {} } { return }

   # Get loaded MPI, no defaulting to intelmpi
   set loaded [ GetLoadedMPI 0 ]

   # If wanted is nompi, no need to do anything unless forceNoMpi set
   if { $wanted eq {nompi} } {
      if { $forceNoMpi } {
         # Complain if any MPI loaded
         if { $loaded ne {} } {
            PrintLoadError "MPI Mismatch
You have an MPI library loaded ($loaded), but package $modTag
really insists that no MPI library be loaded.

Please unload the MPI library and try again."
         }
      }
      return
   }

   # If no MPI is loaded, then load it if $loadIt (this is valid even
   # in edge cases of nompi or intelmpi), prereq it if $requireIt,
   # abd return
   if { $loaded eq {} } {
      if { $loadIt } {
         RequireMPI $wanted $noLoadIntel
         if { $requireIt } { prereq $wanted }
      }
      return
   }

   # Have a loaded MPI, split into family and version
   set tmpLoaded [ GetPackageFamilyVersion $loaded ]
   set loadedFam [ lindex $tmpLoaded 0 ]
   set loadedVer [ lindex $tmpLoaded 1 ]

   # Split wanted into family and version
   set tmpWanted [ split $wanted / ]
   set wantedLen [ llength $tmpWanted ]
   #Nothing to do if no components to wanted
   if { $wantedLen < 1 } { return }
   set wantedFam [ lindex $tmpWanted 0 ]

   # Ensure families match
   if { $loadedFam ne $wantedFam } {
      PrintLoadError "MPI Mismatch

Package $modTag does not appear to be built for currently
loaded MPI library $loaded."
   }

   # OK, families match
   # If no version specified in $wanted, we are basically done
   if { $wantedLen < 2 } {
      if { $requireIt } { prereq $wanted }
      return
   }

   # Ensure versions match
   set wantedVer [ lindex $tmpWanted 1 ]
   if { $loadedVer eq $wantedVer } {
      if { $requireIt } { prereq $wanted }
      return
   }
   # Versions don't match
   PrintLoadError "MPI Mismatch

Package $modTag does not appear to be built for currently
loaded MPI library $loaded."
}

Basically, it determines what if any MPI library was previously module loaded. If one was loaded, it ensures the family and version match what was expected, and if not exit with an error. The use of the GetPackageFamilyVersion procedure is important, because as discussed previously packages can be represented by more than one name (e.g. the modulefile for version 4.0 of openmpi for gcc/9.1.0 could be named openmpi/4.0/gcc/9.1.0 or openmpi/gcc/9.1.0/4.0) and we need to ensure either is recognized.

Although we only showed cases for compilers and MPI libraries, the techniques above can be adapted to other cases as well. For simple cases, where everything falls under a single package name, one can use GetTagOfModuleLoaded to determine the version of the package loaded in the .modulerc and in the modulefile to ensure the version matches.

For bar, we added variants on CPU vectorization support. Rather than add a simd modulefile, we just add variants to the bar modules. So the bar modulefile tree would look like:

bar
├── 4.7
│   ├── gcc
│   │   ├── 8.2.0
│   │   │   ├── avx
│   │   │   ├── .modulerc -> symlink to modulerc.default_lowest_simd
│   │   │   └── sse4.1
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   └── .modulerc -> symlink to modulerc.select_compiler_family
├── 5.4
│   ├── gcc
│   │   ├── 9.1.0
│   │   │   ├── avx
│   │   │   └── avx2
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   └── .modulerc -> symlink to modulerc.select_compiler_family
├── avx
│   ├── gcc
│   │   ├── 8.2.0
│   │   │   └── 4.7 -> ../../../4.7/gcc/8.2.0/avx
│   │   ├── 9.1.0
│   │   │   └── 5.4 -> ../../../5.4/gcc/9.1.0/avx
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   └── .modulerc -> symlink to modulerc.select_compiler_family
├── avx2
│   ├── gcc
│   │   ├── 9.1.0
│   │   │   └── 5.4 -> ../../../5.4/gcc/9.1.0/avx2
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   └── .modulerc -> symlink to modulerc.select_compiler_family
├── common
├── gcc
│   ├── 8.2.0
│   │   ├── avx
│   │   │   └── 4.7 -> ../../../4.7/gcc/8.2.0/avx
│   │   ├── .modulerc -> symlink to modulerc.default_lowest_simd
│   │   └── sse4.1
│   │       └── 4.7 -> ../../../4.7/gcc/8.2.0/sse4.1
│   ├── 9.1.0
│   │   ├── avx
│   │   │   └── 5.4 -> ../../../5.4/gcc/9.1.0/avx
│   │   ├── avx2
│   │   │   └── 5.4 -> ../../../5.4/gcc/9.1.0/avx2
│   │   └── .modulerc -> symlink to modulerc.default_lowest_simd
│   └── .modulerc -> symlink to modulerc.select_compiler_version
├── .modulerc -> symlink to modulerc.select_compiler_family
└── sse4.1
    ├── gcc
    │   ├── 8.2.0
    │   │   └── 4.7 -> ../../../4.7/gcc/8.2.0/sse4.1
    │   └── .modulerc -> symlink to modulerc.select_compiler_version
    └── .modulerc -> symlink to modulerc.select_compiler_family

Here we added yet another alternate set of module names. So the module for bar version 5.4 built with gcc version 9.1.0 and avx2 support can be called:

  • bar/5.4/gcc/9.1.0/avx2

  • bar/gcc/9.1.0/avx2/5.4

  • bar/avx2/gcc/9.1.0/5.4

Due to this multiplicity of names, if an user does a module load bar, the .modulerc immediately under bar will cause modulecmd to default along the path for the loaded (or defaulted) compiler, and then the modulerc.default_lowest_simd will default to the lowest simd level. If the user module loads bar/VERSION, it will find the build for that version of bar with the appropriate compiler (and again, default to the lowest simd level). And if the user module loads bar/avx or another simd level, then it will load the latest version of bar built for the loaded compiler and simd specified.

The modulerc.default_lowest_simd script looks like:

#%Module
# Default to lowest SIMD support available
#
# Defaults to first of sse4.1, avx, or avx2 found
#
# Usage:
# In most cases, can simply symlink .modulerc to this file
#
# In more complicated cases, .modulerc can source this file, and can
# then test the variable _did_default, which will be true if we set
# a default for modules for the next level, or false otherwise (in which
# case your .modulerc can set one)

# Source some required Tcl procedures here.  Hack for cookbook
# making use of environment variable for location.
# In production, this should ideally be in a site config file.
# At minimum, hardcode the path
set rootdir $::env(MOD_GIT_ROOTDIR)
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

set moduledir [file dirname $ModulesCurrentModulefile]
set baseComponent [ file tail $moduledir ]

# _did_default will be true if we actually default something
# Useful in case a .modulerc sources us, and wants to set a default if we
# did not
set _did_default false

set simdList {sse4.1 avx avx2}
set firstChild [ FirstChildModuleInList $simdList ]
if { $firstChild ne {} } {
   module-version $baseComponent/$firstChild default
   set _did_default true
}

It will default to the lowest SIMD level, but could easily be adapted to do something else.

Examples

We now look at the example modulefiles for the Modulerc-based strategy. As noted previously, the best choice of whether to set the requireIt flag to LoadedCompilerMatches and LoadedMPIMatches (instructing them to to a prereq on the requesting compiler/MPI module) depends on whether one is using a 3.x or 4.x version of Environment Modules. Due to this, we provide two modulefile trees, one for 3.x and one for 4.x; they are basically identical except for that matter. This leads to slightly different instructions on how to use the examples, depending on which version of Environment Modules is being used, namely:

  1. Set (and export) MOD_GIT_ROOTDIR to where you git-cloned the modules source

  2. Do a module purge

  3. If using Environment Modules 3.x, set your MODULEPATH to $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc3

  4. If using Environment Modules 4.x, set your MODULEPATH to $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4

As with the previous cases, we start with a module avail command, and here we see the first big difference:

[mod4 (modulerc)]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4 -
bar/4.7/gcc/(default)                 foo/intel/2019/intelmpi/2.4
bar/4.7/gcc/8.2.0/(default)           foo/intel/2019/mvapich/2.3.1/2.4
bar/4.7/gcc/8.2.0/avx                 foo/intel/2019/nompi/2.4
bar/4.7/gcc/8.2.0/sse4.1(default)     foo/intel/2019/openmpi/4.0/2.4
bar/5.4/gcc/(default)                 foo/pgi/18.4/mvapich/2.1/1.1
bar/5.4/gcc/9.1.0/avx                 foo/pgi/18.4/nompi/(default)
bar/5.4/gcc/9.1.0/avx2                foo/pgi/18.4/nompi/1.1
bar/avx/gcc/(default)                 foo/pgi/18.4/openmpi/3.1/1.1
bar/avx/gcc/8.2.0/(default)           foo/pgi/19.4/nompi/(default)
bar/avx/gcc/8.2.0/4.7                 foo/pgi/19.4/nompi/2.4
bar/avx/gcc/9.1.0/5.4                 foo/pgi/19.4/openmpi/3.1/2.4
bar/avx2/gcc/(default)                gcc/8.2.0
bar/avx2/gcc/9.1.0/5.4                gcc/9.1.0
bar/gcc/(default)                     intel/2018
bar/gcc/8.2.0/(default)               intel/2019
bar/gcc/8.2.0/avx/4.7                 mvapich/2.1/gcc/(default)
bar/gcc/8.2.0/sse4.1/(default)        mvapich/2.1/gcc/8.2.0(default)
bar/gcc/8.2.0/sse4.1/4.7              mvapich/2.1/gcc/9.1.0
bar/gcc/9.1.0/avx/(default)           mvapich/2.1/intel/2018
bar/gcc/9.1.0/avx/5.4                 mvapich/2.1/intel/2019
bar/gcc/9.1.0/avx2/5.4                mvapich/2.1/pgi/18.4
bar/sse4.1/gcc/(default)              mvapich/2.1/pgi/19.4
bar/sse4.1/gcc/8.2.0/(default)        mvapich/2.3.1/gcc/(default)
bar/sse4.1/gcc/8.2.0/4.7              mvapich/2.3.1/gcc/9.1.0
foo/1.1/gcc/(default)                 mvapich/2.3.1/intel/2019
foo/1.1/gcc/8.2.0/(default)           mvapich/2.3.1/pgi/19.4
foo/1.1/gcc/8.2.0/mvapich/2.1         mvapich/gcc/(default)
foo/1.1/gcc/8.2.0/nompi(default)      mvapich/gcc/8.2.0/(default)
foo/1.1/gcc/8.2.0/openmpi/3.1         mvapich/gcc/8.2.0/2.1
foo/1.1/intel/2018/intelmpi(default)  mvapich/gcc/9.1.0/2.1
foo/1.1/intel/2018/mvapich/2.1        mvapich/gcc/9.1.0/2.3.1
foo/1.1/intel/2018/nompi              mvapich/intel/2018/2.1
foo/1.1/intel/2018/openmpi/3.1        mvapich/intel/2019/2.1
foo/1.1/pgi/18.4/mvapich/2.1          mvapich/intel/2019/2.3.1
foo/1.1/pgi/18.4/nompi(default)       mvapich/pgi/18.4/2.1
foo/1.1/pgi/18.4/openmpi/3.1          mvapich/pgi/19.4/2.1
foo/2.4/gcc/(default)                 mvapich/pgi/19.4/2.3.1
foo/2.4/gcc/9.1.0/mvapich/2.3.1       openmpi/3.1/gcc/(default)
foo/2.4/gcc/9.1.0/nompi(default)      openmpi/3.1/gcc/8.2.0(default)
foo/2.4/gcc/9.1.0/openmpi/4.0         openmpi/3.1/gcc/9.1.0
foo/2.4/intel/2019/intelmpi(default)  openmpi/3.1/intel/2018
foo/2.4/intel/2019/mvapich/2.3.1      openmpi/3.1/intel/2019
foo/2.4/intel/2019/nompi              openmpi/3.1/pgi/18.4
foo/2.4/intel/2019/openmpi/4.0        openmpi/3.1/pgi/19.4
foo/2.4/pgi/19.4/nompi(default)       openmpi/4.0/gcc/(default)
foo/2.4/pgi/19.4/openmpi/3.1          openmpi/4.0/gcc/9.1.0
foo/gcc/(default)                     openmpi/4.0/intel/2019
foo/gcc/8.2.0/(default)               openmpi/4.0/pgi/19.4
foo/gcc/8.2.0/mvapich/2.1/1.1         openmpi/gcc/(default)
foo/gcc/8.2.0/nompi/(default)         openmpi/gcc/8.2.0/(default)
foo/gcc/8.2.0/nompi/1.1               openmpi/gcc/8.2.0/3.1
foo/gcc/8.2.0/openmpi/3.1/1.1         openmpi/gcc/9.1.0/3.1
foo/gcc/9.1.0/mvapich/2.3.1/2.4       openmpi/gcc/9.1.0/4.0
foo/gcc/9.1.0/nompi/(default)         openmpi/intel/2018/3.1
foo/gcc/9.1.0/nompi/2.4               openmpi/intel/2019/3.1
foo/gcc/9.1.0/openmpi/4.0/2.4         openmpi/intel/2019/4.0
foo/intel/2018/intelmpi/(default)     openmpi/pgi/18.4/3.1
foo/intel/2018/intelmpi/1.1           openmpi/pgi/19.4/3.1
foo/intel/2018/mvapich/2.1/1.1        openmpi/pgi/19.4/4.0
foo/intel/2018/nompi/1.1              pgi/18.4
foo/intel/2018/openmpi/3.1/1.1        pgi/19.4
foo/intel/2019/intelmpi/(default)
[mod4 (modulerc)]$ module avail mvapich
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4 -
mvapich/2.1/gcc/(default)       mvapich/gcc/(default)
mvapich/2.1/gcc/8.2.0(default)  mvapich/gcc/8.2.0/(default)
mvapich/2.1/gcc/9.1.0           mvapich/gcc/8.2.0/2.1
mvapich/2.1/intel/2018          mvapich/gcc/9.1.0/2.1
mvapich/2.1/intel/2019          mvapich/gcc/9.1.0/2.3.1
mvapich/2.1/pgi/18.4            mvapich/intel/2018/2.1
mvapich/2.1/pgi/19.4            mvapich/intel/2019/2.1
mvapich/2.3.1/gcc/(default)     mvapich/intel/2019/2.3.1
mvapich/2.3.1/gcc/9.1.0         mvapich/pgi/18.4/2.1
mvapich/2.3.1/intel/2019        mvapich/pgi/19.4/2.1
mvapich/2.3.1/pgi/19.4          mvapich/pgi/19.4/2.3.1

Unlike the previous cases wherein only package names and versions were shown (because a single modulefile handled all the builds for that package and version), here we see a listing for every build of a package and version. Indeed, we not only see one such listing, but multiple listings per build in many cases (e.g. openmpi/3.1/intel/2018 and openmpi/intel/2018/3.1).

While admittedly the output of a module avail command without specifying any package is rather overwhelming, when a package is specified the output tends to be more reasonable, informing one of which builds of the package are available. This strategy deliberately opts for the presentation of more rather than less information.

The standard functionality of selecting the correct build of a package based on the loaded compiler, e.g.

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load pgi/19.4
[mod4 (modulerc)]$ module load openmpi/4.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/4.0/pgi/19.4(default)
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (modulerc)]$ module unload openmpi
[mod4 (modulerc)]$ module switch --auto pgi intel/2019
[mod4 (modulerc)]$ module load openmpi/4.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0/intel/2019(default)
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4 (modulerc)]$ module unload openmpi
[mod4 (modulerc)]$ module switch --auto intel gcc/9.1.0
[mod4 (modulerc)]$ module load openmpi/4.0
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/4.0, gcc/9.1.0)
[mod4 (modulerc)]$ module unload openmpi
[mod4 (modulerc)]$ module switch --auto gcc gcc/8.2.0
[mod4 (modulerc)]$ module load openmpi/4.0

**** ERROR *****:
Compiler Mismatch
Package openmpi/4.0 does not appear to be built for currently
loaded compiler gcc/8.2.0.


Loading openmpi/4.0/gcc/9.1.0
  ERROR: Module evaluation aborted
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0

works as expected (and fails with the expected error when one attempts to load a module not compatible with the loaded compiler). The only significant difference between the previously examined strategies is that the module list command provides information about what variant of each package is loaded.

The module switch command, however, does not work as well as one would like. While it indeed switches the specified module, it does not successfully reload the modules which depend on the replaced module, even with the Automated module handling feature enabled. As currently implemented, the automated module handling feature attempts to reload dependent modules using the fully qualified module name, and as in this strategy the fully qualified modulename includes the information about the module that was switched out (e.g. pgi/19.4 in the example below), it will be incompatible with the replacement module (e.g. intel/2019 in example below). It is hoped a future version of modulecmd will allow for reloading based on the name specified when the module was loaded. But as things currently stand, the automatic module handling will throw an error attempting to reload the depend module, resulting in the the dependent modules being unloaded.

Without automated module handling mode (i.e. for older Environment Modules or without the --auto flag), the dependent modules remain loaded and there is inconsistency in the loaded modules. But at least module list clearly shows such.

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load pgi/19.4
[mod4 (modulerc)]$ module load openmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/pgi/19.4/4.0
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (modulerc)]$ module switch --auto pgi intel/2019

**** ERROR *****:
Compiler Mismatch
Package openmpi/pgi/19.4/4.0 does not appear to be built for currently
loaded compiler intel/2019.


Loading openmpi/pgi/19.4/4.0
  ERROR: Module evaluation aborted

Switching from pgi/19.4 to intel/2019
  WARNING: Reload of dependent openmpi/pgi/19.4/4.0 failed
  Unloading dependent: openmpi/pgi/19.4/4.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019
[mod4 (modulerc)]$ mpirun
mpirun: command not found

The defaulting of modules is more successful, however, as seen below:

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load openmpi/3.1
Loading openmpi/3.1/gcc/8.2.0
  Loading requirement: gcc/8.2.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) openmpi/3.1/gcc/8.2.0(default)
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/3.1, gcc/8.2.0)
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load gcc/8.2.0
[mod4 (modulerc)]$ module load openmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) openmpi/gcc/8.2.0/3.1
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/3.1, gcc/8.2.0)

Here it not only defaults to the default compiler, but if one tries to load openmpi without specifying a version, it will find and load the latest version compatible with the loaded compiler.

The situation is similar for foo:

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load pgi/19.4
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) foo/2.4/pgi/19.4/nompi(default)
[mod4 (modulerc)]$ foo
foo 2.4 (pgi/19.4, nompi)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module load openmpi/3.1
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4                        3) foo/2.4/pgi/19.4/openmpi/3.1(default)
 2) openmpi/3.1/pgi/19.4(default)
[mod4 (modulerc)]$ foo
foo 2.4 (pgi/19.4, openmpi/3.1)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module unload openmpi
[mod4 (modulerc)]$ module switch --auto pgi intel/2019
[mod4 (modulerc)]$ module load foo/2.4
Loading foo/2.4/intel/2019/intelmpi
  ERROR: foo/2.4/intel/2019/intelmpi cannot be loaded due to missing prereq.
    HINT: the following module must be loaded first: intelmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019
[mod4 (modulerc)]$ foo
foo: command not found
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module load foo/2.4/intel/2019/nompi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) foo/2.4/intel/2019/nompi
[mod4 (modulerc)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module switch --auto intelmpi mvapich/2.3.1
[mod4 (modulerc)]$ module load foo/2.4
Loading foo/2.4/intel/2019/intelmpi
  ERROR: foo/2.4/intel/2019/intelmpi cannot be loaded due to missing prereq.
    HINT: the following module must be loaded first: intelmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) mvapich/2.3.1/intel/2019(default)
[mod4 (modulerc)]$ foo
foo: command not found
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module switch --auto mvapich openmpi/4.0
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019
 2) openmpi/4.0/intel/2019(default)
 3) foo/2.4/intel/2019/openmpi/4.0(default)
[mod4 (modulerc)]$ foo
foo 2.4 (intel/2019, openmpi/4.0)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module unload openmpi
[mod4 (modulerc)]$ module switch --auto intel/2019 gcc/9.1.0
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) foo/2.4/gcc/9.1.0/nompi(default)
[mod4 (modulerc)]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module load mvapich/2.3.1
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0                          3) foo/2.4/gcc/9.1.0/nompi(default)
 2) mvapich/2.3.1/gcc/9.1.0(default)
[mod4 (modulerc)]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module switch --auto mvapich openmpi/4.0
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0                        3) foo/2.4/gcc/9.1.0/openmpi/4.0(default)
 2) openmpi/4.0/gcc/9.1.0(default)
[mod4 (modulerc)]$ foo
foo 2.4 (gcc/9.1.0, openmpi/4.0)

As expected, the correct version of foo is loaded depending on the previously loaded compiler and MPI libraries. Note however that there is no dummy module for Intel MPI, and that when the intel compiler is loaded but no MPI library is explicitly loaded, module load foo/2.4 gets the Intel MPI build. Such only occurs if there is an Intel MPI build in the tree of modulefiles; otherwise a nompi build will be loaded if present. If both versions are present, as in this case, one needs to give the explicit foo/2.4/intel/2019/nompi specification.

Again, the module switch command only succeeds in switching the specified module, and does not successfully reload the modules depending on the switched module. With automated module handling mode, there will be errors reported and the dependent modules will be unloaded; without it the dependent modules will not get unloaded, and there will be inconsistent dependencies (but at least module list will show such).

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load pgi/18.4
[mod4 (modulerc)]$ module load openmpi/3.1
[mod4 (modulerc)]$ module load foo/1.1
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) pgi/18.4                        3) foo/1.1/pgi/18.4/openmpi/3.1(default)
 2) openmpi/3.1/pgi/18.4(default)
[mod4 (modulerc)]$ foo
foo 1.1 (pgi/18.4, openmpi/3.1)
[mod4 (modulerc)]$ module switch --auto pgi intel/2018

**** ERROR *****:
Compiler Mismatch
Package openmpi/3.1/pgi/18.4 does not appear to be built for currently
loaded compiler intel/2018.


Loading openmpi/3.1/pgi/18.4
  ERROR: Module evaluation aborted

**** ERROR *****:
Compiler Mismatch
Package foo/1.1/pgi/18.4/openmpi/3.1 does not appear to be built for currently
loaded compiler intel/2018.


Loading foo/1.1/pgi/18.4/openmpi/3.1
  ERROR: Module evaluation aborted

Switching from pgi/18.4 to intel/2018
  WARNING: Reload of dependent openmpi/3.1/pgi/18.4 failed
  WARNING: Reload of dependent foo/1.1/pgi/18.4/openmpi/3.1 failed
  Unloading dependent: foo/1.1/pgi/18.4/openmpi/3.1 openmpi/3.1/pgi/18.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2018
[mod4 (modulerc)]$ foo
foo: command not found
[mod4 (modulerc)]$ mpirun
mpirun: command not found
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load intel/2019
[mod4 (modulerc)]$ module load foo
Loading foo/intel/2019/intelmpi/2.4
  ERROR: foo/intel/2019/intelmpi/2.4 cannot be loaded due to missing prereq.
    HINT: the following module must be loaded first: intelmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019
[mod4 (modulerc)]$ foo
foo: command not found
[mod4 (modulerc)]$ module load openmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/intel/2019/4.0
[mod4 (modulerc)]$ foo
foo: command not found

The defaulting of modules works relatively well, as shown below:

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load foo
Loading foo/gcc/8.2.0/nompi/1.1
  Loading requirement: gcc/8.2.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) foo/gcc/8.2.0/nompi/1.1
[mod4 (modulerc)]$ foo
foo 1.1 (gcc/8.2.0, nompi)
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load foo/1.1
Loading foo/1.1/gcc/8.2.0/nompi
  Loading requirement: gcc/8.2.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) foo/1.1/gcc/8.2.0/nompi(default)
[mod4 (modulerc)]$ foo
foo 1.1 (gcc/8.2.0, nompi)
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load pgi/18.4
[mod4 (modulerc)]$ module load foo
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) pgi/18.4   2) foo/pgi/18.4/nompi/1.1
[mod4 (modulerc)]$ foo
foo 1.1 (pgi/18.4, nompi)

If one attempts to load foo without specifying a version or having previously loaded a compiler module, the compiler will be defaulted (to gcc/8.2.0 compiler, as that is what we declared via the GetDefaultCompiler Tcl procedure), and the latest version of foo compatible with that compiler (and no MPI) will be loaded.

The situation with bar is similar. We do not have a dummy simd module, so the builds with different CPU vectorization support are specified by appending /avx, etc. to the bar package name. As with previous strategies, if one attempts to load a simd variant which was not built for the compiler loaded, an error will occur.

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load gcc/9.1.0
[mod4 (modulerc)]$ module load bar/avx2
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) bar/avx2/gcc/9.1.0/5.4
[mod4 (modulerc)]$ bar
bar 5.4 (gcc/9.1.0, avx2)
[mod4 (modulerc)]$ module unload bar
[mod4 (modulerc)]$ module load bar/avx
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) bar/avx/gcc/9.1.0/5.4
[mod4 (modulerc)]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4 (modulerc)]$ module unload bar
[mod4 (modulerc)]$ module load bar/sse4.1

**** ERROR *****:
Compiler Mismatch
Package bar/sse4.1 does not appear to be built for currently
loaded compiler gcc/9.1.0.


Loading bar/sse4.1/gcc/8.2.0/4.7
  ERROR: Module evaluation aborted
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0

Summary of Modulerc-based strategy

  • The modulefiles are fairly standard. The only extra logic needed is to ensure that any loaded compiler/MPI or other dependency matches what the modulefile expects, and that can be handled fairly easily with the addition of one or two calls to Tcl procedures.

  • The .modulerc files do the work of routing partial specifications to the correct modulefile, and follow a fairly standard pattern. For the most part, these are generic/independent of the package, and typically can be written once and symlinked to the appropriate places in the directory tree.

  • This strategy involves the use of many more modulefiles than the previously examined strategies, having at least one modulefile per build, and in some cases multiple per build. The output of the module avail command without any module specified is rather overwhelming, although when a module is specified, module avail will provide information about what builds are available.

  • Because each build has a specific module, the module list command shows exactly what builds of the various modules are loaded.

  • The module switch command does not work very well. In particular, when one switches out a module on which other modules depend, the dependent modules are not successfully reloaded. Even with the Automated module handling feature (introduced in 4.2), the dependent modules are unloaded but do not get reloaded properly since currently this feature attempts to reload based on the fully qualified name of the loaded module (which includes a dependency on the module switched out), not the specification used when the module was loaded.

  • In general, when the user tries to load a module based on a partially specified modulename, the .modulerc files handle it well, and the latest version of the module consistent with the specification and previously loaded modules will be used. If no compiler was previously loaded, will use the default compiler, or the latest compiler if nothing matches the default compiler.

  • Modules will fail with an error message if user tries to load a package which was not built for the values of the dependency packages loaded.

Modulepath-based Strategy

This strategy makes use of the ability of modules to support multiple directories in the MODULEPATH environment variable. Every time a module is loaded on which other modules might depend, a new path is added to MODULEPATH containing the modulefiles which depend on the newly added module.

This is basically similar to the strategy that is used in the hierarchical modulefile approach of Lmod.

Implementation

The Homebrewed flavors and Modulerc-based strategies were based on modifying the modulefiles of modules which depended on other modules. The Modulepath-based strategy, in contrast, is based on modifying the modulefiles for modules which other modules might depend on.

To begin with, we set MODULEPATH to a Core directory containing modulefiles for modules which do not depend on any other modules. The modulefiles for compilers will typically be put in here. For this example, we opt not to use a dummy simd module (for reasons explained later), but instead add avx, avx2, sse4.1 variants to the bar modules, but otherwise they might belong here. Other modules which do not depend on compilers, etc. could be place in here as well, e.g. applications which do not expose libraries, etc. So the directory structure would look like:

Core
├── gcc
│   ├── 8.2.0
│   ├── 9.1.0
│   └── common
├── intel
│   ├── 2018
│   ├── 2019
│   └── common
└── pgi
    ├── 18.4
    ├── 19.4
    └── common

A typical common file for the gcc compiler would be something like

# Example modulefiles for compiler-etc-dependency cookbook
# For "modulepath based" strategyy
#
# Common stuff for gcc
#
# Expects version to have been previously set

proc ModulesHelp { } {
   global version

   puts stderr "
This is the dummy GNU compiler suite modulefile for the cookbook
	Handling Compiler and other Package Dependencies

It does not actually do anything

gcc version: $version

"
}

module-whatis "Dummy Gnu $version for cookbook"

# Find the software root.  In production, you should
# hardcode to your real software root
set gitroot $::env(MOD_GIT_ROOTDIR)
set swroot $gitroot/doc/example/compiler-etc-dependencies/dummy-sw-root

set pkgroot $swroot/gcc
set vroot $pkgroot/$version
set bindir $vroot/bin

prepend-path PATH $bindir

# don't load multiple versions of this module (or other compilers)
conflict gcc
conflict gnu
conflict pgi
conflict intel

# Add the proper modulepath
# In production this should be hard-coded, but using $gitroot for cookbook
set modpathroot $gitroot/doc/example/compiler-etc-dependencies/modulepath
set newmodpath $modpathroot/Compiler/gcc/$version
module use $newmodpath

The most interesting aspect is the module use at the end. We add to MODULEPATH a directory under Compiler for this specific compiler and version. Other compilers/versions would add their own specific path to MODULEPATH.

In each of these compiler specific directories, modulefile directory trees exist for the supported MPI libraries and bar. We also include a foo directory containing the non-MPI variants of foo. It would look like:

Compiler
├── gcc
│   ├── 8.2.0
│   │   ├── bar
│   │   │   ├── 4.7
│   │   │   │   ├── avx
│   │   │   │   ├── .modulerc -> symlink to modulerc.default_lowest_simd
│   │   │   │   └── sse4.1
│   │   │   └── common -> symlink to bar/common
│   │   ├── foo
│   │   │   ├── 1.1
│   │   │   └── common -> symlink to foo/common
│   │   ├── mvapich
│   │   │   ├── 2.1
│   │   │   └── common -> symlink to mvapich/common
│   │   └── openmpi
│   │       ├── 3.1
│   │       └── common -> symlink to openmpi/common
│   └── 9.1.0
│       ├── bar
│       │   ├── 5.4
│       │   │   ├── avx
│       │   │   ├── avx2
│       │   │   └── .modulerc -> symlink to modulerc.default_lowest_simd
│       │   └── common -> symlink to bar/common
│       ├── foo
│       │   ├── 2.4
│       │   └── common -> symlink to foo/common
│       ├── mvapich
│       │   ├── 2.1
│       │   ├── 2.3.1
│       │   └── common -> symlink to mvapich/common
│       └── openmpi
│           ├── 3.1
│           ├── 4.0
│           └── common -> symlink to openmpi/common
├── intel
│   ├── 2018
│   │   └── ... modules depending on compiler=intel/2018
│   └── 2019
│       └── ... modules depending on compiler=intel/2019
└── pgi
    ├── 18.4
    │   └── ... modules depending on compiler=pgi/18.4
    └── 19.4
        └── ... modules depending on compiler=pgi/18.4

For brevity, we only show the directory structure for the modules depending on the two versions of gcc in detail; the other compilers will have similar structures.

Basically, there is a separate modulepath for each compiler, containing only the modulefiles depending on that specific build of the compiler (and not also depending on something else, like MPI library). This certainly enforces the consistency of loaded modules; one could not load a specific version of gcc (say gcc/9.1.0) and an incompatible version of foo (e.g. foo/1.1), because all of the foo modulefiles are in compiler specific module trees and there is no foo/1.1 in the gcc/9.1.0 module tree. Conflict statements in the compiler modulefiles will prevent one from loading multiple compilers, thereby preventing multiple compiler specific modulepaths (unless the user explicitly does a module use or similar, and there is only so far one can go in preventing users from shooting themselves in the foot).

The modulefiles for foo and bar are fairly straightforward; we have a common file which does the heavy lifting (and since this can be made independent of version and compiler, this is actually a symlink to a package specific common file in a common directory external to the modulepath trees). We then add some small stubfiles which set variables for the specific build information, which mostly are determined by their position in the tree structure (e.g. the stubfiles under Compiler/gcc/9.1.0 are all going to set compilerTag to gcc/9.1.0, etc), and then invoke the common file.

The modulefiles for openmpi are largely similar. E.g., for gcc/9.1.0 we have a small stubfile like the following (for version 4.0)

#%Module
# Dummy modulefile for openmpi

set version 4.0
set compilerTag gcc/9.1.0
set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/common

which defines the version and compilerTag variables for the OpenMPI version and compiler version, and then invokes the common script

# Common modulefile for openmpi
# Using "modulepath" strategy
# Expects the following variables to have been
# previously defined:
#	version: version of openmpi
#	compilerTag: compiler being used with

# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root

proc ModulesHelp { } {
   global version compilerTag
   puts stderr "
openmpi: Test dummy version of OpenMPI $version

For testing packages depending on compilers/MPI
Version: $version
Compiler: $compilerTag

"
}

module-whatis "Dummy openmpi $version (for $compilerTag)"

# Find the software root.  In production, you should
# hardcode to your real software root
set gitroot $::env(MOD_GIT_ROOTDIR)
set swroot $gitroot/doc/example/compiler-etc-dependencies/dummy-sw-root

# Compute the installation prefix
set pkgroot $swroot/openmpi
set vroot $pkgroot/$version
set prefix $vroot/$compilerTag

# We need to prereq the compiler to allow autohandling to work
prereq $compilerTag

# Set environment variables
setenv MPI_DIR $prefix

set bindir $prefix/bin
set libdir $prefix/lib
set incdir $prefix/include

prepend-path PATH            $bindir
prepend-path LIBRARY_PATH    $libdir
prepend-path LD_LIBRARY_PATH $libdir
prepend-path CPATH           $incdir

# Add the proper modulepath
# In production this should be hard-coded, but using $gitroot for cookbook
set modpathroot $gitroot/doc/example/compiler-etc-dependencies/modulepath
set newmodpath $modpathroot/CompilerMPI/$compilerTag/openmpi/$version
module use $newmodpath

which does the usual stuff (define a help function, whatis string, and sets assorted environment variables like PATH, LIBRARY_PATH, etc. for using OpenMPI), and then adds another directory to the MODULEPATH. This new directory, under CompilerMPI is for modulefiles which depend on the compiler AND the MPI library. (It is assumed that all modulefiles depending on MPI libraries also depend on the compiler). This new branch in the modulepath would have a structure like

CompilerMPI
├── gcc
│   ├── 8.2.0
│   │   ├── mvapich
│   │   │   └── 2.1
│   │   │       └── foo
│   │   │           ├── 1.1
│   │   │           └── common -> symlink to foo/common
│   │   └── openmpi
│   │       └── 3.1
│   │           └── foo
│   │               ├── 1.1
│   │               └── common -> symlink to foo/common
│   └── 9.1.0
│       ├── mvapich
│       │   └── 2.3.1
│       │       └── ... modules depending on gcc/9.1.0 and mvapich/2.3.1
│       └── openmpi
│           └── 4.0
│               └── ... modules depending on gcc/9.1.0 and openmpi/4.0
├── intel
│   ├── 2018
│   │   ├── intelmpi
│   │   │   └── default
│   │   │       └── ... modules depending on intel/2018 and its included MPI lib
│   │   ├── mvapich
│   │   │   └── 2.1
│   │   │       └── ... modules depending on intel/2018 and mvapich/2.1
│   │   └── openmpi
│   │       └── 3.1
│   │           └── ... modules depending on intel/2018 and openmpi/3.1
│   └── 2019
│       ├── intelmpi
│       │   └── default
│       │       └── ... modules depending on intel/2019 and its included MPI lib
│       ├── mvapich
│       │   └── 2.3.1
│       │       └── ... modules depending on intel/2019 and mvapich/2.3.1
│       └── openmpi
│           └── 4.0
│               └── ... modules depending on intel/2019 and openmpi/4.0
└── pgi
    ├── 18.4
    │   ├── mvapich
    │   │   └── 2.1
    │   │       └── ... modules depending on pgi/18.4 and mvapich/2.1
    │   └── openmpi
    │       └── 3.1
    │           └── ... modules depending on pgi/18.4 and openmpi/3.1
    └── 19.4
        └── openmpi
            ├── 3.1
            │   └── ... modules depending on pgi/19.4 and openmpi/3.1
            └── 4.0
                └── ... modules depending on pgi/19.4 and openmpi/3.1

Basically, there is a separate modulepath for each combination of compiler and MPI library, listing all modulefiles depending on that compiler and MPI library. The modulefiles underneath each directory simply do what is needed to load that version of the package built with the specified compiler and MPI library.

This process could be continued further, as needed. E.g., if you had packages which depended on NetCDF, and you had multiple builds of NetCDF for a given compiler/MPI combination, you could add another modulepath tree, e.g. CompilerMPINetCDF, branching on each compiler, MPI, and NetCDF triplet. But things can also get unwieldy if carried too far.

One could also add modulepath trees for disjoint features like SIMD level, but this turns out to be a bit tricky. E.g., if you had a simd branch as well as a compiler branch, with both simd and compiler appearing in Core, things are fine for the Simd and Compiler trees. However, if there were to be modules depending on both, e.g. a CompilerSimd branch, then because compiler and simd are disjoint and could be loaded in either order, the modulefiles for both would need to handle adding the CompilerSimd branch depending on whether the other was previously loaded.

Examples

We now look at the example usage for the Modulepath-based strategy. To use these examples, you must:

  1. Set (and export) MOD_GIT_ROOTDIR to where you git-cloned the modules source

  2. Do a module purge

  3. Set your MODULEPATH to $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Core

Note that we set the MODULEPATH to the Core subdirectory; the Core branch is for those modulefiles that do not depend on compiler or MPI library, and that should be available from the start.

As with the previous cases, we start with a module avail command, and here we see the first big difference:

[mod4 (modulepath)]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Core -
gcc/8.2.0  gcc/9.1.0  intel/2018  intel/2019  pgi/18.4  pgi/19.4
[mod4 (modulepath)]$ module load gcc/9.1.0
[mod4 (modulepath)]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Compiler/gcc/9.1.0 -
bar/5.4/avx(default)  foo/2.4      mvapich/2.3.1  openmpi/4.0
bar/5.4/avx2          mvapich/2.1  openmpi/3.1

- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Core -
gcc/8.2.0  gcc/9.1.0  intel/2018  intel/2019  pgi/18.4  pgi/19.4
[mod4 (modulepath)]$ module load openmpi/4.0
[mod4 (modulepath)]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/CompilerMPI/gcc/9.1.0/openmpi/4.0 -
foo/2.4

- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Compiler/gcc/9.1.0 -
bar/5.4/avx(default)  foo/2.4      mvapich/2.3.1  openmpi/4.0
bar/5.4/avx2          mvapich/2.1  openmpi/3.1

- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Core -
gcc/8.2.0  gcc/9.1.0  intel/2018  intel/2019  pgi/18.4  pgi/19.4

When we first do a module avail, we only see the modulefiles for the compilers. The MPI libraries, foo, bar, etc. all depend on at least the compiler, and so will not become "available" until a compiler (and possibly MPI library as well) is loaded. In a production environment there would likely be other modulefiles available in Core (i.e. an application which is only used as a standalone application and does not provide an API/libraries to link against would likely be placed in Core), but in our simple example, only the compilers appear in Core. As we load a compiler and then an MPI library, additional modulefiles appear available. In a production environment, one might wish to set things up so that a compiler (and maybe MPI library) modulefile is automatically loaded when the user logs in.

The downside of this is that it becomes difficult for an user to know what software is available, at least via the module command. I.e., if application foobar is only built for a single compiler/MPI combination, it will not show up in module avail unless that specific compiler/MPI combination were previously loaded. Lmod adds a module spider command which allows the user to list all packages installed for any compiler/MPI combination, and can be used to also find out which compiler/MPI combinations are needed to access a specific modulefile, but Environment Modules does not have a comparable function at this time. If you were to use this strategy in a production environment, you would likely need to generate lists of available packages (and their compiler/MPI/etc dependencies) in a web page or other documentation area which can be frequently updated.

The standard functionality of selecting the correct build of a package based on the loaded compiler, e.g.

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load pgi/19.4
[mod4 (modulepath)]$ module load openmpi/4.0
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/4.0
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (modulepath)]$ module unload openmpi
[mod4 (modulepath)]$ module switch --auto pgi intel/2019
[mod4 (modulepath)]$ module load openmpi/4.0
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4 (modulepath)]$ module unload openmpi
[mod4 (modulepath)]$ module switch --auto intel gcc/9.1.0
[mod4 (modulepath)]$ module load openmpi/4.0
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/4.0, gcc/9.1.0)
[mod4 (modulepath)]$ module unload openmpi
[mod4 (modulepath)]$ module switch --auto gcc gcc/8.2.0
[mod4 (modulepath)]$ module load openmpi/4.0
ERROR: Unable to locate a modulefile for 'openmpi/4.0'
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0

works as expected. As shown in the last command above, when requesting a module not built for the loaded compiler (e.g. openmpi/4.0 when gcc/8.2.0 is loaded), the module command simply returns Unable to locate a modulefile as there is no corresponding modulefile in the current list of MODULEPATHS.

The functionality of the module switch command depends on version and/or settings of Environment Modules. For version 3.x, or 4.x with the Automated module handling feature disabled, it does not work well, modules which depend on the switched out module are not reloaded, leading to inconsistent environments. But for 4.x versions with the automated module handling feature enabled (as indicated by the --auto after the switch, although that can be made the default), it works as expected. Modules depending on the module being switched out get unloaded and reloaded if possible, as shown below:

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load pgi/19.4
[mod4 (modulepath)]$ module load openmpi
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/4.0
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (modulepath)]$ module switch --auto pgi intel/2019
Switching from pgi/19.4 to intel/2019
  Unloading dependent: openmpi/4.0
  Reloading dependent: openmpi/4.0
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4 (modulepath)]$ module switch --auto intel intel/2018
Switching from intel/2019 to intel/2018
  ERROR: Unable to locate a modulefile for 'openmpi/4.0'
  WARNING: Reload of dependent openmpi/4.0 failed
  Unloading dependent: openmpi/4.0
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2018
[mod4 (modulepath)]$ mpirun
mpirun: command not found

The modulefiles themselves basically have no support for defaulting a compiler; the modulefiles for the various MPI libraries are simply not even available until a modulefile for a compiler is loaded, as seen below:

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load openmpi/3.1
ERROR: Unable to locate a modulefile for 'openmpi/3.1'
[mod4 (modulepath)]$ module list
No Modulefiles Currently Loaded.
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load gcc/8.2.0
[mod4 (modulepath)]$ module load openmpi
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) openmpi/3.1
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/3.1, gcc/8.2.0)

However, this could be somewhat mitigated by having the modulefile for the default compiler automatically loaded for the user upon login (e.g. in their dot files).

The situation is similar for foo:

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load pgi/19.4
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (pgi/19.4, nompi)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module load openmpi/3.1
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/3.1   3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (pgi/19.4, openmpi/3.1)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module unload openmpi
[mod4 (modulepath)]$ module switch --auto pgi intel/2019
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module load intelmpi
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) intelmpi/default   3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, intelmpi)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module switch --auto intelmpi mvapich/2.3.1
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) mvapich/2.3.1   3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, mvapich/2.3.1)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module switch --auto mvapich openmpi/4.0
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0   3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, openmpi/4.0)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module unload openmpi
[mod4 (modulepath)]$ module switch --auto intel/2019 gcc/9.1.0
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module load mvapich/2.3.1
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) mvapich/2.3.1   3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (gcc/9.1.0, mvapich/2.3.1)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module switch --auto mvapich openmpi/4.0
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) openmpi/4.0   3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (gcc/9.1.0, openmpi/4.0)

As expected, the correct version of foo is loaded depending on the previously loaded compiler and MPI libraries. Here again we use an intelmpi modulefile to indicate when we wish to use Intel MPI libraries, even though they are enabled by the intel module. The intelmpi modulefile basically just adds the modulepath for the intel-compiler and intelmpi dependent modules.

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load pgi/18.4
[mod4 (modulepath)]$ module load openmpi/3.1
[mod4 (modulepath)]$ module load foo/1.1
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) pgi/18.4   2) openmpi/3.1   3) foo/1.1
[mod4 (modulepath)]$ foo
foo 1.1 (pgi/18.4, openmpi/3.1)
[mod4 (modulepath)]$ module switch --auto pgi intel/2018
Switching from pgi/18.4 to intel/2018
  Unloading dependent: foo/1.1 openmpi/3.1
  Reloading dependent: openmpi/3.1 foo/1.1
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2018   2) openmpi/3.1   3) foo/1.1
[mod4 (modulepath)]$ foo
foo 1.1 (intel/2018, openmpi/3.1)
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/3.1, intel/2018)
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load intel/2019
[mod4 (modulepath)]$ module load foo
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (modulepath)]$ module load openmpi
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) foo/2.4   3) openmpi/4.0
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, nompi)

Note again the same deficiency in the switch report as the Homebrewed Flavors Strategy had; in the last case above wherein we loaded foo with intel/2019 loaded but no MPI module. As expected, a non-MPI build of foo was loaded. However, when an openmpi module is subsequently loaded, foo does not get reloaded and we still have the non-MPI build of foo, as evidenced by the output of the foo command. And that this fact is not obvious from the module list output.

The behavior when defaulting is nicer. Without any compiler or MPI libraries Here we note a deficiency in the switch support as compared to Flavours. In the last example after loading intel/2019 and foo, we have the non-MPI build of foo as expected. However, upon subsequently loading the openmpi module, we still have the non-MPI version of foo loaded, as evidenced by the output of the dummy foo command. I.e., the foo package was not automatically reloaded, as there was no prereq in the foo modulefile on an MPI library (as in the non-MPI build there is no MPI library to prereq). Also note that module list does not really inform one of this fact.

The ability to default partial modulenames in this strategy is mixed. Without any compiler loaded, most modulefiles are not even visible/available. However, once a compiler is loaded, trying to load a module without specifying the version will end up loading the latest version compatible with the loaded compiler (as no incompatible versions are visible) as seen below:

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load foo
ERROR: Unable to locate a modulefile for 'foo'
[mod4 (modulepath)]$ module load gcc/8.2.0
[mod4 (modulepath)]$ module load foo
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) foo/1.1
[mod4 (modulepath)]$ foo
foo 1.1 (gcc/8.2.0, nompi)
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load gcc/8.2.0
[mod4 (modulepath)]$ module load openmpi
[mod4 (modulepath)]$ module load foo
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) openmpi/3.1   3) foo/1.1
[mod4 (modulepath)]$ foo
foo 1.1 (gcc/8.2.0, openmpi/3.1)
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load foo/1.1
ERROR: Unable to locate a modulefile for 'foo/1.1'
[mod4 (modulepath)]$ module list
No Modulefiles Currently Loaded.
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load pgi/18.4
[mod4 (modulepath)]$ module load foo
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) pgi/18.4   2) foo/1.1
[mod4 (modulepath)]$ foo
foo 1.1 (pgi/18.4, nompi)

This is better than in the Flavours or Homebrewed flavors strategies. If one were to use this strategy in production, it is recommended to have a default compiler module (and maybe even an MPI library, etc) automatically loaded when the user logs in, thereby allowing for a reasonable defaulting ability (and more reasonable module avail output).

The situation with bar is similar. We do not have a dummy simd module, so the builds with different CPU vectorization support are specified by appending /avx, etc. to the bar package name. As with previous strategies, if one attempts to load a simd variant which was not built for the compiler loaded, an error will occur.

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load gcc/9.1.0
[mod4 (modulepath)]$ module load bar/5.4/avx2
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) bar/5.4/avx2
[mod4 (modulepath)]$ bar
bar 5.4 (gcc/9.1.0, avx2)
[mod4 (modulepath)]$ module unload bar
[mod4 (modulepath)]$ module load bar/5.4/avx
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) bar/5.4/avx(default)
[mod4 (modulepath)]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4 (modulepath)]$ module unload bar
[mod4 (modulepath)]$ module load bar/5.4/sse4.1
ERROR: Unable to locate a modulefile for 'bar/5.4/sse4.1'
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0

Defaulting is handled well, as shown

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load gcc/9.1.0
[mod4 (modulepath)]$ module load bar
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) bar/5.4/avx(default)
[mod4 (modulepath)]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load gcc/8.2.0
[mod4 (modulepath)]$ module load bar/4.7
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) bar/4.7/sse4.1(default)
[mod4 (modulepath)]$ bar
bar 4.7 (gcc/8.2.0, sse4.1)

In particular, one case specify bar/avx2 or bar/avx or bar/sse4.1 and the latest version of bar consistent with the specification and any previously loaded compiler (or default compiler) will be loaded

Summary of Modulepath-based strategy

  • It is quite difficult for an user to get an inconsistent environment, at least when automated module handling mode is used. The modulepaths are managed so that only modulefiles consistent with the previously loaded compiler/MPI library are visible.

  • With automated module handling mode, the module switch command is very nicely supported.

  • The module command does not lend itself to readily seeing all the packages which are installed on the system, nor seeing for which compiler/MPI combinations a certain package is built. The module avail command just does not show any modules not compatible with the currently loaded compiler and/or MPI libraries. Lmod needed to add a module spider command to address this, but no such functionality currently exists in Environment Modules. If one were to use this in production, you would need to provide something similar to the Lmod spider sub-command, or at least provide frequently updated web pages or similar with this information.

  • This strategy involves the use of many more modulepaths than the previously examined strategies, having at least one modulepath per compiler installed, and typically also one for each compiler/MPI library combination. If you add additional layers of dependency, things get even more complicated, which can be limiting. It is especially problematic if you have two disjoint dependencies -- i.e. neither dependencies is dependent on the other; in this case you might load one or the other, or both (in either order). If there are modulefiles dependent on both of these dependencies, the last one loaded (and only the last one) needs to handle adding the appropriate modulepath for the modulefiles depending on both.

  • In general, a module will fail to load with an error message if any dependent module is not already loaded. The error will actually state that the module cannot be found, as the modulepath containing it will not be added to the MODULEPATH until the modules depended on are loaded. So if you need to default a compiler, etc., you will need to have the user's initialization scripts automatically load the appropriate modulefile upon login.

  • However, what we have been referring to as "more intelligent defaulting" is effectively supported. I.e., if a user requests to load a package without specifying the version desired, the version will be defaulted to the latest version compatible with the currently loaded compiler and other dependencies. This is because the incompatible versions are not visible in the module tree.

Comparison of Strategies

All of the strategies discussed above have their peculiar strengths and weaknesses. The decision of which strategy to use will depend on how these strengths and weaknesses impact your design goals. It is advised to play around in the sandbox environments a bit, as actual use tends to help make clear which downsides you are willing to accept and which you are not.

With the exception of the Flavours, which works best with Environment Modules version 3.x, all of the other strategies work as well or better on the newer 4.x versions of Environment Modules. This difference is most visible in the discussion of features around the module switch command. So in the following discussions we will assume Environment Modules version 3.x for the Flavours strategies, as that is the version it works best with. For all other strategies, we assume Environment Modules version 4.x, with automated module handling mode enabled, as these strategies work best in that scenario.

The Homebrewed flavors, Modulerc-based, and Modulepath-based strategies all require a significant amount of "assembly" in order to get them working for a production environment; the example scripts and procedures provided here should help significantly, but should not be considered as a polished product. You will likely need to customize and extend for your environment. The Flavours Strategy, on the other hand, has been packaged to present as a finished product and so requires less "assembly" to get working, but does not appear to be actively maintained, so the reduced up-front work might be neutralized by the costs of self support.

Basic Dependency Handling

All of the strategies discussed support a basic level of dependency handling. If a user attempts to load a package, they get the build of the package appropriate for the previously loaded compiler or other dependencies, or, if no appropriate build is found, an error message indicating such. E.g, if the user does:

module load gcc/8.2.0
module load foo/1.1

their environment will be set up to run foo version 1.1 built with gcc version 8.2.0.

All of the strategies discussed meet this criterion, with both 3.x and 4.x versions of Environment Modules.

Advanced Dependency Handling (e.g. the module switch sub-command)

Things are more complicated when we allow for the modules upon which other loaded modules might depend to be changed. This generally involves the module switch command.

While all of the strategies handle well the easy case, when we switch a module upon which no other loaded modules depend, the trick comes when one or more currently loaded modules depend on the module being switched out. E.g., if we assume that module foo depends on the previously loaded compiler, then ideally, we would like for a sequence like:

module load intel/2019
module load foo
module switch intel pgi/18.4

to have the same effective outcome as if the user issued the commands:

module load pgi/18.4
module load foo

I.e., switching out the compiler will cause foo to be reloaded with the new compiler dependency. And ideally in this example we would have pgi/18.4 loaded along with the latest version of foo compatible with that compiler.

This is handled reasonably well with the Flavours (using Environment Modules 3.x) and the Homebrewed flavors and Modulepath-based (both using Environment Modules 4.x) strategies. For the Homebrewed flavors and Modulepath-based strategies, this relies on the Automated module handling feature (As of version 4, this is disabled by default. It can be enabled in modulecmd.tcl, or by running module config auto_handling 1, or by adding the --auto flag to the modules command.)

For these strategies, swapping out a compiler or other module upon which a loaded module depends will cause the the module with a dependency to be unloaded, the module being switched being swapped out, and the module that had a dependency being reloaded.

Our example sequence above actually exposes a subtle issue. For the Flavours and Homebrewed flavors strategies, the sequence of loading pgi/18.4 followed by foo (using our example software library) would fail, as foo would be defaulted to version 2.4 and as there was no build of foo/2.4 for pgi/18.4, the module load foo would fail. And indeed, the switch sequence above would not be completely successful under either strategy: for the Homebrewed Flavors Strategy, the compiler would be swapped out and the reload of foo would fail with an error (so effectively equivalent to the second sequence). For the Flavours Strategy, it detects that it would be unable to reload foo, and the switch command fails (i.e. the compiler remains intel/2019).

For the Modulepath-based Strategy, the second sequence (loading pgi/18.4 then foo) would actually work, as the modulepath exposed by loading pgi/18.4 only has versions of foo that are compatible with pgi/18.4. But the switch command does not work as well as would be desired. This is because that as currently implemented, when the automated module handling code does the reload of the module that was unloaded due to dependencies, it attempts to reload based on the fully qualified name of the module that was loaded, and not based on the (possibly partial) name specified when the module was originally loaded. So, after the module load intel/2019, the module load foo will result in foo/2.4 being loaded. When we switch out intel/2019 for pgi/18.4, foo gets unloaded, but after the compiler swap, it tries to reload foo/2.4. As there is no build of foo/2.4 for pgi/18.4 (and thus no modulefile in the current modulepath), the module is not found and is unable to be loaded. As in the Homebrewed flavors case, the compiler is switched but foo is left unloaded (with an appropriate warning to that effect).

This same issue prevents the Automated module handling feature as currently implemented from being very useful with the Modulerc-based. Basically, whenever you have a module loaded that depends on another module, switching out the module depended upon will fail to reload the other module. This is true even in less pathological cases, like:

module load intel/2019
module load foo
module switch intel pgi/19.4

which succeeds under the other three strategies (as foo/2.4 is built for pgi/19.4). This is because when foo is initially loaded in the above scenario, the actual modulename loaded is foo/intel/2019/2.4. With the switch command, foo gets unloaded, the compilers are switched, and the automated modules handling code tries to reload foo/intel/2019/2.4. That modulename clearly depends on intel/2019, not pgi/19.4, and so will fail to load (with an error message). The net result is that the compiler gets switched, but any modules depending on the compiler (or whatever module being switched) get unloaded (with a warning to that effect). It is hoped that a future version of the Automated module handling feature will have an option to reload the dependent modules using the modulename they were loaded with; this should allow for the switch command to work well in the Modulerc-based Strategy, as well as make edges case (like the pgi/18.4 example discussed above) work better with the Modulepath-based Strategy.

There is another edge case which only the Flavours Strategy handles well. Consider a scenario like:

module purge
module load intel/2019
module load foo
foo
# this should be foo/2.4 built for intel/2019 without MPI
module load openmpi
foo
# What build is this, with or without MPI?

With the Flavours Strategy, the final foo will be built for intel/2019 with openmpi support; i.e. Flavours supports the concept of optional prerequisites, and foo has an optional prerequisite on the MPI libraries, so when openmpi is loaded, any previously loaded modules which depend on it (through a regular or optional prerequisite) get reloaded.

All of the other strategies rely on automated module handling mode for automatic reloads, and in this case no reload of foo will be initiated as foo does not have MPI libraries listed as a prerequisite (otherwise a no MPI version could not be loaded). This is especially problematic for the Homebrewed flavors and Modulepath-based strategies, since it is not easily to tell from a module list or similar command which version of foo the environment is set for (for the Modulerc-based Strategy, the full modulename from module list will indicate that).

However, despite some potential issues with some edge cases, the Flavours, Homebrewed flavors, and Modulepath-based strategies all handle this challenge well. The Modulerc-based approach has a poor showing (although at least the modules depending on the module switched out will be unloaded, so the set of loaded modules is consistent).

Note also that this topic shows the most dependence on the version of Environment Modules. For the Flavours Strategy, the switching of a module upon which other modules depended does not work (the wrapper script returned weird errors). The other strategies all rely on the Automated module handling feature to enable the unload and reload of modules which depend on the module being switched out. Thus for 3.x versions of Environment Modules, the switch of a compiler, etc. will only result in the compiler being replaced, and any modules which depend on the compiler will not be reloaded. This means they will still be pointing to versions built for the old compiler, leading to an inconsistent set of modules being loaded. This is particularly bad in the Homebrewed flavors case, as a module load will not even inform one of that fact. Even if automated module handling mode is disabled on version 4.x of Environment Modules situation is better as loaded environment consistency is assured: the switch of a compiler will be denied since a dependent module is detected loaded.

Visibility into what packages are available

Another set of criteria to weigh has to do with visibility into the available packages. We are interested in

  1. Viewing all of the packages installed on a system

  2. Determining what versions of a specific package are available for a given compiler/MPI/etc combination.

  3. Seeing what compiler/MPI/etc combinations a specific version of a package

  4. For packages that we have currently loaded, determining which compiler/MPI/etc they were built for.

In terms of seeing all of packages that are available, the Flavours and Homebrewed flavors strategies are probably the best. The module avail command will list all modulefiles available, which will typically just be a list of packagename/version for each installed version. On a production system with many packages installed, even this can be a bit overwhelming to the user.

The module avail command will also list all modulefiles for every installed package in the Modulerc-based Strategy, but here every build of every version of every package will have a distinct modulefile. I.e., you will not just have foo/1.1 and foo/2.4 listings, but foo/1.1/gcc/8.2.0/nompi, foo/1.1/gcc/8.2.0/mvapich/2.1, foo/1.1/gcc/8.2.0/openmpi/3.1, foo/1.1/intel/2018/nompi, foo/1.1/intel/2018/intelmpi, etc. modulefiles. Actually, the situation is even worse than that, as in this strategy there are often multiple modulefiles for the same build in order to ease navigation of the module tree, e.g. we could have foo/1.1/gcc/8.2.0/nompi and foo/gcc/8.2.0/nompi/1.1 representing the same build of the same package. Whereas an unqualified module avail in the Flavours or Homebrewed flavors strategies can be a bit overwhelming in a large environment, with the Modulerc-based Strategy it can become practically unusable.

While an unqualified module avail in the Flavours, Homebrewed flavors, and especially Modulerc-based strategies can inundate the user with modulenames, the Modulepath-based Strategy has the opposite problem. With the Modulepath-based Strategy strategy, the modulefiles are split across multiple, often mutually incompatible, modulepaths, so the module avail command will never return a list of all modulefiles installed, only those available given the previously loaded compiler/MPI libraries/etc. E.g., if a package foobar is only installed for a particular compiler/MPI combination, it will not appear in any module avail listing unless that particular compiler and MPI were previously loaded.

While the number of modulefiles listed in an unqualified module avail command in the Modulerc-based Strategy is unwieldy, if one adds a partial package specification argument, the number of modulefiles returned is greatly reduced. This smaller list can provide information about which compilers/MPI/etc. are compatible with a specific version of a package. For example, to see the compilers for which foo/2.4 is built, we can do something like:

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module avail foo/2.4
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4 -
foo/2.4/gcc/(default)                 foo/2.4/intel/2019/mvapich/2.3.1
foo/2.4/gcc/9.1.0/mvapich/2.3.1       foo/2.4/intel/2019/nompi
foo/2.4/gcc/9.1.0/nompi(default)      foo/2.4/intel/2019/openmpi/4.0
foo/2.4/gcc/9.1.0/openmpi/4.0         foo/2.4/pgi/19.4/nompi(default)
foo/2.4/intel/2019/intelmpi(default)  foo/2.4/pgi/19.4/openmpi/3.1

Similarly, to see the builds of foo using gcc compilers, one can do something like:

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module avail foo/gcc
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4 -
foo/gcc/(default)              foo/gcc/8.2.0/openmpi/3.1/1.1
foo/gcc/8.2.0/(default)        foo/gcc/9.1.0/mvapich/2.3.1/2.4
foo/gcc/8.2.0/mvapich/2.1/1.1  foo/gcc/9.1.0/nompi/(default)
foo/gcc/8.2.0/nompi/(default)  foo/gcc/9.1.0/nompi/2.4
foo/gcc/8.2.0/nompi/1.1        foo/gcc/9.1.0/openmpi/4.0/2.4

The other strategies do not readily provide that information. The next best case is the Modulepath-based Strategy, because here at least a module avail will tell you what versions of a package are compatible with the currently loaded compiler/MPI/etc, e.g.:

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load intel/2019
[mod4 (modulepath)]$ module avail foo/2.4
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Compiler/intel/2019 -
foo/2.4

But the Flavours and Homebrewed flavors strategies do not readily show what versions of packages are built for which compilers, etc. You would need to load a compiler/MPI/etc combination, and then try loading a particular version of a package.

Lastly, with the Modulerc-based Strategy, the module list command explicitly shows information about the compiler/MPI/etc used to build the loaded modules, as that is part of the full modulename. This information is not explicitly visible in the other three strategies, but that is usually not a problem. As long as the various user commands result in a set of modules wherein every module depending on either a compiler or MPI library, etc. is built for the currently loaded compiler/MPI library/etc., that information is redundant.

The only cases where this is potentially an issue are the sort of edge cases described in the previous section, e.g. if one were to do something like:

module purge
module load intel/2019
module load foo
foo
# this should be foo/2.4 built for intel/2019 without MPI
module load openmpi
foo
# What build is this, with or without MPI?

In the above example, for the Flavours, Homebrewed flavors, and Modulepath-based strategies a module list at the end would simply list something like:

Currently Loaded Modulefiles:
1) intel/2019   2) foo/2.4   3) openmpi/4.0

For the Flavours Strategy, foo would be built with MPI support, but for the other two, foo would still be the non-MPI build, which is not readily apparent from the above output (although possibly could be inferred by the ordering of the modules).

Ease of Navigating the Module Tree

The final criterion to be discussed is the ease of navigating the module tree. Ideally, we would like it so that when an user gives a partial modulename the system resolves in a sensible manner. Typically a partial modulename will be a package name without a version, in which case ideally the latest version of the package (compatible with previously loaded compiler, etc), should be loaded. And indeed, for a simple case like:

module purge
module load intel
module load foo

all the strategies do just that. The intel module will default to intel/2019, and foo to version 2.4 built for intel/2019. However, with even a slight modification things begin to diverge, e.g.:

module purge
module load intel/2018
module load foo

In this second case, the Flavours and Homebrewed flavors strategies still default the version of foo to version 2.4, despite there being no build of foo/2.4 for intel/2018. That is because they do not do intelligent defaulting, and will simply default the version of foo to the highest version they see even though it is not compatible with the loaded compiler. Thus the load of foo will fail for Flavours and Homebrewed flavors strategies. However, for the Modulerc-based and Modulepath-based strategies, the module load of foo will successfully load foo version 1.1 built for intel/2018. For the Modulepath-based Strategy, this is because the only foo modules currently visible are compatible with the loaded intel/2018 compiler, and it simply defaults to the highest version seen. For the Modulerc-based Strategy, the .modulerc file directly under foo directs it down the subdirectory for the intel compiler family, which has a .modulerc defaulting to the 2018 subdirectory (as the currently loaded compiler version), which has a .modulerc defaulting to the nompi subdirectory (the MPI family as no MPI library was loaded). At this point, there is not another .modulerc file, so the latest version will be defaulted to, and foo/intel/2018/nompi/1.1 module will be loaded.

The situation is analogous when there are chained dependencies (e.g. compiler and MPI libraries); the Flavours and Homebrewed flavors strategies will default to the latest version of the package they know about, regardless of whether it is compatible with the loaded environment. If it is compatible, it will be loaded and all is well, if not the load command will fail with an error message. Not horrible; the set of loaded modules is still consistent, but not terribly user-friendly either.

For the Modulerc-based and Modulepath-based strategies, the latest build compatible with the various dependencies will be loaded (or if no build is compatible, the load will fail with an appropriate error).

For the Flavours, Homebrewed flavors, and Modulepath-based strategies, typically modulenames only consist of the package name and version components, so generally only partial modulenames seen are packages without a version. The Modulerc-based Strategy has much longer modulenames, and therefore a greater variety of partial modulenames, which can allow for interesting possibilities. For example, one could do module load foo/pgi, which (assuming no compiler was previously loaded), would load the latest version of foo built with the latest version of the pgi compiler. The module command will start looking for a modulefile in the foo/pgi subdirectory of the foo module structure, where the .modulerc file will attempt to default to the version of pgi which was loaded, but as no compiler was loaded, the modulerc file will terminate without setting a default, and the modulecmd will default to the latest version, and descend into that directory (19.4). Here, the .modulerc will attempt to default to the MPI family, but since none is loaded (as that would require a compiler to have been loaded), it defaults to the nompi directory, and then to the latest version in the nompi subdirectory. So the module foo/pgi/19.4/nompi/2.4 will get loaded, which will also cause pgi/19.4 to get loaded. Should we have issued that module load after having loaded a non-pgi compiler, the same module file would get loaded, but it would detect an incompatible previously loaded compiler and abort with an error. Should a pgi compiler have been previously loaded, we would have gotten the latest version of foo built for that compiler (and possibly with any previously loaded MPI library).

The above is supported because we create a PACKAGE/COMPILER_FAMILY/COMPILER_VERSION/.../PKGVERSION tree in the package's modulefile directory for packages depending on the compiler. Possibly with additional components to the modulename for MPI libraries, etc. These are created to allow for the more basic module load foo case, but because the module path is descended component by component, it works if you jump in in the middle as well (and will fail if you try something stupid like :

module load intel/2019
module load foo/pgi

But a more interesting case is when there are somewhat disjoint options on the builds. Consider the case of the bar package from the examples, which in addition to the compiler and builds supporting different SIMD levels. For the Flavours and Homebrewed flavors strategies, we handled this with a pseudo-simd package and modules. This worked, and allowed for one to select a build that matched both compiler and SIMD level, but you likely need to manually select a bar version compatible with both of these, as these strategies will not do intelligent defaulting of versions.

For the Modulepath-based Strategy, we added the SIMD label as a final component to the modulename, but the defaulting was somewhat simple (e.g. always default to lowest SIMD).

With the Modulerc-based Strategy, we added an additional naming scheme for the bar modulefiles, so they can start with bar/BARVERSION, or bar/COMPILER_FAMILY, or bar/SIMD_LEVEL. This not only makes it easier to view which versions of bar for which compilers are built with support for a given SIMD level (e.g. module avail bar/avx2), but also one can do something like:

module load pgi/19.4
module load bar/avx

to load the latest version of bar built for pgi/19.4 with AVX support. While there is overhead associated with adding a new naming scheme like that (you add another set of modulefiles to bar and a new subdirectory tree), this can be done on a per package basis and when done judiciously it can be convenient for the users.

Conclusions

We have presented four strategies for dealing with modulefiles for packages with multiple builds depending on compiler, MPI, and/or other factors. All four strategies can deal with the basic requirement of loading of the correct build of a package depending on the previously loaded dependencies, or failing with a reasonable error message if no such build is available. They all have their own strengths and weaknesses. This document tried to present these strategies objectively and has been made to help you to evaluate how to handle this issue at your site.