Handling Compiler and other Package Dependencies
When creating a collection of software (applications and libraries) for users to use, there is the problem of ensuring that the user is using the correct builds of everything. Generally, if an user is attempting to compile code making use of the system software collection, you want to ensure that the user is compiling his code with the same compiler that was used to compile the library. This tends to be particularly true of C++ and Fortran code using modules, and parallel codes using MPI libraries.
As a result, in environments supporting multiple compilers, software libraries often end up with multiple installs of libraries and applications of the same version, depending on the compiler and other libraries used to build them. Sometimes there are even additional installs for variants with different threading models, number formats, level of vectorization support, etc. This cookbook describes various strategies for handling the modulefiles to support all these different builds for each package.
For each strategy, we will provide an overview of how it works, and then show how an user might interact with it, usually a similar sequence for each case. We wish to explore how, and how well, each strategy succeeds in handling the multiple builds of the same version of a package, including
Basic dependency handling: seeing how well the strategy supports the loading of the correct build of a package depending on the previously loaded dependencies. And if no appropriate build is available, they should error accordingly.
The
module switchcommand and more advanced dependency handling: how well the strategy supports more advanced cases. E.g. a case wherein several modules are loaded and the user replaces a module upon which other modules currently loaded depend. In general, how well the strategy prevents the user's set of loaded modules from being incompatible.Visibility into what packages are available. This includes being able to readily see all of the packages installed, seeing what versions of packages are available for a given compiler/MPI/etc combination, and seeing for which compiler/MPI/etc combinations a specific version of a package is available.
How easily the user can navigate the modules for the builds. This includes how well partial modulenames (e.g. omitting version, etc) are handled by the different strategies.
We then try to summarize the strengths, weaknesses, and other attributes of each strategy. We also try to discuss differences in using on older (3.x) and newer (4.x) Environment Modules versions.
In addition to displaying examples for each strategy in this document, we have set up a the test environment as a playground in which you can explore.
Contents
Overview of Examples
The examples are a bit more elaborate than in some other cookbooks, so
the directory structure under doc/example/compiler-etc-dependencies
is similarly more complicated.
Example Software Library
For the purpose of the examples and the playground, we have created
a fake example software library, rooted at the subdirectory
doc/example/compiler-etc-dependencies/dummy-sw-root beneath where
you placed the modules source files. This software tree is intended
to represent some of the features you might see in a real software
tree, which supports various compiler and MPI libraries, and that has
been added to over time, and not always in the most systematic way.
The example software library does not contain any real code; there are
dummy scripts for e.g. gcc, mpirun, etc. which just echo then
name of the code and what version, compiler, etc. it was supposed to be
built for, which is handy to show in the examples that the modulefiles
are working as expected. It also shows how such a directory tree might
be laid out --- the details of the layout will affect some of the code
in the modulefiles, etc. The directory structure can be altered to
fit your standards, but would require some minor modification to the
modulefiles, etc.
Note that there are also a bunch of subdirectories named 1
containing symlinks, these are for the strategy using the Flavours add-on and are
discussed in that section.
The software in the example software library consists of:
GNU compiler versions
8.2.0and9.1.0Intel Parallel Studio suite versions
2018and2019(includes compilers, MPI and MKL)PGI compiler suite versions
18.4and19.4OpenMPI version
4.0, built for:gcc/9.1.0intel/2019pgi/19.4
OpenMPI version
3.1, built for:gccversions8.2.0and9.1.0intelversions2018and2019pgiversions18.4and19.4
mvapich version 2.3.1, built for:
gcc/9.1.0intel/2019pgi/19.4
mvapich version
2.1, built for:gccversions8.2.0and9.1.0intelversions2018and2019pgiversions18.4and19.4
foo version
2.4, built for:gcc/9.1.0andopenmpi/4.0gcc/9.1.0andmvapich/2.3.1gcc/9.1.0and no MPIintel/2019andopenmpi/4.0intel/2019andmvapich/2.3.1intel/2019andintelmpiintel/2019and no MPIpgi/19.4andopenmpi/3.1pgi/19.4and no MPI
foo version
1.1, built for:gcc/8.2.0andopenmpi/3.1gcc/8.2.0andmvapich/2.1gcc/8.2.0and no MPIintel/2018andopenmpi/3.1intel/2018andmvapich/2.1intel/2018andintelmpiintel/2018and no MPIpgi/18.4andopenmpi/3.1pgi/18.4andmvapich/2.1pgi/18.4and no MPI
bar version
5.4, built with:gcc/9.1.0and supportingavx2gcc/9.1.0and supportingavx
bar version
4.7, built for:gcc/8.2.0and supportingavxgcc/8.2.0and supportingsse4.1
I.e., we have 3 families of compiler suites with 2 different versions each. And two MPI families (openmpi and mvapich) with two versions each, with the most recent version only built with the latest compiler version of each family, and the older version built with both versions of each compiler family. In addition, it is assumed that the intel compiler suites include Intel's MPI library built for that compiler. The application foo depends on the compiler and optionally on MPI libraries and has two versions; the newer version mostly has builds for the latest compiler and MPI (for pgi it only supports the latest compiler and older openmpi), and the older version mostly has builds for the older compiler and MPI. The bar application depends on compiler and has variants depending on size of integers used in the API.
We also assume that the gcc/8.2.0 compiler is the system default; i.e. it is
the compiler provided by default by the Linux distro used by the system, and
therefore might potentially be available to users without loading any modules.
More directories under doc/example/compiler-etc-dependencies
The modulefiles for the different strategies do not play well with each
other, in part because we use the same names for many of the modules
between strategies. So in addition to the dummy-sw-root subdirectory,
each strategy has its own modulepath tree subdirectory
underneath doc/example/compiler-etc-dependencies. There are some minor
differences between the modulefiles for the Modulerc-based Strategy
depending on whether Environment Modules 3.x or 4.x is being used,
so we actually have two trees for that case (modulerc3 and modulerc4).
As there are a fair number of modulefiles, we make use of various tricks
in the cookbook Tips for Code Reuse in Modulefiles to minimize the amount of repeated
code. In general, the actual modulefiles are small "stubfiles", setting one or a few
Tcl variables, and then sourcing a common tcl file which does all the
real work. Symlinks are used where possible to avoid duplicating files.
The Modulerc-based Strategy also uses some complicated .modulerc files; these
are fairly generic and to avoid redundancy are symlinked into the appropriate
places in modulepath tree from the modrc_common directory.
We also in some cases use Tcl procedures; for the sake of the
examples these our sourced in the files as needed, but if one were to
use the strategies needing such in production it would be better to follow
the suggestions in Expose procedures and variables to modulefiles and
place the required procedures in site config script.
The various tcl procedures are placed in the tcllib sub-directory, outside
of the modulepaths. These are actually broken up into multiple files
for the purpose of this cookbook (so that smaller chunks of code can be
looked at in this document).
The example-sessions subdirectory contains various shell scripts
used for the usage examples for each strategy (shell scripts are used
because there are some slight variations required between the strategies),
as well as the outputs of running such scripts. Subdirectories exist
for each strategy, and beneath them for each of the two Environment
Modules versions (3.2.10 and 4.3.1) used; for brevity not all of them
are shown in this document, especially as the 3.x and 4.x differences
are often small. In the example outputs, the Environment Modules
version and the strategy being employed is indicated in the shell prompt
(e.g. mod3-flavours or mod4 (modulerc)).
Using the playground environments
Although we strive to provide a decent discussion in this cookbook, you are encouraged to try things out in the playground in order to get a better feel for things.
Because we use some modulefile names (e.g. gcc, intel, pgi, openmpi, etc)
that likely are present on your system as well, it is recommended
that if you wish to explore the playground environment that you
spawn a new shell, do a "module purge", and then set your MODULEPATH
environment variable appropriately for the specific strategy.
The Flavours_strategy, as will be discussed, requires some modifications to your Environment Modules installation. It is recommended that you you make a copy or new installation (for Flavours, a 3.x install works best), and then spawn a new shell and initialize the new Flavours install in that first. Flavours code is not provided with this cookbook.
Some of the modulefiles, etc. require knowledge of where they were
installed. To avoid requiring you to update lines in numerous
files, we require you to set the environment variable
MOD_GIT_ROOTDIR to location where the modules git working directory
was cloned. E.g., if you issued the command
git clone https://github.com/cea-hpc/modules.git ~/modules.test
you should set MOD_GIT_ROOTDIR to ~/modules.test. Please
ensure it is exported (use setenv in csh and related shells,
or export in Bourne derived shells like bash). This is just
a hack to make the examples work better; if you opt to use one of
these strategies in production, you will want to hard code some
relevant paths; the comments in the modulefiles will describe
what needs to be done.
Some more detail on setting up the playground is given at the start of the Examples section for each strategy.
Flavours Strategy
The Flavours strategy uses the Flavours extension to Environment Modules. Unlike the other strategies discussed, this requires the separate download and installation of an extension to Environment Modules.
Installation and Implementation
More details can be found at the website for this extension, but to install this you basically just need to:
Clone the git repo somewhere (
git clone https://git.code.sf.net/p/flavours/code flavours-code)Rename the standard Environment Modules
modulecmdfile (in thebinsubdirectory under the installation root) tomodulecmd.wrapped. (It is recommended that you do this in a copy of your production installation, or better yet, in a new install of the 3.x Environment Modules (as Flavours has been developed for Modules 3.x))Copy the
modulecmd.wrapperfile from Flavours to thebinsubdirectory above. Make sure themodulecmd.wrapperfile is executable.Symlink
modulecmd.wrappertomodulecmdEdit
modulecmd.wrapperwhere indicated to give fully qualified path tomodulecmd.wrappedCopy the
flavours.tclandpkgIndex.tclfiles to some (possibly new) directory under the modules installation root, and setTCLLIBPATHto that directory (you probably will want to add that to the various modules init scripts)
The module command invokes modulecmd, which in this case is results
in the Flavours wrapper bash script modulecmd.wrapper being invoked. This
calls the renamed standard modulecmd.wrapped command. This wrapper
command catches and processes certain output from the modulefile evaluation
intended for its consumption.
The modulefiles themselves make use of various commands in the Tcl module flavours.
Many of these are just flavours variants of standard modulefile commands, e.g.
flavours prepend-path versus prepend-path. Some important flavours
commands:
package require flavours: This loads the Tcl package flavours, and should occur near the top of your modulefileflavours init: This initializes the flavours package, and should be the first of the flavours commands issued. Typically call right after the package load.flavours prereq: Like the standardprereqcommand, this declares a prerequisite. But it also does quite a bit more, as is discussed further below.flavours root: This is used to set the root for where the package is actually installed. This is used when generating theflavours pathflavours revision: seem intended to allow for changes in the path format in future versions offlavours. It is used in constructing the final path to the package.flavours conflict: This is similar to the standard conflict command, but enhanced to recognize the flavours prereqs above.flavours commit: This should be called after theroot,revision, andprereqsub-commands offlavoursare called, and before any of thepathsub-commands. It seems to be responsible for taking all those values to above and constructing the path to the package.flavours path: This returns a string with the path to the specific build of the package.flavours prepend-path,flavour append-path: These work much like the standardprepend-pathandappend-path, except that the value being prepended/appended to the environment variable has the path (as returned byflavours path) prepended to it with the appropriate directory separator. E.g., to add to thePATHvariable the bin subdirectory of the root directory where the specific build was installed, useflavours prepend-path PATH binflavours cleanup: This should be called after allflavourssub-commands are finished and before exiting the script to ensure proper cleanup. Among other things, it ensures that any packages that depend on this package will get reloaded if this package is switched out.
The flavours prereq command accepts the new -class parameter, allowing
it to require a class of packages; e.g. one could use -class compiler to indicate
that it has a prereq on a compiler (any of the modules gnu, intel, or pgi).
The allowable classes, and the package basenames that are in each class, is defined in
flavours.tcl in the Tcl associative array _class. The ones shipped by default are
compiler: consisting of gnu, intel, and pgi
mpi: consisting of openmpi, mvapich2, mvapich, intelmpi
linalg: consisting of mkl, atlas, acml, netlib
You will likely want to adjust these if you go with flavours in production.
The flavours prereq command also accepts the parameter -optional, which declares
optional prerequisites. Although it sounds a little oxymoronic, this comes into play
with the secondary purpose of the command in declaring the components of the path, as
discussed below. If a prereq is not optional, the modulefile will complain if nothing
satisfying the prereq has been module loaded previously. If the prereq is optional,
the modulefile will not complain if it was not loaded, but will use the prereq in
constructing the path to the build of the package if it was loaded.
The flavours prereq command also defines the components which will comprise the final
path to the directory containing the specific build of the package. The order of the
prereq commands controls the order of the components in the path.
The modulefile will check that all non-optional flavours prereq commands are satisfied,
and then construct a path to the installation root for this build of the package using
the packages satisfying the prereqs. The resultant path is composed of:
the value from
flavours rootdirectory separator (
/)the value from
flavours revisiondirectory separator (
/)a
prefixcreated by concatenating the package names satisfying the prereqs, in order. The package name and version will be separated by a hyphen (-), as will the different components.
So if flavours root was set to /local/software/foo/1.7, revision to 1, and the
package had prereqs compiler and mpi, and gnu/9.1.0 and openmpi/4.0 were loaded, the
resulting path would be /local/software/foo/1.7/1/gnu-9.1.0-openmpi-4.0.
The modulefile actually will test for the existence of that directory, and if not found will return an error to the that the package was not built for that combination of prereqs. You either need to install your packages using the above directory schema, or create symlinks linking that scheme to where you actually install the packages.
Examples
We now look at the example modulefiles for flavours. To use the examples, you must
Have Flavours extension installed. NOTE these examples will NOT work without the Flavours installed.
Set (and export)
MOD_GIT_ROOTDIRto where you git-cloned the modules sourceDo a
module purge, and then set yourMODULEPATHto$MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/flavours
We start with the module avail command:
[mod4-flavours]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/flavours -
bar/4.7 foo/2.4 intel/2018 mvapich/2.1 openmpi/4.0 simd/avx
bar/5.4 gnu/8.2.0 intel/2019 mvapich/2.3.1 pgi/18.4 simd/avx2
foo/1.1 gnu/9.1.0 intelmpi/default openmpi/3.1 pgi/19.4 simd/sse4.1
We note that we only see the package names and versions; e.g. foo/2.4, without any mention
of the compilers and MPI libraries for which it is built. This terser type was an intentional
design goal of the authors. Also of note are the intelmpi and simd packages. The Flavours
approach relies on seeing what modules have been loaded previously in order to determine what
'flavor' of the requested package should be loaded. To support the different builds of bar
which depend on the CPU vectorization commands supported, we need to add a "dummy" package simd.
The module definition is quite trivial; a simple stub file like
#%Module
# Modulefile for CPU vectorization support
set simd avx
set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/common
and the main content in the common file:
# Common stuff for "simd" modulefiles
# Using "flavours" strategy
#
# This file expects the following Tcl variable to have been previously defined:
# simd: The level of simd support, eg. avx, avx2, sse4.1
# Initialise "flavours"
package require flavours
flavours init
proc ModulesHelp { } {
global simd
puts stderr "
This is a dummy modulefile to indicate that the CPU vectorization
should be set to '$simd' where possible.
"
}
module-whatis "CPU Vectorization support level: $simd"
# Even in production, this modulefile would not do anything
conflict simd
# Reload any modules with this as a prerequisite
flavours cleanup
Basically it just declares a help procedure and whatis text. This way, an user can
load the appropriate simd module to control which variant of bar they will get. The
only interesting aspect is that near the beginning of the file we do a
package require flavours and flavours init, and add a flavours cleanup
near the bottom. The lines at the beginning instruct Tcl command to load the Flavours
package, and then initialize the package.
The flavours cleanup is required so that if the simd module is switched out,
any modulefiles that depend on it get reloaded.
In our example, we assumed that the Intel MPI libraries are automatically set up properly if one
were to load the intel module, and we assumed the Intel MPI libraries were not supported
for either the GNU or PGI compilers. However, we also wished to allow for foo to
be used without any MPI support. So we need a way to distinguish if someone wants to
use an Intel compiler build of foo without MPI or with the Intel MPI libraries. Our
choice for this example was to require one to explicitly module load intelmpi if one
wished to use the Intel MPI variant --- we do not bother with a real version number because
assuming the version is determined by the version of intel (the Intel Parallel Studio version).
So the intelmpi modulefile is similar to the simd modulefiles, a dummy modulefile. Again,
it includes the flavours init and flavours cleanup wrapping to ensure proper reloading
of dependent modules should it be switched out.
If you were to support Intel MPI for non-intel compilers, you could create your intelmpi
modulefiles as usual, and then add a default or intel "dummy" version to use the
version that is part of the intel Parallel Studio. Or you could separate the intelmpi
bits from the intel modulefile so both non-intel and intel compilers need to explicitly
module load intelmpi.
The modulefiles for the various compilers are all pretty much standard, except for the
same three flavours lines as the simd modulefile: package require flavours,
flavours init, and flavours cleanup. These are required to
ensure dependent modulefiles get reloaded if the compiler is switched out.
We also note that the modulefile for the GNU Compiler Collection is referred to as gnu, not gcc
(this is due to how the compiler class is defined in flavours.tcl, and we did not bother
to change that for the purposes of this cookbook).
With the openmpi and mvapich MPI libraries, things start to get interesting. These all
should setup the environment for a different build depending on the compiler loaded. The
real work is done in the common tcl file, as shown below:
# Common stuff for "openmpi" modulefiles
# Using "flavours" strategy
#
# Expects the following Tcl variables to have been previously set:
# version: version of openmpi
# Common parts of modulefile for openmpi
# Initialise "flavours"
package require flavours
flavours init
proc ModulesHelp { } {
global version
puts stderr "
openmpi: Test dummy version of OpenMPI $version
For testing packages depending on compilers/MPI
"
}
module-whatis "Dummy openmpi $version"
# Construct flavour name
flavours prereq -class compiler
flavours conflict openmpi
# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root
flavours root $swroot/openmpi/$version
flavours revision 1
flavours commit
# Set environment variables
setenv MPI_DIR [flavours path]
# Prepend to environment variables (paths relative to
# the directory containing the flavour)
flavours prepend-path PATH bin
flavours prepend-path LIBRARY_PATH lib
flavours prepend-path LD_LIBRARY_PATH lib
flavours prepend-path CPATH include
# Reload any modules with this as a prerequisite
flavours cleanup
Like the previous cases, the file starts with the Tcl command to load the
package, followed by the flavours init command.
The flavours prereq command states that this package requires a compiler to have been previously
loaded, and that the path to the specific build to use will depend on that. We note the use
of the -class parameter; the exact definition of the compiler class is in the compiler
field of the Tcl associative hash _class defined in flavours.tcl.
The flavours root sets the root directory of where the builds for this package is installed.
We use the MOD_GIT_ROOTDIR environment variable for convenience in this example, but in production
you would generally hardcode a path. The result of all the directives is that the build will be
found in a path named after the compiler (since in this case there is only one flavour prereq);
e.g. for gcc version 9.1.0, we expect to find the build in $swroot/openmpi/4.0/1/gnu-9.1.0.
If you do not use that naming convention for your installation directories, you can use symlinks
to fake it.
The flavours path command in the setenv MPI_DIR statement sets MPI_DIR to the aforementioned
build path. The flavours prepend-path commands prepend to the environment variable specified
by the first argument the result of prepending the flavours path to their second argument. E.g.,
the first such, assuming openmpi version 4.0 was requested and gnu/9.1.0 loaded, would be basically
the same as a standard Modules command:
prepend-path PATH $swroot/openmpi/4.0/1/gnu-9.1.0/bin
The following shows how this would appear to the user:
[mod4-flavours]$ module purge
[mod4-flavours]$ module load pgi/19.4
[mod4-flavours]$ module load openmpi/4.0
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 2) openmpi/4.0
[mod4-flavours]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4-flavours]$ module unload openmpi
[mod4-flavours]$ module switch pgi intel/2019
[mod4-flavours]$ module load openmpi/4.0
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) openmpi/4.0
[mod4-flavours]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4-flavours]$ module unload openmpi
[mod4-flavours]$ module switch intel gnu/9.1.0
[mod4-flavours]$ module load openmpi/4.0
[mod4-flavours]$ mpirun
mpirun (openmpi/4.0, gcc/9.1.0)
[mod4-flavours]$ module unload openmpi
[mod4-flavours]$ module switch gnu gnu/8.2.0
[mod4-flavours]$ module load openmpi/4.0
openmpi/4.0 - no flavour compatible with modules 'gnu/8.2.0'
Loading openmpi/4.0
ERROR: Module evaluation aborted
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
1) gnu/8.2.0
Here we note that once a compiler is loaded, the PATH and the other environment
variables are set appropriately to point to the bin dir for the particular build of
openmpi/4.0, as evidenced by the output of our dummy mpirun command. At the end, we attempt
to load openmpi/4.0 for gnu/8.2.0, and receive an error because our dummy SW library does not contain
a matching build. This is determined from the flavours path; if the
path does not exist (in this example $swroot/openmpi/4.0/1/gnu-8.2.0) it will
abort in this fashion.
In the above, we have explicitly unloaded openmpi, switched the compilers, and then reloaded openmpi. A nice feature of Flavours is that it can handle the switching out of compilers or other modulefiles which other modulefiles depend on, as:
[mod3-flavours]$ module purge
[mod3-flavours]$ module load pgi/19.4
[mod3-flavours]$ module load openmpi
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 2) openmpi/4.0
[mod3-flavours]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod3-flavours]$ module switch pgi intel/2019
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) openmpi/4.0
[mod3-flavours]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod3-flavours]$ module switch intel intel/2018
openmpi/4.0 - no flavour compatible with modules 'intel/2018'
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) openmpi/4.0
[mod3-flavours]$ mpirun
mpirun (openmpi/4.0, intel/2019)
Note that when we switched between the pgi and intel compilers above, Flavours
automatically "unloaded" and "reloaded" the openmpi module. This happens in
the flavours cleanup portion of the compiler modulefiles, and is due to
openmpi declaring a flavours prereq on the compiler class.
Note
The above behavior with switch was done with version 3.2.10
of Environment Modules; it does not appear to work with 4.3.1.
Note that when we further tried to replace version 2019 of the intel compiler with the 2018 version,
the module switch of the compilers failed because openmpi/4.0 was not built with
intel/2018. Since the user never explicitly requested version 4.0 of openmpi (it was
defaulted in the initial load as the latest version of openmpi available for pgi/19.4),
it would have been nicer had the attempted reload of openmpi allowed it t:o default to the 3.1
version (as the latest version available for intel/2018). Nevertheless, it behaved well
in this situation; the module switch failed with a reasonable error message and the resulting
set of modules was still consistent.
We also note that if we attempt to load openmpi without having previously loading a compiler, we will get an error:
[mod4-flavours]$ module purge
[mod4-flavours]$ module load openmpi/3.1
openmpi/3.1 depends on one of the module(s) 'gnu intel pgi'
Loading openmpi/3.1
ERROR: Module evaluation aborted
[mod4-flavours]$ module list
No Modulefiles Currently Loaded.
[mod4-flavours]$ module purge
[mod4-flavours]$ module load gnu/8.2.0
[mod4-flavours]$ module load openmpi
openmpi/4.0 - no flavour compatible with modules 'gnu/8.2.0'
Loading openmpi/4.0
ERROR: Module evaluation aborted
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
1) gnu/8.2.0
In particular, there is no support for a "default" compiler; if e.g. you wished to make the
distribution supplied gcc the default compiler, you will need to have the initializations
scripts automatically do a module load of that compiler (possibly
a dummy modulefile like simd/intelmpi if the compiler is already in the user's path)
in your user's start up dot files or similar. We also note that there is no additional
intelligence in the version defaulting --- in the last example, we have gnu/8.2.0 loaded and if
we try to load openmpi without specifying a version, it defaults to version 4.0
as that is the latest version of openmpi without regard for the fact that there
is no build of openmpi version 4.0 for gnu/8.2.0 (but there is such for openmpi/3.1).
The situation for foo is more complicated, as it depends both on the compiler and
optionally on the MPI library. But with Flavours, the modulefile is only slightly more
complicated, e.g. for the common file is:
# Common stuff for "foo" modulefiles
# Using "flavours" strategy
#
# Expects the following Tcl variables to have been previously set:
# version: version of foo
# Initialise "flavours"
package require flavours
flavours init
proc ModulesHelp { } {
global version
puts stderr "
foo: Test dummy version of foo $version
For testing packages depending on compilers/MPI
"
}
module-whatis "Dummy foo $version"
# Construct flavour name
flavours prereq -class compiler
flavours prereq -optional -class mpi
flavours conflict foo
# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root
flavours root $swroot/foo/$version
flavours revision 1
flavours commit
# Set environment variables
setenv FOO_DIR [flavours path]
# Prepend to environment variables (paths relative to
# the directory containing the flavour)
flavours prepend-path PATH bin
flavours prepend-path LIBRARY_PATH lib
flavours prepend-path LD_LIBRARY_PATH lib
flavours prepend-path CPATH include
# Reload any modules with this as a prerequisite
flavours cleanup
Basically, the main difference is the addition of the
line flavours prereq -optional -class mpi.
This instructs Flavours that there is an additional, optional prereq. The
order of the prereq lines matter, as that controls the resultant flavors path.
With the current configuration, assuming gnu/9.1.0 and openmpi/4.0 were loaded,
the path would become $swroot/foo/2.4/1/gnu-9.1.0-openmpi-4.0. If the order were
reversed, the openmpi-4.0 would precede the gnu-9.1.0. Because the MPI requirement
is optional, if gnu/9.1.0 was loaded and no MPI library loaded, the path would
evaluate to $swroot/foo/2.4/1/gnu-9.1.0.
We show how it works below:
[mod4-flavours]$ module purge
[mod4-flavours]$ module load pgi/19.4
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 2) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (pgi/19.4, nompi)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module load openmpi/3.1
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 2) openmpi/3.1 3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (pgi/19.4, openmpi/3.1)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module unload openmpi
[mod4-flavours]$ module switch pgi intel/2019
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (intel/2019, nompi)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module load intelmpi
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) intelmpi/default 3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (intel/2019, intelmpi)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module switch intelmpi mvapich/2.3.1
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) mvapich/2.3.1 3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (intel/2019, mvapich/2.3.1)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module switch mvapich openmpi/4.0
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) openmpi/4.0 3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (intel/2019, openmpi/4.0)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module unload openmpi
[mod4-flavours]$ module switch intel/2019 gnu/9.1.0
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
1) gnu/9.1.0 2) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module load mvapich/2.3.1
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
1) gnu/9.1.0 2) mvapich/2.3.1 3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (gcc/9.1.0, mvapich/2.3.1)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module switch mvapich openmpi/4.0
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
1) gnu/9.1.0 2) openmpi/4.0 3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (gcc/9.1.0, openmpi/4.0)
So basically, if the user loads a compiler, the
the environment variables (PATH, etc) are set up for the correct build of foo.
If no MPI library was loaded, a version of foo built without MPI will be loaded,
otherwise, a version of foo built with the loaded MPI library will be loaded.
This is shown by the output of the foo command. Note
also how we use the dummy intelmpi package to indicate a desire for the
intelmpi enabled version.
The 3.x version of Environment Modules supports using the switch command on either the compiler or MPI library, and will result in reloading of foo and the MPI library.
[mod3-flavours]$ module purge
[mod3-flavours]$ module load pgi/18.4
[mod3-flavours]$ module load openmpi/3.1
[mod3-flavours]$ module load foo/1.1
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
1) pgi/18.4 2) openmpi/3.1 3) foo/1.1
[mod3-flavours]$ foo
foo 1.1 (pgi/18.4, openmpi/3.1)
[mod3-flavours]$ module switch pgi intel/2018
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
1) intel/2018 2) openmpi/3.1 3) foo/1.1
[mod3-flavours]$ foo
foo 1.1 (intel/2018, openmpi/3.1)
[mod3-flavours]$ mpirun
mpirun (openmpi/3.1, intel/2018)
[mod3-flavours]$ module purge
[mod3-flavours]$ module load intel/2019
[mod3-flavours]$ module load foo
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) foo/2.4
[mod3-flavours]$ foo
foo 2.4 (intel/2019, nompi)
[mod3-flavours]$ module load openmpi
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) foo/2.4 3) openmpi/4.0
[mod3-flavours]$ foo
foo 2.4 (intel/2019, openmpi/4.0)
In particular note the final case, wherein we load intel/2019 then foo, and get the
version of foo built without MPI. When we subsequently load openmpi, foo is reloaded
to be the openmpi version (this is because the hooks to reload foo are in the flavours cleanup
part of the openmpi modulefile, and foo declared its optional dependency on MPI).
Also, we don't bother showing it, but if you were to attempt to load foo without
at least a compiler loaded, it would display an error.
Our final example for flavours is the bar command. Here in addition to the
compiler dependency, we have versions for different SIMD vectorization supported.
Again, the difference in the modulefile is small, e.g.
# Common stuff for "bar" modulefiles
# Using "flavours" strategy
#
# Expects the following Tcl variables to have been previously set:
# version: version of bar
# Initialise "flavours"
package require flavours
flavours init
proc ModulesHelp { } {
global version
puts stderr "
bar: Test dummy version of bar $version
For testing packages depending on compilers/MPI
"
}
module-whatis "Dummy bar $version"
# Construct flavour name
flavours prereq -class compiler
flavours prereq simd
flavours conflict bar
# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root
flavours root $swroot/bar/$version
flavours revision 1
flavours commit
# Set environment variables
setenv BAR_DIR [flavours path]
# Prepend to environment variables (paths relative to
# the directory containing the flavour)
flavours prepend-path PATH bin
flavours prepend-path LIBRARY_PATH lib
flavours prepend-path LD_LIBRARY_PATH lib
flavours prepend-path CPATH include
# Reload any modules with this as a prerequisite
flavours cleanup
Basically, the optional flavours prereq on the mpi class from the foo package
is replaced by a (mandatory) flavours prereq on the simd dummy package.
We note that Flavours package knows nothing about our simd dummy package until
we add it as a prereq for bar. (This is in contrast to the compiler and mpi classes).
Usage would be like:
[mod4-flavours]$ module purge
[mod4-flavours]$ module load gnu/9.1.0
[mod4-flavours]$ module load simd/avx2
[mod4-flavours]$ module load bar/5.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
1) gnu/9.1.0 2) simd/avx2 3) bar/5.4
[mod4-flavours]$ bar
bar 5.4 (gcc/9.1.0, avx2)
[mod4-flavours]$ module unload bar
[mod4-flavours]$ module switch simd simd/avx
[mod4-flavours]$ module load bar/5.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
1) gnu/9.1.0 2) simd/avx 3) bar/5.4
[mod4-flavours]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4-flavours]$ module unload bar
[mod4-flavours]$ module switch simd simd/sse4.1
[mod4-flavours]$ module load bar/5.4
bar/5.4 - no flavour compatible with modules 'gnu/9.1.0 simd/sse4.1'
Loading bar/5.4
ERROR: Module evaluation aborted
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
1) gnu/9.1.0 2) simd/sse4.1
Here we note that as both the compiler and simd prereqs are non-optional, it complains
unless both have been previously loaded. When both have been loaded, the PATH and
other environment variables are set appropriately for the requested build; and if
it does not exist and error is produced.
Summary of Flavours
It is an external extension to Environment Modules, requiring additional installation steps.
The git repository appears to have been last updated in 2013; although which means that it has not been updated for Environment Modules 4.x, a simple experimentation indicates that it still works, with the exception of automatic reloading of a module if any of the modules it depends on are switched. However, the Flavours extension does not appear to be actively supported.
The Flavours package (using Environment Modules 3.x) fully supports the
module switchsyntax, with the switching out of a dependency (e.g. a compiler) causing the reload of all modulefiles depending on it. (however the test of this feature failed when using Environment Modules 4.x.)The syntax for modulefiles is elegant, and one can easily extend the basic compiler dependency modulefile to add additional dependencies. Even for packages/dummy packages that the Flavours extension knows nothing about (e.g. simd in the above example).
The Flavours package will provide a shorter module avail output, only e.g. giving package name and version and not listing a separate modulefile for each combination of package, version, compiler+version, MPI library+version, etc.
The Flavours package will fail with an error message if user tries to load a package which was not built for the values of the dependency packages loaded.
The Flavours package will fail to load with an error message if any dependent package is not already loaded. In particular, it will not attempt to default these. So if such defaults are desired, you will need to have initialization scripts automatically load the appropriate modules.
The Flavours package does not include any mechanism for more intelligent defaulting. I.e., if an user requests to load a package without specifying the version desired, the version will be defaulted to the latest version (or whatever the
.modulercfile specifies) without regard for which versions support the versions of the compiler and other prereq-ed packages the user has loaded. While one could write custom.modulercfiles for such, Flavours does not provide any tools for simplifying such.
Homebrewed Flavors Strategy
Although the Flavours extension described above has an elegance about it, one can achieve much of the same functionality in modulefiles using standard Environment Modules and Tcl commands. This can be facilitated by the definition of some useful Tcl procedures. For lack of a better name, we will refer to this strategy as Homebrewed flavors.
Implementation
This strategy just makes use of standard Environment Modules and Tcl procedures
to query what modules of a given type are loaded and to construct the path to the
software package accordingly. To avoid needless (and error prone) repetition of
code, we collect these into several Tcl procedures of our own. Ideally, these
should be placed in a site configuration Tcl file and exposed to modulefiles
as explained in the cookbook Expose procedures and variables to modulefiles.
However, to avoid the need for that in these examples, we instead have placed them
into a file and use the MOD_GIT_ROOTDIR to locate and source that file in the
relevant modulefiles. (Actually, we have a single tcl file that is sourced both
for this and some other strategies, and it sources several files so that we can
break up the discussion of the the Tcl procedures. All of that is just for the
purposes of this cookbook; normally you just put the procedures you need in the
one site config file).
We discuss the various Tcl procedures here, as they are what provide most of the functionality. We start with the routines for generic loaded modules:
#--------------------------------------------------------------------
# GetLoadedModules
#
# Returns a tcl list of all modules loaded. From $ENV{LOADEDMODULES}
proc GetLoadedModules { } {
global env
#Handle case if no modules loaded
if {![info exists env(LOADEDMODULES)]} {
#No modules loaded, return empty list
return [ list ]
}
set loadedenv $env(LOADEDMODULES)
set loaded [ split $loadedenv : ]
return $loaded
}
#--------------------------------------------------------------------
# GetTagOfModuleLoaded(pkg)
#
# Looks for a loaded module matching ^$pkg/, and returns the tag matched.
# Returns {} if no tags matched (i.e. module pkg not loaded)
proc GetTagOfModuleLoaded { mymodule } {
set loadedlist [ GetLoadedModules ]
set regex "^$mymodule/"
set fndidx [ lsearch -regex $loadedlist $regex ]
if { $fndidx == -1 } { return {} }
set found [ lindex $loadedlist $fndidx ]
return $found
}
This defines the two Tcl procedures:
GetLoadedModules : this returns the list of loaded modules, from the
LOADEDMODULESGetTagOfModuleLoaded : this takes as argument the base name of a package, and returns the first full spec for the matching package, or an empty string if no matching package found.
The Tcl procedure GetTagOfModuleLoaded can be used to find out what version of a given package is loaded, and is enough for many packages. However, for compilers, and similar, a bit more is needed. For compilers:
#--------------------------------------------------------------------
# GetDefaultCompiler:
#
# Returns the default compiler, gcc/8.2.0
proc GetDefaultCompiler { } {
return "gcc/8.2.0"
}
#--------------------------------------------------------------------
# RequireCompiler:
#
# Does a module load of specified compiler $mycomp.
# Includes special handling if $mycomp is the default compiler
proc RequireCompiler { mycomp } {
# If your module tree is set up so that there is no module for the
# default compiler (because e.g. it is available w/out loading a module
# anyway), you can uncomment the following block which will cause
# RequireCompiler to do nothing if mycomp is the default compiler
#set defComp [GetDefaultCompiler]
#if { $mycomp eq $defComp } {
#return
#}
module load $mycomp
}
#--------------------------------------------------------------------
# GetKnownCompilerFamilies:
#
# Returns a list of recognized compiler family names
#E.g. gcc, intel, pgi
proc GetKnownCompilerFamilies { } {
set cfamilies {gcc intel pgi}
return $cfamilies
}
#--------------------------------------------------------------------
# GetLoadedCompiler:
#
# Returns the string for the compiler we are using (i.e. was previously
# module loaded). E.g., gcc/8.2.0
# If no compiler was previously loaded, then if the optional parameter
# $pathDefault is set, it will look for a compiler family and
# version in the last components to the path to the current modulefile or
# .modulerc, and if found, uses that.
# If still no path found, it will return the value of [GetDefaultCompiler]
# if $useDefault is set.
# Otherwise, returns empty string.
#
# Takes the following arguments:
# pathDefault: boolean, default false. If set, attempt to determine
# the compiler from the full path to the modulefile if
# no compiler was loaded.
# useDefault: boolean, default false. If set, return the value of
# GetDefaultCompiler if no compiler is loaded or found from
# path (if $pathDefault).
# loadIt: boolean, default false. If set and a compiler
# was defaulted from path of GetDefaultCompiler, we will
# module load that compiler.
# Ignored unless either pathDefault or useDefault is set
# requireIt: boolean, default false. If set, we will prereq the
# compiler before returning.
proc GetLoadedCompiler {{pathDefault 0} { useDefault 0}
{loadIt 0 } { requireIt 0 } } {
global ModulesCurrentModulefile
set ctag {}
set cfams [ GetKnownCompilerFamilies ]
foreach cfam $cfams {
if { [ is-loaded $cfam ] } {
set ctag [ GetTagOfModuleLoaded $cfam ]
if { $requireIt } { prereq $ctag }
return $ctag
}
}
# No loaded compiler found, try to default from path to modulefile?
if { $pathDefault} {
set moduledir [file dirname $ModulesCurrentModulefile ]
set cversion [file tail $moduledir]
set tmppath [file dirname $moduledir]
set cfamily [file tail $tmppath]
if { [lsearch $cfams $cfamily] > -1 } {
# We matched a known compiler family in our path
set ctag "$cfamily/$cversion"
if { $loadIt } { RequireCompiler $ctag }
if { $requireIt } { prereq $ctag }
return $ctag
}
}
# Still no compiler, default to GetDefaultCompiler>
if { $useDefault } {
set ctag [ GetDefaultCompiler ]
if { $loadIt } { RequireCompiler $ctag }
if { $requireIt } { prereq $ctag }
return $ctag
}
#Nothing found, and not defaulting
return $ctag
}
We defined four procedures above:
GetDefaultCompiler : this simply returns the name of our default compiler, which for this example is
gcc/8.2.0RequireCompiler : this simply does a module load on the specified compiler. It is kept as a separate procedure just in case you wish to intercept and prevent the loading of the default compiler (e.g. because no modulefile exists for it). In our example, there is a modulefile for it and so it is just a wrapper for
module load.GetKnownCompilerFamilies : this simply returns a Tcl list of known compiler families.
GetLoadedCompiler: this is the procedure that does the main work, and is described in detail below.
The GetLoadedCompiler procedure basically checks if any packages matching
the names in GetKnownCompilerFamilies have been previously loaded. If so,
it returns the modulefile specification for the first one found, and returns.
If not, if pathDefault is set and there is a recognized compiler name and version
in the last two components of the module specification, it will return
that compiler. Otherwise, if the optional flag useDefault is set, it will return the
value from GetDefaultCompiler. If all else fails, returns the empty string.
If the optional parameter loadIt is set, if a compiler was defaulted (i.e.
not returned because it was already loaded), the procedure will call RequireCompiler
to module load it.
If the optional parameter requireIt is set, we invoke prereq on the compiler
found before returning.
A similar set of procedures exist for the MPI libraries, namely:
#--------------------------------------------------------------------
# RequireMPI:
#
# Does a module load of specified MPI library $mympi
# Includes special handling if $mympi is nompi or intelmpi (or one of its aliases)
# If $mympi is nompi, nothing is loaded.
# If the optional parameter noLoadIntel is set (default false), and if
# $mympi is intelmpi (or intel or impi), then we do not load module if
# the loaded compiler is intel (as we assume that provides intel MPI as well)
proc RequireMPI { mympi {noLoadIntel 0} } {
# We do not do anything if mympi is nompi
if { $mympi eq {nompi} } { return }
if { $noLoadIntel } {
# Get the basename of requested MPI library
set mympiSplit [ split $mympi / ]
set mympiBase [ lindex $mympiSplit 0 ]
# Check if we requested an intel MPI
set intelList "intelmpi impi intel intelmpi-mt impi-mt intel-mt"
if { [lsearch $intelList $mympiBase ] > -1 } {
# We requested intelmpi in some form
# Check if an intel compiler was loaded
set curComp [ GetLoadedCompiler 1]
if { $curComp ne {} } {
set curCompSplit [ split $curComp / ]
set curCompBase [ lindex $curCompSplit 0 ]
if { $curCompBase eq {intel} } {
# $noLoadIntel is set, requested MPI is intel MPI, and intel compiler loaded
return
}
}
}
}
module load $mympi
}
#--------------------------------------------------------------------
# GetKnownMpiFamilies:
#
# Returns a list of recognized MPI library family names
#E.g. gcc, intel, pgi
proc GetKnownMpiFamilies { } {
set mfamilies {openmpi mpavich intelmpi}
return $mfamilies
}
#--------------------------------------------------------------------
# GetLoadedMPI:
#
# Returns the string for the MPI library we are using (i.e. was previously
# module loaded), or empty string if nothing loaded (from is-loaded command)
# E.g., intel/2013.1.117
# Takes an optional argument,
# useIntel: boolean, default false. If set, returns 'intelmpi' if
# no MPI library is loaded but intel compiler is loaded
# forceIt: boolean, default false. If set, prereq MPI lib before returning.
# requireIt: boolean, default false. If set, prereq the MPI library
proc GetLoadedMPI { { useIntel 0} {forceIt 0} {requireIt 0} } {
set mtag {}
foreach mfam [ GetKnownMpiFamilies ] {
if { [ is-loaded $mfam ] } {
set mtag [ GetTagOfModuleLoaded $mfam ]
if { $requireIt } { prereq $ctag }
return $mtag
}
}
# No loaded compiler found, should we check for Intel compiler and return intelmpi?
if { $useIntel } {
#Yes
set ctag [ GetCompilerLoaded ]
set cSplit [ split $ctag / ]
set cBase [ lindex $cSplit 0 ]
if { $cBase eq intel } { return intelmpi }
}
return $mtag
}
The three procedures here are analogues of the compiler versions:
RequireMPI : this basically does a module load of the specified MPI library. It has some added logic so that it will not do a module load if the MPI library is
nompi. Also, if the optional parameternoLoadIntelis set, if the MPI library isintelmpi(or a variant of that name) and the loaded compiler inintel, we assume that no additional module needs to be loaded. For this strategy, we want to loadintelmpimodules, because, just like in the Flavours Strategy, we need to provide dummyintelmpimodules to allow one to request the use of the Intel MPI library.GetKnownMpiFamilies : this returns a list of known MPI library family names. Used in GetLoadedMPI
GetLoadedMPI : This is the analogue of GetLoadedCompiler. If an MPI library is loaded, it will return the name of that module. If the optional
requireItflag is set, it will do aprereqon the MPI library before returning. The first optional argument,useIntel, indicates whether this module should returnintelmpiif no MPI library is loaded but an Intel compiler is loaded.
The modulefiles for the compilers are basically standard; unlike the
Flavours Strategy there is nothing special needed in these. Likewise
for the dummy simd and intelmpi modules (the latter is only this
basic because we assume intelmpi is only available if an Intel compiler
is loaded. If one allowed for intelmpi with other compilers, it would
more closely resemble the other MPI libraries).
We also define some Tcl procedures for generating warning and error messages, namely
#--------------------------------------------------------------------
# PrintIfLoading:
#
# Prints supplied text to stderr but only if in "load" mode
proc PrintIfLoading { args } {
if [ module-info mode load ] {
set tmp [ join $args ]
puts stderr "$tmp"
}
}
#--------------------------------------------------------------------
# PrintLoadInfo:
#
# Prints supplied text to stderr as informational message, but only
# if actually trying to load the module.
proc PrintLoadInfo { args } {
set tmp [ join $args ]
PrintIfLoading "
\[INFO\] $tmp
"
}
#--------------------------------------------------------------------
# PrintLoadWarning:
#
# Prints supplied text to stderr as warning message, but only
# if actually trying to load the module.
proc PrintLoadWarning { args } {
set tmp [ join $args ]
PrintIfLoading "
WARNING:
$tmp
"
}
#--------------------------------------------------------------------
# PrintLoadError:
#
# Like PrintLoadWarning, but as error message and does a "break"
proc PrintLoadError { args } {
set tmp [ join $args ]
PrintIfLoading "
**** ERROR *****:
$tmp
"
if [ module-info mode load ] {
break
}
}
These procedures:
PrintIfLoading: will print supplied text to stderr only when in
loadmodePrintLoadInfo: will print supplied text as informational text, but only when trying to load a module
PrintLoadWarning: will print supplied text as warning text, but only when trying to load a module
PrintLoadError: will print supplied text as error text and abort, but only when trying to load a module
The gist of this is that we might wish to print errors if an user tries to load an incompatible modulefile, but do not wish to print errors if they are merely doing a help, display, or whatis command.
The interesting bit begins with the openmpi and mvapich modulefiles. These both depend on the compiler, we show the main part of the openmpi modulefile below:
# Common modulefile for openmpi
# Using "homebrewed flavors" strategy
# Expects the following variables to have been
# previously defined:
# version: version of openmpi
# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root
# Also get location of and load common procedures
# This is a hack for the cookbook examples, in production
# one should either
# 1) declare the procedures in a site config file (preferred)
# 2) hardcode the path to $tcllibdir and common_utilities.tcl
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl
proc ModulesHelp { } {
global version
puts stderr "
openmpi: Test dummy version of OpenMPI $version
For testing packages depending on compilers/MPI
"
}
module-whatis "Dummy openmpi $version"
# Figure out what compiler we have loaded
# Optional args ensure will default to and load/prereq default compiler
# if no compiler currently loaded
set ctag [ GetLoadedCompiler 1 1 1 ]
# Compute the installation prefix
set pkgroot $swroot/openmpi
set vroot $pkgroot/$version
set prefix $vroot/$ctag
# Make sure there is a build for this openmpi version/compiler
# I.e. that the prefix exists
if ![ file exists $prefix ] {
# Not built for this compiler, alert user and abort
PrintLoadError "
openmpi/$version does not appear to be built for compiler $ctag
Please select a different openmpi version or different compiler.
"
}
# We need to prereq the compiler to allow autohandling to work
prereq $ctag
# Set environment variables
setenv MPI_DIR $prefix
set bindir $prefix/bin
set libdir $prefix/lib
set incdir $prefix/include
prepend-path PATH $bindir
prepend-path LIBRARY_PATH $libdir
prepend-path LD_LIBRARY_PATH $libdir
prepend-path CPATH $incdir
We begin by sourcing the common_utilities file which defined the previously
described Tcl procedures. Normally it is recommended that you put those
procedures in a site config Tcl script and expose them to the modulefiles
using the techniques described in the cookbook Expose procedures and variables to modulefiles.
Even if you opt against that and decide to source a Tcl file, it is recommended
to hard code the path.
The next interesting bit comes when we set the local Tcl variable ctag
by calling the GetLoadedCompiler procedure. We allow the procedure to
default the compiler, and because we have a default compiler defined we
should always get a value. (If no default compiler was defined, one would
have to handle the error if no compiler was loaded/defaulted.) We then use
the value of ctag to set the path to the build of the package. To ensure
that the package is built for this compiler, we do a quick check that the
package installation path exists.
The modulefile for foo is a bit more complex:
# Common stuff for "foo" modulefiles
# Using "homebrewed flavors" strategy
#
# This file expects the following Tcl variables to have been
# previously set:
# version: version of foo
# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root
# Also get location of and load common procedures
# This is a hack for the cookbook examples, in production
# one should either
# 1) declare the procedures in a site config file (preferred)
# 2) hardcode the path to $tcllibdir and common_utilities.tcl
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl
proc ModulesHelp { } {
global version
puts stderr "
foo: Test dummy version of foo $version
For testing packages depending on compilers/MPI
"
}
module-whatis "Dummy foo $version"
# Figure out what compiler we have loaded
# Will default to and load default compiler if none loaded
set ctag [ GetLoadedCompiler 1 1 1 ]
# Figure out what MPI we have loaded
# We do NOT want to default to intelmpi if intel compiler loaded
set mtag [ GetLoadedMPI ]
# Set mtag it nompi if no MPI library found loaded
if { $mtag eq {} } { set mtag nompi }
# Set mtag to intelmpi if intelmpi/default found loaded
if { $mtag eq {intelmpi/default} } { set mtag intelmpi }
# Compute the installation prefix
set pkgroot $swroot/foo
set vroot $pkgroot/$version
set prefix $vroot/$ctag/$mtag
# Make sure there is a build for this foo version/compiler/MPI library
# I.e. that the prefix exists
if ![ file exists $prefix ] {
# Not built for this compiler/MPI, alert user and abort
PrintLoadError "
foo/$version does not appear to be built for compiler $ctag and MPI $mtag
Please select a different openmpi version or different compiler/MPI library combination.
"
}
# We need to prereq the compiler to allow autohandling to work
prereq $ctag
# and the MPI library if used
if { $mtag ne {nompi} } {
# We currently require one to load intelmpi to use Intel MPI with
# intel compilers. So we prereq on intelmpi as well
# But if not, we could place the prereq below in an if-then block
# so is only executed if mtag != intelmpi
prereq $mtag
}
# Set environment variables
setenv FOO_DIR $prefix
set bindir $prefix/bin
set libdir $prefix/lib
set incdir $prefix/include
prepend-path PATH $bindir
prepend-path LIBRARY_PATH $libdir
prepend-path LD_LIBRARY_PATH $libdir
prepend-path CPATH $incdir
conflict foo
The main difference between this modulefile, depending on both compiler and
optionally MPI, and the openmpi modulefile above, is that in addition to
detecting which compiler is loaded, we call GetLoadedMPI to determine
the MPI library which was loaded, and use both of them in constructing the
prefix to the installed foo.
Examples
We now look at the example modulefiles for the Homebrewed flavors strategy.
To use the examples, you must
#. Set (and export) MOD_GIT_ROOTDIR to where you git-cloned the modules source
#. Do a module purge, and then set your MODULEPATH to:
$MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/homebrewed
The Homebrewed flavors strategy behaves much like the Flavours Strategy in practice. The module avail command,
[mod4 (homebrewed)]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/homebrewed -
bar/4.7 foo/2.4 intel/2018 mvapich/2.1 openmpi/4.0 simd/avx
bar/5.4 gcc/8.2.0 intel/2019 mvapich/2.3.1 pgi/18.4 simd/avx2
foo/1.1 gcc/9.1.0 intelmpi/default openmpi/3.1 pgi/19.4 simd/sse4.1
looks basically the same, showing the a concise listing of packages and versions without information on the compilers and MPI libraries they were built with.
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load pgi/19.4
[mod4 (homebrewed)]$ module load openmpi/4.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 2) openmpi/4.0
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (homebrewed)]$ module unload openmpi
[mod4 (homebrewed)]$ module switch --auto pgi intel/2019
[mod4 (homebrewed)]$ module load openmpi/4.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) openmpi/4.0
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4 (homebrewed)]$ module unload openmpi
[mod4 (homebrewed)]$ module switch --auto intel gcc/9.1.0
[mod4 (homebrewed)]$ module load openmpi/4.0
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, gcc/9.1.0)
[mod4 (homebrewed)]$ module unload openmpi
[mod4 (homebrewed)]$ module switch --auto gcc gcc/8.2.0
[mod4 (homebrewed)]$ module load openmpi/4.0
**** ERROR *****:
openmpi/4.0 does not appear to be built for compiler gcc/8.2.0
Please select a different openmpi version or different compiler.
Loading openmpi/4.0
ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) gcc/8.2.0
Again, once a compiler is loaded, loading openmpi will set the PATH, etc. for
the correct build of openmpi, as evidenced by the output of the dummy
mpirun command. Notice that the module list command only shows
the version of openmpi loaded, and contains no information about what
compiler it was built with (you just have to assume it matches the loaded
compiler). And again we note that if one attempts to load a version
of openmpi (e.g. 4.0) that was not built for the specified compiler (e.g.
gcc/8.2.0), an error is generated.
Unlike in Flavours Strategy, we did not put any code in the modulefiles to cause
dependent modulefiles to be reloaded if a module they depend on gets switched
out. However, starting with Environment Modules 4.2, a feature called
Automated module handling was added. Without this feature, attempting to
switch out a module upon which other modules depended could be problematic,
as evidenced in this sequence below (using Environment Modules 3.2.10 and
so without automated module handling mode):
[mod3 (homebrewed)]$ module purge
[mod3 (homebrewed)]$ module load pgi/19.4
[mod3 (homebrewed)]$ module load openmpi
[mod3 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 2) openmpi/4.0
[mod3 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod3 (homebrewed)]$ module switch pgi intel/2019
[mod3 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) openmpi/4.0
[mod3 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
Here we note that we were able to switch out the pgi compiler for the intel compiler, but the openmpi module was not reloaded and the environment is still set for openmpi compiled with the pgi compiler, and that this inconsistency is not readily determined from the module list command.
Environment Modules 4.x, even with automated module handling disabled, is better ---
in a command sequence as above the module switch from pgi to intel would fail due to the
prereq module. However, with the Automated module handling
enabled (this feature is disabled by default but could be enabled on a per-command basis with the --auto
flag), things work much better, as evidenced below:
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load pgi/19.4
[mod4 (homebrewed)]$ module load openmpi
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 2) openmpi/4.0
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (homebrewed)]$ module switch --auto pgi intel/2019
Switching from pgi/19.4 to intel/2019
Unloading dependent: openmpi/4.0
Reloading dependent: openmpi/4.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) openmpi/4.0
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4 (homebrewed)]$ module switch --auto intel intel/2018
**** ERROR *****:
openmpi/4.0 does not appear to be built for compiler intel/2018
Please select a different openmpi version or different compiler.
Loading openmpi/4.0
ERROR: Module evaluation aborted
Switching from intel/2019 to intel/2018
WARNING: Reload of dependent openmpi/4.0 failed
Unloading dependent: openmpi/4.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) intel/2018
[mod4 (homebrewed)]$ mpirun
mpirun: command not found
When we switch out the pgi compiler for the intel/2019 compiler, the openmpi
module is automatically unloaded before the compiler switch and reload afterwards,
so we end up with the correct build of openmpi for the newly loaded intel/2019
compiler. If one then switches out intel/2019 replacing it with intel/2018,
the openmpi module is first unloaded, the compilers are switched, and as there
is no openmpi/4.0 build for intel/2018, a warning is given and the openmpi module
is left unloaded. Since the user never specifically requested version 4.0 of openmpi
(it was defaulted in the original module load of openmpi as that was the latest version
available for pgi/19.4), it would have been nicer if on the switch of intel compiler
versions the reload only attempted a module load openmpi instead of module load openmpi/4.0,
but nevertheless this well behaved. The openmpi module is dropped with a warning and the
user has a consistent set of modules loaded.
We note that the modulefile is able to default the compiler, so when we attempt to load openmpi without having previously loaded a compiler, as in
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load openmpi/3.1
Loading openmpi/3.1
Loading requirement: gcc/8.2.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) gcc/8.2.0 2) openmpi/3.1
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/3.1, gcc/8.2.0)
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load gcc/8.2.0
[mod4 (homebrewed)]$ module load openmpi
**** ERROR *****:
openmpi/4.0 does not appear to be built for compiler gcc/8.2.0
Please select a different openmpi version or different compiler.
Loading openmpi/4.0
ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) gcc/8.2.0
it will default to the default compiler, gcc/8.2.0. Note however, that if
one does not specify version 3.1 of openmpi, it will still default to 4.0
and fail to load as there is no build of openmpi/4.0 for gcc/8.2.0.
The situation is similar for foo:
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load pgi/19.4
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 2) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (pgi/19.4, nompi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module load openmpi/3.1
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 2) openmpi/3.1 3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (pgi/19.4, openmpi/3.1)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module unload openmpi
[mod4 (homebrewed)]$ module switch --auto pgi intel/2019
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module load intelmpi
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) intelmpi/default 3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, intelmpi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module switch --auto intelmpi mvapich/2.3.1
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) mvapich/2.3.1 3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module switch --auto mvapich openmpi/4.0
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) openmpi/4.0 3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, openmpi/4.0)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module unload openmpi
[mod4 (homebrewed)]$ module switch --auto intel/2019 gcc/9.1.0
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module load mvapich/2.3.1
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) mvapich/2.3.1 3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module switch --auto mvapich openmpi/4.0
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) openmpi/4.0 3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (gcc/9.1.0, openmpi/4.0)
Again, one can load a compiler without an MPI library to get the non-MPI version of foo, or a compiler and MPI library to get the MPI version. The dummy intelmpi modulefile is used to allow one to indicate that the Intel MPI library is desired. The automated module handling mode can again allow the switch functionality work properly, as in
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load pgi/18.4
[mod4 (homebrewed)]$ module load openmpi/3.1
[mod4 (homebrewed)]$ module load foo/1.1
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) pgi/18.4 2) openmpi/3.1 3) foo/1.1
[mod4 (homebrewed)]$ foo
foo 1.1 (pgi/18.4, openmpi/3.1)
[mod4 (homebrewed)]$ module switch --auto pgi intel/2018
Switching from pgi/18.4 to intel/2018
Unloading dependent: foo/1.1 openmpi/3.1
Reloading dependent: openmpi/3.1 foo/1.1
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) intel/2018 2) openmpi/3.1 3) foo/1.1
[mod4 (homebrewed)]$ foo
foo 1.1 (intel/2018, openmpi/3.1)
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/3.1, intel/2018)
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load intel/2019
[mod4 (homebrewed)]$ module load foo
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (homebrewed)]$ module load openmpi
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) foo/2.4 3) openmpi/4.0
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, nompi)
Here we note a deficiency in the switch support as compared to Flavours Strategy. In the last example
after loading intel/2019 and foo, we have the non-MPI build of foo as expected. However, upon
subsequently loading the openmpi module, we still have the non-MPI version of foo loaded, as evidenced
by the output of the dummy foo command. I.e., the foo package was not automatically reloaded, as
there was no prereq in the foo modulefile on an MPI library (as in the non-MPI build there is no MPI
library to prereq). Also note that module list does not really inform one of this fact.
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load foo
**** ERROR *****:
foo/2.4 does not appear to be built for compiler gcc/8.2.0 and MPI nompi
Please select a different openmpi version or different compiler/MPI library combination.
Loading foo/2.4
ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
No Modulefiles Currently Loaded.
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load foo/1.1
Loading foo/1.1
Loading requirement: gcc/8.2.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) gcc/8.2.0 2) foo/1.1
[mod4 (homebrewed)]$ foo
foo 1.1 (gcc/8.2.0, nompi)
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load pgi/18.4
[mod4 (homebrewed)]$ module load foo
**** ERROR *****:
foo/2.4 does not appear to be built for compiler pgi/18.4 and MPI nompi
Please select a different openmpi version or different compiler/MPI library combination.
Loading foo/2.4
ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) pgi/18.4
Above, we see once more that the compiler can be defaulted, but that the defaulting mechanism is not smart enough to default the version of foo based on the compiler loaded (or defaulted to).
The situation with bar is basically the same; with a compiler and simd module loaded, the environment for the appropriate build of bar is loaded when you module load bar.
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load gcc/9.1.0
[mod4 (homebrewed)]$ module load simd/avx2
[mod4 (homebrewed)]$ module load bar/5.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) simd/avx2 3) bar/5.4
[mod4 (homebrewed)]$ bar
bar 5.4 (gcc/9.1.0, avx2)
[mod4 (homebrewed)]$ module unload bar
[mod4 (homebrewed)]$ module switch --auto simd simd/avx
[mod4 (homebrewed)]$ module load bar/5.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) simd/avx 3) bar/5.4
[mod4 (homebrewed)]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4 (homebrewed)]$ module unload bar
[mod4 (homebrewed)]$ module switch --auto simd simd/sse4.1
[mod4 (homebrewed)]$ module load bar/5.4
**** ERROR *****:
foo/5.4 does not appear to be built for compiler gcc/9.1.0 and simd/sse4.1
Please select a different openmpi version or different compiler/simd combination.
Loading bar/5.4
ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) simd/sse4.1
And an error is generated if there is no build for that combination of compiler and simd. The automatic handling of modules again allows the switch command to work as expected:
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load gcc/9.1.0
[mod4 (homebrewed)]$ module load simd/avx
[mod4 (homebrewed)]$ module load bar/5.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) simd/avx 3) bar/5.4
[mod4 (homebrewed)]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4 (homebrewed)]$ module switch --auto simd simd/avx2
Switching from simd/avx to simd/avx2
Unloading dependent: bar/5.4
Reloading dependent: bar/5.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) simd/avx2 3) bar/5.4
[mod4 (homebrewed)]$ bar
bar 5.4 (gcc/9.1.0, avx2)
and both the simd level and compiler can be defaulted, but one still has to choose a version of bar which supports the defaults.
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load bar
[INFO] Setting simd to simd/sse4.1
**** ERROR *****:
foo/5.4 does not appear to be built for compiler gcc/8.2.0 and simd/sse4.1
Please select a different openmpi version or different compiler/simd combination.
Loading bar/5.4
ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
No Modulefiles Currently Loaded.
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load bar/4.7
[INFO] Setting simd to simd/sse4.1
Loading bar/4.7
Loading requirement: gcc/8.2.0 simd/sse4.1
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
1) gcc/8.2.0 2) simd/sse4.1 3) bar/4.7
[mod4 (homebrewed)]$ bar
bar 4.7 (gcc/8.2.0, sse4.1)
Summary of Homebrewed flavors strategy
The Automated module handling feature (introduced in Environment Modules
4.2) allows for the switching out of a dependency (e.g. a compiler) to cause the reload of all modulefiles depending on it. Without the automated module handling (as is default for 4.x, and the only option for 3.x), the switching of a compiler only changes the compiler and leaves the modulefiles that depend on the compiler unchanged.The various Tcl procedures make it somewhat easy to determine which compiler, MPI, etc. modules have been loaded and set the paths appropriately. Not as elegant or easy to use as Flavours Strategy, but not difficult
Like the Flavours package, the module avail output is concise, only e.g. giving package name and version rather than listing a separate modulefile for each combination of package, version, compiler+version, MPI library+version, etc.
Modules will fail with an error message if user tries to load a package which was not built for the values of the dependency packages loaded.
With Automated module handling mode, the dependencies of a module asked for load could automatically be loaded. But it requires that these dependencies are clearly specified with mention of the versions for which a build is available.
It does not include a mechanism for more intelligent defaulting. I.e., if an user requests to load a package without specifying the version desired, the version will be defaulted to the latest version (or whatever the
.modulercfile specifies) without regard for which versions support the versions of the compiler and other prereq-ed packages the user has loaded.Note that future releases of Environment Modules will introduce additional mechanisms to the Automated module handling mode, which will improve the user experience on such Homebrewed flavors setup.
Modulerc-based Strategy
The previous two strategies used additional code in the modulefile to
determine which compiler, etc. was loaded and adjust the values for
PATH, etc. accordingly. The Modulerc-based strategy instead uses
.modulerc files to direct the module command to the proper modulefile
depending on what compiler, etc. was previously loaded. Because of
this, there are a number of differences in behavior and what is seen
by the user, most notably many, many more modulefiles. Whether this is good
or bad is a matter of taste.
Implementation
Whereas the Homebrewed Flavors Strategy had the modulefile invoke
a Tcl procedure to determine which, if any, version of a module like a compiler
was loaded and then adjust paths, the Modulerc-based strategy instead
uses the same Tcl procedures to default the modulefile which will be loaded.
This implies that there is a distinct modulefile for every build of the package,
and an immediate consequence is that this strategy has many more modulefiles
than the others. We make use of the techniques in the cookbook
Tips for Code Reuse in Modulefiles to reduce the total amount of code; the actual modulefiles
for each build are typically small stubfiles defining a couple of Tcl variables
and then sourcing a common script (unique to each package) which does all
the real work.
The modules will be named with components for the different dependencies,
so the one for openmpi version 4.0 built with gcc version 9.1.0 would
be openmpi/4.0/gcc/9.1.0; similarly the module for foo version 1.1
built for pgi version 18.4 and mvapich 2.1 would be
foo/1.1/pgi/18.4/mvapich/2.1.
The .modulerc files themselves are not trivial, but these
can generally be written in a generic fashion, usable by multiple packages,
and can just be symlinked to the appropriate locations.
We define five such files which can be linked as .modulerc:
Two of these files can be linked in the module tree
at various places as .modulerc for defaulting the compiler. One to
default to the family portion of the compiler (e.g. gcc, intel, or pgi),
and one for the version. For the family portion of the compiler,
we have the file modulerc.select_compiler_family as below:
#%Module
# Choose compiler family
#
# This check if any compiler was previously "module loaded", or if no compiler
# previously loaded, will use the default compiler. Either way, it will check
# if there is a subdirectory matching the compiler family name, and if so
# will default to that.
#
# If no subdirectory matching compiler family name is found, we just return
# without defaulting. The modulecmd will default based on its internal rules,
# and it is up to the resulting module file to either load an appropriate
# compiler if no compiler is loaded or to abort with appropriate error
# messages if it wants an incompatible compiler.
#
# Usage:
# In most cases, can simply symlink .modulerc to this file
#
# In more complicated cases, .modulerc can source this file, and can
# then test the variable _did_default, which will be true if we set
# a default for modules for the next level, or false otherwise (in which
# case your .modulerc can set one)
# Source some required Tcl procedures here. Hack for cookbook
# making use of environment variable for location.
# In production, this should ideally be in a site config file.
# At minimum, hardcode the path
set rootdir $::env(MOD_GIT_ROOTDIR)
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl
set moduledir [file dirname $ModulesCurrentModulefile]
set baseComponent [ file tail $moduledir ]
# Get the currently loaded compiler, or default comp. Don't module load/prereq it
set fullCompilerTag [ GetLoadedCompiler 1 1 ]
set tmpCompTag [ GetPackageFamilyVersion $fullCompilerTag ]
set compilerFamily [ lindex $tmpCompTag 0 ]
set compFamList "$compilerFamily"
# Allow either gnu or gcc for GCC compilers
if { $compilerFamily eq {gcc} || $compilerFamily eq {gnu} } {
set compFamList "gnu gcc"
}
# _did_default will be true if we actually default something
# Useful in case a .modulerc sources us, and wants to set a default if we
# did not
set _did_default false
if { $compilerFamily ne {} } {
# Default to the family of currently loaded compiler
set firstChild [ FirstChildModuleInList $compFamList ]
if { $firstChild ne {} } {
# Compiler family found, so default to it
module-version $baseComponent/$firstChild default
set _did_default true
}
}
The file starts by sourcing a set of useful Tcl procedures. For the purpose
of the example for this cookbook this is done based on the MOD_GIT_ROOTDIR
environment variable. If you were to use this in production, it is
recommended that the Tcl procedures be placed in a site configuration script
and exposed to modulefiles via the techniques described in the cookbook
Expose procedures and variables to modulefiles. At the minimum, it is
recommended to hardcode the path to the common_utilities.tcl file.
The modulerc script then determines the directory it is in using the
Tcl variable ModulesCurrentModulefile. It then uses the
GetLoadedCompiler Tcl procedure (which was discussed in the section
on the Homebrewed Flavors Strategy. We then parse the resulting module
name into family and version pieces (we will discuss the procedure
GetPackageFamilyVersion later; for now suffice to say it takes
a module name and returns a Tcl list with family and version).
We do some trickery to support the use of either gcc or gnu as family names for
the GNU compiler suite, and then see if there is a directory or modulefile
underneath the directory containing the .modulerc file, and if so defaults
to it. For this purpose, we use the Tcl procedure FirstChildModuleInList
(defined in common_utilities.tcl). This and a related procedure are
defined as:
#--------------------------------------------------------------------
# FirstChildModuleInList
#
# Given a list of names of modulefile components, returns the
# first one found in the directory of the current modulefile
# (typically a .modulerc file). Returns empty string if none
# of them were found.
# NOTE: we do NOT check for module magic signature or even that symlink
# targets exist.
proc FirstChildModuleInList { modlist } {
global ModulesCurrentModulefile
# Get directory for current modulefile
set moduledir [ file dirname $ModulesCurrentModulefile ]
# See if any names in $modlist in moduledir
foreach mcomp $modlist {
if [ file exists $moduledir/$mcomp ] { return $mcomp }
}
# Nothing found
return
}
#--------------------------------------------------------------------
# ChildModuleExists
#
# Takes a module path component, and returns true if that path component
# exists beneath the current level, i.e. in the directory from which
# this .modulerc file was called.
# NOTE: we do NOT check for module magic signature or even that symlink
# targets exist.
proc ChildModuleExists { pathcomponent } {
global ModulesCurrentModulefile
# Get directory for current modulefile
set moduledir [ file dirname $ModulesCurrentModulefile ]
# See if file exists in moduledir
return [ file exists $moduledir/$pathcomponent ]
}
and basically make use of the Tcl variable ModulesCurrentModulefile. They
are limited, however, in that they will not work properly if the module is
split across multiple modulepaths. That is, ChildModuleExists and
FirstChildModuleInList will only detect children in the same directory
as the modulerc file they are called from; e.g., if two paths in the MODULEPATH
both have directories for openmpi, one with an intel subdirectory and one
with a pgi subdirectory and the modulerc invoking ChildModuleExists, the
ChildModuleExists procedure will not see the intel directory.
The modulerc.select_compiler_version file is similar,
#%Module
# Choose compiler version
#
# This check if any compiler was previously "module loaded", or if no compiler
# previously loaded, will use the default compiler. Either way, it will check
# if there is a subdirectory matching the compiler family name, and if so
# will default to that.
#
# If no subdirectory matching compiler family name is found, we just return
# without defaulting. The modulecmd will default based on its internal rules,
# and it is up to the resulting module file to either load an appropriate
# compiler if no compiler is loaded or to abort with appropriate error
# messages if it wants an incompatible compiler.
#
# Usage:
# In most cases, can simply symlink .modulerc to this file
#
# In more complicated cases, .modulerc can source this file, and can
# then test the variable _did_default, which will be true if we set
# a default for modules for the next level, or false otherwise (in which
# case your .modulerc can set one)
# Source some required Tcl procedures here. Hack for cookbook
# making use of environment variable for location.
# In production, this should ideally be in a site config file.
# At minimum, hardcode the path
set rootdir $::env(MOD_GIT_ROOTDIR)
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl
set moduledir [file dirname $ModulesCurrentModulefile]
set parentdir [ file tail $moduledir ]
# _did_default will be true if we actually default something
# Useful in case a .modulerc sources us, and wants to set a default if we
# did not
set _did_default false
# Get the currently loaded compiler or default, but do not load/prereq it
set compilerTag [ GetLoadedCompiler 1 1 ]
if { $compilerTag ne {} } {
# Compiler loaded or defaulted
# Make sure it matches our parent dir
set tmpCompTag [ GetPackageFamilyVersion $compilerTag ]
set compilerFamily [ lindex $tmpCompTag 0 ]
set compilerVersion [ lindex $tmpCompTag 1 ]
# Convert any gnus to gccs
set tmp1 $compilerFamily
if { $tmp1 eq {gnu} } { set tmp1 gcc }
set tmp2 $parentdir
if { $tmp2 eq {gnu} } { set tmp2 gcc }
if { $tmp1 eq $tmp2 } {
# If reach here, our parent dir agrees with
# the family of our loaded/defaulted compiler
if { $compilerVersion ne {} } {
# We have a version, does a modulefile/dir exist for it
if [ ChildModuleExists $compilerVersion ] {
# There is a modulefile/dir for this compiler version, default to it
module-version $compilerFamily/$compilerVersion default
set _did_default true
}
}
}
}
Again, we source the common_utitilies.tcl file and use
ModulesCurrentModulefile to get the directory in which the .modulerc
script resides. It then uses GetLoadedCompiler to determine what
if any compiler was loaded. If one was, we split into family and version,
and ensure that the directory containing the .modulerc file matches the
family name. If so, it checks if there is a directory or modulefile in that
directory matching the version name, and if so defaults to it.
For both of the .modulerc scripts we examined, we note that in case of
errors, etc., the script just exits without defaulting. One might be
tempted to have the .modulerc script actually return an error in such
cases (perhaps using module-info mode to only have it output during
load operations). This, however, runs into issues:
For Environment Modules 3.x: for some reason, when modulecmd processes
.modulercscripts,module-info mode loadalways returns true. Because of this, users would get spurious errors when doing amodule availif the.modulercscripts error-ed, because they all get processed regardless of what compiler is loaded.For versions 4.x: the modulecmd process tends to process all
.modulercfiles underneath the package root during a load, which will similarly generate spurious errors if the.modulercscripts throw errors.
Because of this, if an user explicitly requests a modulefile that conflicts
with the loaded compiler (e.g. the user does a module load pgi/19.4
followed by a module load openmpi/4.0/intel/2019), the modulefile needs
to detect this and error appropriately. A similar situation can happen if
one of the .modulerc scripts fail to default, presumably because there is no
appropriate build for the requested package; the modulecmd will default to
something, but likely not what is wanted. To facilitate this, we define
the Tcl procedure LoadedCompilerMatches:
#--------------------------------------------------------------------
# LoadedCompilerMatches:
#
# Takes the tag of the desired compiler $wanted (in form of FAMILY/VERSION)
# Checks if any compiler is loaded, and will throw error if one is
# loaded and it does not match $wanted.
# If compilers match, it will prereq the compiler if $requireIt set
# If no compiler is loaded, will load if optional boolean parameter
# $loadIt is set (default unset), and prereq if $requireIt is set, and
# then return.
# Option parameters:
# requireIt: boolean, default false. If set, require $wanted
# loadIt: boolean, default false. If set, load $wanted if no compiler
# already loaded
# modTag: string, defaults go [module-info specified]. Used in error messages
proc LoadedCompilerMatches { wanted {requireIt 0} { loadIt 0 } {modTag {} } } {
# Default modTag
if { $modTag eq {} } { set modTag [ module-info specified ] }
# If no compiler given in $wanted, just return
if { $wanted eq {} } { return }
# Get loaded compiler (w/out defaults)
set loaded [ GetLoadedCompiler 0 0 ]
# If no compiler is loaded, then require it if asked, and return
if { $loaded eq {} } {
if { $loadIt } {
RequireCompiler $wanted
if { $requireIt } { prereq $wanted }
}
return
}
# Have a loaded compiler, split into family and version
set tmpLoaded [ GetPackageFamilyVersion $loaded ]
set loadedFam [ lindex $tmpLoaded 0 ]
set loadedVer [ lindex $tmpLoaded 1 ]
# Always use 'gcc' not 'gnu'
if { $loadedFam eq {gnu} } { set loadedFam gcc }
# Split wanted into family and version
set tmpWanted [ split $wanted / ]
set wantedLen [ llength $tmpWanted ]
#Nothing to do if no components to wanted
if { $wantedLen < 1 } { return }
set wantedFam [ lindex $tmpWanted 0 ]
# Always use 'gcc' not 'gnu'
if { $wantedFam eq {gnu} } { set wantedFam gcc }
# Ensure families match
if { $loadedFam ne $wantedFam } {
PrintLoadError "Compiler Mismatch
Package $modTag does not appear to be built for currently
loaded compiler $loaded."
}
# OK, families match
# If no version specified in $wanted, we are basically done
if { $wantedLen < 2 } {
if { $requireIt } { prereq $wanted }
return
}
# Ensure versions match
set wantedVer [ lindex $tmpWanted 1 ]
if { $loadedVer eq $wantedVer } {
if { $requireIt } { prereq $wanted }
return
}
# Versions don't match
PrintLoadError "Compiler Mismatch
Package $modTag does not appear to be built for currently
loaded compiler $loaded."
}
The procedure takes a string for the family and version of the compiler
that the modulefile expects, and ensures that if a compiler is loaded, they
match. If they match, the procedure just returns, and if not, it spits out
an error indicating a compiler mismatch. It also takes optional boolean flags:
requireIt to determine if the procedure should prereq the compiler
before returning, and loadIt to determine if, when no compiler was
previously loaded, whether the procedure should load it (using
RequireCompiler so it can properly handle the default compiler specially
if needed). The requireIt flag should generally be set for Environment
Modules 4.x (as that will allow the automatic handling of modules to
properly recognize dependencies, even though as we will see that does not
work too well for this strategy as yet), and should not be set for versions
3.x (as the loading of modules with loadIt does not occur until after
the prereq, causing module loads to fail).
The resulting modulefile for something depending only on the compiler, using mvapich as an example, then would look like:
# Common modulefile for mvapich
# Using "modulerc based" strategy
# Expects the following variables to have been
# previously defined:
# version: version of mvapich
# compilerTag: the compiler to use
# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root
# Also get location of and load common procedures
# This is a hack for the cookbook examples, in production
# one should either
# 1) declare the procedures in a site config file (preferred)
# 2) hardcode the path to $tcllibdir and common_utilities.tcl
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl
proc ModulesHelp { } {
global version compilerTag
puts stderr "
mvapich: Test dummy version of mvapich $version
mvapich version: $version
Compiler: $compilerTag
For testing packages depending on compilers/MPI
"
}
module-whatis "Dummy mvapich $version (built for $compilerTag)"
# Make sure loadedCompiler matches what we want. And
# load compiler if no compiler loaded
LoadedCompilerMatches $compilerTag 1 1
# For Env Modules 3.x, set requireIt to 0
#LoadedCompilerMatches $compilerTag 0 1
# Compute the installation prefix
set pkgroot $swroot/mvapich
set vroot $pkgroot/$version
set prefix $vroot/$compilerTag
# Set environment variables
setenv MPI_DIR $prefix
set bindir $prefix/bin
set libdir $prefix/lib
set incdir $prefix/include
prepend-path PATH $bindir
prepend-path LIBRARY_PATH $libdir
prepend-path LD_LIBRARY_PATH $libdir
prepend-path CPATH $incdir
Basically, the modulefile knows what compiler it wants (in the above example,
that is set by the stubfile and passed as the Tcl variable compilerTag into the common
script above), and then calls the LoadedCompilerMatches procedure
above to ensure the loaded compiler matches what the modulefile wants (and
to load it if no compiler is loaded).
So the mvapich directory in the MODULEPATH consists of subdirectories for
each version of mvapich supported (e.g. 2.1 and 2.3.1). In each of the
version subdirectories, we symlink modulerc.select_compiler_family as
.modulerc, and create an additional layer of subdirectories, one for
each compiler family for which the version of mvapich is built (e.g.
gcc, intel, pgi). In each compiler family directory under each mvapich
version, we symlink modulerc.select_compiler_version as .modulerc,
and create a stub modulefile named for the compiler version. The stub
modulefile then defines some Tcl variables for the version of mvapich
and the compiler family/version, and sources the common file above.
E.g., for mvapich/2.3.1/intel/2019, the stubfile would look like
#%Module
# Dummy modulefile for mvapich
set version 2.3.1
set compilerTag intel/2019
set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/../../common
So the mvapich directory in MODULEPATH would have a structure like
mvapich
├── 2.1
│ ├── gcc
│ │ ├── 8.2.0
│ │ ├── 9.1.0
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version
│ ├── intel
│ │ ├── 2018
│ │ ├── 2019
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version
│ ├── .modulerc -> symlink to modulerc.select_compiler_family
│ └── pgi
│ ├── 18.4
│ ├── 19.4
│ └── .modulerc -> symlink to modulerc.select_compiler_version
├── 2.3.1
│ ├── gcc
│ │ ├── 9.1.0
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version
│ ├── intel
│ │ ├── 2019
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version
│ ├── .modulerc -> symlink to modulerc.select_compiler_family
│ └── pgi
│ ├── 19.4
│ └── .modulerc -> symlink to modulerc.select_compiler_version
└── common
With this directory structure, an user can load a compiler and then module load a specific version of mvapich, and if a build of that version of mvapich exists for that compiler, the environment will be set up properly, and otherwise an error reported. However, if the user module loads mvapich without specifying a version, it will simply default the version to the latest version of mvapich available, regardless of whether there is a build of that version of mvapich for the loaded compiler. Ideally, we would like it so that if one issues a module load of a package without specifying a version, it should load the latest version available that is compatible with any loaded compilers or other modules.
A simple change can enable that. We add a symlink to
modulerc.select_compiler_family as .modulerc directly under the
mvapich directory, and create an additional subdirectory under the mvapich
directory for each compiler family. We symlink
modulerc.select_compiler_version as .modulerc under each compiler
family subdirectory, along with a subdirectory for each version of that
compiler family that a version of mvapich is built for. We then symlink
the various stub modulefiles under the mvapich/VERSION into the
corresponding mvapich/COMPILER_FAMILY/COMPILER_VERSION directory, with
the symlink's name being the mvapich version number. So the mvapich
directory tree will now look like:
mvapich
├── 2.1
│ ├── gcc
│ │ ├── 8.2.0
│ │ ├── 9.1.0
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version
│ ├── intel
│ │ ├── 2018
│ │ ├── 2019
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version
│ ├── .modulerc -> symlink to modulerc.select_compiler_family
│ └── pgi
│ ├── 18.4
│ ├── 19.4
│ └── .modulerc -> symlink to modulerc.select_compiler_version
├── 2.3.1
│ ├── gcc
│ │ ├── 9.1.0
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version
│ ├── intel
│ │ ├── 2019
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version
│ ├── .modulerc -> symlink to modulerc.select_compiler_family
│ └── pgi
│ ├── 19.4
│ └── .modulerc -> symlink to modulerc.select_compiler_version
├── common
├── gcc
│ ├── 8.2.0
│ │ └── 2.1 -> symlink to ../../2.1/gcc/8.2.0
│ ├── 9.1.0
│ │ ├── 2.1 -> symlink to ../../2.1/gcc/9.1.0
│ │ └── 2.3.1 -> symlink to ../../2.3.1/gcc/9.1.0
│ └── .modulerc -> symlink to modulerc.select_compiler_version
├── intel
│ ├── 2018
│ │ └── 2.1 -> symlink to ../../2.1/intel/2018
│ ├── 2019
│ │ ├── 2.1 -> symlink to ../../2.1/intel/2019
│ │ └── 2.3.1 -> symlink to ../../2.3.1/intel/2019
│ └── .modulerc -> symlink to modulerc.select_compiler_version
├── .modulerc -> symlink to modulerc.select_compiler_family
└── pgi
├── 18.4
│ └── 2.1 -> symlink to ../../2.1/pgi/18.4
├── 19.4
│ ├── 2.1 -> symlink to ../../2.1/pgi/19.4
│ └── 2.3.1 -> symlink to ../../2.3.1/pgi/19.4
└── .modulerc -> symlink to modulerc.select_compiler_version
When one attempts to module load mvapich without specifying a version,
the .modulerc file will default to the family of the loaded compiler (or
the default compiler if none loaded). It then descends into the corresponding
compiler family directory, where the .modulerc there will default to the
version of the loaded or defaulted compiler. After descending into the
compiler version directory, there is no .modulerc, so the standard modulecmd
defaulting mechanism will select the latest version.
We now have two ways to refer to the same build of mvapich, namely one using the traditional form
PACKAGE/PKGVERSION/COMPILER_FAMILY/COMPILER_VERSION
(e.g. mvapich/2.3.1/pgi/19.4)
and a new one giving the compiler family and version immediately after package name, with the version of the package coming last
PACKAGE/COMPILER_FAMILY/COMPILER_VERSION/PKGVERSION
(e.g. mvapich/pgi/19.4/2.3.1)
These both refer to the same build of the package (e.g. version 2.3.1
of mvapich, built for version 19.4 of the PGI compiler suite).
Indeed, through the judicious use of symlinks, these actually refer to the
modulefile on disk, but with two distinct names depending on the path
used to get to it. In general, we allow for
modules to be named with multiple dependency packages and/or flags if
doing so would make it easier for a user to default to the correct package.
This is the primary reason for the procedure GetPackageFamilyVersion ---
it splits a module name into components and takes the first component
as the family name. It then examines the second component, and if it
matches the name of a known package (like a compiler family), it uses
the last component as the version. Otherwise, the second component is
assumed to be the version. The code is as follows:
#--------------------------------------------------------------------
# GetPackageFamilyVersion
#
# Given the fully qualified spec for a modulefile, returns a list
# with the package family name and version.
#
# E.g., for foo/1.1/gcc/8.2.0/openmpi/3.1 would return {foo 1.1}
# But also handles stuff like foo/gcc/8.2.0/openmpi/3.1/1.1 (returning
# again {foo 1.1})
proc GetPackageFamilyVersion { tag } {
# Return empty list if given empty tag
if { $tag eq {} } { return {} }
# Split tag into components
set components [ split $tag / ]
set compLen [ llength $components ]
if { $compLen < 1 } { return {} }
# Family should always be first
set family [ lindex $components 0 ]
if { $compLen < 2 } { return $family }
# First guess, version immediately follows family
set version [ lindex $components 1 ]
# Check if it matches known non-version stuff following family
# Compilers, MPIs, other things branch on
set nonVers { gnu gcc pgi intel openmpi ompi intelmpi impi mvapich
mvapich2 avx avx2 sse4.1 sse3 python perl hdf hdf4 hdf5 }
set tmp [ lsearch $nonVers $version ]
if { $tmp > -1 } {
# The component immediately following family was NOT a version
# Use last component as version
set version [ lindex $components end ]
}
return "$family $version"
}
Similarly, there are two corresponding .modulerc scripts for defaulting
the MPI library: one to default the family (e.g. openmpi, mvapich, intelmpi)
and one to default the version. The family version, shown below:
#%Module
# Choose MPI family
#
# This check if any MPI lib was previously "module loaded"
#
# If no MPI library was previously module loaded (and intel compiler suite
# was not loaded, see below) and nompiFlag is set (default is set), then
# if a "nompi" subdirectory exists, we will default to that.
# If nompiFlag is not set, then we just return w/out setting any default
# (allowing modulecmd to default on its own. It is the responsibility
# and the resulting modulefile to lado/require the MPI lib it needs or err
# as appropriate).
#
# If an MPI library was loaded, then we check for a subdirectory named after
# the family of the loaded MPI library, and if such exists, default to that.
# If it does not exist, we return w/out defaulting, allowing modulecmd to
# default according to its own rules. The resulting modulefile is responsible
# for aborting/complaining about the MPI library mismatch.
#
# Special handling is needed for intel MPI, as it does not get explicitly
# loaded if the Intel compiler suite is being used. So if no MPI library
# is loaded, we check to see if an Intel compiler was loaded. If so, we
# check for an "intelmpi" or similar subdirectory, and if found default to
# that. If not found, we proceed as above for the no MPI library case
# (defaulting to "nompi" if found and nompiFlag set, or just returning w/out
# defaulting otherwise).
#
# Usage:
# In most cases, can simply symlink .modulerc to this file
#
# In more complicated cases, .modulerc can source this file, and can
# then test the variable _did_default, which will be true if we set
# a default for modules for the next level, or false otherwise (in which
# case your .modulerc can set one)
# Source some required Tcl procedures here. Hack for cookbook
# making use of environment variable for location.
# In production, this should ideally be in a site config file.
# At minimum, hardcode the path
set rootdir $::env(MOD_GIT_ROOTDIR)
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl
#Default nompiFlag to set
if ![info exists nompiFlag] { set nompiFlag true }
# _did_default will be true if we actually default something
# Useful in case a .modulerc sources us, and wants to set a default if we
# did not
set _did_default false
set moduledir [file dirname $ModulesCurrentModulefile]
set baseComponent [ file tail $moduledir ]
# Get the currently loaded MPI library
set fullMpiTag [ GetLoadedMPI 0 ]
set tmpMpiTag [ GetPackageFamilyVersion $fullMpiTag ]
set mpiFamily [ lindex $tmpMpiTag 0 ]
if { $mpiFamily eq {} } {
# No MPI module loaded
# What compiler is loaded (default from path if needed, but not GetDefaultCompiler)
set fullComp [ GetLoadedCompiler 1 0 ]
set tmpComp [ GetPackageFamilyVersion $fullComp ]
set compFam [ lindex $tmpComp 0 ]
if { $compFam eq {intel} } {
# OK, no MPI explicitly loaded, but using Intel compiler
# So set mpiFamily to first found in list below
set mpiList {intelmpi impi intel intelmpi-mt impi-mt intel-mp}
set mpiFamily [ FirstChildModuleInList $mpiList ]
}
}
if { $mpiFamily eq {} } {
# No MPI lib was explicitly loaded, and if intel compiler was loaded,
# did not find a subdir matching intelmpi or one of its variants
# Looks for a "nompi" dir if $nompiFlag is set
if { $nompiFlag } {
if [ ChildModuleExists nompi ] {
# nompi subdir exists, default to it
module-version $baseComponent/nompi default
set _did_default true
}
}
} else {
# Either MPI lib was explicitly loaded, or implied intelmpi
# Default to the subdir named after MPI family, if it exists
if [ ChildModuleExists $mpiFamily ] {
module-version $baseComponent/$mpiFamily default
set _did_default true
}
}
is similar to the corresponding compiler version, but contains additional logic to handle the intelmpi case. We treat intelmpi specially, because in our example here we assume the environment for Intel MPI is setup properly when the user loads the intel module. If no MPI library is explicitly loaded, but the intel module is loaded, than we look for a subdirectory named intelmpi (or one of its aliases) and default to it if it exists. Otherwise, if no MPI module is loaded, it looks for and will default to a directory or modulefile named nompi if it exists.
The modulerc.select_mpi_version script is also similar to its compiler
counterpart,
#%Module
# Choose MPI version
#
# This check if any MPI lib was previously "module loaded"
#
# If an MPI library was loaded, and the family of loaded MPI lib matches
# the name of the parent dir for this .modulerc, and there is a subdir
# of that directory matching the MPI version, then we default to that
# subdir.
#
# If no MPI library was previously module loaded, or the loaded MPI family
# does not match the name of the parent directory, or no subdir matching
# the MPI version is found, then we just return w/out defaulting.
# The modulecmd will then default according to its own rules, and it is
# up to the resulting modulefile to either load the appropriate MPI library
# or abort/complain about an MPI mismatch as appropriate.
#
# NOTE: no special handling is needed for intelmpi, as we assume that if
# intelmpi is selected because the intel compiler was loaded, then we
# are to use the intelmpi that shipped with the compiler suite, and so
# there is no needed for additional versioning of the intelmpi beneath
# the intelmpi subdir. If intelmpi is being used with a non-intel compiler,
# then it is assumed intelmpi was explicitly loaded.
#
# Usage:
# In most cases, can simply symlink .modulerc to this file
#
# In more complicated cases, .modulerc can source this file, and can
# then test the variable _did_default, which will be true if we set
# a default for modules for the next level, or false otherwise (in which
# case your .modulerc can set one)
# Source some required Tcl procedures here. Hack for cookbook
# making use of environment variable for location.
# In production, this should ideally be in a site config file.
# At minimum, hardcode the path
set rootdir $::env(MOD_GIT_ROOTDIR)
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl
# _did_default will be true if we actually default something
set _did_default false
set moduledir [file dirname $ModulesCurrentModulefile]
set parentDir [ file tail $moduledir ]
# Get the currently loaded compiler, w/out defaulting
set fullMpiTag [ GetLoadedMPI 0 ]
if { $fullMpiTag ne {} } {
# We have an MPI module loaded
set tmpMpiTag [ GetPackageFamilyVersion $fullMpiTag ]
set mpiFamily [ lindex $tmpMpiTag 0 ]
set mpiVersion [ lindex $tmpMpiTag 1 ]
# Canonicalize intelmpi variants in parentDir and mpiFamily for comparison
set intelList "intelmpi impi intel intelmpi-mt impi-mt intel-mt"
set tmpMpiFamily $mpiFamily
if { [lsearch $intelList $tmpMpiFamily] > -1 } {
set tmpMpiFamily intelmpi
}
set tmpParentDir $parentDir
if { [lsearch $intelList $tmpParentDir] > -1 } {
set tmpParentDir intelmpi
}
if { $tmpParentDir eq $tmpMpiFamily } {
# The loaded MPI lib family name matches parentDir
# So see if have subdir matching version, and if so, default it
if { $mpiVersion ne {} } {
if [ ChildModuleExists $mpiVersion ] {
module-version $mpiFamily/$mpiVersion default
set _did_default true
}
}
}
}
It checks if any MPI library was explicitly loaded, and if so it checks
if the family of the loaded MPI module matches the name of the parent directory
of this .modulerc file. If so, it checks for the existence of a subdirectory
matching the version, and if found it defaults to it. There is no special
handling needed for the nompi case, as since there is no version attached
to the MPI library in the nompi case, this .modulerc is not present there.
Similarly, for the case of Intel MPI when the Intel compiler suite is loaded, we
expect the Intel MPI version to be that from the Intel compiler suite, so
no version or this .modulerc is needed.
As the MPI modules depend themselves on the compiler, it is assumed that any package depending on the MPI libraries also depend on the compiler, and so will have a structure like the one described below for foo
foo
├── 1.1
│ ├── gcc
│ │ ├── 8.2.0
│ │ │ ├── .modulerc -> symlink to modulerc.select_mpi_family
│ │ │ ├── mvapich
│ │ │ │ ├── 2.1
│ │ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ │ │ ├── nompi
│ │ │ └── openmpi
│ │ │ ├── 3.1
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version
│ ├── intel
│ │ ├── 2018
│ │ │ ├── intelmpi
│ │ │ ├── .modulerc -> symlink to modulerc.select_mpi_family
│ │ │ ├── mvapich
│ │ │ │ ├── 2.1
│ │ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ │ │ ├── nompi
│ │ │ └── openmpi
│ │ │ ├── 3.1
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version
│ ├── .modulerc -> symlink to modulerc.select_compiler_family
│ └── pgi
│ ├── 18.4
│ │ ├── .modulerc -> symlink to modulerc.select_mpi_family
│ │ ├── mvapich
│ │ │ ├── 2.1
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ │ ├── nompi
│ │ └── openmpi
│ │ ├── 3.1
│ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ └── .modulerc -> symlink to modulerc.select_compiler_version
├── 2.4
│ ├── gcc
│ │ ├── 9.1.0
│ │ │ ├── .modulerc -> symlink to modulerc.select_mpi_family
│ │ │ ├── mvapich
│ │ │ │ ├── 2.3.1
│ │ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ │ │ ├── nompi
│ │ │ └── openmpi
│ │ │ ├── 4.0
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version
│ ├── intel
│ │ ├── 2019
│ │ │ ├── intelmpi
│ │ │ ├── .modulerc -> symlink to modulerc.select_mpi_family
│ │ │ ├── mvapich
│ │ │ │ ├── 2.3.1
│ │ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ │ │ ├── nompi
│ │ │ └── openmpi
│ │ │ ├── 4.0
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version
│ ├── .modulerc -> symlink to modulerc.select_compiler_family
│ └── pgi
│ ├── 19.4
│ │ ├── .modulerc -> symlink to modulerc.select_mpi_family
│ │ ├── nompi
│ │ └── openmpi
│ │ ├── 3.1
│ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ └── .modulerc -> symlink to modulerc.select_compiler_version
├── common
├── gcc
│ ├── 8.2.0
│ │ ├── .modulerc -> symlink to modulerc.select_mpi_family
│ │ ├── mvapich
│ │ │ ├── 2.1
│ │ │ │ └── 1.1 -> ../../../../1.1/gcc/8.2.0/mvapich/2.1
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ │ ├── nompi
│ │ │ └── 1.1 -> ../../../1.1/gcc/8.2.0/nompi
│ │ └── openmpi
│ │ ├── 3.1
│ │ │ └── 1.1 -> ../../../../1.1/gcc/8.2.0/openmpi/3.1
│ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ ├── 9.1.0
│ │ ├── .modulerc -> symlink to modulerc.select_mpi_family
│ │ ├── mvapich
│ │ │ ├── 2.3.1
│ │ │ │ └── 2.4 -> ../../../../2.4/gcc/9.1.0/mvapich/2.3.1
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ │ ├── nompi
│ │ │ └── 2.4 -> ../../../2.4/gcc/9.1.0/nompi
│ │ └── openmpi
│ │ ├── 4.0
│ │ │ └── 2.4 -> ../../../../2.4/gcc/9.1.0/openmpi/4.0
│ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ └── .modulerc -> symlink to modulerc.select_compiler_version
├── intel
│ ├── 2018
│ │ ├── intelmpi
│ │ │ └── 1.1 -> ../../../1.1/intel/2018/intelmpi
│ │ ├── .modulerc -> symlink to modulerc.select_mpi_family
│ │ ├── mvapich
│ │ │ ├── 2.1
│ │ │ │ └── 1.1 -> ../../../../1.1/intel/2018/mvapich/2.1
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ │ ├── nompi
│ │ │ └── 1.1 -> ../../../1.1/intel/2018/nompi
│ │ └── openmpi
│ │ ├── 3.1
│ │ │ └── 1.1 -> ../../../../1.1/intel/2018/openmpi/3.1
│ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ ├── 2019
│ │ ├── intelmpi
│ │ │ └── 2.4 -> ../../../2.4/intel/2019/intelmpi
│ │ ├── .modulerc -> symlink to modulerc.select_mpi_family
│ │ ├── mvapich
│ │ │ ├── 2.3.1
│ │ │ │ └── 2.4 -> ../../../../2.4/intel/2019/mvapich/2.3.1
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ │ ├── nompi
│ │ │ └── 2.4 -> ../../../2.4/intel/2019/nompi
│ │ └── openmpi
│ │ ├── 4.0
│ │ │ └── 2.4 -> ../../../../2.4/intel/2019/openmpi/4.0
│ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ └── .modulerc -> symlink to modulerc.select_compiler_version
├── .modulerc -> symlink to modulerc.select_compiler_family
└── pgi
├── 18.4
│ ├── .modulerc -> symlink to modulerc.select_mpi_family
│ ├── mvapich
│ │ ├── 2.1
│ │ │ └── 1.1 -> ../../../../1.1/pgi/18.4/mvapich/2.1
│ │ └── .modulerc -> symlink to modulerc.select_mpi_version
│ ├── nompi
│ │ └── 1.1 -> ../../../1.1/pgi/18.4/nompi
│ └── openmpi
│ ├── 3.1
│ │ └── 1.1 -> ../../../../1.1/pgi/18.4/openmpi/3.1
│ └── .modulerc -> symlink to modulerc.select_mpi_version
├── 19.4
│ ├── .modulerc -> symlink to modulerc.select_mpi_family
│ ├── nompi
│ │ └── 2.4 -> ../../../2.4/pgi/19.4/nompi
│ └── openmpi
│ ├── 3.1
│ │ └── 2.4 -> ../../../../2.4/pgi/19.4/openmpi/3.1
│ └── .modulerc -> symlink to modulerc.select_mpi_version
└── .modulerc -> symlink to modulerc.select_compiler_version
Although the directory tree above is somewhat lengthy, it is similar
to the case for a module depending only on the compiler, but expanded
to handle MPI dependencies as well. There is a subdirectory under foo
for each version of foo. Each of these subdirectories have further subdirectories
for the compiler families for which that version of foo was built, and under
the subdirectories for each compiler family are subdirectories for each version
of that compiler family for which foo was built. Underneath these
are stub modulefiles for nompi (and for intel compilers, intelmpi) builds of
the package, and subdirectories for each MPI family, containing stub modulefiles
for each version of the MPI family the package was built for.
Additionally, there are subdirectories directly under foo for each compiler
family the package was built for, with each having subdirectories for the
version of the compiler. Beneath each of those are one or two more layers
of subdirectories for the MPI family and version (the version layer is omitted
in the nompi or intelmpi cases), underneath which are symlinks to the corresponding
stub modulefile from the foo/FOOVERSION path.
By placing the appropriate .modulerc files (actually, symlinks to the correct
modulerc file), when the user enters a partial modulename the modulecmd will
descend this directory tree based on the previously loaded compiler and MPI
library to get to correct build of the package, if it exists. If no MPI library
was previously loaded, it will search for an intelmpi build if an intel compiler
was loaded, or a nompi build if no intelmpi build found or a non-intel compiler
was used. If only MPI builds were made, it will default (using standard
module defaulting rules) to one of the MPI builds, and the appropriate MPI
library will be loaded when the foo module gets loaded (via the LoadedMPIMatches
Tcl procedure described below). If a compiler and MPI library were loaded but no
build for that combination exists, again a modulefile will be defaulted to
(using standard module defaulting rules), but an error will be generated by
the one of the LoadedCompilerMatches or LoadedMPIMatches calls in the
foo modulefile (see below).
The common code of the modulefile is fairly standard, as shown below
# Common stuff for "foo" modulefiles
# Using "modulerc based" strategy
#
# This file expects the following Tcl variables to have been
# previously set:
# version: version of foo
# compilerTag: compiler was built for
# mpiTag: mpi was built for
# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root
# Also get location of and load common procedures
# This is a hack for the cookbook examples, in production
# one should either
# 1) declare the procedures in a site config file (preferred)
# 2) hardcode the path to $tcllibdir and common_utilities.tcl
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl
proc ModulesHelp { } {
global version compilerTag mpiTag
puts stderr "
foo: Test dummy version of foo $version
For testing packages depending on compilers/MPI
Foo: $version
Compiler: $compilerTag
MPI: $mpiTag
"
}
module-whatis "Dummy foo $version ($compilerTag, $mpiTag)"
# Make sure loaded compiler and MPI match what we want.
# And load them if not already loaded
LoadedCompilerMatches $compilerTag 1 1
# But don't load intelmpi if compiler is intel
LoadedMpiMatches $mpiTag 1 1 1
# For Env Modules 3.x, set requireIt to false
#LoadedCompilerMatches $compilerTag 0 1
#LoadedMpiMatches $mpiTag 0 1 1
# Compute the installation prefix
set pkgroot $swroot/foo
set vroot $pkgroot/$version
set prefix $vroot/$compilerTag/$mpiTag
# Set environment variables
setenv FOO_DIR $prefix
set bindir $prefix/bin
set libdir $prefix/lib
set incdir $prefix/include
prepend-path PATH $bindir
prepend-path LIBRARY_PATH $libdir
prepend-path LD_LIBRARY_PATH $libdir
prepend-path CPATH $incdir
conflict foo
The main difference from a standard modulefile is the inclusion of the
invocations of LoadedCompilerMatches and LoadedMpiMatches. This
is needed to ensure that the compiler and MPI for the modulefile being
processed are compatible with any which are loaded. The LoadedMpiMatches
procedure is defined by:
#--------------------------------------------------------------------
# LoadedMpiMatches:
#
# Takes the tag of the desired MPI library $wanted (in form of FAMILY/VERSION)
# Checks if any MPI is loaded, and will throw error if one is
# loaded and it does not match $wanted.
# If MPIs match, it will prereq the MPI lib if $requireIt set
# If no MPI is loaded, will load if optional boolean parameter
# $loadIt is set (default unset), and prereq if $requireIt is set, and
# then return.
# Option parameters:
# requireIt: boolean, default false. If set, will prereq $wanted
# loadIt: boolean, default false. If set, load $wanted
# if no MPI is already loaded
# noLoadIntel: boolean, passed to RequireMPI. If set,
# will not attempt to load intelmpi variants if compiler is intel
# forceNoMpi: boolean, default false. If set and $wanted is 'nompi',
# insist that no MPI module is loaded
# modTag: string, defaults go [module-info specified]. Used in error messages
proc LoadedMpiMatches { wanted {requireIt 0} { loadIt 0 } { noLoadIntel 0 }
{ forceNoMpi 0 } {modTag {} } } {
# Default modTag
if { $modTag eq {} } { set modTag [ module-info specified ] }
# If no MPI given in $wanted, just return
if { $wanted eq {} } { return }
# Get loaded MPI, no defaulting to intelmpi
set loaded [ GetLoadedMPI 0 ]
# If wanted is nompi, no need to do anything unless forceNoMpi set
if { $wanted eq {nompi} } {
if { $forceNoMpi } {
# Complain if any MPI loaded
if { $loaded ne {} } {
PrintLoadError "MPI Mismatch
You have an MPI library loaded ($loaded), but package $modTag
really insists that no MPI library be loaded.
Please unload the MPI library and try again."
}
}
return
}
# If no MPI is loaded, then load it if $loadIt (this is valid even
# in edge cases of nompi or intelmpi), prereq it if $requireIt,
# abd return
if { $loaded eq {} } {
if { $loadIt } {
RequireMPI $wanted $noLoadIntel
if { $requireIt } { prereq $wanted }
}
return
}
# Have a loaded MPI, split into family and version
set tmpLoaded [ GetPackageFamilyVersion $loaded ]
set loadedFam [ lindex $tmpLoaded 0 ]
set loadedVer [ lindex $tmpLoaded 1 ]
# Split wanted into family and version
set tmpWanted [ split $wanted / ]
set wantedLen [ llength $tmpWanted ]
#Nothing to do if no components to wanted
if { $wantedLen < 1 } { return }
set wantedFam [ lindex $tmpWanted 0 ]
# Ensure families match
if { $loadedFam ne $wantedFam } {
PrintLoadError "MPI Mismatch
Package $modTag does not appear to be built for currently
loaded MPI library $loaded."
}
# OK, families match
# If no version specified in $wanted, we are basically done
if { $wantedLen < 2 } {
if { $requireIt } { prereq $wanted }
return
}
# Ensure versions match
set wantedVer [ lindex $tmpWanted 1 ]
if { $loadedVer eq $wantedVer } {
if { $requireIt } { prereq $wanted }
return
}
# Versions don't match
PrintLoadError "MPI Mismatch
Package $modTag does not appear to be built for currently
loaded MPI library $loaded."
}
Basically, it determines what if any MPI library was previously module loaded.
If one was loaded, it ensures the family and version match what was expected,
and if not exit with an error. The use of the GetPackageFamilyVersion
procedure is important, because as discussed previously packages can be
represented by more than one name (e.g. the modulefile for version 4.0 of
openmpi for gcc/9.1.0 could be named openmpi/4.0/gcc/9.1.0 or
openmpi/gcc/9.1.0/4.0) and we need to ensure either is recognized.
Although we only showed cases for compilers and MPI libraries, the techniques
above can be adapted to other cases as well. For simple cases, where everything
falls under a single package name, one can use GetTagOfModuleLoaded to
determine the version of the package loaded in the .modulerc and in the
modulefile to ensure the version matches.
For bar, we added variants on CPU vectorization support. Rather than add a simd modulefile, we just add variants to the bar modules. So the bar modulefile tree would look like:
bar
├── 4.7
│ ├── gcc
│ │ ├── 8.2.0
│ │ │ ├── avx
│ │ │ ├── .modulerc -> symlink to modulerc.default_lowest_simd
│ │ │ └── sse4.1
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version
│ └── .modulerc -> symlink to modulerc.select_compiler_family
├── 5.4
│ ├── gcc
│ │ ├── 9.1.0
│ │ │ ├── avx
│ │ │ └── avx2
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version
│ └── .modulerc -> symlink to modulerc.select_compiler_family
├── avx
│ ├── gcc
│ │ ├── 8.2.0
│ │ │ └── 4.7 -> ../../../4.7/gcc/8.2.0/avx
│ │ ├── 9.1.0
│ │ │ └── 5.4 -> ../../../5.4/gcc/9.1.0/avx
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version
│ └── .modulerc -> symlink to modulerc.select_compiler_family
├── avx2
│ ├── gcc
│ │ ├── 9.1.0
│ │ │ └── 5.4 -> ../../../5.4/gcc/9.1.0/avx2
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version
│ └── .modulerc -> symlink to modulerc.select_compiler_family
├── common
├── gcc
│ ├── 8.2.0
│ │ ├── avx
│ │ │ └── 4.7 -> ../../../4.7/gcc/8.2.0/avx
│ │ ├── .modulerc -> symlink to modulerc.default_lowest_simd
│ │ └── sse4.1
│ │ └── 4.7 -> ../../../4.7/gcc/8.2.0/sse4.1
│ ├── 9.1.0
│ │ ├── avx
│ │ │ └── 5.4 -> ../../../5.4/gcc/9.1.0/avx
│ │ ├── avx2
│ │ │ └── 5.4 -> ../../../5.4/gcc/9.1.0/avx2
│ │ └── .modulerc -> symlink to modulerc.default_lowest_simd
│ └── .modulerc -> symlink to modulerc.select_compiler_version
├── .modulerc -> symlink to modulerc.select_compiler_family
└── sse4.1
├── gcc
│ ├── 8.2.0
│ │ └── 4.7 -> ../../../4.7/gcc/8.2.0/sse4.1
│ └── .modulerc -> symlink to modulerc.select_compiler_version
└── .modulerc -> symlink to modulerc.select_compiler_family
Here we added yet another alternate set of module names. So the module
for bar version 5.4 built with gcc version 9.1.0 and avx2 support can be
called:
bar/5.4/gcc/9.1.0/avx2bar/gcc/9.1.0/avx2/5.4bar/avx2/gcc/9.1.0/5.4
Due to this multiplicity of names, if an user does a module load bar, the
.modulerc immediately under bar will cause modulecmd to default along the
path for the loaded (or defaulted) compiler, and then the
modulerc.default_lowest_simd will default to the lowest simd level.
If the user module loads bar/VERSION, it will find the build for that version
of bar with the appropriate compiler (and again, default to the lowest simd
level). And if the user module loads bar/avx or another simd level, then
it will load the latest version of bar built for the loaded compiler and
simd specified.
The modulerc.default_lowest_simd script looks like:
#%Module
# Default to lowest SIMD support available
#
# Defaults to first of sse4.1, avx, or avx2 found
#
# Usage:
# In most cases, can simply symlink .modulerc to this file
#
# In more complicated cases, .modulerc can source this file, and can
# then test the variable _did_default, which will be true if we set
# a default for modules for the next level, or false otherwise (in which
# case your .modulerc can set one)
# Source some required Tcl procedures here. Hack for cookbook
# making use of environment variable for location.
# In production, this should ideally be in a site config file.
# At minimum, hardcode the path
set rootdir $::env(MOD_GIT_ROOTDIR)
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl
set moduledir [file dirname $ModulesCurrentModulefile]
set baseComponent [ file tail $moduledir ]
# _did_default will be true if we actually default something
# Useful in case a .modulerc sources us, and wants to set a default if we
# did not
set _did_default false
set simdList {sse4.1 avx avx2}
set firstChild [ FirstChildModuleInList $simdList ]
if { $firstChild ne {} } {
module-version $baseComponent/$firstChild default
set _did_default true
}
It will default to the lowest SIMD level, but could easily be adapted to do something else.
Examples
We now look at the example modulefiles for the Modulerc-based strategy.
As noted previously, the best choice of whether to set the requireIt flag
to LoadedCompilerMatches and LoadedMPIMatches (instructing them to
to a prereq on the requesting compiler/MPI module) depends on whether
one is using a 3.x or 4.x version of Environment Modules. Due to this,
we provide two modulefile trees, one for 3.x and one for 4.x; they are
basically identical except for that matter. This leads to slightly
different instructions on how to use the examples, depending on which
version of Environment Modules is being used, namely:
Set (and export)
MOD_GIT_ROOTDIRto where you git-cloned the modules sourceDo a
module purgeIf using Environment Modules 3.x, set your
MODULEPATHto$MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc3If using Environment Modules 4.x, set your
MODULEPATHto$MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4
As with the previous cases, we start with a module avail command, and here
we see the first big difference:
[mod4 (modulerc)]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4 -
bar/4.7/gcc/(default) foo/intel/2019/intelmpi/2.4
bar/4.7/gcc/8.2.0/(default) foo/intel/2019/mvapich/2.3.1/2.4
bar/4.7/gcc/8.2.0/avx foo/intel/2019/nompi/2.4
bar/4.7/gcc/8.2.0/sse4.1(default) foo/intel/2019/openmpi/4.0/2.4
bar/5.4/gcc/(default) foo/pgi/18.4/mvapich/2.1/1.1
bar/5.4/gcc/9.1.0/avx foo/pgi/18.4/nompi/(default)
bar/5.4/gcc/9.1.0/avx2 foo/pgi/18.4/nompi/1.1
bar/avx/gcc/(default) foo/pgi/18.4/openmpi/3.1/1.1
bar/avx/gcc/8.2.0/(default) foo/pgi/19.4/nompi/(default)
bar/avx/gcc/8.2.0/4.7 foo/pgi/19.4/nompi/2.4
bar/avx/gcc/9.1.0/5.4 foo/pgi/19.4/openmpi/3.1/2.4
bar/avx2/gcc/(default) gcc/8.2.0
bar/avx2/gcc/9.1.0/5.4 gcc/9.1.0
bar/gcc/(default) intel/2018
bar/gcc/8.2.0/(default) intel/2019
bar/gcc/8.2.0/avx/4.7 mvapich/2.1/gcc/(default)
bar/gcc/8.2.0/sse4.1/(default) mvapich/2.1/gcc/8.2.0(default)
bar/gcc/8.2.0/sse4.1/4.7 mvapich/2.1/gcc/9.1.0
bar/gcc/9.1.0/avx/(default) mvapich/2.1/intel/2018
bar/gcc/9.1.0/avx/5.4 mvapich/2.1/intel/2019
bar/gcc/9.1.0/avx2/5.4 mvapich/2.1/pgi/18.4
bar/sse4.1/gcc/(default) mvapich/2.1/pgi/19.4
bar/sse4.1/gcc/8.2.0/(default) mvapich/2.3.1/gcc/(default)
bar/sse4.1/gcc/8.2.0/4.7 mvapich/2.3.1/gcc/9.1.0
foo/1.1/gcc/(default) mvapich/2.3.1/intel/2019
foo/1.1/gcc/8.2.0/(default) mvapich/2.3.1/pgi/19.4
foo/1.1/gcc/8.2.0/mvapich/2.1 mvapich/gcc/(default)
foo/1.1/gcc/8.2.0/nompi(default) mvapich/gcc/8.2.0/(default)
foo/1.1/gcc/8.2.0/openmpi/3.1 mvapich/gcc/8.2.0/2.1
foo/1.1/intel/2018/intelmpi(default) mvapich/gcc/9.1.0/2.1
foo/1.1/intel/2018/mvapich/2.1 mvapich/gcc/9.1.0/2.3.1
foo/1.1/intel/2018/nompi mvapich/intel/2018/2.1
foo/1.1/intel/2018/openmpi/3.1 mvapich/intel/2019/2.1
foo/1.1/pgi/18.4/mvapich/2.1 mvapich/intel/2019/2.3.1
foo/1.1/pgi/18.4/nompi(default) mvapich/pgi/18.4/2.1
foo/1.1/pgi/18.4/openmpi/3.1 mvapich/pgi/19.4/2.1
foo/2.4/gcc/(default) mvapich/pgi/19.4/2.3.1
foo/2.4/gcc/9.1.0/mvapich/2.3.1 openmpi/3.1/gcc/(default)
foo/2.4/gcc/9.1.0/nompi(default) openmpi/3.1/gcc/8.2.0(default)
foo/2.4/gcc/9.1.0/openmpi/4.0 openmpi/3.1/gcc/9.1.0
foo/2.4/intel/2019/intelmpi(default) openmpi/3.1/intel/2018
foo/2.4/intel/2019/mvapich/2.3.1 openmpi/3.1/intel/2019
foo/2.4/intel/2019/nompi openmpi/3.1/pgi/18.4
foo/2.4/intel/2019/openmpi/4.0 openmpi/3.1/pgi/19.4
foo/2.4/pgi/19.4/nompi(default) openmpi/4.0/gcc/(default)
foo/2.4/pgi/19.4/openmpi/3.1 openmpi/4.0/gcc/9.1.0
foo/gcc/(default) openmpi/4.0/intel/2019
foo/gcc/8.2.0/(default) openmpi/4.0/pgi/19.4
foo/gcc/8.2.0/mvapich/2.1/1.1 openmpi/gcc/(default)
foo/gcc/8.2.0/nompi/(default) openmpi/gcc/8.2.0/(default)
foo/gcc/8.2.0/nompi/1.1 openmpi/gcc/8.2.0/3.1
foo/gcc/8.2.0/openmpi/3.1/1.1 openmpi/gcc/9.1.0/3.1
foo/gcc/9.1.0/mvapich/2.3.1/2.4 openmpi/gcc/9.1.0/4.0
foo/gcc/9.1.0/nompi/(default) openmpi/intel/2018/3.1
foo/gcc/9.1.0/nompi/2.4 openmpi/intel/2019/3.1
foo/gcc/9.1.0/openmpi/4.0/2.4 openmpi/intel/2019/4.0
foo/intel/2018/intelmpi/(default) openmpi/pgi/18.4/3.1
foo/intel/2018/intelmpi/1.1 openmpi/pgi/19.4/3.1
foo/intel/2018/mvapich/2.1/1.1 openmpi/pgi/19.4/4.0
foo/intel/2018/nompi/1.1 pgi/18.4
foo/intel/2018/openmpi/3.1/1.1 pgi/19.4
foo/intel/2019/intelmpi/(default)
[mod4 (modulerc)]$ module avail mvapich
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4 -
mvapich/2.1/gcc/(default) mvapich/gcc/(default)
mvapich/2.1/gcc/8.2.0(default) mvapich/gcc/8.2.0/(default)
mvapich/2.1/gcc/9.1.0 mvapich/gcc/8.2.0/2.1
mvapich/2.1/intel/2018 mvapich/gcc/9.1.0/2.1
mvapich/2.1/intel/2019 mvapich/gcc/9.1.0/2.3.1
mvapich/2.1/pgi/18.4 mvapich/intel/2018/2.1
mvapich/2.1/pgi/19.4 mvapich/intel/2019/2.1
mvapich/2.3.1/gcc/(default) mvapich/intel/2019/2.3.1
mvapich/2.3.1/gcc/9.1.0 mvapich/pgi/18.4/2.1
mvapich/2.3.1/intel/2019 mvapich/pgi/19.4/2.1
mvapich/2.3.1/pgi/19.4 mvapich/pgi/19.4/2.3.1
Unlike the previous cases wherein only package names and versions were shown
(because a single modulefile handled all the builds for that package and
version), here we see a listing for every build of a package and version.
Indeed, we not only see one such listing, but multiple listings per
build in many cases (e.g. openmpi/3.1/intel/2018 and
openmpi/intel/2018/3.1).
While admittedly the output of a module avail command without
specifying any package is rather overwhelming, when a package is specified
the output tends to be more reasonable, informing one of which builds
of the package are available. This strategy deliberately opts for the
presentation of more rather than less information.
The standard functionality of selecting the correct build of a package based on the loaded compiler, e.g.
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load pgi/19.4
[mod4 (modulerc)]$ module load openmpi/4.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 2) openmpi/4.0/pgi/19.4(default)
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (modulerc)]$ module unload openmpi
[mod4 (modulerc)]$ module switch --auto pgi intel/2019
[mod4 (modulerc)]$ module load openmpi/4.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) openmpi/4.0/intel/2019(default)
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4 (modulerc)]$ module unload openmpi
[mod4 (modulerc)]$ module switch --auto intel gcc/9.1.0
[mod4 (modulerc)]$ module load openmpi/4.0
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/4.0, gcc/9.1.0)
[mod4 (modulerc)]$ module unload openmpi
[mod4 (modulerc)]$ module switch --auto gcc gcc/8.2.0
[mod4 (modulerc)]$ module load openmpi/4.0
**** ERROR *****:
Compiler Mismatch
Package openmpi/4.0 does not appear to be built for currently
loaded compiler gcc/8.2.0.
Loading openmpi/4.0/gcc/9.1.0
ERROR: Module evaluation aborted
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) gcc/8.2.0
works as expected (and fails with the expected error when one attempts to load a module not compatible with the loaded compiler). The only significant difference between the previously examined strategies is that the module list command provides information about what variant of each package is loaded.
The module switch command, however, does not work as well as one would
like. While it indeed switches the specified module, it does not
successfully reload the modules which depend on the replaced module, even
with the Automated module handling
feature enabled. As currently implemented,
the automated module handling feature attempts to reload dependent modules
using the fully qualified module name, and as in this strategy the fully
qualified modulename includes the information about the module that was switched
out (e.g. pgi/19.4 in the example below), it will be incompatible with the
replacement module (e.g. intel/2019 in example below). It
is hoped a future version of modulecmd will allow for reloading based on the
name specified when the module was loaded. But as things currently stand,
the automatic module handling will throw an error attempting to reload the
depend module, resulting in the the dependent modules being unloaded.
Without automated module handling mode (i.e. for older Environment Modules or without the --auto flag), the dependent modules remain loaded and there is inconsistency in the loaded modules. But at least module list clearly shows such.
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load pgi/19.4
[mod4 (modulerc)]$ module load openmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 2) openmpi/pgi/19.4/4.0
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (modulerc)]$ module switch --auto pgi intel/2019
**** ERROR *****:
Compiler Mismatch
Package openmpi/pgi/19.4/4.0 does not appear to be built for currently
loaded compiler intel/2019.
Loading openmpi/pgi/19.4/4.0
ERROR: Module evaluation aborted
Switching from pgi/19.4 to intel/2019
WARNING: Reload of dependent openmpi/pgi/19.4/4.0 failed
Unloading dependent: openmpi/pgi/19.4/4.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) intel/2019
[mod4 (modulerc)]$ mpirun
mpirun: command not found
The defaulting of modules is more successful, however, as seen below:
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load openmpi/3.1
Loading openmpi/3.1/gcc/8.2.0
Loading requirement: gcc/8.2.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) gcc/8.2.0 2) openmpi/3.1/gcc/8.2.0(default)
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/3.1, gcc/8.2.0)
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load gcc/8.2.0
[mod4 (modulerc)]$ module load openmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) gcc/8.2.0 2) openmpi/gcc/8.2.0/3.1
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/3.1, gcc/8.2.0)
Here it not only defaults to the default compiler, but if one tries to load openmpi without specifying a version, it will find and load the latest version compatible with the loaded compiler.
The situation is similar for foo:
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load pgi/19.4
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 2) foo/2.4/pgi/19.4/nompi(default)
[mod4 (modulerc)]$ foo
foo 2.4 (pgi/19.4, nompi)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module load openmpi/3.1
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 3) foo/2.4/pgi/19.4/openmpi/3.1(default)
2) openmpi/3.1/pgi/19.4(default)
[mod4 (modulerc)]$ foo
foo 2.4 (pgi/19.4, openmpi/3.1)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module unload openmpi
[mod4 (modulerc)]$ module switch --auto pgi intel/2019
[mod4 (modulerc)]$ module load foo/2.4
Loading foo/2.4/intel/2019/intelmpi
ERROR: foo/2.4/intel/2019/intelmpi cannot be loaded due to missing prereq.
HINT: the following module must be loaded first: intelmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) intel/2019
[mod4 (modulerc)]$ foo
foo: command not found
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module load foo/2.4/intel/2019/nompi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) foo/2.4/intel/2019/nompi
[mod4 (modulerc)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module switch --auto intelmpi mvapich/2.3.1
[mod4 (modulerc)]$ module load foo/2.4
Loading foo/2.4/intel/2019/intelmpi
ERROR: foo/2.4/intel/2019/intelmpi cannot be loaded due to missing prereq.
HINT: the following module must be loaded first: intelmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) mvapich/2.3.1/intel/2019(default)
[mod4 (modulerc)]$ foo
foo: command not found
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module switch --auto mvapich openmpi/4.0
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) intel/2019
2) openmpi/4.0/intel/2019(default)
3) foo/2.4/intel/2019/openmpi/4.0(default)
[mod4 (modulerc)]$ foo
foo 2.4 (intel/2019, openmpi/4.0)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module unload openmpi
[mod4 (modulerc)]$ module switch --auto intel/2019 gcc/9.1.0
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) foo/2.4/gcc/9.1.0/nompi(default)
[mod4 (modulerc)]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module load mvapich/2.3.1
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 3) foo/2.4/gcc/9.1.0/nompi(default)
2) mvapich/2.3.1/gcc/9.1.0(default)
[mod4 (modulerc)]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module switch --auto mvapich openmpi/4.0
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 3) foo/2.4/gcc/9.1.0/openmpi/4.0(default)
2) openmpi/4.0/gcc/9.1.0(default)
[mod4 (modulerc)]$ foo
foo 2.4 (gcc/9.1.0, openmpi/4.0)
As expected, the correct version of foo is loaded depending on the previously
loaded compiler and MPI libraries. Note however that there is no dummy
module for Intel MPI, and that when the intel compiler is loaded but no MPI
library is explicitly loaded, module load foo/2.4 gets the Intel MPI build.
Such only occurs if there is an Intel MPI build in the tree of modulefiles;
otherwise a nompi build will be loaded if present. If both versions are
present, as in this case, one needs to give the explicit
foo/2.4/intel/2019/nompi specification.
Again, the module switch command only succeeds in switching the
specified module, and does not successfully reload the modules depending
on the switched module. With automated module handling mode, there will be errors
reported and the dependent modules will be unloaded; without it the dependent
modules will not get unloaded, and there will be inconsistent dependencies
(but at least module list will show such).
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load pgi/18.4
[mod4 (modulerc)]$ module load openmpi/3.1
[mod4 (modulerc)]$ module load foo/1.1
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) pgi/18.4 3) foo/1.1/pgi/18.4/openmpi/3.1(default)
2) openmpi/3.1/pgi/18.4(default)
[mod4 (modulerc)]$ foo
foo 1.1 (pgi/18.4, openmpi/3.1)
[mod4 (modulerc)]$ module switch --auto pgi intel/2018
**** ERROR *****:
Compiler Mismatch
Package openmpi/3.1/pgi/18.4 does not appear to be built for currently
loaded compiler intel/2018.
Loading openmpi/3.1/pgi/18.4
ERROR: Module evaluation aborted
**** ERROR *****:
Compiler Mismatch
Package foo/1.1/pgi/18.4/openmpi/3.1 does not appear to be built for currently
loaded compiler intel/2018.
Loading foo/1.1/pgi/18.4/openmpi/3.1
ERROR: Module evaluation aborted
Switching from pgi/18.4 to intel/2018
WARNING: Reload of dependent openmpi/3.1/pgi/18.4 failed
WARNING: Reload of dependent foo/1.1/pgi/18.4/openmpi/3.1 failed
Unloading dependent: foo/1.1/pgi/18.4/openmpi/3.1 openmpi/3.1/pgi/18.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) intel/2018
[mod4 (modulerc)]$ foo
foo: command not found
[mod4 (modulerc)]$ mpirun
mpirun: command not found
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load intel/2019
[mod4 (modulerc)]$ module load foo
Loading foo/intel/2019/intelmpi/2.4
ERROR: foo/intel/2019/intelmpi/2.4 cannot be loaded due to missing prereq.
HINT: the following module must be loaded first: intelmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) intel/2019
[mod4 (modulerc)]$ foo
foo: command not found
[mod4 (modulerc)]$ module load openmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) openmpi/intel/2019/4.0
[mod4 (modulerc)]$ foo
foo: command not found
The defaulting of modules works relatively well, as shown below:
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load foo
Loading foo/gcc/8.2.0/nompi/1.1
Loading requirement: gcc/8.2.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) gcc/8.2.0 2) foo/gcc/8.2.0/nompi/1.1
[mod4 (modulerc)]$ foo
foo 1.1 (gcc/8.2.0, nompi)
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load foo/1.1
Loading foo/1.1/gcc/8.2.0/nompi
Loading requirement: gcc/8.2.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) gcc/8.2.0 2) foo/1.1/gcc/8.2.0/nompi(default)
[mod4 (modulerc)]$ foo
foo 1.1 (gcc/8.2.0, nompi)
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load pgi/18.4
[mod4 (modulerc)]$ module load foo
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) pgi/18.4 2) foo/pgi/18.4/nompi/1.1
[mod4 (modulerc)]$ foo
foo 1.1 (pgi/18.4, nompi)
If one attempts to load foo without specifying a version or having previously loaded a compiler
module, the compiler will be defaulted (to gcc/8.2.0 compiler, as that is what
we declared via the GetDefaultCompiler Tcl procedure), and the latest
version of foo compatible with that compiler (and no MPI) will be loaded.
The situation with bar is similar. We do not have a dummy simd module,
so the builds with different CPU vectorization support are specified by
appending /avx, etc. to the bar package name. As with previous strategies,
if one attempts to load a simd variant which was not built for the compiler
loaded, an error will occur.
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load gcc/9.1.0
[mod4 (modulerc)]$ module load bar/avx2
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) bar/avx2/gcc/9.1.0/5.4
[mod4 (modulerc)]$ bar
bar 5.4 (gcc/9.1.0, avx2)
[mod4 (modulerc)]$ module unload bar
[mod4 (modulerc)]$ module load bar/avx
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) bar/avx/gcc/9.1.0/5.4
[mod4 (modulerc)]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4 (modulerc)]$ module unload bar
[mod4 (modulerc)]$ module load bar/sse4.1
**** ERROR *****:
Compiler Mismatch
Package bar/sse4.1 does not appear to be built for currently
loaded compiler gcc/9.1.0.
Loading bar/sse4.1/gcc/8.2.0/4.7
ERROR: Module evaluation aborted
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0
Summary of Modulerc-based strategy
The modulefiles are fairly standard. The only extra logic needed is to ensure that any loaded compiler/MPI or other dependency matches what the modulefile expects, and that can be handled fairly easily with the addition of one or two calls to Tcl procedures.
The
.modulercfiles do the work of routing partial specifications to the correct modulefile, and follow a fairly standard pattern. For the most part, these are generic/independent of the package, and typically can be written once and symlinked to the appropriate places in the directory tree.This strategy involves the use of many more modulefiles than the previously examined strategies, having at least one modulefile per build, and in some cases multiple per build. The output of the module avail command without any module specified is rather overwhelming, although when a module is specified, module avail will provide information about what builds are available.
Because each build has a specific module, the module list command shows exactly what builds of the various modules are loaded.
The module switch command does not work very well. In particular, when one switches out a module on which other modules depend, the dependent modules are not successfully reloaded. Even with the Automated module handling feature (introduced in
4.2), the dependent modules are unloaded but do not get reloaded properly since currently this feature attempts to reload based on the fully qualified name of the loaded module (which includes a dependency on the module switched out), not the specification used when the module was loaded.In general, when the user tries to load a module based on a partially specified modulename, the
.modulercfiles handle it well, and the latest version of the module consistent with the specification and previously loaded modules will be used. If no compiler was previously loaded, will use the default compiler, or the latest compiler if nothing matches the default compiler.Modules will fail with an error message if user tries to load a package which was not built for the values of the dependency packages loaded.
Modulepath-based Strategy
This strategy makes use of the ability of modules to support multiple
directories in the MODULEPATH environment variable. Every time a module is loaded
on which other modules might depend, a new path is added to MODULEPATH
containing the modulefiles which depend on the newly added module.
This is basically similar to the strategy that is used in the hierarchical modulefile approach of Lmod.
Implementation
The Homebrewed flavors and Modulerc-based strategies were based on modifying the modulefiles of modules which depended on other modules. The Modulepath-based strategy, in contrast, is based on modifying the modulefiles for modules which other modules might depend on.
To begin with, we set MODULEPATH to a Core directory containing
modulefiles for modules which do not depend on any other modules.
The modulefiles for compilers will typically be put in here.
For this example, we opt not to use a dummy simd module (for reasons
explained later), but instead add avx, avx2, sse4.1 variants to the
bar modules, but otherwise they might belong here. Other modules which
do not depend on compilers, etc. could be place in here as well, e.g.
applications which do not expose libraries, etc. So the directory
structure would look like:
Core
├── gcc
│ ├── 8.2.0
│ ├── 9.1.0
│ └── common
├── intel
│ ├── 2018
│ ├── 2019
│ └── common
└── pgi
├── 18.4
├── 19.4
└── common
A typical common file for the gcc compiler would be something like
# Example modulefiles for compiler-etc-dependency cookbook
# For "modulepath based" strategyy
#
# Common stuff for gcc
#
# Expects version to have been previously set
proc ModulesHelp { } {
global version
puts stderr "
This is the dummy GNU compiler suite modulefile for the cookbook
Handling Compiler and other Package Dependencies
It does not actually do anything
gcc version: $version
"
}
module-whatis "Dummy Gnu $version for cookbook"
# Find the software root. In production, you should
# hardcode to your real software root
set gitroot $::env(MOD_GIT_ROOTDIR)
set swroot $gitroot/doc/example/compiler-etc-dependencies/dummy-sw-root
set pkgroot $swroot/gcc
set vroot $pkgroot/$version
set bindir $vroot/bin
prepend-path PATH $bindir
# don't load multiple versions of this module (or other compilers)
conflict gcc
conflict gnu
conflict pgi
conflict intel
# Add the proper modulepath
# In production this should be hard-coded, but using $gitroot for cookbook
set modpathroot $gitroot/doc/example/compiler-etc-dependencies/modulepath
set newmodpath $modpathroot/Compiler/gcc/$version
module use $newmodpath
The most interesting aspect is the module use at the end. We add
to MODULEPATH a directory under Compiler for this specific compiler
and version. Other compilers/versions would add their own specific path
to MODULEPATH.
In each of these compiler specific directories, modulefile directory trees exist for the supported MPI libraries and bar. We also include a foo directory containing the non-MPI variants of foo. It would look like:
Compiler
├── gcc
│ ├── 8.2.0
│ │ ├── bar
│ │ │ ├── 4.7
│ │ │ │ ├── avx
│ │ │ │ ├── .modulerc -> symlink to modulerc.default_lowest_simd
│ │ │ │ └── sse4.1
│ │ │ └── common -> symlink to bar/common
│ │ ├── foo
│ │ │ ├── 1.1
│ │ │ └── common -> symlink to foo/common
│ │ ├── mvapich
│ │ │ ├── 2.1
│ │ │ └── common -> symlink to mvapich/common
│ │ └── openmpi
│ │ ├── 3.1
│ │ └── common -> symlink to openmpi/common
│ └── 9.1.0
│ ├── bar
│ │ ├── 5.4
│ │ │ ├── avx
│ │ │ ├── avx2
│ │ │ └── .modulerc -> symlink to modulerc.default_lowest_simd
│ │ └── common -> symlink to bar/common
│ ├── foo
│ │ ├── 2.4
│ │ └── common -> symlink to foo/common
│ ├── mvapich
│ │ ├── 2.1
│ │ ├── 2.3.1
│ │ └── common -> symlink to mvapich/common
│ └── openmpi
│ ├── 3.1
│ ├── 4.0
│ └── common -> symlink to openmpi/common
├── intel
│ ├── 2018
│ │ └── ... modules depending on compiler=intel/2018
│ └── 2019
│ └── ... modules depending on compiler=intel/2019
└── pgi
├── 18.4
│ └── ... modules depending on compiler=pgi/18.4
└── 19.4
└── ... modules depending on compiler=pgi/18.4
For brevity, we only show the directory structure for the modules depending on the two versions of gcc in detail; the other compilers will have similar structures.
Basically, there is a separate modulepath for each compiler, containing only the
modulefiles depending on that specific build of the compiler (and not also depending
on something else, like MPI library).
This certainly enforces the consistency of loaded modules; one could not load
a specific version of gcc (say gcc/9.1.0) and an incompatible version of foo (e.g. foo/1.1),
because all of the foo modulefiles are in compiler specific module trees and there is
no foo/1.1 in the gcc/9.1.0 module tree. Conflict statements in the compiler modulefiles
will prevent one from loading multiple compilers, thereby preventing multiple compiler
specific modulepaths (unless the user explicitly does a module use or similar, and
there is only so far one can go in preventing users from shooting themselves in the foot).
The modulefiles for foo and bar are fairly straightforward; we have a common file which
does the heavy lifting (and since this can be made independent of version and compiler, this
is actually a symlink to a package specific common file in a common directory external
to the modulepath trees).
We then add some small stubfiles which set variables for the specific build information, which
mostly are determined by their position in the tree structure (e.g. the stubfiles under
Compiler/gcc/9.1.0 are all going to set compilerTag to gcc/9.1.0, etc), and
then invoke the common file.
The modulefiles for openmpi are largely similar. E.g., for gcc/9.1.0 we have a small
stubfile like the following (for version 4.0)
#%Module
# Dummy modulefile for openmpi
set version 4.0
set compilerTag gcc/9.1.0
set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/common
which defines the version and compilerTag variables for the OpenMPI version and compiler version, and then invokes the common script
# Common modulefile for openmpi
# Using "modulepath" strategy
# Expects the following variables to have been
# previously defined:
# version: version of openmpi
# compilerTag: compiler being used with
# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root
proc ModulesHelp { } {
global version compilerTag
puts stderr "
openmpi: Test dummy version of OpenMPI $version
For testing packages depending on compilers/MPI
Version: $version
Compiler: $compilerTag
"
}
module-whatis "Dummy openmpi $version (for $compilerTag)"
# Find the software root. In production, you should
# hardcode to your real software root
set gitroot $::env(MOD_GIT_ROOTDIR)
set swroot $gitroot/doc/example/compiler-etc-dependencies/dummy-sw-root
# Compute the installation prefix
set pkgroot $swroot/openmpi
set vroot $pkgroot/$version
set prefix $vroot/$compilerTag
# We need to prereq the compiler to allow autohandling to work
prereq $compilerTag
# Set environment variables
setenv MPI_DIR $prefix
set bindir $prefix/bin
set libdir $prefix/lib
set incdir $prefix/include
prepend-path PATH $bindir
prepend-path LIBRARY_PATH $libdir
prepend-path LD_LIBRARY_PATH $libdir
prepend-path CPATH $incdir
# Add the proper modulepath
# In production this should be hard-coded, but using $gitroot for cookbook
set modpathroot $gitroot/doc/example/compiler-etc-dependencies/modulepath
set newmodpath $modpathroot/CompilerMPI/$compilerTag/openmpi/$version
module use $newmodpath
which does the usual stuff (define a help function, whatis string, and sets assorted environment
variables like PATH, LIBRARY_PATH, etc. for using OpenMPI), and then adds another directory
to the MODULEPATH. This new directory, under CompilerMPI is for modulefiles which
depend on the compiler AND the MPI library. (It is assumed that all modulefiles depending on
MPI libraries also depend on the compiler). This new branch in the modulepath would have
a structure like
CompilerMPI
├── gcc
│ ├── 8.2.0
│ │ ├── mvapich
│ │ │ └── 2.1
│ │ │ └── foo
│ │ │ ├── 1.1
│ │ │ └── common -> symlink to foo/common
│ │ └── openmpi
│ │ └── 3.1
│ │ └── foo
│ │ ├── 1.1
│ │ └── common -> symlink to foo/common
│ └── 9.1.0
│ ├── mvapich
│ │ └── 2.3.1
│ │ └── ... modules depending on gcc/9.1.0 and mvapich/2.3.1
│ └── openmpi
│ └── 4.0
│ └── ... modules depending on gcc/9.1.0 and openmpi/4.0
├── intel
│ ├── 2018
│ │ ├── intelmpi
│ │ │ └── default
│ │ │ └── ... modules depending on intel/2018 and its included MPI lib
│ │ ├── mvapich
│ │ │ └── 2.1
│ │ │ └── ... modules depending on intel/2018 and mvapich/2.1
│ │ └── openmpi
│ │ └── 3.1
│ │ └── ... modules depending on intel/2018 and openmpi/3.1
│ └── 2019
│ ├── intelmpi
│ │ └── default
│ │ └── ... modules depending on intel/2019 and its included MPI lib
│ ├── mvapich
│ │ └── 2.3.1
│ │ └── ... modules depending on intel/2019 and mvapich/2.3.1
│ └── openmpi
│ └── 4.0
│ └── ... modules depending on intel/2019 and openmpi/4.0
└── pgi
├── 18.4
│ ├── mvapich
│ │ └── 2.1
│ │ └── ... modules depending on pgi/18.4 and mvapich/2.1
│ └── openmpi
│ └── 3.1
│ └── ... modules depending on pgi/18.4 and openmpi/3.1
└── 19.4
└── openmpi
├── 3.1
│ └── ... modules depending on pgi/19.4 and openmpi/3.1
└── 4.0
└── ... modules depending on pgi/19.4 and openmpi/3.1
Basically, there is a separate modulepath for each combination of compiler and MPI library, listing all modulefiles depending on that compiler and MPI library. The modulefiles underneath each directory simply do what is needed to load that version of the package built with the specified compiler and MPI library.
This process could be continued further, as needed. E.g., if you had packages which
depended on NetCDF, and you had multiple builds of NetCDF for a given compiler/MPI
combination, you could add another modulepath tree, e.g. CompilerMPINetCDF,
branching on each compiler, MPI, and NetCDF triplet. But things can also get
unwieldy if carried too far.
One could also add modulepath trees for disjoint features like SIMD level, but this turns out to be a bit tricky. E.g., if you had a simd branch as well as a compiler branch, with both simd and compiler appearing in Core, things are fine for the Simd and Compiler trees. However, if there were to be modules depending on both, e.g. a CompilerSimd branch, then because compiler and simd are disjoint and could be loaded in either order, the modulefiles for both would need to handle adding the CompilerSimd branch depending on whether the other was previously loaded.
Examples
We now look at the example usage for the Modulepath-based strategy. To use these examples, you must:
Set (and export)
MOD_GIT_ROOTDIRto where you git-cloned the modules sourceDo a
module purgeSet your
MODULEPATHto$MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Core
Note that we set the MODULEPATH to the Core subdirectory; the Core branch is for those modulefiles
that do not depend on compiler or MPI library, and that should be available from the start.
As with the previous cases, we start with a module avail command, and here
we see the first big difference:
[mod4 (modulepath)]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Core -
gcc/8.2.0 gcc/9.1.0 intel/2018 intel/2019 pgi/18.4 pgi/19.4
[mod4 (modulepath)]$ module load gcc/9.1.0
[mod4 (modulepath)]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Compiler/gcc/9.1.0 -
bar/5.4/avx(default) foo/2.4 mvapich/2.3.1 openmpi/4.0
bar/5.4/avx2 mvapich/2.1 openmpi/3.1
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Core -
gcc/8.2.0 gcc/9.1.0 intel/2018 intel/2019 pgi/18.4 pgi/19.4
[mod4 (modulepath)]$ module load openmpi/4.0
[mod4 (modulepath)]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/CompilerMPI/gcc/9.1.0/openmpi/4.0 -
foo/2.4
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Compiler/gcc/9.1.0 -
bar/5.4/avx(default) foo/2.4 mvapich/2.3.1 openmpi/4.0
bar/5.4/avx2 mvapich/2.1 openmpi/3.1
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Core -
gcc/8.2.0 gcc/9.1.0 intel/2018 intel/2019 pgi/18.4 pgi/19.4
When we first do a module avail, we only see the modulefiles for the compilers.
The MPI libraries, foo, bar, etc. all depend on at least the compiler, and so
will not become "available" until a compiler (and possibly MPI library as well) is loaded.
In a production environment there would likely be other modulefiles available in Core
(i.e. an application which is only used as a standalone application and does not
provide an API/libraries to link against would likely be placed in Core), but in our
simple example, only the compilers appear in Core. As we load a compiler and then an
MPI library, additional modulefiles appear available. In a production environment, one
might wish to set things up so that a compiler (and maybe MPI library) modulefile is
automatically loaded when the user logs in.
The downside of this is that it becomes difficult for an user to know what software
is available, at least via the module command. I.e., if application foobar is only built
for a single compiler/MPI combination, it will not show up in module avail unless that
specific compiler/MPI combination were previously loaded. Lmod adds a module spider
command which allows the user to list all packages installed for any compiler/MPI
combination, and can be used to also find out which compiler/MPI combinations are needed
to access a specific modulefile, but Environment Modules does not have a comparable function
at this time. If you were to use this strategy in a production environment, you would
likely need to generate lists of available packages (and their compiler/MPI/etc dependencies)
in a web page or other documentation area which can be frequently updated.
The standard functionality of selecting the correct build of a package based on the loaded compiler, e.g.
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load pgi/19.4
[mod4 (modulepath)]$ module load openmpi/4.0
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 2) openmpi/4.0
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (modulepath)]$ module unload openmpi
[mod4 (modulepath)]$ module switch --auto pgi intel/2019
[mod4 (modulepath)]$ module load openmpi/4.0
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) openmpi/4.0
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4 (modulepath)]$ module unload openmpi
[mod4 (modulepath)]$ module switch --auto intel gcc/9.1.0
[mod4 (modulepath)]$ module load openmpi/4.0
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/4.0, gcc/9.1.0)
[mod4 (modulepath)]$ module unload openmpi
[mod4 (modulepath)]$ module switch --auto gcc gcc/8.2.0
[mod4 (modulepath)]$ module load openmpi/4.0
ERROR: Unable to locate a modulefile for 'openmpi/4.0'
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) gcc/8.2.0
works as expected. As shown in the last command above, when requesting a module not
built for the loaded compiler (e.g. openmpi/4.0 when gcc/8.2.0 is loaded), the module command
simply returns Unable to locate a modulefile as there is no corresponding modulefile
in the current list of MODULEPATHS.
The functionality of the module switch command depends on version and/or settings
of Environment Modules. For version 3.x, or 4.x with the Automated module handling
feature disabled, it does not work well, modules which depend on the switched out module
are not reloaded, leading to inconsistent environments. But for 4.x versions with
the automated module handling feature enabled (as indicated by the --auto after the
switch, although that can be made the default), it works as expected. Modules depending
on the module being switched out get unloaded and reloaded if possible, as shown below:
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load pgi/19.4
[mod4 (modulepath)]$ module load openmpi
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 2) openmpi/4.0
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (modulepath)]$ module switch --auto pgi intel/2019
Switching from pgi/19.4 to intel/2019
Unloading dependent: openmpi/4.0
Reloading dependent: openmpi/4.0
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) openmpi/4.0
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4 (modulepath)]$ module switch --auto intel intel/2018
Switching from intel/2019 to intel/2018
ERROR: Unable to locate a modulefile for 'openmpi/4.0'
WARNING: Reload of dependent openmpi/4.0 failed
Unloading dependent: openmpi/4.0
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) intel/2018
[mod4 (modulepath)]$ mpirun
mpirun: command not found
The modulefiles themselves basically have no support for defaulting a compiler; the modulefiles for the various MPI libraries are simply not even available until a modulefile for a compiler is loaded, as seen below:
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load openmpi/3.1
ERROR: Unable to locate a modulefile for 'openmpi/3.1'
[mod4 (modulepath)]$ module list
No Modulefiles Currently Loaded.
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load gcc/8.2.0
[mod4 (modulepath)]$ module load openmpi
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) gcc/8.2.0 2) openmpi/3.1
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/3.1, gcc/8.2.0)
However, this could be somewhat mitigated by having the modulefile for the default compiler automatically loaded for the user upon login (e.g. in their dot files).
The situation is similar for foo:
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load pgi/19.4
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 2) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (pgi/19.4, nompi)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module load openmpi/3.1
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) pgi/19.4 2) openmpi/3.1 3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (pgi/19.4, openmpi/3.1)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module unload openmpi
[mod4 (modulepath)]$ module switch --auto pgi intel/2019
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module load intelmpi
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) intelmpi/default 3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, intelmpi)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module switch --auto intelmpi mvapich/2.3.1
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) mvapich/2.3.1 3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, mvapich/2.3.1)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module switch --auto mvapich openmpi/4.0
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) openmpi/4.0 3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, openmpi/4.0)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module unload openmpi
[mod4 (modulepath)]$ module switch --auto intel/2019 gcc/9.1.0
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module load mvapich/2.3.1
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) mvapich/2.3.1 3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (gcc/9.1.0, mvapich/2.3.1)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module switch --auto mvapich openmpi/4.0
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) openmpi/4.0 3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (gcc/9.1.0, openmpi/4.0)
As expected, the correct version of foo is loaded depending on the previously
loaded compiler and MPI libraries. Here again we use an intelmpi modulefile
to indicate when we wish to use Intel MPI libraries, even though they are enabled
by the intel module. The intelmpi modulefile basically just adds the modulepath
for the intel-compiler and intelmpi dependent modules.
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load pgi/18.4
[mod4 (modulepath)]$ module load openmpi/3.1
[mod4 (modulepath)]$ module load foo/1.1
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) pgi/18.4 2) openmpi/3.1 3) foo/1.1
[mod4 (modulepath)]$ foo
foo 1.1 (pgi/18.4, openmpi/3.1)
[mod4 (modulepath)]$ module switch --auto pgi intel/2018
Switching from pgi/18.4 to intel/2018
Unloading dependent: foo/1.1 openmpi/3.1
Reloading dependent: openmpi/3.1 foo/1.1
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) intel/2018 2) openmpi/3.1 3) foo/1.1
[mod4 (modulepath)]$ foo
foo 1.1 (intel/2018, openmpi/3.1)
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/3.1, intel/2018)
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load intel/2019
[mod4 (modulepath)]$ module load foo
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (modulepath)]$ module load openmpi
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) intel/2019 2) foo/2.4 3) openmpi/4.0
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, nompi)
Note again the same deficiency in the switch report as the Homebrewed Flavors Strategy
had; in the last case above wherein we loaded foo with intel/2019 loaded
but no MPI module. As expected, a non-MPI build of foo was loaded. However, when an
openmpi module is subsequently loaded, foo does not get reloaded and we still have
the non-MPI build of foo, as evidenced by the output of the foo command. And that
this fact is not obvious from the module list output.
The behavior when defaulting is nicer. Without any compiler or MPI libraries
Here we note a deficiency in the switch support as compared to Flavours. In the last example
after loading intel/2019 and foo, we have the non-MPI build of foo as expected. However, upon
subsequently loading the openmpi module, we still have the non-MPI version of foo loaded, as evidenced
by the output of the dummy foo command. I.e., the foo package was not automatically reloaded, as
there was no prereq in the foo modulefile on an MPI library (as in the non-MPI build there is no MPI
library to prereq). Also note that module list does not really inform one of this fact.
The ability to default partial modulenames in this strategy is mixed. Without any compiler loaded, most modulefiles are not even visible/available. However, once a compiler is loaded, trying to load a module without specifying the version will end up loading the latest version compatible with the loaded compiler (as no incompatible versions are visible) as seen below:
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load foo
ERROR: Unable to locate a modulefile for 'foo'
[mod4 (modulepath)]$ module load gcc/8.2.0
[mod4 (modulepath)]$ module load foo
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) gcc/8.2.0 2) foo/1.1
[mod4 (modulepath)]$ foo
foo 1.1 (gcc/8.2.0, nompi)
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load gcc/8.2.0
[mod4 (modulepath)]$ module load openmpi
[mod4 (modulepath)]$ module load foo
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) gcc/8.2.0 2) openmpi/3.1 3) foo/1.1
[mod4 (modulepath)]$ foo
foo 1.1 (gcc/8.2.0, openmpi/3.1)
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load foo/1.1
ERROR: Unable to locate a modulefile for 'foo/1.1'
[mod4 (modulepath)]$ module list
No Modulefiles Currently Loaded.
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load pgi/18.4
[mod4 (modulepath)]$ module load foo
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) pgi/18.4 2) foo/1.1
[mod4 (modulepath)]$ foo
foo 1.1 (pgi/18.4, nompi)
This is better than in the Flavours or Homebrewed flavors strategies. If one were to use this strategy in production, it is recommended to have a default compiler module (and maybe even an MPI library, etc) automatically loaded when the user logs in, thereby allowing for a reasonable defaulting ability (and more reasonable module avail output).
The situation with bar is similar. We do not have a dummy simd module,
so the builds with different CPU vectorization support are specified by
appending /avx, etc. to the bar package name. As with previous strategies,
if one attempts to load a simd variant which was not built for the compiler
loaded, an error will occur.
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load gcc/9.1.0
[mod4 (modulepath)]$ module load bar/5.4/avx2
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) bar/5.4/avx2
[mod4 (modulepath)]$ bar
bar 5.4 (gcc/9.1.0, avx2)
[mod4 (modulepath)]$ module unload bar
[mod4 (modulepath)]$ module load bar/5.4/avx
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) bar/5.4/avx(default)
[mod4 (modulepath)]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4 (modulepath)]$ module unload bar
[mod4 (modulepath)]$ module load bar/5.4/sse4.1
ERROR: Unable to locate a modulefile for 'bar/5.4/sse4.1'
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0
Defaulting is handled well, as shown
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load gcc/9.1.0
[mod4 (modulepath)]$ module load bar
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) gcc/9.1.0 2) bar/5.4/avx(default)
[mod4 (modulepath)]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load gcc/8.2.0
[mod4 (modulepath)]$ module load bar/4.7
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
1) gcc/8.2.0 2) bar/4.7/sse4.1(default)
[mod4 (modulepath)]$ bar
bar 4.7 (gcc/8.2.0, sse4.1)
In particular, one case specify bar/avx2 or bar/avx or bar/sse4.1 and the
latest version of bar consistent with the specification and any previously
loaded compiler (or default compiler) will be loaded
Summary of Modulepath-based strategy
It is quite difficult for an user to get an inconsistent environment, at least when automated module handling mode is used. The modulepaths are managed so that only modulefiles consistent with the previously loaded compiler/MPI library are visible.
With automated module handling mode, the module switch command is very nicely supported.
The module command does not lend itself to readily seeing all the packages which are installed on the system, nor seeing for which compiler/MPI combinations a certain package is built. The module avail command just does not show any modules not compatible with the currently loaded compiler and/or MPI libraries. Lmod needed to add a
module spidercommand to address this, but no such functionality currently exists in Environment Modules. If one were to use this in production, you would need to provide something similar to the Lmod spider sub-command, or at least provide frequently updated web pages or similar with this information.This strategy involves the use of many more modulepaths than the previously examined strategies, having at least one modulepath per compiler installed, and typically also one for each compiler/MPI library combination. If you add additional layers of dependency, things get even more complicated, which can be limiting. It is especially problematic if you have two disjoint dependencies -- i.e. neither dependencies is dependent on the other; in this case you might load one or the other, or both (in either order). If there are modulefiles dependent on both of these dependencies, the last one loaded (and only the last one) needs to handle adding the appropriate modulepath for the modulefiles depending on both.
In general, a module will fail to load with an error message if any dependent module is not already loaded. The error will actually state that the module cannot be found, as the modulepath containing it will not be added to the
MODULEPATHuntil the modules depended on are loaded. So if you need to default a compiler, etc., you will need to have the user's initialization scripts automatically load the appropriate modulefile upon login.However, what we have been referring to as "more intelligent defaulting" is effectively supported. I.e., if a user requests to load a package without specifying the version desired, the version will be defaulted to the latest version compatible with the currently loaded compiler and other dependencies. This is because the incompatible versions are not visible in the module tree.
Comparison of Strategies
All of the strategies discussed above have their peculiar strengths and weaknesses. The decision of which strategy to use will depend on how these strengths and weaknesses impact your design goals. It is advised to play around in the sandbox environments a bit, as actual use tends to help make clear which downsides you are willing to accept and which you are not.
With the exception of the Flavours, which works best with Environment Modules version 3.x, all of the other strategies work as well or better on the newer 4.x versions of Environment Modules. This difference is most visible in the discussion of features around the module switch command. So in the following discussions we will assume Environment Modules version 3.x for the Flavours strategies, as that is the version it works best with. For all other strategies, we assume Environment Modules version 4.x, with automated module handling mode enabled, as these strategies work best in that scenario.
The Homebrewed flavors, Modulerc-based, and Modulepath-based strategies all require a significant amount of "assembly" in order to get them working for a production environment; the example scripts and procedures provided here should help significantly, but should not be considered as a polished product. You will likely need to customize and extend for your environment. The Flavours Strategy, on the other hand, has been packaged to present as a finished product and so requires less "assembly" to get working, but does not appear to be actively maintained, so the reduced up-front work might be neutralized by the costs of self support.
Basic Dependency Handling
All of the strategies discussed support a basic level of dependency handling. If a user attempts to load a package, they get the build of the package appropriate for the previously loaded compiler or other dependencies, or, if no appropriate build is found, an error message indicating such. E.g, if the user does:
module load gcc/8.2.0
module load foo/1.1
their environment will be set up to run foo version 1.1 built
with gcc version 8.2.0.
All of the strategies discussed meet this criterion, with both 3.x and 4.x versions of Environment Modules.
Advanced Dependency Handling (e.g. the module switch sub-command)
Things are more complicated when we allow for the modules upon which other loaded modules might depend to be changed. This generally involves the module switch command.
While all of the strategies handle well the easy case, when we switch a module upon which no other loaded modules depend, the trick comes when one or more currently loaded modules depend on the module being switched out. E.g., if we assume that module foo depends on the previously loaded compiler, then ideally, we would like for a sequence like:
module load intel/2019
module load foo
module switch intel pgi/18.4
to have the same effective outcome as if the user issued the commands:
module load pgi/18.4
module load foo
I.e., switching out the compiler
will cause foo to be reloaded with the new compiler dependency. And ideally
in this example we would have pgi/18.4 loaded along with the latest version
of foo compatible with that compiler.
This is handled reasonably well with the Flavours (using Environment Modules
3.x) and the Homebrewed flavors and
Modulepath-based (both using Environment
Modules 4.x) strategies. For the Homebrewed flavors and
Modulepath-based
strategies, this relies on the Automated module handling feature (As of
version 4, this is disabled by default. It can be enabled in modulecmd.tcl,
or by running module config auto_handling 1, or by adding the --auto flag
to the modules command.)
For these strategies, swapping out a compiler or other module upon which a loaded module depends will cause the the module with a dependency to be unloaded, the module being switched being swapped out, and the module that had a dependency being reloaded.
Our example sequence above actually exposes a subtle issue. For the Flavours
and Homebrewed flavors strategies, the sequence of loading pgi/18.4 followed
by foo (using our example software library) would fail, as foo would be defaulted to
version 2.4 and as there was no build of foo/2.4 for pgi/18.4, the module load foo
would fail. And indeed, the switch sequence above would not be completely successful
under either strategy: for the Homebrewed Flavors Strategy, the compiler would be
swapped out and the reload of foo would fail with an error (so effectively
equivalent to the second sequence). For the Flavours Strategy, it detects that
it would be unable to reload foo, and the switch command fails (i.e. the compiler
remains intel/2019).
For the Modulepath-based Strategy, the second sequence (loading pgi/18.4 then foo) would
actually work, as the modulepath exposed by loading pgi/18.4 only has versions of
foo that are compatible with pgi/18.4. But the switch command does not work
as well as would be desired. This is because that as currently implemented, when the
automated module handling code does the reload of the module that was unloaded
due to dependencies, it attempts to reload based on the fully qualified name of the
module that was loaded, and not based on the (possibly partial) name specified
when the module was originally loaded. So, after the module load intel/2019,
the module load foo will result in foo/2.4 being loaded. When we switch out intel/2019
for pgi/18.4, foo gets unloaded, but after the compiler swap, it tries to reload
foo/2.4. As there is no build of foo/2.4 for pgi/18.4 (and thus no modulefile in
the current modulepath), the module is not found and is unable to be loaded.
As in the Homebrewed flavors case, the compiler is switched but foo is left unloaded
(with an appropriate warning to that effect).
This same issue prevents the Automated module handling feature as currently implemented from being very useful with the Modulerc-based. Basically, whenever you have a module loaded that depends on another module, switching out the module depended upon will fail to reload the other module. This is true even in less pathological cases, like:
module load intel/2019
module load foo
module switch intel pgi/19.4
which succeeds under the other three strategies (as foo/2.4 is built for pgi/19.4).
This is because when foo is initially loaded in the above scenario, the actual modulename
loaded is foo/intel/2019/2.4. With the switch command, foo gets unloaded, the compilers
are switched, and the automated modules handling code tries to reload foo/intel/2019/2.4.
That modulename clearly depends on intel/2019, not pgi/19.4, and so will fail to load
(with an error message). The net result is that the compiler gets switched, but any
modules depending on the compiler (or whatever module being switched) get unloaded
(with a warning to that effect). It is hoped that a future version of the
Automated module handling feature will have an option to reload the dependent
modules using the modulename they were loaded with; this should allow for the
switch command to work well in the Modulerc-based Strategy, as well as make
edges case (like the pgi/18.4 example discussed above) work better with the
Modulepath-based Strategy.
There is another edge case which only the Flavours Strategy handles well. Consider a scenario like:
module purge
module load intel/2019
module load foo
foo
# this should be foo/2.4 built for intel/2019 without MPI
module load openmpi
foo
# What build is this, with or without MPI?
With the Flavours Strategy, the final foo will be built for intel/2019
with openmpi support; i.e. Flavours supports the concept of optional prerequisites, and foo has
an optional prerequisite on the MPI libraries, so when openmpi is loaded, any previously loaded modules
which depend on it (through a regular or optional prerequisite) get reloaded.
All of the other strategies rely on automated module handling mode for automatic reloads, and in this case no reload of foo will be initiated as foo does not have MPI libraries listed as a prerequisite (otherwise a no MPI version could not be loaded). This is especially problematic for the Homebrewed flavors and Modulepath-based strategies, since it is not easily to tell from a module list or similar command which version of foo the environment is set for (for the Modulerc-based Strategy, the full modulename from module list will indicate that).
However, despite some potential issues with some edge cases, the Flavours, Homebrewed flavors, and Modulepath-based strategies all handle this challenge well. The Modulerc-based approach has a poor showing (although at least the modules depending on the module switched out will be unloaded, so the set of loaded modules is consistent).
Note also that this topic shows the most dependence on the version of Environment Modules. For the Flavours Strategy, the switching of a module upon which other modules depended does not work (the wrapper script returned weird errors). The other strategies all rely on the Automated module handling feature to enable the unload and reload of modules which depend on the module being switched out. Thus for 3.x versions of Environment Modules, the switch of a compiler, etc. will only result in the compiler being replaced, and any modules which depend on the compiler will not be reloaded. This means they will still be pointing to versions built for the old compiler, leading to an inconsistent set of modules being loaded. This is particularly bad in the Homebrewed flavors case, as a module load will not even inform one of that fact. Even if automated module handling mode is disabled on version 4.x of Environment Modules situation is better as loaded environment consistency is assured: the switch of a compiler will be denied since a dependent module is detected loaded.
Visibility into what packages are available
Another set of criteria to weigh has to do with visibility into the available packages. We are interested in
Viewing all of the packages installed on a system
Determining what versions of a specific package are available for a given compiler/MPI/etc combination.
Seeing what compiler/MPI/etc combinations a specific version of a package
For packages that we have currently loaded, determining which compiler/MPI/etc they were built for.
In terms of seeing all of packages that are available, the Flavours and
Homebrewed flavors strategies
are probably the best. The module avail command will list all modulefiles available, which will
typically just be a list of packagename/version for each installed version. On a production system
with many packages installed, even this can be a bit overwhelming to the user.
The module avail command will also list all modulefiles for every installed package in the
Modulerc-based Strategy, but here every build of every version of every package will have a distinct modulefile. I.e.,
you will not just have foo/1.1 and foo/2.4 listings, but foo/1.1/gcc/8.2.0/nompi, foo/1.1/gcc/8.2.0/mvapich/2.1,
foo/1.1/gcc/8.2.0/openmpi/3.1, foo/1.1/intel/2018/nompi, foo/1.1/intel/2018/intelmpi, etc. modulefiles.
Actually, the situation is even worse than that, as in this strategy there are often multiple
modulefiles for the same build in order to ease navigation of the module tree, e.g. we could have
foo/1.1/gcc/8.2.0/nompi and foo/gcc/8.2.0/nompi/1.1 representing the same build of the same package.
Whereas an unqualified module avail in the Flavours or
Homebrewed flavors strategies can be a bit overwhelming
in a large environment, with the Modulerc-based Strategy it can become practically unusable.
While an unqualified module avail in the Flavours,
Homebrewed flavors, and especially Modulerc-based strategies
can inundate the user with modulenames, the Modulepath-based Strategy has the opposite problem.
With the Modulepath-based Strategy strategy, the modulefiles are split across multiple, often mutually incompatible, modulepaths, so the module avail
command will never return a list of all modulefiles installed, only those available given the previously loaded
compiler/MPI libraries/etc. E.g., if a package foobar is only installed for a particular compiler/MPI combination,
it will not appear in any module avail listing unless that particular compiler and MPI were previously loaded.
While the number of modulefiles listed in an unqualified module avail command in the Modulerc-based Strategy
is unwieldy, if one adds a partial package specification argument, the number of modulefiles returned is greatly
reduced. This smaller list can provide information about which compilers/MPI/etc. are compatible with
a specific version of a package. For example, to see the compilers for which foo/2.4 is built, we can
do something like:
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module avail foo/2.4
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4 -
foo/2.4/gcc/(default) foo/2.4/intel/2019/mvapich/2.3.1
foo/2.4/gcc/9.1.0/mvapich/2.3.1 foo/2.4/intel/2019/nompi
foo/2.4/gcc/9.1.0/nompi(default) foo/2.4/intel/2019/openmpi/4.0
foo/2.4/gcc/9.1.0/openmpi/4.0 foo/2.4/pgi/19.4/nompi(default)
foo/2.4/intel/2019/intelmpi(default) foo/2.4/pgi/19.4/openmpi/3.1
Similarly, to see the builds of foo using gcc compilers, one can do something like:
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module avail foo/gcc
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4 -
foo/gcc/(default) foo/gcc/8.2.0/openmpi/3.1/1.1
foo/gcc/8.2.0/(default) foo/gcc/9.1.0/mvapich/2.3.1/2.4
foo/gcc/8.2.0/mvapich/2.1/1.1 foo/gcc/9.1.0/nompi/(default)
foo/gcc/8.2.0/nompi/(default) foo/gcc/9.1.0/nompi/2.4
foo/gcc/8.2.0/nompi/1.1 foo/gcc/9.1.0/openmpi/4.0/2.4
The other strategies do not readily provide that information. The next best case is the Modulepath-based Strategy, because here at least a module avail will tell you what versions of a package are compatible with the currently loaded compiler/MPI/etc, e.g.:
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load intel/2019
[mod4 (modulepath)]$ module avail foo/2.4
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Compiler/intel/2019 -
foo/2.4
But the Flavours and Homebrewed flavors strategies do not readily show what versions of packages are built for which compilers, etc. You would need to load a compiler/MPI/etc combination, and then try loading a particular version of a package.
Lastly, with the Modulerc-based Strategy, the module list command explicitly shows information about the compiler/MPI/etc used to build the loaded modules, as that is part of the full modulename. This information is not explicitly visible in the other three strategies, but that is usually not a problem. As long as the various user commands result in a set of modules wherein every module depending on either a compiler or MPI library, etc. is built for the currently loaded compiler/MPI library/etc., that information is redundant.
The only cases where this is potentially an issue are the sort of edge cases described in the previous section, e.g. if one were to do something like:
module purge
module load intel/2019
module load foo
foo
# this should be foo/2.4 built for intel/2019 without MPI
module load openmpi
foo
# What build is this, with or without MPI?
In the above example, for the Flavours, Homebrewed flavors, and Modulepath-based strategies a module list at the end would simply list something like:
Currently Loaded Modulefiles:
1) intel/2019 2) foo/2.4 3) openmpi/4.0
For the Flavours Strategy, foo would be built with MPI support, but for the other two, foo would still be the non-MPI build, which is not readily apparent from the above output (although possibly could be inferred by the ordering of the modules).
Conclusions
We have presented four strategies for dealing with modulefiles for packages with multiple builds depending on compiler, MPI, and/or other factors. All four strategies can deal with the basic requirement of loading of the correct build of a package depending on the previously loaded dependencies, or failing with a reasonable error message if no such build is available. They all have their own strengths and weaknesses. This document tried to present these strategies objectively and has been made to help you to evaluate how to handle this issue at your site.