Environment Modules

Welcome to the Environment Modules documentation portal. The Environment Modules package provides for the dynamic modification of a user's environment via modulefiles.

The Modules package is a tool that simplifies shell initialization and lets users easily modify their environment during a session using modulefiles.

Each modulefile contains the information needed to configure the shell for an application. Once the Modules package is initialized, the environment can be modified on a per-module basis using the module command which interprets modulefiles. Typically modulefiles instruct the module command to alter or set shell environment variables such as PATH, MANPATH, etc. modulefiles may be shared by many users on a system and users may have their own collection to supplement or replace the shared modulefiles.

Modules can be loaded and unloaded dynamically and atomically, in an clean fashion. All popular shells are supported, including bash, ksh, zsh, sh, csh, tcsh, fish, as well as some scripting languages such as tcl, perl, python, ruby, cmake and r.

Modules are useful in managing different versions of applications. Modules can also be bundled into metamodules that will load an entire suite of different applications.

Note

Modules presented here are ones that modify the shell or script execution environment. They should not be confused with language-specific modules (e.g., Perl modules, Python modules or R modules) that add specific capabilities to scripts.

Quick examples

Here is an example of loading a module on a Linux machine under bash.

$ module load gcc/6.1.1
$ which gcc
$ /usr/local/gcc/6.1.1/linux-x86_64/bin/gcc

Now we'll switch to a different version of the module

$ module switch gcc gcc/6.3.1
$ which gcc
/usr/local/gcc/6.3.1/linux-x86_64/bin/gcc

And now we'll unload the module altogether

$ module unload gcc
$ which gcc
gcc not found

Now we'll log into a different machine, using a different shell (tcsh).

% module load gcc/6.3.1
% which gcc
/usr/local/gcc/6.3.1/linux-aarch64/bin/gcc

Note that the command line is exactly the same, but the path has automatically configured to the correct architecture.

Installing Modules on Unix

This document is an overview of building and installing Modules on a Unix system.

Requirements

Modules consists of one Tcl script so to run it from a user shell the only requirement is to have a working version of tclsh (version 8.4 or later) available on your system. tclsh is a part of Tcl (http://www.tcl.tk/software/tcltk/).

To install Modules from a distribution tarball or a clone of the git repository, a build step is there to adapt the initialization scripts to your configuration and create the documentation files. This build step requires the tools to be found on your system:

  • bash
  • make
  • sed
  • runtest

When also installing Modules Tcl extension library or the bundled compatibility version of Modules (both enabled by default), these additional tools are needed:

  • grep
  • gcc
  • tcl-devel >= 8.4

When installing from a distribution tarball, documentation is pre-built and scripts to configure Modules Tcl extension library and compatibility version builds are already generated. Thus no additional software is required. When installing from a clone of the git repository or from a git archive export, documentation and scripts to prepare for compilation have to be built and the following tools are required:

  • autoconf
  • automake
  • autopoint
  • python
  • sphinx >= 1.0

Get Modules

Modules can usually be installed with the package manager of your Unix system. It it is available by default on most Linux distributions, on OS X and FreeBSD either under the name of modules or environment-modules.

If you want to install Modules from sources, tarballs from all Modules' releases can be retrieved from one of the following link:

For instance to download then unpack the last release of Modules:

$ curl -LJO https://github.com/cea-hpc/modules/releases/download/v4.8.0/modules-4.8.0.tar.gz
$ tar xfz modules-4.8.0.tar.gz

Installation instructions

The simplest way to build and install Modules is:

$ ./configure
$ make
$ make install

Some explanation, step by step:

  1. cd to the directory containing the package's source code. Your system must have the above requirements installed to properly build scripts, compatibility version of Modules if enabled, and documentation if build occurs from a clone of the git repository.
  2. Type ./configure to adapt the installation for your system. At this step you can choose the installation paths and the features you want to enable in the initialization scripts (see Build and installation options section below for a complete overview of the available options)
  3. Type make to adapt scripts to the configuration, build Tcl extension library and compatibility version if enabled and build documentation if working from git repository.
  4. Optionally, type make test to run the test suite.
  5. Type make install to install modulecmd.tcl, initialization scripts, compatibility version if built and documentation.
  6. Optionally, type make testinstall to run the installation test suite.
  7. You can remove the built files from the source code directory by typing make clean. To also remove the files that configure created, type make distclean.

A default installation process like described above will install Modules under /usr/local/Modules. You can change this with the --prefix option. By default, /usr/local/Modules/modulefiles will be setup as the default directory containing modulefiles. --modulefilesdir option enables to change this directory location. For example:

$ ./configure --prefix=/usr/share/Modules \
              --modulefilesdir=/etc/modulefiles

See Build and installation options section to discover all ./configure option available.

Note

GNU Make is excepted to be used for this build and installation process. On non-Linux systems, the gmake should be called instead of make.

Configuration

Once installed you should review and adapt the configuration to make it fit your needs. The following steps are provided for example. They are not necessarily mandatory as it depends of the kind of setup you want to achieve.

  1. Tune the initialization scripts. Review of these scripts is highly encouraged as you may add or adapt specific stuff to get Modules initialized the way you want.

  2. Enable Modules initialization at shell startup. An easy way to get module function defined and its associated configuration setup at shell startup is to make the initialization scripts part of the system-wide environment setup in /etc/profile.d. To do so, make a link in this directory to the profile scripts that can be found in your Modules installation init directory:

    $ ln -s PREFIX/init/profile.sh /etc/profile.d/modules.sh
$ ln -s PREFIX/init/profile.csh /etc/profile.d/modules.csh

    These profile scripts will automatically adapt to the kind of sh or csh shell you are running.

    Another approach may be to get the Modules initialization script sourced from the shell configuration startup file. For instance following line could be added to the end of the ~/.bashrc file if running Bash shell:

    source PREFIX/init/bash

    Beware that shells have multiple ways to initialize depending if they are a login shell or not and if they are launched in interactive mode or not.

  3. Define module paths to enable by default. Edit modulerc configuration file or .modulespath if you have chosen --enable-dotmodulespath at configure time. If you have set --with-initconf-in to etcdir to install these Modules initialization configurations in the configuration directory designated by the --etcdir option, these configuration files are respectively named initrc and modulespath. If you use .modulespath (or modulespath) configuration file, add one line mentioning each modulefile directory:

    /path/to/regular/modulefiles
/path/to/other/modulefiles

    If you use modulerc (or initrc) configuration file, add one line mentioning each modulefile directory prefixed by the module use command:

    module use /path/to/regular/modulefiles
module use /path/to/other/modulefiles

  4. Define modulefiles to load by default. Edit modulerc (or initrc) configuration file. Modulefiles to load cannot be specified in .modulespath (or modulespath) file. Add there all the modulefiles you want to load by default at Modules initialization time.

    Add one line mentioning each modulefile to load prefixed by the module load command:

    module load foo
module load bar

    In fact you can add to the modulerc (or initrc) configuration file any kind of supported module command, like module config commands to tune module's default behaviors.

If you go through the above steps you should have a valid setup tuned to your needs. After that you still have to write modulefiles to get something to load and unload in your newly configured Modules setup. Please have a look at the doc/example.txt that explains how the user environment is setup with Modules at the University of Minnesota computer science department.

Build and installation options

Options available at the ./configure installation step are described below. These options enable to choose the installation paths and the features to enable or disable. You can also get a description of these options by typing ./configure --help.

Fine tuning of the installation directories (the default value for each option is displayed within brakets):

--prefix=PREFIX

Installation root directory [/usr/local/Modules]

--bindir=DIR

Directory for executables reachable by users [PREFIX/bin]

--libdir=DIR

Directory for object code libraries like libtclenvmodules.so [PREFIX/lib]

--libexecdir=DIR

Directory for executables called by other executables like modulecmd.tcl [PREFIX/libexec]

--etcdir=DIR

Directory for the executable configuration scripts [PREFIX/etc]

New in version 4.1.

--initdir=DIR

Directory for the per-shell environment initialization scripts [PREFIX/init]

--datarootdir=DIR

Base directory to set the man and doc directories [PREFIX/share]

--mandir=DIR

Directory to host man pages [DATAROOTDIR/man]

--docdir=DIR

Directory to host documentation other than man pages like README, license file, etc [DATAROOTDIR/doc]

--vimdatadir=DIR

Directory to host Vim addon files [DATAROOTDIR/vim/vimfiles]

New in version 4.3.

--modulefilesdir=DIR

Directory of main modulefiles also called system modulefiles [PREFIX/modulefiles]

New in version 4.0.

Optional Features (the default for each option is displayed within parenthesis, to disable an option replace enable by disable for instance --disable-set-manpath):

--enable-set-manpath

Prepend man page directory defined by the --mandir option to the MANPATH environment variable in the shell initialization scripts. (default=yes)

New in version 4.0.

--enable-append-manpath

Append rather prepend man page directory to the MANPATH environment variable when the --enable-set-manpath option is enabled. (default=no)

New in version 4.2.

--enable-set-binpath

Prepend binary directory defined by the --bindir option to the PATH environment variable in the shell initialization scripts. (default=yes)

New in version 4.0.

--enable-append-binpath

Append rather prepend binary directory to the PATH environment variable when the --enable-set-binpath option is enabled. (default=no)

New in version 4.2.

--enable-dotmodulespath, --enable-modulespath

Set the module paths defined by --with-modulepath option in a .modulespath file (following C version fashion) within the initialization directory defined by the --initdir option rather than within the modulerc file. Or respectively, if option --with-initconf-in has been set to etcdir, in a modulespath file within the configuration directory defined by the --etcdir option rather than within the initrc file. (default=no)

New in version 4.0.

Changed in version 4.3: Option --enable-modulespath added

--enable-doc-install

Install the documentation files in the documentation directory defined with the --docdir option. This feature has no impact on manual pages installation. Disabling documentation file installation is useful in case of installation process handled via a package manager which handles by itself the installation of this kind of documents. (default=yes)

New in version 4.0.

--enable-vim-addons

Install the Vim addon files in the Vim addons directory defined with the --vimdatadir option. (default=yes)

New in version 4.3.

--enable-example-modulefiles

Install some modulefiles provided as example in the system modulefiles directory defined with the --modulefilesdir option. (default=yes)

New in version 4.0.

--enable-compat-version

Build and install the Modules compatibility (C) version in addition to the main released version. This feature also enables switching capabilities from initialization script between the two installed version of Modules (by setting-up the switchml shell function or alias). (default=no)

New in version 4.0.

--enable-libtclenvmodules

Build and install the Modules Tcl extension library which provides optimized Tcl commands for the modulecmd.tcl script.

New in version 4.3.

--enable-multilib-support

Support multilib systems by looking in modulecmd.tcl at an alternative location where to find the Modules Tcl extension library depending on current machine architecture.

New in version 4.6.

--enable-versioning

Append Modules version to installation prefix and deploy a versions modulepath shared between all versioning enabled Modules installation. A modulefile corresponding to Modules version is added to the shared modulepath and enables to switch from one Modules version to another. (default=no)

--enable-silent-shell-debug-support

Generate code in module function definition and initialization scripts to add support for silencing shell debugging properties (default=yes)

New in version 4.2.

--enable-set-shell-startup

Set when module function is defined the shell startup file to ensure that the module function is still defined in sub-shells. (default=yes)

New in version 4.3.

--enable-quarantine-support

Generate code in module function definition and initialization scripts to add support for the environment variable quarantine mechanism (default=yes)

New in version 4.2.

--enable-auto-handling

Set modulecmd.tcl to automatically apply automated modulefiles handling actions, like loading the pre-requisites of a modulefile when loading this modulefile. (default=no)

New in version 4.2.

--enable-implicit-requirement

Implicitly define a prereq or a conflict requirement toward modules specified respectively on module load or module unload commands in modulefile. (default=yes)

New in version 4.7.

--enable-avail-indepth

When performing an avail sub-command, include in search results the matching modulefiles and directories and recursively the modulefiles and directories contained in these matching directories when enabled or limit search results to the matching modulefiles and directories found at the depth level expressed by the search query if disabled. (default=yes)

New in version 4.3.

--enable-implicit-default

Define an implicit default version, for modules with none explicitly defined, to select when the name of the module to evaluate is passed without the mention of a specific version. When this option is disabled the name of the module passed for evaluation should be fully qualified elsewhere an error is returned. (default=yes)

New in version 4.3.

--enable-extended-default

Allow to specify module versions by their starting part, i.e. substring separated from the rest of the version string by a . character. (default=no)

New in version 4.4.

--enable-advanced-version-spec

Activate the advanced module version specifiers which enables to finely select module versions by specifying after the module name a version constraint prefixed by the @ character. (default=no)

New in version 4.4.

--enable-ml

Define the ml command, a handy frontend to the module command, when Modules initializes. (default=yes)

New in version 4.5.

--enable-color

Control if output should be colored by default or not. A value of yes equals to the auto color mode. no equals to the never color mode. (default=no)

New in version 4.3.

--enable-wa-277

Activate workaround for issue #277 related to Tcsh history mechanism which does not cope well with default module alias definition. Note that enabling this workaround solves Tcsh history issue but weakens shell evaluation of the code produced by modulefiles.

New in version 4.3.

--enable-windows-support

Install all required files for Windows platform (module, ml and envml command batch file and cmd.cmd initialization script). (default=no)

New in version 4.5.

--enable-new-features

Enable all new features that are disabled by default due to the substantial behavior changes they imply on Modules 4. This option is equivalent to the cumulative use of --enable-auto-handling, --enable-color, --with-icase=search, --enable-extended-default and --enable-advanced-version-spec. (default=no)

New in version 4.7.

--enable-mcookie-version-check

Enable check of the version specified right after Modules magic cookie (#%Module) in modulefiles, which defines the minimal version of the Modules tool to use in order to evaluate the modulefile. (default=yes)

New in version 4.7.

Optional Packages (the default for each option is displayed within parenthesis, to disable an option replace with by without for instance --without-modulepath):

--with-bin-search-path=PATHLIST

List of paths to look at when searching the location of tools required to build and configure Modules (default=/usr/bin:/bin:/usr/local/bin)

New in version 4.2.

--with-moduleshome

Location of the main Modules package file directory (default=PREFIX)

New in version 4.4.

--with-initconf-in=VALUE

Location where to install Modules initialization configuration files. Either initdir or etcdir (default=initdir)

New in version 4.1.

--with-tclsh=BIN

Name or full path of Tcl interpreter shell (default=tclsh)

New in version 4.0.

--with-pager=BIN

Name or full path of default pager program to use to paginate informational message output (can be superseded at run-time by environment variable) (default=less)

New in version 4.1.

--with-pager-opts=OPTLIST

Settings to apply to default pager program (default=-eFKRX)

New in version 4.1.

--with-verbosity=VALUE

Specify default message verbosity. accepted values are silent, concise, normal, verbose, verbose2, trace, debug and debug2. (default=normal)

New in version 4.3.

--with-dark-background-colors=SGRLIST

Default color set to apply if terminal background color is defined to dark. SGRLIST follows the same syntax than used in LS_COLORS. Each element in SGRLIST is an output item associated to a Select Graphic Rendition (SGR) code. Elements in SGRLIST are separated by :. Output items are designated by keys.

Items able to be colorized are: highlighted element (hi), debug information (db), trace information (tr) tag separator (se); Error (er), warning (wa), module error (me) and info (in) message prefixes; Modulepath (mp), directory (di), module alias (al), module variant (va), module symbolic version (sy) and module default version (de).

Module tags can also be colorized. The key to set in the color palette to get a graphical rendering of a tag is the tag name or the tag abbreviation if one is defined for tag. The SGR code applied to a tag name is ignored if an abbreviation is set for this tag thus the SGR code should be defined for this abbreviation to get a graphical rendering. Each basic tag has by default a key set in the color palette, based on its abbreviated string: auto-loaded (aL), forbidden (F), hidden and hidden-loaded (H), loaded (L), nearly-forbidden (nF), sticky (S) and super-sticky (sS).

For a complete SGR code reference, see https://en.wikipedia.org/wiki/ANSI_escape_code#SGR_(Select_Graphic_Rendition)_parameters. (default=hi=1:db=2:tr=2:se=2:er=91:wa=93:me=95:in=94:mp=1;94:di=94:al=96:va=93:sy=95:de=4:cm=92:aL=100:L=90;47:H=2:F=41:nF=43:S=46:sS=44)

New in version 4.3.

Changed in version 4.6: Output item for trace information (tr) added

Changed in version 4.7: Output items for module tags auto-loaded (aL), forbidden (F), hidden and hidden-loaded (H), loaded (L), nearly-forbidden (nF), sticky (S) and super-sticky (sS) added

Changed in version 4.8: Output item for module variant (va) added

--with-light-background-colors=SGRLIST

Default color set to apply if terminal background color is defined to light. Expect the same syntax than described for --with-dark-background-colors. (default=hi=1:db=2:tr=2:se=2:er=31:wa=33:me=35:in=34:mp=1;34:di=34:al=36:va=33:sy=35:de=4:cm=32:aL=107:L=47:H=2:F=101:nF=103:S=106:sS=104)

New in version 4.3.

Changed in version 4.6: Output item for trace information (tr) added

Changed in version 4.7: Output items for module tags auto-loaded (aL), forbidden (F), hidden and hidden-loaded (H), loaded (L), nearly-forbidden (nF), sticky (S) and super-sticky (sS) added

Changed in version 4.8: Output item for module variant (va) added

--with-terminal-background=VALUE

The terminal background color that determines the color set to apply by default between the dark background colors or the light background colors (default=dark)

New in version 4.3.

--with-locked-configs=CONFIGLIST

Ignore environment variable superseding value for the listed configuration options. Accepted option names in CONFIGLIST are extra_siteconfig and implicit_default (each option name should be separated by whitespace character). (default=no)

New in version 4.3.

--with-unload-match-order=VALUE

When unloading a module if multiple loaded modules match the request, unload module loaded first (returnfirst) or module loaded last (returnlast) (default=returnlast)

New in version 4.3.

--with-search-match=VALUE

When searching for a module with avail sub-command, match query string against module name start (starts_with) or any part of module name string (contains). (default=starts_with)

New in version 4.3.

--with-icase=VALUE

Apply a case insensitive match to module specification on avail, whatis and paths sub-commands (when set to search) or on all module sub-commands and modulefile Tcl commands for the module specification they receive as argument (when set to always). Case insensitive match is disabled when this option is set to never. (default=never)

New in version 4.4.

--with-nearly-forbidden-days=VALUE

Define the number of days a module is considered nearly forbidden prior reaching its expiry date. VALUE should be an integer comprised between 0 and 365. (default=14)

New in version 4.6.

--with-tag-abbrev=ABBRVLIST

Define the abbreviation to use when reporting each module tag. Each element in ABBRVLIST is a tag name associated to an abbreviation string (elements in ABBRVLIST are separated by :). (default=auto-loaded=aL:loaded=L:hidden=H:hidden-loaded=H:forbidden=F:nearly-forbidden=nF:sticky=S:super-sticky=sS)

New in version 4.7.

--with-tag-color-name=TAGLIST

Define the tags whose graphical rendering should be applied over their name instead of over the name of the module they are attached to. Each element in TAGLIST is a tag name or abbreviation (elements in TAGLIST are separated by :). (default=)

New in version 4.7.

--with-avail-output=LIST

Specify the content to report on avail sub-command regular output in addition to the available module names. Elements accepted in LIST are: modulepath, alias, dirwsym,``sym``, tag and key (elements in LIST are separated by :). The order of the elements in LIST does not matter. (default=modulepath:alias:dirwsym:sym:tag:key)

New in version 4.7.

--with-avail-terse-output=LIST

Specify the content to report on avail sub-command terse output in addition addition to the available module names. Elements accepted in LIST are: modulepath, alias, dirwsym,``sym``, tag and key (elements in LIST are separated by :). The order of the elements in LIST does not matter. (default=modulepath:alias:dirwsym:sym:tag)

New in version 4.7.

--with-list-output=LIST

Specify the content to report on list sub-command regular output in addition to the loaded module names. Elements accepted in LIST are: header, idx, variant, sym, tag and key (elements in LIST are separated by :). The order of the elements in LIST does not matter. (default=header:idx:variant:sym:tag:key)

New in version 4.7.

Changed in version 4.8: Element variant added and set by default

--with-list-terse-output=LIST

Specify the content to report on list sub-command terse output in addition to the loaded module names. Elements accepted in LIST are: header, idx, variant, sym, tag and key (elements in LIST are separated by :). The order of the elements in LIST does not matter. (default=header)

New in version 4.7.

Changed in version 4.8: Element variant added

--with-variant-shortcut=SHORTCUTLIST

Define the shortcut characters that could be used to specify variant names. Each element in SHORTCUTLIST is a variant name associated to a shortcut character (e.g., foo=%). Shortcuts cannot exceed a length of 1 character and cannot be alphanumeric characters ([A-Za-z0-9]) or characters with already a special meaning ([+~/@=-]). Elements in SHORTCUTLIST are separated by :. (default=)

New in version 4.8.

--with-editor=BIN

Name or full path of default editor program to use to open modulefile through the edit sub-command. (default=vi)

New in version 4.8.

--with-modulepath=PATHLIST

Default path list to setup as the default modulepaths. Each path in this list should be separated by :. Defined value is registered in the modulerc or .modulespath configuration file, depending on the --enable-dotmodulespath option. These files are respectively called initrc and modulespath if --with-initconf-in is set to etcdir. The path list value is read at initialization time to populate the MODULEPATH environment variable. By default, this modulepath is composed of the directory set for the system modulefiles (default=PREFIX/modulefiles or BASEPREFIX/$MODULE_VERSION/modulefiles if versioning installation mode enabled)

New in version 4.0.

--with-loadedmodules=MODLIST

Default modulefiles to load at Modules initialization time. Each modulefile in this list should be separated by :. Defined value is registered in the modulerc configuration file or in the initrc file if --with-initconf-in is set to etcdir. (default=no)

New in version 4.0.

--with-quarantine-vars=<VARNAME[=VALUE] ...>

Environment variables to put in quarantine when running the module command to ensure it a sane execution environment (each variable should be separated by space character). A value can eventually be set to a quarantine variable instead of emptying it. (default=no)

New in version 4.1.

--with-tcl

Directory containing the Tcl configuration script tclConfig.sh. Useful to compile Modules compatibility version or Modules Tcl extension library if this file cannot be automatically found in default locations.

--with-tclinclude

Directory containing the Tcl header files. Useful to compile Modules compatibility version or Modules Tcl extension library if these headers cannot be automatically found in default locations.

--with-python=BIN

Name or full path of Python interpreter command to set as shebang for helper scripts. (default=python)

New in version 4.5.

Installing Modules on Windows

This document is an overview of building and installing Modules on a Windows platform.

Requirements

Modules consists of one Tcl script so to run it from a user shell the only requirement is to have a working version of tclsh (version 8.4 or later) available on your system. tclsh is a part of Tcl.

A specific distribution zipball is provided to install Modules on a Windows platform. Content of this distribution zipball is ready for use and does not require a specific build step. All scripts and documentation found in this zipball are pre-built so there is no specific tools are required to install Modules from the Windows-specific distribution zipball.

Installation instructions

  1. Install a Tcl binary distribution for Windows like ActiveTcl or Magicsplat Tcl/Tk for Windows. Follow instructions provided with the chosen distribution to install it.

  1. Once installed, verify that the tclsh command is correctly found in defined PATH by typing the following command from a Windows cmd shell (windows string should be obtained as result):

    > echo puts $tcl_platform(platform) | tclsh
windows

  2. Download Modules specific distribution zipball for Windows from SourceForge or GitHub. Such distribution archives are available for Modules release starting version 4.5.0 and can be distinguished from the source tarball by the -win suffix in their name.

  1. Unpack downloaded zip file then enter deflated directory and execute the INSTALL.bat script file found in it. This script installs files by default in C:\Program Files\Environment Modules\ directory and adds the bin directory in this installation location to the system-wide PATH environment variable.

Note

INSTALL.bat script may require to be run with administrator rights to perform installation correctly.

  1. Once installed, verify that the module command is correctly found in defined PATH by typing the following command from a Windows cmd shell:

    > module -V
Modules Release 4.4.1 (2020-01-03)

Installation location can be adapted by running the INSTALL.bat script from a cmd console shell and passing desired installation target as argument. For instance to install Modules in C:\EnvironmentModules directory:

> INSTALL.bat C:\EnvironmentModules

Modules installation is now operational and you can setup your modulefiles. By default, the modulefiles directory in installation directory is defined as a modulepath and contains few modulefile examples:

> module avail
------- C:/Program Files/Environment Modules/modulefiles --------
module-git  module-info  null

Documentation of the module and ml commands and modulefile syntax can be found in the doc directory in installation directory.

MIGRATING

This document describes the major changes occurring between versions of Modules. It provides an overview of the new features and changed behaviors that will be encountered when upgrading.

From v4.7 to v4.8

This new version is backward-compatible with v4.7 and primarily fixes bugs and adds new features. Version 4.8 introduces new functionalities that are described in this section. See the 4.8 release notes for a complete list of the changes between Modules v4.7 and v4.8.

Editing modulefiles

edit sub-command is introduced to give the ability to open modulefiles in a text editor. Modulefiles can be specified like with any other sub-command: using regular, symbolic or aliased names or using advanced version specifiers.

$ ml edit foo

edit sub-command resolves the path toward the designated modulefile then call configured text editor to open this modulefile with it. Below, the modulefile is opened with the vi command:

#%Module
module-whatis [module-info name]
setenv PATH /path/to/foo-1.0/bin
~
~
~
"/path/to/modulefiles/foo/1.0" 3L, 42B 1,1           All

The editor configuration option controls the editor command to use. This option can be configured at installation time with the --with-editor installation option. If not set, editor configuration option is set by default to vi.

editor configuration option can be changed with the config sub-command. Which sets the MODULES_EDITOR environment variable.

The VISUAL or the EDITOR environment variables override the default value of editor configuration option but are overridden by the MODULES_EDITOR environment variable.

Using version range in version list

The Advanced module version specifiers mechanism has been improved to allow the use of version range (@:version, @vers1:vers2 or @version:) within version list (@version1,version2,...).

It is now possible to write for instance mod@:1.2,1.4:1.6,1.8: to designate all versions of module mod, except versions 1.3 and 1.7.

This improvement is available where the advanced version specifier syntax is supported. Thus it can be either used from the command-line or when writing modulefiles, for instance to hide or tag modules or to declare requirements.

Try module load with no complain if not found

Add the try-load sub-command that tries to load the modulefile passed as argument, like the load sub-command, but does not raise an error if this modulefile cannot be found.

$ module load unknown
ERROR: Unable to locate a modulefile for 'unknown'
$ echo $?
1
$ module try-load unknown
$ echo $?
0
$ module list
No Modulefiles Currently Loaded.

This sub-command first introduced by the Lmod project is added to Modules to improve the compatibility between the two module implementations.

try-load is also available within modulefile context to continue the evaluation of a modulefile in case no module is found in its attempt to load another modulefile

$ module display foo/1.0
-------------------------------------------------------------------
/path/to/modulefiles/foo/1.0:

module   try-load unknown/1.0
-------------------------------------------------------------------
$ module load foo/1.0
$ module list
Currently Loaded Modulefiles:
 1) foo/1.0
Module variants

Module variants is a new mechanism that allows to pass arguments to evaluated modulefiles in order to achieve different environment variable or module requirement setup with a single modulefile.

Variant specification relies on the Advanced module version specifiers mechanism, which leverages the variant syntax of the Spack package manager:

$ module config advanced_version_spec 1
$ module load -v bar/1.2 toolchain=a -debug
Loading bar/1.2{-debug:toolchain=a}

Variants are defined in modulefile with the variant command, which defines the variant type and its accepted values:

#%Module4.8
variant toolchain a b c
variant --boolean --default off debug

# select software build depending on variant values
set suffix -[getvariant toolchain]
if {$ModuleVariant(debug)} {
    append suffix -dbg
}

prepend-path PATH /path/to/bar-1.2$suffix/bin
prepend-path LD_LIBRARY_PATH /path/to/bar-1.2$suffix/lib

The bar/1.2 modulefile defines a toolchain variant, which accepts the a, b and c values, and a debug Boolean variant, which is set off by default. Once these two variants are declared, their value specified on module designation are instantiated in the ModuleVariant array variable which could also be queried with the getvariant modulefile command. Selected variant values enable to define a specific installation build path for the bar/1.2 software.

If a variant is not specified when designating module and if this variant is not declared with a default value, an error is obtained:

$ module purge
$ module load bar@1.2
Loading bar/1.2
  ERROR: No value specified for variant 'toolchain'
    Allowed values are: a b c

Once module is loaded, selected variants are reported on the list sub-command output:

$ module load bar@1.2 toolchain=b
$ module list
Currently Loaded Modulefiles:
 1) bar/1.2{-debug:toolchain=b}

Key:
{-variant}={variant=off}  {variant=value}

Note

The default value of the --with-list-output installation option has been updated to include variant information.

Variant specification could be used where the Advanced module version specifiers is supported. For instance a module may express a dependency over a specific module variant:

$ module show foo/2.1 toolchain=c
-------------------------------------------------------------------
/path/to/modulfiles/foo/2.1:

variant         toolchain a b c
prereq          bar@1.2 toolchain={toolchain}
prepend-path    PATH /path/to/foo-2.1-{toolchain}/bin
prepend-path    LD_LIBRARY_PATH /path/to/foo-2.1-{toolchain}/lib
-------------------------------------------------------------------

In this example, foo/2.1 module depends on bar/1.2 and the same toolchain variant should be selected for both modules in order to load two software builds that are compatible between each other.

$ module purge
$ module config auto_handling 1
$ module load foo/2.1 toolchain=a
Loading foo/2.1{toolchain=a}
  Loading requirement: bar/1.2{-debug:toolchain=a}
Variant shortcuts

The variant_shortcut configuration option is added to define shortcut characters for easily specifying variants. Instead of writing the variant name to specify it in module designation (e.g., name=value), the shortcut associated to this variant could be used (i.e., <shortcut>value):

$ module purge
$ module config variant_shortcut toolchain=%
$ module load foo/2.1 %a
Loading foo/2.1{%a}
  Loading requirement: bar/1.2{-debug:%a}

Configured shortcuts are also used to report the loaded variant on list sub-command output (shortcuts are explained in key section):

$ module list
Currently Loaded Modulefiles:
 1) bar/1.2{-debug:%a}  2) foo/2.1{%a}

Key:
auto-loaded  {-variant}={variant=off}  {%value}={toolchain=value}  {variant=value}

From v4.6 to v4.7

This new version is backward-compatible with v4.6 and primarily fixes bugs and adds new features. Version 4.7 introduces new functionalities that are described in this section. See the 4.7 release notes for a complete list of the changes between Modules v4.6 and v4.7.

Determining module implementation and version

New Modules variables are introduced to determine during the evaluation of a modulefile or a modulerc what module implementation is currently in use. The ModuleTool variable corresponds to the name of the module implementation and is set to Modules for this project. The ModuleToolVersion variable corresponds to the version number of the implementation (e.g. 4.7.0).

With these new variables it is possible to precisely know what module command is in use then adapt modulefile code to handle a specific behavior or leverage a new feature.

The modulefile command versioncmp is also introduced to provide a simple way to compare two version strings and return if first version string is less than, equal to or greater than second one.

if {[info exists ModuleTool] && $ModuleTool eq {Modules}
    && [versioncmp $ModuleToolVersion 4.7] >= 0} {
    # here some code specific for Modules 4.7 and later versions
}

The ModuleTool and ModuleToolVersion variables and the versioncmp modulefile command are supported by the Lmod project starting version 8.4.8.

Symbolic version to designate module loaded version

When the Advanced module version specifiers is enabled, the loaded symbolic version may be used to designate the currently loaded version of specified module.

$ ml display foo@loaded
-------------------------------------------------------------------
/path/to/modulefiles/foo/1.0:

module-whatis   foo/1.0
-------------------------------------------------------------------

If no version of specified module can be found loaded, an error is returned.

$ ml display foo@loaded
ERROR: No loaded version found for 'foo' module
Module tags

Module tags are piece of information that can be associated to individual modulefiles. Tags could be purely informational or may lead to specific behaviors.

Module tags may be inherited from the module state set by a modulefile command or consequence of a module action. Tags may also be associated to modules by using the new module-tag modulefile command.

Module tags are reported along the module they are associated to on avail and list sub-command results. Tags could be reported either:

  • along the module name, all tags set within angle brackets, each tag separated from the others with a colon character (e.g., foo/1.2 <tag1:tag2>).
$ cat /path/to/modulefiles/foo/.modulerc
#%Module
module-tag mytag foo
module-tag othertag foo/1.0
$ ml av
--------------- /path/to/modulefiles ---------------
foo/1.0 <mytag:othertag>  foo/2.0 <mytag>
$ ml foo/1.0
$ ml
Currently Loaded Modulefiles:
 1) foo/1.0 <mytag:othertag>
  • graphically rendered over the module name for each tag associated to a Select Graphic Rendition (SGR) code in the color palette (see MODULES_COLORS)
$ # set SGR code to report 'mytag' with blue background color
$ ml config colors "hi=1:di=94:L=90;47:mytag=102"
$ ml av
--------------- /path/to/modulefiles ---------------
foo/1.0 <othertag>  foo/2.0
$ ml
Currently Loaded Modulefiles:
 1) foo/1.0 <othertag>

The tag_abbrev configuration option is available to define abbreviated strings for module tags and then use these abbreviations instead of tag names when reporting tags on avail and list command results.

$ # add abbreviation for 'othertag' tag
$ ml config tag_abbrev loaded=L:othertag=oT
$ ml av
--------------- /path/to/modulefiles ---------------
foo/1.0 <oT>  foo/2.0
$ ml
Currently Loaded Modulefiles:
 1) foo/1.0 <oT>

When a SGR code is set for a tag in the color palette, this graphical rendition is applied by default over the module name and the tag name or its abbreviation is not displayed. If tag name or abbreviation is added to the tag_color_name configuration option, graphical rendering is applied to the tag name or abbreviation rather than over the module name they are attached to.

$ # add SGR code for 'oT' tag and set rendition over tag name
$ ml config colors "hi=1:di=94:L=90;47:mytag=44:oT=41"
$ ml config tag_color_name oT
$ ml av
--------------- /path/to/modulefiles ---------------
foo/1.0 <oT>  foo/2.0
$ ml
Currently Loaded Modulefiles:
 1) foo/1.0 <oT>

Tags inherited from module state, consequence of a module action or set by using module-tag but that have a special meaning currently are:

Tag Description Set with Abbr. Color
auto-loaded Module has been loaded automatically Inherited aL 
mod/1.0
forbidden Module cannot be loaded Inherited from module-forbid F 
mod/1.0
hidden Module is not visible on avail Inherited from module-hide H 
mod/1.0
hidden-loaded See Hiding loaded modules Inherited from module-hide H 
mod/1.0
loaded Module is currently loaded Inherited L 
mod/1.0
nearly-forbidden Module will soon not be able to load anymore Inherited from module-forbid nL 
mod/1.0
sticky See Sticky modules module-tag S 
mod/1.0
super-sticky See Sticky modules module-tag sS 
mod/1.0
Hiding loaded modules

The --hidden-loaded option has been added to the module-hide modulefile command and it indicates that designated hidden modules remain hidden after being loaded.

$ cat /path/to/modulefiles/foo/1.0
#%Module
module load bar
$ cat /path/to/modulefiles/bar/.modulerc
#%Module4.7
module-hide --soft --hidden-loaded bar

In this example, foo depends on bar which is set soft hidden and hidden once loaded. As a consequence, automated load of bar module will not be reported and bar/1.0 will not appear in loaded module list by default:

$ ml foo
$ ml
Currently Loaded Modulefiles:
 1) foo/1.0

However bar/1.0 is loaded. Hidden loaded modules can be unveiled with the --all/-a option set on the list sub-command. hidden-loaded tag (abbreviated by default to H when colored output is disabled) applies to such modules.

$ ml -a
Currently Loaded Modulefiles:
 1) bar/1.0   2) foo/1.0

To also get the informational messages about hidden loaded module automated load or unload, the new verbosity level verbose2 can be used (with -vv option for instance):

$ ml purge
$ ml -vv foo
Loading bar/1.0

Loading foo/1.0
  Loading requirement: bar/1.0
Sticky modules

Module stickyness is introduced, in a similar fashion than on the Lmod project, to allow to glue modules to the loaded environment. A sticky module cannot be unloaded, unless if the unload action is forced or if the module reloads after being unloaded.

A modulefile is declared sticky by applying it the sticky tag with the module-tag modulefile command.

$ cat mp/foo/.modulerc
#%Module4.7
module-tag sticky foo/1.0
$ ml
Currently Loaded Modulefiles:
 1) foo/1.0
$ ml -foo
Unloading foo/1.0
  ERROR: Unload of sticky module 'foo/1.0' skipped
$ ml
Currently Loaded Modulefiles:
 1) foo/1.0
$ ml --force -foo
Unloading foo/1.0
  WARNING: Unload of sticky module 'foo/1.0' forced
$ ml
No Modulefiles Currently Loaded.

Modulefile can also be defined super-sticky by applying the corresponding module tag. Super-sticky module cannot be unloaded even if the unload action is forced. It can only be unloaded if the module reloads afterward.

$ cat mp/bar/.modulerc
#%Module4.7
module-tag super-sticky bar/1.0
$ ml
Currently Loaded Modulefiles:
 1) bar/1.0
$ ml purge
Unloading bar/1.0
  ERROR: Unload of super-sticky module 'bar/1.0' skipped
$ ml purge -f
Unloading bar/1.0
  ERROR: Unload of super-sticky module 'bar/1.0' skipped
$ ml
Currently Loaded Modulefiles:
 1) bar/1.0

Modulefiles targeted by a sticky or a super-sticky tag are colored on avail and list sub-command outputs to indicate such tag applies. If colored output is disabled a tag abbreviation is reported along module designation (respectively S and sS).

In case the stickyness applies to the generic module name (and does not target a specific module version or version-set), one version of the sticky or super-sticty module can be swapped by another version of this same module:

$ cat mp/baz/.modulerc
#%Module4.7
module-tag sticky baz
$ ml
Currently Loaded Modulefiles:
 1) baz/2.0
$ ml switch baz/1.0
$ ml
Currently Loaded Modulefiles:
 1) baz/1.0
Explaining avail/list output

A Key section is added at the end of the avail and list sub-commands output to give hints on the meaning of the graphical rendition applied to elements or what the elements set in parentheses or chevrons along module name stand for.

$ ml av
------------------ /path/to/modulefiles ------------------
foo/1.0 <oT>  foo/2.0  foo/3.0

Key:
loaded      default-version  sticky        <oT>=othertag
modulepath  module-alias     <module-tag>
Configuring avail/list output

New configuration options are introduced to control what content to output in addition to modules names on the regular and terse output modes of the avail and list sub-commands.

These new configuration options named avail_output, avail_terse_output, list_output and list_terse_output can be updated using the config sub-command or set at installation time respectively with the --with-avail-output, --with-avail-terse-output, --with-list-output and --with-list-terse-output configure options.

The four options accept a colon separated list of elements as value. Accepted elements for the avail-related options are: modulepath, alias, dirwsym, sym, tag and key. Accepted elements for the list-related options are: header, idx, sym, tag and key.

In the following example, default output configuration for the avail sub-command is checked then module tags and key section are removed to get a simpler output:

$ ml config avail_output
Modules Release 4.7.0 (2021-02-19)

- Config. name ---------.- Value (set by if default overridden) ---------------
avail_output              modulepath:alias:dirwsym:sym:tag:key
$ ml av
------------------ /path/to/modulefiles ------------------
bar/1.0  bar/2.0  foo/1.0  foo/2.0  foo/2.2

Key:
modulepath       module-alias  sticky
default-version  forbidden
$ ml config avail_output modulepath:alias:dirwsym:sym
$ ml av
------------------ /path/to/modulefiles ------------------
bar/1.0  bar/2.0  foo/1.0  foo/2.0  foo/2.2

The --output/-o switches are added to define a specific output configuration for the duration of the associated command line. The following example shows how to limit the content reported on a module list to the loaded index and the symbolic versions in addition to the module names:

$ ml
Currently Loaded Modulefiles:
 1) bar/1.0   2) foo/2.0

Key:
default-version  sticky
$ ml -o idx:sym
 1) bar/1.0   2) foo/2.0

When the new configuration options or command line switches are set to an empty value, the module names are the sole information reported:

$ ml -t -o ""
bar/1.0
foo/2.0

In case the modulepath element is withdrawn from the avail sub-command output configuration, the available modules from all enabled modulepaths are reported as a single list:

$ ml av
--------------- /path/to/other/modulefiles ---------------
baz/1.0  baz/2.0

------------------ /path/to/modulefiles ------------------
bar/1.0  bar/2.0  foo/1.0  foo/2.0  foo/2.2

Key:
modulepath       module-alias  sticky
default-version  forbidden
$ ml av --output=alias:tag
bar/1.0  baz/1.0  foo/1.0  foo/2.2
bar/2.0  baz/2.0  foo/2.0

Note

The avail_report_dir_sym and avail_report_mfile_sym locked configuration options have been removed. Their behaviors can now be obtained by respectively adding the dirwsym and sym elements to the avail_output or avail_terse_output configuration options.

From v4.5 to v4.6

This new version is backward-compatible with v4.5 and primarily fixes bugs and adds new features. Version 4.6 introduces new functionalities that are described in this section. See the 4.6 release notes for a complete list of the changes between Modules v4.5 and v4.6.

sh-to-mod sub-command

The sh-to-mod sub-command is added to output as a modulefile content the environment changes done by the evaluation of a shell script passed as argument. sh-to-mod is especially useful for software providing a shell script for their enablement in shell session: it can convert these scripts into modulefiles.

Say for instance, a foo software has been installed and it provides a foo-setup.sh script to activate foo software in user environment:

$ cat /path/to/foo-1.2/foo-setup.sh
#!/bin/sh
export FOOENV="$1"
export PATH=/path/to/foo-1.2/bin:$PATH
alias foo='foobin -q -l'

Calling module sh-to-mod on this shell script outputs the environment changes it performs as a modulefile content:

$ module sh-to-mod sh /path/to/foo-1.2/foo-setup.sh arg1
#%Module
prepend-path    PATH /path/to/foo-1.2/bin
set-alias       foo {foobin -q -l}
setenv          FOOENV arg1

Changes on environment variables, shell aliases, shell functions and current working directory are tracked. The following shells are supported: sh, dash, csh, tcsh, bash, ksh, ksh93, zsh and fish.

sh-to-mod acts as a full replacement for the standalone createmodule.sh and createmodule.py scripts. However those two scripts are currently still provided for compatibility purpose.

source-sh modulefile command

The source-sh modulefile command is introduced to source environment changes done by the evaluation of a shell script passed as argument. With newly introduced sh-to-mod sub-command resulting environment changes done by script are output as modulefile commands. source-sh applies those modulefile commands as if they were directly written in loading modulefile.

source-sh is useful for software providing a shell script for their enablement. If you want to enable such software with module yet using shell script provided by software for this task, just write a modulefile using source-sh command to call the shell script.

Keeping the same example used to describe sh-to-mod sub-command: foo software provides a foo-setup.sh script for its activation. Create a modulefile foo/1.2 that calls this script:

$ cat /path/to/modulefiles/foo/1.2
#%Module4.6
source-sh sh /path/to/foo-1.2/foo-setup.sh arg1

Displaying this modulefile indicates the environment changes done by script:

$ module display foo/1.2
-------------------------------------------------------------------
/path/to/modulefiles/foo/1.2:

prepend-path    PATH /path/to/foo-1.2/bin
set-alias       foo {foobin -q -l}
setenv          FOOENV arg1
-------------------------------------------------------------------

Loading the modulefile applies the environment changes seen above:

$ module load -v foo/1.2
Loading foo/1.2
$ echo $FOOENV
arg1
$ alias foo
alias foo='foobin -q -l'

Track of these changes is kept in user environment to be able to undo them when modulefile is unloaded:

$ module unload -v foo/1.2
Unloading foo/1.2
$ echo $FOOENV

$ alias foo
bash: alias: foo: not found

Changes on environment variables, shell aliases, shell functions and current working directory are tracked. The following shells are supported: sh, dash, csh, tcsh, bash, ksh, ksh93, zsh and fish.

Querying user's name and groups membership

Two new sub-commands are introduced for the module-info modulefile command: username and usergroups. They respectively fetch the name of the user currently running modulecmd.tcl or the name of all the groups this user is member of.

These two new modulefile commands can help to adapt code to specific users or groups. Like for instance to instantiate a modulefile for each group the user is member of:

$ cat /path/to/modulefiles/foo/.modulerc
#%Module4.6
foreach grp [module-info usergroups] {
    module-virtual foo/$grp .common
}
$ id -G -n
grp1 grp2 grp3
$ module avail
--------------- /path/to/modulefiles ---------------
foo/grp1  foo/grp2  foo/grp3

username and usergroups sub-commands of module-info modulefile command are only supported on Unix platform.

Hiding modules

The newly introduced module-hide modulefile command enables to dynamically hide modulefiles, module aliases or symbolic versions specified to it:

$ cat /path/to/modulefiles/bar/.modulerc
#%Module4.6
module-version bar/1.0 old
# hide 'old' symbolic version
module-hide bar/old
# hide all version 2 and above
module-hide bar@2:
$ cat /path/to/modulefiles/.modulerc
#%Module4.6
# hide all versions of foo module
module-hide foo

module-hide commands should be placed in module rc files and can leverage the Advanced module version specifiers syntax as shown in the above example.

Hidden modules are excluded from available module search or module selection unless query refers to hidden module by its exact name:

$ ml av
--------------- /path/to/modulefiles ---------------
bar/1.0  bar/2.0
$ module load -v foo
ERROR: Unable to locate a modulefile for 'foo'
$ module load -v foo/1.0
Loading foo/1.0
$ module avail bar/old
--------------- /path/to/modulefiles ---------------
bar/1.0(old)

module-hide command accepts a --soft option to apply a lighter of hiding to modules:

$ cat /path/to/modulefiles/qux/.modulerc
#%Module4.6
# softly hide all qux modules
module-hide --soft qux

The soft hiding mode enables to hide modules from full availability listing yet keeping the ability to select such module for load without having to use module exact name:

$ ml av
--------------- /path/to/modulefiles ---------------
bar/1.0  bar/2.0
$ ml av qux
--------------- /path/to/modulefiles ---------------
qux/1.0  qux/2.0
$ module load -v qux
Loading qux/2.0

Alternatively, a --hard option can be set on module-hide command to ensure designated modules do not unveil even if referred by their exact name:

$ cat /path/to/modulefiles/qux/.modulerc
#%Module4.6
# softly hide all qux modules
module-hide --soft qux
# set highest version of qux hard hidden
module-hide --hard qux/3.0
$ ml av qux/3.0
$ ml qux/3.0
ERROR: Unable to locate a modulefile for 'qux/3.0'

Some users or groups can be set unaffected by hiding mechanism with the --not-user or --not-group options:

$ cat /path/to/modulefiles/quuz/.modulerc
#%Module4.6
# hiding does not apply to grp1 and grp2 groups
module-hide --not-group {grp1 grp2} quuz

$ id --groups --name
grp1 grp7
$ ml av quuz
--------------- /path/to/modulefiles ---------------
quuz/1.0  quuz/2.0
$ ml -v quuz
Loading quuz/2.0

Hiding mechanism can also be set effective only before or after a given date time with the --before and --after options. Accepted date time format is YYYY-MM-DD[THH:MM].

$ cat /path/to/modulefiles/fum/.modulerc
#%Module4.6
# hide only before a given date
module-hide --hard --before 2020-09-01T12:00 fum/1.0
# hide only after a given date
module-hide --hard --after 2020-09-01 fum/2.0

$ date
Fri 04 Sep 2020 06:21:48 AM CEST
$ ml av fum
--------------- /path/to/modulefiles ---------------
fum/1.0

Hidden modules can be included in available module searches if option --all/-a is set on avail, aliases, whatis or search sub-commands. Hard hidden modules are unaffected by this option and stay hidden.

$ ml av -a
--------------- /path/to/modulefiles ---------------
bar/1.0(old)  foo/1.0  fum/1.0   quuz/2.0  qux/2.0
bar/2.0       foo/2.0  quuz/1.0  qux/1.0
Forbidding use of modules

The module-forbid modulefile command is added to dynamically forbid the evaluation of modulefiles it specifies. When forbidden, a module cannot be loaded and an access error is returned when an attempt is made to evaluate it.

$ cat /path/to/modulefiles/foo/.modulerc
#%Module4.6
module-forbid foo@1:
$ ml foo/1.0
ERROR: Access to module 'foo/1.0' is denied
$ ml
No Modulefiles Currently Loaded.

module-forbid statements can be coupled with module-hide statements to hide modules in addition to forbid their use. module-forbid supports the --not-user, --not-group, --before and --after options to still allow some users or forbid modules before or after a given date time.

An additional error message can be defined with the --message option to guide for instance users when they try to evaluate a forbidden module:

$ cat /path/to/modulefiles/bar/.modulerc
#%Module4.6
module-forbid --message {Software bar/1.0 is decommissioned, please now use\
    bar/2.0} --after 2020-09-01 bar/1.0
$ ml bar/1.0
ERROR: Access to module 'bar/1.0' is denied
  Software bar/1.0 is decommissioned, please now use bar/2.0

When an evaluated module will soon be forbidden, a message is returned to the user to warn him/her of the near limit. An additional warning message can also be defined here with the --nearly-message option to guide users.

$ cat /path/to/modulefiles/qux/.modulerc
#%Module4.6
module-forbid --nearly-message {Version 1.0 will soon expire, please now use\
    version 2.0} --after 2020-09-15 qux/1.0
$ date
Tue 08 Sep 2020 06:49:43 AM CEST
$ ml qux/1.0
Loading qux/1.0
  WARNING: Access to module will be denied starting '2020-09-15'
    Version 1.0 will soon expire, please now use version 2.0

The range of time the nearly forbidden warning appears can be controlled with the nearly_forbidden_days configuration option, whose value equals to the number of days prior the module starts to be forbidden. This configuration is set to 14 (days) by default and this value can be controlled at configure time with --with-nearly-forbidden-days option. When the nearly_forbidden_days configuration is set through the config sub-command, the MODULES_NEARLY_FORBIDDEN_DAYS environment variable is set.

Tracing module execution

The trace verbosity is introduced between the verbose and debug levels to report details on module searches, resolutions, selections and evaluations. Trace mode can be enabled by setting the verbosity config to the trace value or by using the -T/--trace command-line switches.

To specifically render trace messages, the tr key is added to the color palette with a default value of 2 (decreased intensity).

$ ml -T foo
Evaluate modulerc: '/path/to/modulefiles/.modulerc'
Get modules: {foo} matching 'foo' in '/path/to/modulefiles'
Resolve: 'foo' into 'bar'
Get modules: {bar bar/1.0} matching 'bar' in '/path/to/modulefiles'
Select module: 'bar/1.0' (/path/to/modulefiles/bar/1.0) matching 'bar/1.0'

Loading bar/1.0
  Evaluate modulefile: '/path/to/modulefiles/bar/1.0' as 'bar/1.0'

From v4.4 to v4.5

This new version is backward-compatible with v4.4 and primarily fixes bugs and adds new features. Version 4.5 introduces new functionalities that are described in this section. See the 4.5 release notes for a complete list of the changes between Modules v4.4 and v4.5.

ml command

The ml command is added to Modules. ml is a frontend to the module command that reduces the number of characters to type to trigger module actions.

With no argument provided ml is equivalent to module list, ml foo corresponds to module load foo and ml -foo means module unload foo:

$ ml foo
$ ml
Currently Loaded Modulefiles:
 1) foo/2
$ ml -foo
$ ml
No Modulefiles Currently Loaded.

Multiple modules to either load or unload can be combined on a single command. The unloads are first processed then the loads.

ml accepts all command-line switches and sub-commands accepted by module command:

$ ml avail -t foo
foo/1
foo/2

This handy interface has been originally developed by the Lmod project. Having this command line interface also supported on Modules helps to provide a similar user experience whatever the module implementation used.

JSON format output

The -j and --json command line switches are added for the avail, list, savelist, whatis and search module sub-commands. When set, the output result of these sub-commands is rendered in JSON format:

$ module avail --json bar | python -mjson.tool
{
    "/path/to/modulefiles": {
        "bar/2.3": {
            "name": "bar/2.3",
            "pathname": "/path/to/modulefiles/bar/2.3",
            "symbols": [
                "default"
            ],
            "type": "modulefile"
        },
        "bar/3.4": {
            "name": "bar/3.4",
            "pathname": "/path/to/modulefiles/bar/3.4",
            "symbols": [],
            "type": "modulefile"
        }
    }
}
$ ml whatis -j foo/1.2.3 | python -mjson.tool
{
    "/path/to/modulefiles": {
        "foo/1.2.3": {
            "name": "foo/1.2.3",
            "whatis": [
                "The foo/1.2.3 modulefile"
            ]
        }
    }
}
Improved Windows support

A new option to the ./configure script named --enable-windows-support is introduced to install additional files relative to the enablement of Modules on the Windows platform. When set, this option installs module.cmd, ml.cmd and envml.cmd scripts in bindir and initialization script cmd.cmd in initdir. With these four files the Modules installation may be used from either a Unix or a Windows platform.

module.cmd, ml.cmd and envml.cmd scripts respectively provide the module, ml and envml commands for Windows cmd terminal shell, relying on modulecmd.tcl script which was already able to produce shell code for this Windows shell. Initialization script cmd.cmd adds the directory of module.cmd, ml.cmd and envml.cmd to PATH.

These Windows-specific files are relocatable: module.cmd, ml.cmd and envml.cmd scripts expect to find initialization script cmd.cmd in the init directory next to them (to setup Modules-specific variables in current environment) and cmd.cmd expects modulecmd.tcl to be found in libexec directory and the 3 commands in bin directory next to it.

Starting from this 4.5 release a distribution zipball is published to install Modules on Windows. This zip archive ships an install and an uninstall scripts (INSTALL.bat and UNINSTALL.bat). The zipball can be built locally from Modules sources by running make dist-win.

The Installing Modules on Windows document describes how to install Modules on Windows from the distribution zipball.

Error stack trace

Error messages will now embed a stack trace for unknown errors to help localize the root cause of issues. This change applies to modulefile evaluation:

Loading foo/1.2
  Module ERROR: add-path cannot handle path equals to separator string
        while executing
    "append-path PATH :"
        (file "/path/to/modulefiles/foo/1.2" line 24)
    Please contact <root@localhost>

A stack trace is also returned when an unknown error occurs in modulecmd.tcl script, which facilitates issue report and analysis:

$ module load bar
ERROR: invalid command name "badcommand"
      while executing
  "badcommand"
      (procedure "module" line 14)
      invoked from within
  "module load bar"
      ("eval" body line 1)
      invoked from within
  "eval $execcmdlist"
  Please report this issue at https://github.com/cea-hpc/modules/issues
Automatic default and latest symbolic versions

When the implicit default mechanism and the Advanced module version specifiers are both enabled, a default and a latest symbolic versions are automatically defined for each module name.

This new feature gives the ability to select the highest version available for a module, without knowing beforehand this version name:

$ module load -v foo@latest
Loading foo/1.10

The symbolic versions are automatically defined unless a symbolic version, an alias or a regular module version already exists for these default or latest version names.

From v4.3 to v4.4

This new version is backward-compatible with v4.3 and primarily fixes bugs and adds new features. Version 4.4 introduces new functionalities that are described in this section. See the 4.4 release notes for a complete list of the changes between Modules v4.3 and v4.4.

Warning

Modules configuration option handling has been reworked internally to provide a unified way for all options to get initialized, retrieved or set. Existing site-specific configuration script should be reviewed to make use of the new getConf, setConf, unsetConf and lappendConf procedures to manipulate configuration options.

Specify modules in a case insensitive manner

The ability to match module name in a case insensitive manner has been added. This feature can be enabled at different level with the following values set to the icase configuration option:

  • never: a case sensitive match is applied in any cases
  • search: a case insensitive match is applied to the avail, whatis and paths sub-commands
  • always: a case insensitive match is applied to search contexts and also to the other module sub-commands and modulefile Tcl commands for the module specification they receive as argument.

It can help for instance to load a module without knowing the case used to name its relative modulefile:

$ module config icase always
$ module load -v mysoftware
Loading MySoftware/1.0

Insensitive case match activation can be controlled at configure time with the --with-icase option, which could be passed any of the above activation levels. This option could be superseded with the MODULES_ICASE environment variable, which could be set through the config sub-command with the icase option. Command-line switch --icase supersedes in turns any other icase configurations. When this command-line switch is passed, icase mode equals always.

Extended default

The extended default mechanism has been introduced to help selecting a module when only the first numbers in its version are specified. Starting portion of the version, part separated from the rest of the version string by a . character, could be used to refer to a more precise version number.

This mechanism is activated through the new configuration option extended_default. It enables to refer to a module named foo/1.2.3 as foo/1.2 or foo/1:

$ module config extended_default 1
$ module load -v foo/1
Loading foo/1.2.3

When multiple versions match partial version specified and only one module should be selected, the default version (whether implicitly or explicitly defined) among matches is returned. The following example shows that foo/1.1.1, the foo module default version, is selected when it matches query. Elsewhere the highest version (also called the latest version or the implicit default) among matching modules is returned:

$ module av foo
--------------- /path/to/modulefiles ---------------
foo/1.1.1(default)  foo/1.2.1  foo/1.10
foo/1.1.10          foo/1.2.3
$ module load -v foo/1.1
Loading foo/1.1.1
$ module purge
$ module load -v foo/1.2
Loading foo/1.2.3
$ module purge
$ module load -v foo/1
Loading foo/1.1.1

In case implicit_default option is disabled and no explicit default is found among matches, an error is returned:

$ module config implicit_default 0
$ module load -v foo/1.2
ERROR: No default version defined for 'foo/1.2'

When it is enabled, extended default applies everywhere a module could be specified, which means it could be used with any module sub-command or any modulefile Tcl command receiving a module specification as argument. It may help for instance to declare dependencies between modules:

$ module show bar/3
----------------------------------------------------------
/path/to/modulefiles/bar/3.4:

prereq             foo/1.2
----------------------------------------------------------
$ module load --auto bar/3
Loading bar/3.4
  Loading requirement: foo/1.2.3

Extended default activation can be controlled at configure time with the --enable-extended-default option. This option could be superseded with the MODULES_EXTENDED_DEFAULT environment variable, which could be set through the config sub-command with the extended_default option.

Advanced module version specifiers

The ability to specify finer constraints on module version has been added to Modules. It enables to filter the module selection to a given version list or range by specifying after the module name a version constraint prefixed by the @ character.

This new feature leverages the version specifier syntax of the Spack package manager as this syntax covers all the needs for a fine-grained selection of module versions. It copes very well with command-line typing, by avoiding characters having a special meaning on shells. Moreover the users of Spack that also are users of Modules may already be familiar with this syntax.

The mechanism introduced here is called advanced module version specifier and it can be activated through the new configuration option advanced_version_spec. Constraints can be expressed to refine the selection of module version to:

  • a single version with the @version syntax, for instance foo@1.2.3 syntax will select module foo/1.2.3
  • a list of versions with the @version1,version2,... syntax, for instance foo@1.2.3,1.10 will match modules foo/1.2.3 and foo/1.10
  • a range of versions with the @version1:, @:version2 and @version1:version2 syntaxes, for instance foo@1.2: will select all versions of module foo greater than or equal to 1.2, foo@:1.3 will select all versions less than or equal to 1.3 and foo@1.2:1.3 matches all versions between 1.2 and 1.3 including 1.2 and 1.3 versions

This new feature enables for instance to list available versions of module foo higher or equal to 1.2:

$ module config advanced_version_spec 1
$ module av foo
--------------- /path/to/modulefiles ---------------
foo/1.1.1(default)  foo/1.2.1  foo/1.10
foo/1.1.10          foo/1.2.3
$ module av foo@1.2:
--------------- /path/to/modulefiles ---------------
foo/1.2.1  foo/1.2.3  foo/1.10

Then choose to load for instance a version higher than or equal to 1.2 and less than or equal to 1.3. Default version is selected if it corresponds to a version included in the range, elsewhere the highest version (also called latest version or implicit default) is selected:

$ module load -v foo@1.2:1.3
Loading foo/1.2.3

In case implicit_default option is disabled and no explicit default is found among version specifier matches, an error is returned:

$ module config implicit_default 0
$ module load -v foo@1.2:1.3
ERROR: No default version defined for 'foo@1.2:1.3'

When advanced module version specifier is enabled, it applies everywhere a module could be specified, which means it could be used with any module sub-command or any modulefile Tcl command receiving a module specification as argument. It may help for instance to declare smoother dependencies between modules:

$ module show bar@:2
----------------------------------------------------------
/path/to/modulefiles/bar/2.3:

prereq          foo@1.1.10,1.2.1
----------------------------------------------------------
$ module load --auto bar@:2
Loading bar/2.3
  Loading requirement: foo/1.2.1

Advanced specification of single version or list of versions may benefit from the activation of the Extended default mechanism (range of versions natively handles abbreviated versions):

$ module config extended_default 1
$ module load -v foo@1.2
Loading foo/1.2.3
$ module unload -v foo @1.2,1.5
Unloading foo/1.2.3

Advanced module version specifier activation can be controlled at configure time with the --enable-advanced-version-spec option. This option could be superseded with the MODULES_ADVANCED_VERSION_SPEC environment variable, which could be set through the config sub-command with the advanced_version_spec option.

From v4.2 to v4.3

This new version is backward-compatible with v4.2 and primarily fixes bugs and adds new features. Version 4.3 introduces new functionalities that are described in this section. See the 4.3 release notes for a complete list of the changes between Modules v4.2 and v4.3.

Modulepath rc file

A .modulerc file found at the root of an enabled modulepath directory is now evaluated when modulepath is walked through to locate modulefiles. This modulepath rc file gives for instance the ability to define module alias whose name does not correspond to any module directory in this modulepath. Thus this kind of module alias would not be found unless if it is defined at the modulepath global scope.

Further I/O operations optimization

Additional work has been performed to save a significant number of filesystem I/O operations made to search and evaluate modulefiles.

When fully read, the content of a modulefile is now cached in memory to avoid new I/O operations in case this modulefile should be read one more time during the same module command evaluation.

Except for path, paths, list, avail and aliases module commands always fully read a modulefile whether its full content is needed or just its header to verify its validity. This way modulefiles are only read once on commands that first check modulefile validity then read again valid files to get their full content.

Last but not least, Modules Tcl extension library is introduced to extend the Tcl language in order to provide more optimized I/O commands to read a file or a directory content than native Tcl commands do. This library is built and enabled in modulecmd.tcl script with --enable-libtclenvmodules configure argument (it is enabled by default). As this library is written in C, it must be compiled and --with-tcl or --with-tclinclude configure arguments may be used to indicate where to find Tcl development files.

Modules Tcl extension library greatly reduces the number of filesystem I/O operations by removing unneeded ioctl, fcntl and lstat system calls done (by Tcl open command) to read each modulefile. Directory content read is also improved by fetching hidden and regular files in one pass. Moreover .modulerc and .version read access is tested only if these files are found in the directory.

Colored output

The ability to graphically enhance some part of the produced output has been added to improve readability. Among others, error, warning and info message prefixes can be colored as well as modulepath, module alias and symbolic version.

Color mode can be set to never, auto or always. When color mode is set to auto, output is colored only if the standard error output channel is attached to a terminal.

Default color mode could be controlled at configure time with the --enable-color and the --disable-color option, which respectively correspond to the auto and never color mode. This default mode could be superseded with the CLICOLOR, CLICOLOR_FORCE and MODULES_COLOR environment variables and the --color command-line switch.

Color to apply to each element can be controlled with the MODULES_COLORS environment variable or the --with-dark-background-colors and --with-light-background-colors configure options. These variable and options take as value a colon-separated list in the same fashion LS_COLORS does. In this list, output item that should be highlighted is designated by a key which is associated to a Select Graphic Rendition (SGR) code.

The MODULES_TERM_BACKGROUND environment variable and the --with-terminal-background configure option help Modules to determine if the color set for dark background or the color set for light background should be used to color output in case no specific color set is defined with the MODULES_COLORS variable.

Output items able to be colorized and their relative key are: highlighted element (hi), debug information (db), tag separator (se); Error (er), warning (wa), module error (me) and info (in) message prefixes; Modulepath (mp), directory (di), module alias (al), module symbolic version (sy), module default version (de) and modulefile command (cm).

For instance the default color set for a terminal with dark background is defined to:

hi=1:db=2:se=2:er=91:wa=93:me=95:in=94:mp=1;94:di=94:al=96:sy=95:de=4:cm=92

When colored output is enabled and a specific graphical rendition is defined for module default version, the default symbol is omitted and instead the defined graphical rendition is applied to the relative modulefile. When colored output is enabled and a specific graphical rendition is defined for module alias, the @ symbol is omitted.

CLICOLOR and CLICOLOR_FORCE environment variables are also honored to define color mode. The never mode is set if CLICOLOR equals to 0. If CLICOLOR is set to another value, it corresponds to the auto mode. The always mode is set if CLICOLOR_FORCE is set to a value different than 0. Color mode set with these two variables is superseded by mode set with MODULES_COLOR environment variable.

Configure modulecmd with config sub-command

The config sub-command has been added to module to help getting or setting the modulecmd.tcl options. With no additional command-line argument, this sub-command reports the current value of all existing options with a mention to indicate if this value has been overridden from a command-line switch or from an environment variable.

See the description of this sub-command in the module man page for a complete reference on existing configuration options.

Most of the options can be altered by passing the option name and a value to the sub-command. Setting an option by this mean overrides its default value, set at installation time in modulecmd.tcl script, by defining the environment variable which supersedes this default.:

$ module config auto_handling 1
$ module config auto_handling
Modules Release 4.3.0 (2019-07-26)

- Config. name ---------.- Value (set by if default overridden) ---------------
auto_handling             1 (env-var)

Setting options with module config could be done in the Modules initialization RC file to change default value of options when module command is initialized.

When command-line switch --reset and an option name is passed to the config sub-command, it restores default value for configuration option by unsetting related environment variable.

With command-line switch --dump-state, the config sub-command reports, in addition to currently set options, the current state of modulecmd.tcl script and Modules-related environment variables. Providing the output of the module config --dump-state command when submitting an issue to the Modules project will help to analyze the situation.

Control module command verbosity

The ability to control message verbosity has been added so module command can be configured whether it should display more or less information. Available verbosity levels from the least to the most verbose are:

  • silent: turn off error, warning and informational messages but does not affect module command output result.
  • concise: enable error and warning messages but disable informational messages.
  • normal: turn on informational messages, like a report of the additional module evaluations triggered by loading or unloading modules, aborted evaluation issues or a report of each module evaluation occurring during a restore or source sub-commands.
  • verbose: add additional informational messages, like a systematic report of the loading or unloading module evaluations.
  • debug: print debugging messages about module command execution.

Default verbosity level can be controlled at configure time with the --with-verbosity option, which could be passed any of the above level names. This default verbosity level could be superseded with the MODULES_VERBOSITY environment variable, which could be set through the config sub-command with the verbosity option. Command-line switches --silent, --verbose and --debug supersede in turns any other verbosity configuration to respectively set module command silent, verbose or in debug mode.

Other new sub-commands, command-line switches and environment variables
  • The avail sub-command gets two new command-line switches: --indepth and --no-indepth. These options control whether search results should recursively include or not modulefiles from directories matching search query. Shell completion scripts have been updated to complete available modulefiles in the no in depth mode.
  • The MODULES_AVAIL_INDEPTH environment variable defines if the avail sub-command should include or exclude by default the modulefiles from directories matching search query. Its value is superseded by the use of the --indepth and --no-indepth command-line switches.
  • The clear sub-command, which was available on Modules version 3.2, has been reintroduced. This sub-command resets the Modules runtime information but does not apply further changes to the environment at all. This sub-command now leverages the --force command-line switch to skip its confirmation dialog.
  • The MODULES_SITECONFIG environment variable defines an additional siteconfig script which is loaded if it exists after the siteconfig script configured at build time in modulecmd.tcl. This ability is enabled by default and could be disabled with configure option --with-locked-configs=extra_siteconfig.
  • The MODULES_UNLOAD_MATCH_ORDER environment variable sets whether the firstly or the lastly loaded module should be selected for unload when multiple loaded modules match unload request. Configure option --with-unload-match-order defines this setting which can be superseded by the environment variable. By default, lastly loaded module is selected and it is recommended to keep this behavior when used modulefiles express dependencies between each other.
  • The MODULES_IMPLICIT_DEFAULT environment variable sets whether an implicit default version should be defined for modules with no default version explicitly defined. When enabled, which is the default behavior, a module version is automatically selected (latest one) when the generic name of the module is passed. When implicit default is disabled and no default version is explicitly defined for a module, the name of this module to evaluate should be fully qualified elsewhere an error is returned. Configure option --enable-implicit-default defines this setting which can be superseded by the environment variable. This superseding mechanism can be disabled with configure option --with-locked-configs=implicit_default.
  • The MODULES_SEARCH_MATCH environment variable defines the matching style to perform when searching for available modules. With starts_with value, modules whose name begins by search query string are returned. When search match style is set to contains, modules returned are those whose fully qualified name contains search query string. Configure option --with-search-match defines this setting which can be superseded by the environment variable, which in turns can be superseded by the --starts-with and --contains command-line switches of avail module sub-command.
  • The MODULES_SET_SHELL_STARTUP environment variable controls whether or not shell startup file should be set to ensure module command is defined once shell has been initialized. When enabled, the ENV and BASH_ENV environment variables are set, when module function is defined, to the Modules bourne shell initialization script. Configure options --enable-set-shell-startup and --disable-set-shell-startup define this setting which can be superseded by the environment variable.
  • When initializing the module command in a shell session, initialization configuration files stored in the defined configuration directory are taken into account if present instead of the configuration files stored in the initialization script directory. When they are stored in the configuration directory, these configuration files are named initrc and modulespath instead of respectively modulerc and .modulespath. The location of the installation of those files can be controlled with configure option --with-initconf-in, which accepts etcdir and initdir values.
  • The MODULES_WA_277 environment variable helps to define an alternative module alias on Tcsh shell when set to 1. It workarounds an issue on Tcsh history mechanism occurring with default module command alias: erroneous history entries are recorded each time the module command is called. However the alternative definition of the module alias weakens shell evaluation of the code produced by modulefiles. Characters with special meaning for Tcsh shell (like { and }) may not be used anymore in shell alias definition elsewhere the evaluation of the code produced by modulefiles will return a syntax error.

From v4.1 to v4.2

This new version is backward-compatible with v4.1 and primarily fixes bugs and adds new features. Version 4.2 introduces new functionalities that are described in this section. See the 4.2 release notes for a complete list of the changes between Modules v4.1 and v4.2.

Modulefile conflict constraints consistency

With the conflict modulefile command, a given modulefile can list the other modulefiles it conflicts with. To load this modulefile, the modulefiles it conflicts with cannot be loaded.

This constraint was until now satisfied when loading the modulefile declaring the conflict but it vanished as soon as this modulefile was loaded. In the following example a modulefile declares a conflict with b:

$ module load b a
WARNING: a cannot be loaded due to a conflict.
HINT: Might try "module unload b" first.
$ module list
Currently Loaded Modulefiles:
 1) b
$ module purge
$ module load a b
$ module list
Currently Loaded Modulefiles:
 1) a   2) b

Consistency of the declared conflict is now ensured to satisfy this constraint even after the load of the modulefile declaring it. This is achieved by keeping track of the conflict constraints of the loaded modulefiles in an environment variable called MODULES_LMCONFLICT:

$ module load a b
ERROR: WARNING: b cannot be loaded due to a conflict.
HINT: Might try "module unload a" first.
$ module list
Currently Loaded Modulefiles:
 1) a

An environment variable is used to keep track of this conflict information to proceed the same way than used to keep track of the loaded modulefiles with the LOADEDMODULES environment variable.

In case a conflict constraint toward a modulefile is set by an already loaded modulefile, loading the conflicting modulefile will lead to a load evaluation attempt in order for this modulefile to get the chance to solve the constraint violation. If at the end of the load evaluation, the conflict has not been solved, modulefile load will be discarded.

Warning

On versions 4.2.0 and 4.2.1, a conflict constraint set by an already loaded modulefile forbade the load of the conflicting modulefile. This has been changed starting version 4.2.2 to better cope with behaviors of previous Modules version: an evaluation attempt of the conflicting modulefile is made to give it the opportunity to solve this conflict by using module unload modulefile command.

Modulefile prereq constraints consistency

With the prereq modulefile command, a given modulefile can list the other modulefiles it pre-requires. To load this modulefile, the modulefiles it pre-requires must be loaded prior its own load.

This constraint was until now satisfied when loading the modulefile declaring the prereq but, as for the declared conflict, it vanished as soon as this modulefile was loaded. In the following example c modulefile declares a prereq on a:

$ module load c
WARNING: c cannot be loaded due to missing prereq.
HINT: the following module must be loaded first: a
$ module list
No Modulefiles Currently Loaded.
$ module load a c
$ module list
Currently Loaded Modulefiles:
 1) a   2) c
$ module unload a
$ module list
Currently Loaded Modulefiles:
 1) c

Consistency of the declared prereq is now ensured to satisfy this constraint even after the load of the modulefile declaring it. This is achieved, like for the conflict consistency, by keeping track of the prereq constraints of the loaded modulefiles in an environment variable called MODULES_LMPREREQ:

$ module load a c
$ module list
Currently Loaded Modulefiles:
 1) a   2) c
$ module unload a
ERROR: WARNING: a cannot be unloaded due to a prereq.
HINT: Might try "module unload c" first.
$ module list
Currently Loaded Modulefiles:
 1) a   2) c
By-passing module defined constraints

The ability to by-pass a conflict or a prereq constraint defined by modulefiles is introduced with the --force command line switch (-f for short notation) for the load, unload and switch sub-commands.

With this new command line switch, a given modulefile is loaded even if it conflicts with other loaded modulefiles or even if the modulefiles it pre-requires are not loaded. Some example reusing the same modulefiles a, b and c than above:

$ module load b
$ module load --force a
WARNING: a conflicts with b
$ module list
Currently Loaded Modulefiles:
 1) b   2) a
$ module purge
$ module load --force c
WARNING: c requires a loaded
$ module list
Currently Loaded Modulefiles:
 1) c

--force also enables to unload a modulefile required by another loaded modulefiles:

$ module load a c
$ module list
Currently Loaded Modulefiles:
 1) a   2) c
$ module unload --force a
WARNING: a is required by c
$ module list
Currently Loaded Modulefiles:
 1) c

In a situation where some of the loaded modulefiles have unsatisfied constraints corresponding to the prereq and conflict they declare, the save and reload sub-commands do not perform and return an error.

Automated module handling mode

An automatic management of the dependencies between modulefiles has been added and it is called automated module handling mode. This new mode consists in additional actions triggered when loading or unloading a modulefile to satisfy the constraints it declares.

When loading a modulefile, following actions are triggered:

  • Requirement Load (ReqLo): load of the modulefiles declared as a prereq of the loading modulefile.
  • Dependent Reload (DepRe): reload of the modulefiles declaring a prereq onto loaded modulefile or declaring a prereq onto a modulefile part of this reloading batch.

When unloading a modulefile, following actions are triggered:

  • Dependent Unload (DepUn): unload of the modulefiles declaring a non-optional prereq onto unloaded modulefile or declaring a non-optional prereq onto a modulefile part of this unloading batch. A prereq modulefile is considered optional if the prereq definition order is made of multiple modulefiles and at least one alternative modulefile is loaded.
  • Useless Requirement Unload (UReqUn): unload of the prereq modulefiles that have been automatically loaded for either the unloaded modulefile, an unloaded dependent modulefile or a modulefile part of this useless requirement unloading batch. Modulefiles are added to this unloading batch only if they are not required by any other loaded modulefiles. MODULES_LMNOTUASKED environment variable helps to keep track of these automatically loaded modulefiles and to distinguish them from modulefiles asked by user.
  • Dependent Reload (DepRe): reload of the modulefiles declaring a conflict or an optional prereq onto either the unloaded modulefile, an unloaded dependent or an unloaded useless requirement or declaring a prereq onto a modulefile part of this reloading batch.

In case a loaded modulefile has some of its declared constraints unsatisfied (pre-required modulefile not loaded or conflicting modulefile loaded for instance), this loaded modulefile is excluded from the automatic reload actions described above.

For the specific case of the switch sub-command, where a modulefile is unloaded to then load another modulefile. Dependent modulefiles to Unload are merged into the Dependent modulefiles to Reload that are reloaded after the load of the switched-to modulefile.

This automated module handling mode integrates concepts (like the Dependent Reload mechanism) of the Flavours extension, which was designed for Modules compatibility version. As a whole, automated module handling mode can be seen as a generalization and as an expansion of the Flavours concepts.

This new feature can be controlled at build time with the --enable-auto-handling configure option. This default configuration can be superseded at run-time with the MODULES_AUTO_HANDLING environment variable or the command line switches --auto and --no-auto.

By default, automated module handling mode is disabled and will stay so until the next major release version (5.0) where it will be enabled by default. This new feature is currently considered experimental and the set of triggered actions will be refined over the next feature releases.

Consistency of module load/unload commands in modulefile

With the module load modulefile command, a given modulefile can automatically load a modulefile it pre-requires. Similarly with the module unload modulefile command, a given modulefile can automatically unload a modulefile it conflicts with.

Both commands imply additional actions on the loaded environment (loading or unloading extra modulefiles) that should cope with the constraints defined by the loaded environment.

Additionally module load and module unload modulefile commands express themselves constraints on loaded environment that should stay satisfied to ensure consistency.

To ensure the consistency of module load modulefile command once the modulefile defining it has been loaded, this command is assimilated to a prereq command. Thus the defined constraint is recorded in the MODULES_LMPREREQ environment variable. Same approach is used for module unload modulefile command which is assimilated to a conflict command. Thus the defined constraint is recorded in the MODULES_LMCONFLICT environment variable.

To ensure the consistency of the loaded environment, the additional actions of the module load and module unload modulefile commands have been adapted in particular situations:

  • When unloading modulefile, module load command will unload the modulefile it targets only if no other loaded modulefile requires it and if this target has not been explicitly loaded by user.
  • When unloading modulefile, module unload command does nothing as the relative conflict registered at load time ensure environment consistency and will forbid conflicting modulefile load.

Please note that loading and unloading results may differ than from previous Modules version now that consistency is checked:

  • Modulefile targeted by a module load modulefile command may not be able to load due to a registered conflict in the currently loaded environment. Which in turn will break the load of the modulefile declaring the module load command.
  • Modulefile targeted by a module unload modulefile command may not be able to unload due to a registered prereq in the loaded environment. Which in turn will break the load of the modulefile declaring the module unload command.
  • If automated module handling mode is enabled, module load modulefile command is interpreted when unloading modulefile as part of the Useless Requirement Unload (UReqUn) mechanism not through modulefile evaluation. As a consequence, an error occurring when unloading the modulefile targeted by the module load command does not break the unload of the modulefile declaring this command. Moreover unload of the module load targets is done in the reverse loaded order, not in the module load command definition order.
Modulefile alias and symbolic modulefile name consistency

With the module-alias and module-version modulefile commands, alternative names can be given to a modulefile. When these names are used to load for instance a modulefile, they are resolved to the modulefile they target which is then processed for the load action.

Until now, the alias and symbolic version names were correctly resolved for the load and unload actions and also for the querying sub-commands (like avail or whatis). However this alternative name information vanishes once the modulefile it resolves to is loaded. As a consequence there was no consistency over these alternative designations. In the following example f modulefile declares a conflict on e alias which resolves to d modulefile:

$ module load e
$ module list
Currently Loaded Modulefiles:
 1) d
$ module info-loaded e
$ module load f
$ module list
Currently Loaded Modulefiles:
 1) d   2) f

Consistency of the alternative names set on a modulefile with module-alias and module-version commands is now ensured to enable modulefile commands prereq, conflict, is-loaded and module-info loaded using these alternative designations as argument. This consistency is achieved, like for the conflict and prereq consistencies, by keeping track of the alternative names of the loaded modulefiles in an environment variable called MODULES_LMALTNAME:

$ module load e
$ module list
Currently Loaded Modulefiles:
 1) d
$ module info-loaded e
d
$ module load f
WARNING: f cannot be loaded due to a conflict.
HINT: Might try "module unload e" first.
$ module list
Currently Loaded Modulefiles:
 1) d
Environment variable change through modulefile evaluation context

All environment variable edition commands (setenv, unsetenv, append-path, prepend-path and remove-path) have been updated to:

  • Reflect environment variable value change on the environment of the current modulefile Tcl interpreter. So using $env(VAR) will return the currently defined value for environment variable VAR, not the one found prior modulefile evaluation.
  • Clear environment variable content instead of unsetting it on the environment of the current modulefile Tcl interpreter to avoid raising error about accessing an undefined element in $env(). Code is still produced to purely unset environment variable in shell environment.

Exception is made for the whatis evaluation mode: environment variables targeted by variable edition commands are not set to the defined value in the evaluation context during this whatis evaluation. These variables are only initialized to an empty value if undefined. This exception is made to save performances on this global evaluation mode.

Improved module message report

Module sub-commands like load, unload or switch, may perform multiple load or unload modulefile evaluations in a row. Also these kind of evaluation modes may sometimes trigger additional load or unload evaluations, when for instance a modulefile contains a module load command.

To improve the readability of the module messages produced relatively to a load or an unload evaluation, these messages are now stacked under a Loading or an Unloading message block that gathers all the messages produced for a given modulefile evaluation:

$ module load --no-auto foo
Loading foo/1.2
  ERROR: foo/1.2 cannot be loaded due to missing prereq.
    HINT: the following module must be loaded first: bar/4.5

In addition, foreground load, unload, switch and restore actions (ie. asked on the command-line) now report a summary of the additional load and unload evaluations that were eventually triggered in the process:

$ module load --auto foo
Loading foo/1.2
  Loading requirement: bar/4.5
New modulefile commands

2 new modulefile Tcl commands have been introduced:

  • set-function: define a shell function on sh-kind and fish shells.
  • unset-function: unset a shell function on sh-kind and fish shells.

From v4.0 to v4.1

This new version is backward-compatible with v4.0 and primarily fixes bugs and adds new features. Version 4.1 introduces new functionalities that are described in this section. See the 4.1 release notes for a complete list of the changes between Modules v4.0 and v4.1.

Virtual modules

A virtual module stands for a module name associated to a modulefile. The modulefile is the script interpreted when loading or unloading the virtual module which appears or can be found with its virtual name.

The module-virtual modulefile command is introduced to give the ability to define these virtual modules. This new command takes a module name as first argument and a modulefile location as second argument:

module-virtual app/1.2.3 /path/to/virtualmod/app

With this feature it is now possible to dynamically define modulefiles depending on the context.

Extend module command with site-specific Tcl code

module command can now be extended with site-specific Tcl code. modulecmd.tcl now looks at a siteconfig.tcl file in an etcdir defined at configure time (by default $prefix/etc). If it finds this Tcl script file, it is sourced within modulecmd.tcl at the beginning of the main procedure code.

siteconfig.tcl enables to supersede any global variable or procedure definitions made in modulecmd.tcl with site-specific code. A module sub-command can for instance be redefined to make it fit local needs without having to touch the main modulecmd.tcl.

Quarantine mechanism to protect module execution

To protect the module command run-time environment from side effect coming from the current environment definition a quarantine mechanism is introduced. This mechanism, sets within module function definition and shell initialization script, modifies the modulecmd.tcl run-time environment to sanitize it.

The mechanism is piloted by environment variables. First of all MODULES_RUN_QUARANTINE, a space-separated list of environment variable names. Every variable found in MODULES_RUN_QUARANTINE will be set in quarantine during the modulecmd.tcl run-time. Their value will be set empty or set to the value of the corresponding MODULES_RUNENV_<VAR> environment variable if defined. Once modulecmd.tcl is started it restores quarantine variables to their original values.

MODULES_RUN_QUARANTINE and MODULES_RUNENV_<VAR> environment variables can be defined at build time by using the following configure option:

--with-quarantine-vars='VARNAME[=VALUE] ...'

Quarantine mechanism is available for all supported shells except csh and tcsh.

Pager support

The informational messages Modules sends on the stderr channel may sometimes be quite long. This is especially the case for the avail sub-command when hundreds of modulefiles are handled. To improve the readability of those messages, stderr output can now be piped into a paging command.

This new feature can be controlled at build time with the --with-pager and --with-pager-opts configure options. Default pager command is set to less and its relative options are by default -eFKRX. Default configuration can be supersedes at run-time with MODULES_PAGER environment variables or command-line switches (--no-pager, --paginate).

Warning

On version 4.1.0, the PAGER environment variable was taken in consideration to supersede pager configuration at run-time. Since version 4.1.1, PAGER environment variable is ignored to avoid side effects coming from the system general pager configuration.

Module function to return value in scripting languages

On Tcl, Perl, Python, Ruby, CMake and R scripting shells, module function was not returning value and until now an occurred error led to raising a fatal exception.

To make module function more friendly to use on these scripting shells it now returns a value. False in case of error, true if everything goes well.

As a consequence, returned value of a module sub-command can be checked. For instance in Python:

if module('load', 'foo'):
  # success
else:
  # failure
New modulefile commands

4 new modulefile Tcl commands have been introduced:

  • is-saved: returns true or false whether a collection, corresponding to currently set collection target, exists or not.
  • is-used: returns true or false whether a given directory is currently enabled in MODULEPATH.
  • is-avail: returns true or false whether a given modulefile exists in currently enabled module paths.
  • module-info loaded: returns the exact name of the modulefile currently loaded corresponding to the name argument.

Multiple collections, paths or modulefiles can be passed respectively to is-saved, is-used and is-avail in which case true is returned if at least one argument matches condition (acts as a OR boolean operation). No argument may be passed to is-loaded, is-saved and is-used commands to return if anything is respectively loaded, saved or used.

If no loaded modulefile matches the module-info loaded query, an empty string is returned.

New module sub-commands

Modulefile-specific commands are sometimes wished to be used outside of a modulefile context. Especially for the commands managing path variables or commands querying current environment context. So the following modulefile-specific commands have been made reachable as module sub-commands with same arguments and properties as if called from within a modulefile:

  • append-path
  • prepend-path
  • remove-path
  • is-loaded
  • info-loaded

The is-loaded sub-command returns a boolean value. Small Python example:

if module('is-loaded', 'app'):
  print 'app is loaded'
else:
  print 'app not loaded'

info-loaded returns a string value and is the sub-command counterpart of the module-info loaded modulefile command:

$ module load app/0.8
$ module info-loaded app
app/0.8

From v3.2 to v4.0

Major evolution occurs with this v4.0 release as the traditional module command implemented in C is replaced by the native Tcl version. This full Tcl rewrite of the Modules package was started in 2002 and has now reached maturity to take over the binary version. This flavor change enables to refine and push forward the module concept.

This document provides an outlook of what is changing when migrating from v3.2 to v4.0 by first describing the introduced new features. Both v3.2 and v4.0 are quite similar and transition to the new major version should be smooth. Slights differences may be noticed in a few use-cases. So the second part of the document will help to learn about them by listing the features that have been discontinued in this new major release or the features where a behavior change can be noticed.

New features

On its overall this major release brings a lot more robustness to the module command with now more than 4000 non-regression tests crafted to ensure correct operations over the time. This version 4.0 also comes with fair amount of improved functionalities. The major new features are described in this section.

Additional shells supported

Modules v4 introduces support for fish, lisp, tcl and R code output.

Non-zero exit code in case of error

All module sub-commands will now return a non-zero exit code in case of error whereas Modules v3.2 always returned zero exit code even if issue occurred.

Output redirect

Traditionally the module command output text that should be seen by the user on stderr since shell commands are output to stdout to change shell's environment. Now on sh, bash, ksh, zsh and fish shells, output text is redirected to stdout after shell command evaluation if shell is in interactive mode.

Filtering avail output

Results obtained from the avail sub-command can now be filtered to only get the default version of each module name with use of the --default or -d command line switch. Default version is either the explicitly set default version or the highest numerically sorted modulefile or module alias if no default version set.

It is also possible to filter results to only get the highest numerically sorted version of each module name with use of the --latest or -L command line switch.

Extended support for module alias and symbolic version

Module aliases are now included in the result of the avail, whatis and apropos sub-commands. They are displayed in the module path section where they are defined or in a global/user modulerc section for aliases set in user's or global modulerc file. A @ symbol is added in parenthesis next to their name to distinguish them from modulefiles.

Search may be performed with an alias or a symbolic version-name passed as argument on avail, whatis and apropos sub-commands.

Modules v4 resolves module alias or symbolic version passed to unload command to then remove the loaded modulefile pointed by the mentioned alias or symbolic version.

A symbolic version sets on a module alias is now propagated toward the resolution path to also apply to the relative modulefile if it still correspond to the same module name.

Hiding modulefiles

Visibility of modulefiles can be adapted by use of file mode bits or file ownership. If a modulefile should only be used by a given subset of persons, its mode an ownership can be tailored to provide read rights to this group of people only. In this situation, module only reports the modulefile, during an avail command for instance, if this modulefile can be read by the current user.

These hidden modulefiles are simply ignored when walking through the modulepath content. Access issues (permission denied) occur only when trying to access directly a hidden modulefile or when accessing a symbol or an alias targeting a hidden modulefile.

Improved modulefiles location

When looking for an implicit default in a modulefile directory, aliases are now taken into account in addition to modulefiles and directories to determine the highest numerically sorted element.

Modules v4 resolves module alias or symbolic version when it points to a modulefile located in another modulepath.

Access issues (permission denied) are now distinguished from find issues (cannot locate) when trying to access directly a directory or a modulefile as done on load, display or whatis commands. In addition, on this kind of access not readable .modulerc or .version files are ignored rather producing a missing magic cookie error.

Module collection

Modules v4 introduces support for module collections. Collections describe a sequence of module use then module load commands that are interpreted by Modules to set the user environment as described by this sequence. When a collection is activated, with the restore sub-command, modulepaths and loaded modules are unused or unloaded if they are not part or if they are not ordered the same way as in the collection.

Collections are generated by the save sub-command that dumps the current user environment state in terms of modulepaths and loaded modules. By default collections are saved under the $HOME/.module directory. Collections can be listed with savelist sub-command, displayed with saveshow and removed with saverm.

Collections may be valid for a given target if they are suffixed. In this case these collections can only be restored if their suffix correspond to the current value of the MODULES_COLLECTION_TARGET environment variable. Saving collection registers the target footprint by suffixing the collection filename with .$MODULES_COLLECTION_TARGET.

Path variable element counter

Modules 4 provides path element counting feature which increases a reference counter each time a given path entry is added to a given path-like environment variable. As consequence a path entry element is removed from a path-like variable only if the related element counter is equal to 1. If this counter is greater than 1, path element is kept in variable and reference counter is decreased by 1.

This feature allows shared usage of particular path elements. For instance, modulefiles can append /usr/local/bin to PATH, which is not unloaded until all the modulefiles that loaded it unload too.

Optimized I/O operations

Substantial work has been done to reduce the number of I/O operations done during global modulefile analysis commands like avail or whatis. stat, open, read and close I/O operations have been cut down to the minimum required when walking through the modulepath directories to check if files are modulefiles or to resolve module aliases.

Interpretation of modulefiles and modulerc are handled by the minimum required Tcl interpreters. Which means a configured Tcl interpreter is reused as much as possible between each modulefile interpretation or between each modulerc interpretation.

Sourcing modulefiles

Modules 4 introduces the possibility to source a modulefile rather loading it. When it is sourced, a modulefile is interpreted into the shell environment but then it is not marked loaded in shell environment which differ from load sub-command.

This functionality is used in shell initialization scripts once module function is defined. There the etc/modulerc modulefile is sourced to setup the initial state of the environment, composed of module use and module load commands.

Removed features and substantial behavior changes

Following sections provide list of Modules v3.2 features that are discontinued on Modules v4 or features with a substantial behavior change that should be taken in consideration when migrating to v4.

Package initialization

MODULESBEGINENV environment snapshot functionality is not supported anymore on Modules v4. Modules collection mechanism should be used instead to save and restore sets of enabled modulepaths and loaded modulefiles.

Command line switches

Some command line switches are not supported anymore on v4.0. When still using them, a warning message is displayed and the command is ran with these unsupported switches ignored. Following command line switches are concerned:

  • --force, -f
  • --human
  • --verbose, -v
  • --silent, -s
  • --create, -c
  • --icase, -i
  • --userlvl lvl, -u lvl
Module sub-commands

During an help sub-command, Modules v4 does not redirect output made on stdout in ModulesHelp Tcl procedure to stderr. Moreover when running help, version 4 interprets all the content of the modulefile, then call the ModulesHelp procedure if it exists, whereas Modules 3.2 only interprets the ModulesHelp procedure and not the rest of the modulefile content.

When load is asked on an already loaded modulefiles, Modules v4 ignores this new load order whereas v3.2 refreshed shell alias definitions found in this modulefile.

When switching on version 4 an old modulefile by a new one, no error is raised if old modulefile is not currently loaded. In this situation v3.2 threw an error and abort switch action. Additionally on switch sub-command, new modulefile does not keep the position held by old modulefile in loaded modules list on Modules v4 as it was the case on v3.2. Same goes for path-like environment variables: replaced path component is appended to the end or prepended to the beginning of the relative path-like variable, not appended or prepended relatively to the position hold by the swapped path component.

During a switch command, version 4 interprets the swapped-out modulefile in unload mode, so the sub-modulefiles loaded, with module load order in the swapped-out modulefile are also unloaded during the switch.

Modules 4 provides path element counting feature which increases a reference counter each time a given path entry is added to a given environment variable. This feature also applies to the MODULEPATH environment variable. As consequence a modulepath entry element is removed from the modulepath enabled list only if the related element counter is equal to 1. When unusing a modulepath if its reference counter is greater than 1, modulepath is kept enabled and reference counter is decreased by 1.

On Modules 3.2 paths composing the MODULEPATH environment variable may contain reference to environment variable. These variable references are resolved dynamically when MODULEPATH is looked at during module sub-command action. This feature has been discontinued on Modules v4.

Following Modules sub-commands are not supported anymore on v4.0:

  • clear
  • update
Modules specific Tcl commands

Modules v4 provides path element counting feature which increases a reference counter each time a given path entry is added to a given environment variable. As a consequence a path entry element is not always removed from a path-like variable when calling to remove-path or calling to append-path or append-path at unloading time. The path element is removed only if its related element counter is equal to 1. If this counter is greater than 1, path element is kept in variable and reference counter is decreased by 1.

On Modules v4, module-info mode returns during an unload sub-command the unload value instead of remove on Modules v3.2. However if mode is tested against remove value, true will be returned. During a switch sub-command on Modules v4, unload then load is returned instead of switch1 then switch2 then switch3 on Modules v3.2. However if mode is tested against switch value, true will be returned.

When using set-alias, Modules v3.2 defines a shell function when variables are in use in alias value on Bourne shell derivatives, Modules 4 always defines a shell alias never a shell function.

Some Modules specific Tcl commands are not supported anymore on v4.0. When still using them, a warning message is displayed and these unsupported Tcl commands are ignored. Following Modules specific Tcl commands are concerned:

  • module-info flags
  • module-info trace
  • module-info tracepat
  • module-info user
  • module-log
  • module-trace
  • module-user
  • module-verbosity
Further reading

To get a complete list of the differences between Modules v3.2 and v4, please read the Differences between versions 3.2 and 4 document.

A significant number of issues reported for v3.2 have been closed on v4. List of these closed issues can be found at:

https://github.com/cea-hpc/modules/milestone/1?closed=1

Release notes

This file describes changes in recent versions of Modules. It primarily documents those changes that are of interest to users and admins.

Modules 4.8.0 (2021-07-14)

  • Introduce the edit sub-command that opens modulefile passed as argument in a text editor. Modulefile can be specified like with any other sub-command, leveraging defined symbolic versions, aliases or using advanced version specifiers.
  • Add the editor configuration option to select the text editor to use with edit sub-command. When this option is set through the config sub-command, the MODULES_EDITOR environment variable is set. The --with-editor installation option controls the default value of editor configuration option. If not set at installation time, vi is set as default editor.
  • Default value of editor configuration option is overridden by the VISUAL or the EDITOR environment variables, which are both in turn overridden by the MODULES_EDITOR environment variable.
  • Doc: fix modulecmd.tcl internal state check in recipes example codes. (fix issue #396)
  • The Advanced module version specifiers mechanism now allows the use of version range in version list (for instance mod@:1.2,1.4:1.6,1.8:). Such specification helps to exclude specific versions. (fix issue #397)
  • Install: fix installation scripts to allow building Modules when its repository is set as a git submodule. (fix issue #398)
  • Doc: demonstrate in the Source shell script in modulefile recipe how to use the source-sh command when software provide a specific initialization script for each shell it supports. (fix issue #399)
  • When defining a shell function with the set-function modulefile command, only export this function when using the Bash shell (using the export -f shell command) to make it available in sub-shell contexts. Shell function export is not supported on other kind of sh shell (sh, ksh and zsh). (fix issue #401)
  • Doc: add Variants design notes.
  • Add the variant modulefile command that enables to pass down arguments, specified when designating the module to evaluate, within modulefile evaluation context. This command defines a variant name and a list of allowed values. When evaluated, variant instantiates an element in the ModuleVariant array whose name equals variant name and value is set with value specified for variant when module is designated. If specified value does not correspond to an allowed value or if no value is specified for variant an error is raised.
  • Enhance the Advanced module version specifiers to handle variant specification following Spack's syntax (e.g., name=value). When the advanced_version_spec configuration is enabled, variant could be specified anywhere a module can be specified.
  • Add the --default option to the variant modulefile command to indicate the default value of the variant to apply when the designation of the evaluating module does not mention this variant.
  • Add the --boolean option to the variant modulefile command to indicate that the variant defined is of the Boolean type, thus no list of accepted value is expected.
  • Enhance the Advanced module version specifiers to handle Boolean variant specification following Spack's syntax (e.g., +name, ~name and -name). The -name syntax is not supported on ml command as the minus sign already means to unload designated module.
  • Accept any minus argument (-word) set after the sub-command name when the advanced_version_spec configuration is enabled and if sub-command accepts Advanced module version specifiers (like load or unload sub-commands). A false value may be set to Boolean variant this way.
  • Add the variant_shortcut configuration option to define shortcut characters that could be used to specify and report module variants. Default value for this option could be set at installation time with the --with-variant-shortcut option. No variant shortcut is defined by default. This value could be superseded by setting up the variant_shortcut option with config sub-command. Which sets the MODULES_VARIANT_SHORTCUT environment variable.
  • Enhance the Advanced module version specifiers to handle variant shortcut specification (e.g., <shortcut>value).
  • Record in user loaded environment, with MODULES_LMVARIANT environment variable, the value specified for the variants defined in the loaded modulefiles and their properties (if it is a Boolean variant and if the value set is the default one).
  • Add the variant element in the allowed value list of the list_output and list_terse_output configuration options. Set this new element in the default value list of the list_output option. When set, the variant defined for loaded modules are reported on module list command output.
  • Add the va color key in default light and dark color palettes to graphically enhance the report of variant value.
  • Update the key section to explain on list sub-command output the reported variant elements (name=value, +name, -name or <shortcut>value)
  • Record variant specification of loaded modules when saving collections and reload specified variants when restoring these collections.
  • When collection_pin_version configuration is disabled, only record in collections the variants whose value is not the default one.
  • Update module designation in error, warning or informational messages to report variant specification enclosed in curly braces ({}), enclose module name and version or variant specification in single quotes ('') if they contain a space character and highlight the module designation in report message if configured.
  • Introduce the getvariant modulefile command to query for currently evaluating module the value of a given variant name.
  • When translating the @loaded version specifier also retrieve the variant specified for corresponding loaded module.
  • Update hide, forbid and tag mechanisms to apply them only if they match selected module variant.
  • Any variant defined in module specification passed as argument to search sub-commands (avail, whatis, is-avail, path and paths) is ignored.
  • Raise an error if a variant named version is declared in a modulefile to let room for the future implementation of this specific variant.
  • Doc: describe in the Differences between versions 3.2 and 4 document argument handling change on setenv since v3.2. (fix issue #402)
  • Introduce the try-load sub-command which like load sub-command tries to load the modulefile passed as argument, but does not complain if this modulefile cannot be found. (fix issue #392)
  • Init: fix stderr redirection in fish shell initialization script, now that use of the ^ character to redirect stderr is disabled by default (fish >=3.3).
  • Protect quarantine mechanism code from rcexpandparam Zsh option when initializing the module command on this shell. (fix issue #403)

Modules 4.7.1 (2021-04-06)

  • Doc: clarify the license terms used by the project. (fix issue #389)
  • Align all files from the Modules project under the GPLv2+ license. Scripts and libraries that were previously licensed with GPLv3+ have been moved to GPLv2+ with the consent of their respective copyright holders. (fix issue #389)
  • Revert "Install: have configure script assume the . dot directory when invoked without the prepended ./" as consent was not obtained from author to relicense the contribution to GPLv2+.
  • Doc: fixes few typos in module and modulefile.
  • Update the sh-to-mod mechanism to support version 3.2 of the fish shell. Fish 3.2 introduces the . builtin command that should be regexp-escaped when determining the shell functions or aliases defined by the script analyzed by sh-to-mod.
  • Vim: update addon files to highlight modulefile variables ModuleTool, ModuleToolVersion and ModulesCurrentModulefile.
  • Doc: update the description and default value of the --with-dark-background-colors and --with-light-background-colors installation options.
  • Doc: add description of changes that occurred on versions 4.6 and 4.7 for the --with-dark-background-colors and --with-light-background-colors installation options and for the MODULES_COLORS environment variable.
  • Doc: correct the default value of the --with-tag-abbrev installation option.
  • Doc: add Sticky modules cookbook recipe.

Modules 4.7.0 (2021-02-19)

  • Doc: simplify TOC of MIGRATING document
  • Add the ModuleTool and ModuleToolVersion Modules variables to determine during modulefile or modulerc evaluation the name and version of the module implementation currently in use.
  • Introduce the versioncmp modulefile command to compare two version strings passed as argument.
  • Enable the use of wildcard character to designate multiple directories at once in modulespath configuration file. (fix issue #125)
  • Distinguish aliases from symbolic versions in MODULES_LMALTNAME environment variable. Prefix these alias entries with the al| string.
  • Fetch modulefile modification time only if required by list sub-command display format.
  • Use symbolic versions recorded in environment, with MODULES_LMALTNAME variable, to report the symbols applying to loaded modules on list sub-command. Modulerc files are not evaluated anymore when performing a module list.
  • Move the definition of the FPATH environment variable for Modules initialization on ksh shell from the initialization script of this shell to the resulting output of the autoinit sub-command.
  • Introduce the shells_with_ksh_fpath configuration option to define a list of shell where to ensure that any ksh sub-shell will get the module function defined by use of the FPATH environment variable. When the shells_with_ksh_fpath option is set through the config sub-command, the MODULES_SHELLS_WITH_KSH_FPATH environment variable is set. Accepted values are a list of shell among sh, bash, csh, tcsh and fish separated by colon character (:).
  • Add the implicit_requirement configuration option to control whether a prereq or a conflict requirement should be implicitly set onto modules respectively specified on module load or module unload commands in modulefile. Default value for this option could be set at configure time with the --enable-implicit-requirement option (enabled by default). This value could be superseded by setting up the implicit_requirement option with config sub-command. Which sets the MODULES_IMPLICIT_REQUIREMENT environment variable. (fix issue #260)
  • Add the --not-req option to the module modulefile command to inhibit for its load and unload sub-commands the definition of a prereq or conflict requirement onto specified modules.
  • Add the lpopState and currentState procedures to respectively remove or return the last entry from the list of values of a given state.
  • Add the topState and depthState procedures to respectively return the first element from or the number of elements in the list of values of a given state.
  • Remove the pre-definition of runtime states with no specific property. These basic states are defined on-the-fly which implied they are not reported on a module config --dump-state command unless if instanciated.
  • Introduce the loaded symbolic version among advanced version specifiers (e.g. foo@loaded) to designate the currently loaded version of specified module. (fix issue #366)
  • Doc: add Module tags design notes.
  • Report tags applying to the modules returned by the avail sub-command. Adapt the regular, terse and JSON output styles to report these tags along the module they are attached to (enclosed in <>). Reported tags currently are states that apply to modules: auto-loaded, forbidden, hidden, loaded, nearly-forbidden, sticky and super-sticky.
  • Record tags applying to each loaded module in the MODULES_LMTAG environment variable to make this information persist after module being loaded.
  • Report tags applying to the loaded modules returned by the list sub-command. Adapt the regular and JSON output styles to report these tags along the module they are attached to (enclosed in <>). Reported tags currently are states applying to loaded modules: auto-loaded, hidden-loaded, nearly-forbidden, sticky and super-sticky.
  • Introduce the module-info tags modulefile command to query the tags that apply to the currently evaluated modulefile.
  • Add the module-tag modulefile command to associate tag to designated modulefile. Those tags are reported on avail and list sub-commands along the module they are attached to. module-tag supports the advanced module version specifier syntax.
  • Add the tag_abbrev configuration option to define abbreviated strings for module tags and use these abbreviations instead of tag names when reporting tags on avail and list command results. Default value for this option could be set at configure time with the --with-tag-abbrev option. By default the following abbreviations are set: aL for auto-loaded, F for forbidden, H for hidden, H for hidden-loaded, L for loaded, nF for nearly-forbidden, S for sticky, sS for super-sticky. This value could be superseded by setting up the tag_abbrev option with config sub-command. Which sets the MODULES_TAG_ABBREV environment variable.
  • A Select Graphic Rendition (SGR) code can be associated to module tag names or abbreviation strings in the color palette to graphically render these tags over the module name they are associated to. The default light and dark color palettes have been updated to set a color code for all basic module tags. When a color code is set for a tag, it is then graphically rendered over the module names and not reported along module name by its tag name or abbreviation. When multiple colored tags apply to a given module, each tag is graphically rendered over a sub-part of the module name.
  • Add the tag_color_name configuration option to designate module tags whose graphical rendering should be applied to their own name or abbreviation rather than over the module name they are attached to. Default value for this option could be set at configure time with the --with-tag-color-name option (empty by default). This value could be superseded by setting up the tag_color_name option with config sub-command. Which sets the MODULES_TAG_COLOR_NAME environment variable.
  • Add the --hidden-loaded option to the module-hide modulefile command that indicates module should be hidden once loaded. When set, the hidden-loaded module tag applies to module specification set on module-hide command.
  • Do not report on list sub-command results the loaded modules associated with the hidden-loaded tag, unless if the --all option is set.
  • Doc: add an hidden-loaded example in the Hide and forbid modules cookbook recipe.
  • Introduce the verbose2 verbosity level between verbose and trace levels. Verbose2 mode can be enabled by setting the verbosity config to the verbose2 value or by using the -v command-line switch twice.
  • Do not report the load, unload or switch of modules set hidden-loaded if these modules have been loaded, unloaded or switched automatically. Unless the verbosity mode is set to verbose2 or any higher level or if any specific messages have to be reported for these module evaluations.
  • Report when trying to load a module which is already loaded or when trying to unload a module which is not loaded in case the verbosity mode is set to verbose2 or any higher level. (fix issue #187)
  • Doc: improve readability of version 4 improvements in Differences between versions 3.2 and 4 document.
  • Introduce stickyness: module tagged sticky with module-tag command cannot be unloaded unless if the unload is forced or if the module is reloaded. (fix issue #269)
  • Introduce super-stickyness: module tagged super-sticky with module-tag command cannot be unloaded even if the unload is forced unless if the module is reloaded. (fix issue #269)
  • Allow swap of sticky or super-sticky modules by another modulefile version if stickyness definition applies to module parent name. E.g., foo/1.0 can be swapped by foo/2.0 if sticky tag applies to foo.
  • When forcing purge with a purge --force sub-command, also unload the modules that are depended by unloadable modules.
  • Doc: improve readability of Modules installation configuration in Installing Modules on Unix document and enable hypertext reference to these elements.
  • Doc: improve readability of module command configuration option in module document and enable hypertext reference to these elements.
  • Doc: describe in HTML documentation when installation options, module command configuration options and options of modulefile command or module sub-command were introduced.
  • Doc: update HTML documentation Table Of Content.
  • Doc: improve markup of module sub-commands, modulefile commands, installation option, module configuration option across documentation.
  • Doc: colorize terminal output examples in MIGRATING document.
  • Abort modulefile read if first file content chunk does not start with the #%Module magic cookie. (fix issue #375)
  • Install: add installation option --enable-new-features that enables all at once the installation options that are disabled by default due to the substantial behavior changes they imply.
  • Add a Key section at the end of avail and list sub-commands to explain the meaning of graphical renditions or of elements set in parentheses or chevrons along module name.
  • Fix output of avail and list sub-commands on very small termminal width. (fix issue #378)
  • Add mcookie_version_check configuration to define if version set in modulefile magic cookie should be checked against module current version to determine if modulefile can be evaluated. The new configuration, which is enabled by default, can be set at installation time with configure option --enable-mcookie-version-check or can be superseded later on with the MODULES_MCOOKIE_VERSION_CHECK environment variable. (fix issue #377)
  • Fix output of modulefile evaluation error stack trace on very small terminal width. (fix issues #379 and #381)
  • Correct config sub-command to set nearly_forbidden_days configuration. (fix issue #380)
  • Init: reduce usage of helper variables in bash_completion and tcsh_completion that are showing up in the output of the shell's set command. (fix issue #382 with contribution from Colin Marquardt)
  • Consider modulepath starting with a reference to an environment variable as absolute. (fix issue #376)
  • Consider the module load performed in the user or the global RC file like load commands issued from initialization RC file. (fix issue #372)
  • Install: have configure script assume the . dot directory when invoked without the prepended ./. (contribution from R.K. Owen)
  • Install: disable the Makefile rules to build the HTML documentation in case if the documentation is found pre-built in the dist archive.
  • Install: do not flag documentation as pre-built if configure script is ran another time after building docs.
  • Restrict the value accepted by nearly_forbidden_days configuration and --with-nearly-forbidden-days installation option to integers comprised between 0 and 365.
  • Install: color ERROR and WARNING message headers produced by file:configure script if output is sent to a terminal.
  • Install: split error messages produced by file:configure script over an additional line when too long.
  • Doc: add Output configuration design notes.
  • Introduce the avail_output and avail_terse_output configuration options to define the content to report in addition to the available module names respectively for avail sub-command regular and terse output modes. Excepted value for these configuration options is a colon separated list of elements to report. Default value is modulepath:alias:dirwsym:sym:tag:key for avail_output and modulepath:alias:dirwsym:sym:tag for avail_terse_output. These values can be changed at installation time respectively with the --with-avail-output and --with-avail-terse-output options. These values can then be superseded by using the config sub-command which sets the MODULES_AVAIL_OUTPUT and MODULES_AVAIL_TERSE_OUTPUT environment variables.
  • Introduce the list_output and list_terse_output configuration options to define the content to report in addition to the available module names respectively for list sub-command regular and terse output modes. Excepted value for these configuration options is a colon separated list of elements to report. Default value is header:idx:sym:tag:key for list_output and header for list_terse_output. These values can be changed at installation time respectively with the --with-list-output and --with-list-terse-output options. These values can then be superseded by using the config sub-command which sets the MODULES_LIST_OUTPUT and MODULES_LIST_TERSE_OUTPUT environment variables.
  • Add the --output/-o command-line switches to supersede the output configuration of avail or list sub-commands on their regular or terse output modes.
  • Remove the avail_report_dir_sym and avail_report_mfile_sym locked configuration options whose behaviors can now be obtained by respectively adding the dirwsym and sym elements to the avail_output or avail_terse_output configuration options.
  • When modulepath is omitted from the content to report on avail sub-command, available modules collected from global/user rc and enabled modulepaths are aggregated and reported all together.
  • Install: print generated file names rather commands executed to generate these files on Makefile build targets. Output obtained when building Modules is this way simplified. When option V=1 is passed to make, the verbose mode is enabled and run commands are shown. The simplified make output does not apply to the install, test and clean targets or any target similar to them.
  • Install: fix configure and build files of Modules Tcl extension library to make them compatible with autoconf >=2.69.
  • Script: correctly detect previous Modules version number released from a side git branch on mpub command.
  • Install: align RPM spec file syntax with spec file used on Fedora. Add missing build dependency on make package. Also remove obsolete Group RPM tag.
  • Add the term_width configuration option to set the width of the output. This configuration option is set to 0 by default, which means that the output width is the full terminal width. The --width/-w command line switches are added to supersede the value of the configuration option. (fix issue #359 with contribution from Anaïs Gaertner)
  • Doc: add a Get Modules section in Installing Modules on Unix document to provide download links for Modules' sources. (fix issue #387)

Modules 4.6.1 (2020-11-14)

  • Lib: implement initStateClockSeconds as a Tcl command in libtclenvmodules to provide an optimized way to retrieve current Epoch time.
  • Lib: implement parseDateTimeArg as a Tcl command in libtclenvmodules to provide an optimized way to convert a datetime string into an Epoch time.
  • When full module specification is equal to @, raise an error as no module name is provided. (fix issue #362)
  • Optimize internal recording of hidden module and tag specification when parsing modulerc files in order to reduce the time taken to test if a given module is hidden or if a given tag applies to it.
  • Script: add the ability to select the benchmark test to perform on mb utility.
  • Doc: add Use new features without breaking old module command cookbook recipe
  • Doc: rework option description for module-hide and module-forbid commands in modulefile document.
  • Doc: describe in Differences between versions 3.2 and 4 document that shell special characters like backticks are escaped when used in values starting Modules 4.0. (fix issue #365)
  • Doc: make the ENVIRONMENT section from modulefile man page point to the ENVIRONMENT section of module man page.
  • Fix clear sub-command to unset the MODULES_LMSOURCESH environment variable. (fix issue #367)
  • Correctly return on avail sub-command a symbolic version defined in a global RC file when specifically searched. (fix issue #368)
  • Fix module hiding resolution for symbolic versions defined in a global RC file when module-hide statements are set in the modulepath where the modulefiles targeted by these symbols are located. (fix issue #369)
  • When a module fails to unload during a purge sub-command, preserve loaded the modules it requires to keep environment consistent. (fix issue #370)
  • Doc: add Hide and forbid modules cookbook recipe.

Modules 4.6.0 (2020-09-16)

  • Rework internal state handling to gather all state definitions in a global array and use the same initialization and retrieval procedure, named getState, for all these states.
  • Add the setState, unsetState, lappendState, isStateDefined and isStateEqual procedures to provide unified ways to set or check the value of state.
  • Introduce the sh-to-mod sub-command, to evaluate shell script and determine the environment changes it does. Corresponding modulefile content is outputted as a result. Changes on environment variables, shell aliases, shell functions and current working directory are tracked. The following shells are supported: sh, dash, csh, tcsh, bash, ksh, ksh93, zsh and fish.
  • Doc: add Source shell script in modulefile design notes.
  • Introduce the source-sh modulefile command, to evaluate shell script and apply resulting environment changes through modulefile commands. When a modulefile using source-sh modulefile command is loaded, the modulefile commands resulting from shell script evaluation are recorded in the MODULES_LMSOURCESH environment variable to be able to undo these environment changes when modulefile is unloaded and to report the modulefile commands used when modulefile is displayed. The same kind of environment changes than the sh-to-mod sub-command are tracked. The same list of shells than sh-to-mod sub-command are supported. (fix issue #346)
  • Doc: add Source shell script in modulefile cookbook recipe.
  • Doc: embed new Modules logo on website, online README and documentation portal.
  • Install: disable by default the build of Modules compatibility version. From now on, option --enable-compat-version has to be set to trigger this build.
  • Introduce the username sub-command to the module-info modulefile command to get the username of the user currently running modulecmd.tcl or to test a string passed as argument corresponds to this username.
  • Introduce the usergroups sub-command to the module-info modulefile command to get all the groups of the user currently running modulecmd.tcl or to test a string passed as argument corresponds to one of these groups.
  • Doc: improve markup of Release notes and MIGRATING documents starting from this 4.6 version to enable references to module sub-commands, command line switches, environment variables and modulefile Tcl commands.
  • Use inclusive terminology to eliminate master and slave terms as much as possible from code source and documentation.
  • Doc: use a versioned magic cookie in examples that demonstrate new modulefile features. (fix issue #349)
  • Introduce the --enable-multilib-support configure option to add mechanism in modulecmd.tcl to look at an alternative location to find the Modules Tcl extension library in case this library cannot be found at its main location.
  • Lib: remove fetch_hidden argument from getFilesInDirectory procedure of Modules Tcl extension library.
  • Doc: add Hide or forbid modulefile design notes.
  • Add the module-hide modulefile command, to dynamically hide modulefile, module alias or symbolic version matching passed specification. When hidden, a modulefile, an alias or a symbolic version is not reported nor selected unless referred by its exact name, like for module whose name or version starts with a dot character. module-hide supports the advanced module version specifiers. (fix issue #202)
  • Add option --soft to the module-hide modulefile command to introduce a soften level of camouflage: modules targeted by such hide directive are made visible as soon as their root name is part of search query.
  • Add option --hard to the module-hide modulefile command to introduce a hardened level of camouflage: modules targeted by such hide directive keep being hidden even if they are fully matched by search query.
  • Do not report among whatis search result the modulefiles with version name prefixed by a dot character and targeted by a symbolic version unless if they are precisely searched.
  • When a loading module has hidden alternative names (hidden due to their name or version starting with a dot character or because they match a module-hide statement), these alternative names are not recorded in environment unless if they are not hard-hidden and if they have been used in query to select loading module.
  • On avail sub-command, remove hidden symbolic versions from the list to display along modulefile or directory they target, unless these symbols are not hard-hidden and are used in query to search modules.
  • When the --default filter of avail sub-command is set, unhide all the default symbolic versions or modules targeted by these symbols unless if they are hard-hidden.
  • Define the default and latest automatic symbolic versions only if relative module name matches search query to ensure all elements for this module have been processed prior assigning the symbols.
  • In case a symbolic version is transitively applied toward a modulefile, like for instance when this symbol is first set onto a directory, record the resolution of each transitively applied symbol. By doing so, a module load tentative using the transitively applied symbolic version will now correctly resolve to the modulefile targeted by symbol.
  • Fix use of the advanced version specifiers in arguments to the is-avail modulefile command.
  • Introduce the --all/-a option for avail, aliases, whatis and search sub-commands, to include in the search process all hidden modulefiles, module aliases or symbolic versions. Hard-hidden modules stay hidden even if --all/-a option is used.
  • Add the module-forbid modulefile command, to dynamically forbid evaluation of modulefile matching passed specification. When forbidden, a module cannot be loaded and an access error is obtained when trying to evaluate them. module-forbid supports the advanced module version specifiers.
  • Add --not-user and --not-group options to module-hide and module-forbid modulefile commands to ignore hiding or forbidding definition if current user is respectively part of specified username list or member of one of specified group list.
  • Add --before and --after options to module-hide and module-forbid modulefile commands to ignore hiding or forbidding definition respectively after and before a specified date time. Accepted date time format is YYYY-MM-DD[THH:MM].
  • Add --message option to module-forbid modulefile command to supplement the error message obtained when trying to evaluate a forbidden module.
  • When a module that will soon be forbidden (as the date limit specified on the --after option of a matching module-forbid command is near) is evaluated, warn user this module access will soon be denied.
  • The range of time the above warning appears can be controlled with the nearly_forbidden_days configuration option, whose value equals to the number of days prior the module starts to be forbidden. This configuration is set to 14 (days) by default and this value can be controlled at configure time with --with-nearly-forbidden-days option. When the nearly_forbidden_days configuration is set through the config sub-command, the MODULES_NEARLY_FORBIDDEN_DAYS environment variable is set.
  • Add --nearly-message option to module-forbid modulefile command to supplement the warning message obtained when evaluating a nearly forbidden module.
  • Add the debug2 verbosity level, to report each call of modulecmd.tcl internal procedures in addition to debug messages. Debug2 mode can be enabled by setting the verbosity config to the debug2 value or by using the -D command-line switch twice.
  • Install: look for make rather gmake on MinGW and build library with a .dll extension on this platform.
  • Add the trace verbosity level, to report details on module searches, resolutions, selections and evaluations. Trace mode can be enabled by setting the verbosity config to the trace value or by using the -T/--trace command-line switches.
  • Introduce the tr key in the color palette to specifically render trace messages. Default value for tr key is 2 (decreased intensity).
  • When trying to set an environment variable to an empty value on the Windows platform, unset this environment variable instead to cope with the underlying OS behavior.

Modules 4.5.3 (2020-08-31)

  • Install: take into account the --build, --host, --target, --enable-dependency-tracking and --disable-dependency-tracking configure options to transmit them to the configure scripts of Modules Tcl extension library and Modules compatibility version. (fix issue #354)
  • Install: ignore some regular options of an Autoconf configure script that are useless for this project but usually implied in build macros (like RPM %configure macro).
  • Install: ignore unsupported --enable-* and --with-* options on configure script rather raise an error and add support to define environment variable and build system type as configure script arguments to comply with GNU configuration recommendations.
  • Install: fix modulecmd pre-alternatives check in RPM spec file.
  • Install: use %make_build and %make_install macros in RPM spec file.
  • When module switch command is used in modulefile, do not state when processing it a conflict over switched-off module if its specification on the module switch command also matches switched-on module's specification. Allow this way the replacement of any loaded version of a module for a specific one required by currently loading module. (fix issue #355)
  • Correctly report failed attempts to load module requirements expressed with advanced version specifiers. (fix issue #356)

Modules 4.5.2 (2020-07-30)

  • Init: list and source sub-commands do not take available modules as argument in fish completion script.
  • Init: fix option list for search sub-command in bash completion script.
  • Fix double error counter increase when modulefile evaluation breaks.
  • Install: adapt configure script to pass to the configure script of Modules compatibility version only a subset of the options it supports (most commonly used options).
  • Install: raise an error when an unknown option is passed to configure script rather silently ignore it. (fix issue #348)
  • Install: enable the definition of installation directory options of configure script with the --option value syntax in addition to the --option=value syntax. (fix issue #348)
  • Doc: alphabetically sort sub-commands of module-info modulefile Tcl command in modulefile document.
  • Script: clean previously built environment-modules RPMs in mrel.
  • Clearly separate quarantine variable definition from tclsh binary on modulecmd.tcl evaluated command call in _module_raw function for sh, bash, ksh and zsh shells. (fix issue #350)
  • Doc: clarify in documentation index that Environment Modules should not be confused with language-specific modules. (contribution from Rob Hurt)
  • Adapt conflict detection tests to ensure a module loaded by its full pathname will not detect itself as a conflict when declaring a reflexive conflict. (fix issue #352)
  • Adapt the mrel and mpub commands to produce new Modules release from a vZ.Y.x git branch rather than from the repository main branch.

Modules 4.5.1 (2020-06-01)

  • Install: consistently output Makefile warning messages on stderr.
  • Script: add the mrel script, that automates build of the Modules release files and performs tests over these distribution files to guaranty their correctness.
  • Script: add the mpub script, that automates Modules new release publishing over git repositories and websites.
  • Install: remove project-specific tools from git repository export thus from release distribution files.
  • Disable pager when clear sub-command is called from ml shortcut command. (fix issue #338)
  • In case a modulefile evaluation fails, environment context prior this failed evaluation is restored. Fix environment variable restoration mechanism to keep the link that monitors and updates environment variable array env in every Tcl sub-interpreters. (fix issue #340)
  • Ensure environment variable change at the Tcl interpreter level is propagated to every sub-interpreters used to evaluate modulefiles or modulercs. (fix issue #342)
  • Use absolute path to load Modules Tcl extension library. (fix issue #344 with contribution from Roy Storey)
  • Fix formatting of error stack trace not to look for internal commands to withdraw if start-up stack pattern cannot be matched.

Modules 4.5.0 (2020-04-07)

  • Doc: fix typos and grammar mistakes on module, modulefile and Differences between versions 3.2 and 4 documents. (contribution from Colin Marquardt)
  • Doc: update cookbook recipes to highlight code of the Tcl scripts included. (contribution from Colin Marquardt)
  • Doc: improve markup of module, modulefile and Differences between versions 3.2 and 4 documents to enable references to module sub-commands, command line switches, environment variables and modulefile Tcl commands. (contribution from Colin Marquardt)
  • Doc: alphabetically sort module sub-commands, command-line switches, environment variables and modulefile Tcl commands in module and modulefile documents.
  • Introduce the ml command, a handy frontend to the module command. ml reduces the number of characters to type to trigger module. With no argument ml is equivalent to module list, ml mod corresponds to module load mod and ml -mod means module unload mod. Multiple modules to either load or unload can be combined on a single command. ml accepts all command-line switches and sub-commands accepted by module command. ml command is defined by default. Its definition can be controlled at ./configure time with --enable-ml option or later on with ml configuration option (which defines MODULES_ML environment variable when set).
  • Fix module sub-command abbreviation match to ensure passed abbreviated form fully match sub-command, not only its minimal abbreviated form. As an example, lod or loda do not match anymore the load sub-command, lo or loa still do.
  • Add the -j/--json command line switches to the avail, list, savelist, whatis and search module sub-commands to render their output in JSON format. (fix issue #303)
  • Script: remove need to build project management-specific tools (mtreview, mb, mlprof and playdemo) prior using them.
  • Script: gather all distributed and maintained scripts in a script directory at the root of the project repository tree.
  • Install: provide Windows-specific batch files when ./configure option --enable-windows-support is set. module command wrapper module.cmd is installed in bindir and initialization script cmd.cmd in initdir. Those batch files are relocatable and expect modulecmd.tcl in ..\libexec directory. (fix issue #272 with contribution from Jacques Raphanel)
  • Install: add ml command wrapper ml.cmd and install it in bindir when ./configure option --enable-windows-support is set.
  • Install: introduce envml command wrapper envml.cmd for Windows cmd shell and install it in bindir when ./configure option --enable-windows-support is set. (contribution from Jacques Raphanel)
  • Doc: improve documentation portal index.
  • Install: add dist-win target to Makefile in order to build a distribution zipball containing the required files to run Modules on a Windows platform. INSTALL.bat and UNINSTALL.bat Windows batch files are introduced and shipped in the zipball to automate installation and basic configuration of Modules on the Windows platform.
  • Doc: update Installing Modules on Windows document to describe how to install Modules with newly provided Windows-specific distribution zipball.
  • Install: enable build of Modules from git archive tarball or zipball exports (like download source archives automatically provided on GitHub project)
  • Install: ship reStructuredText and MarkDown source documents at the root of Modules distribution tarball rather their built txt counterpart.
  • Script: fix createmodule.sh script to correctly analyses environment when shell functions are found defined in it.
  • Script: inhibit output generated by scripts evaluated by createmodule.sh and createmodule.py to ensure these outputs will not get in the way when analyzing the environment changes. (fix issue #309)
  • Correctly handle symbolic version target including a whitespace in their name.
  • Testsuite: output test error details whatever the testsuite run verbose mode.
  • Install: adapt configure script and Makefile to detect python command location and set it as shebang for createmodule.py and gitlog2changelog.py. If python command is not found, python3 then python2 are searched.
  • Install: enable to pass a specific Python interpreter command name or location at configure step with --with-python option. Specified command name or location should be found on build system only if building from git repository.
  • Install: build createmodule.py script and install it in bindir.
  • Install: update RPM spec file to explicitly define Python interpreter location.
  • Script: fix createmodule.py script for Python3 (fix issue #315 with contribution from Armin Wehrfritz)
  • Lift Perl variable strictness when defining _mlstatus variable in case modulecmd.tcl output is directly evaluated without use of the module sub-routine in Perl script. (with contribution from Andrey Maslennikov)
  • Script: fix path de-duplication in createmodule.sh. (fix issue #316)
  • Doc: add Handling Compiler and other Package Dependencies cookbook recipe, which discusses various strategies for creating modulefiles for packages with multiple builds depending on previously loaded compiler, MPI libraries, etc. (contribution from Tom Payerle)
  • Init: test availability of compopt Bash builtin prior using it in Bash completion script to avoid error with versions of this shell older than 4.0. (fix issue #318)
  • Install: adapt configure step to detect if sed option -E is supported and fallback to -r otherwise in shell completion scripts. (fix issue #317)
  • Add support for the NO_COLOR environment variable (https://no-color.org/) which when set (regardless of its value) prevents the addition of ANSI color. When set, NO_COLOR prevails over CLICOLOR and CLICOLOR_FORCE environment variables. MODULES_COLOR overrides these three variables. (fix issue #310)
  • Script: when analyzing environment variable changes in createmodule.sh applied by shell script passed as argument, produce a setenv modulefile statement for any variable found set prior script evaluation and for which value is completely changed after script evaluation. (fix issue #320)
  • When an error message is composed of multiple lines, render it in the same way whether it is part of a block message or not: lines after the first one are prepended with a 2-space padding. As a result error messages appear clearly separated from each other.
  • Append to the error message the error stack trace when a general unknown error occurs in modulecmd.tcl and provide a link to encourage users to report such error to the GitHub project.
  • Add to the error message the error stack trace for errors occurring during site-specific configuration evaluation. Error stack is expunged from the modulecmd.tcl internals to only report information relevant to site-specific configuration file.
  • When an error occurs during the evaluation of a modulefile or a modulerc, report associated error stack trace expunged from modulecmd.tcl internal references to only output useful information for users.
  • GitHub: add issue templates to guide people submitting a bug report or a feature request.
  • Doc: provide a link toward issues that have been fixed between versions 3.2 and 4.0 in Differences between versions 3.2 and 4 document.
  • Script: introduce envml.cmd script for Windows platform providing similar behavior than envml Bash script. (contribution from Jacques Raphanel)
  • Init: add Bash shell completion for the ml command. (contribution from Adrien Cotte)
  • Fix Fish shell stderr redirection for newer Fish versions. (fix issue #325)
  • Correctly handle modulefiles and modulepaths containing a space character in their name whether they are used from the command-line, in collections, within modulefiles or from loaded environment definitions.
  • Doc: add Default and latest version specifiers design note.
  • An avail search over a symbolic version targeting a directory now correctly returns the special modules (alias and virtual module) lying in this directory. (fix issue #327)
  • whatis and paths searches only return special modules (symbolic version, alias and virtual modules) that fully match search query, not those that partially match it. (fix issue #328)
  • alias and virtual module whose name mention a directory that does not exists are correctly handled. (fix issue #168)
  • Hide special modules (aliases, symbolic versions and virtual modules) whose version name starts with a dot character (.) from avail, whatis and paths searches if their query does not fully match special module name. (fix issue #329)
  • Filter-out from the output of the aliases sub-command all hidden aliases, symbolic versions or hidden modules targeted by a non-hidden symbolic version. (fix issue #330)
  • Enable resolution of default module in module sub-directory when this default symbol targets a hidden directory (whose name starts with a dot character). (fix issue #331)
  • Doc: clarify hidden module location in modulefile man page.
  • Install: define LD_PRELOAD as quarantine var along with LD_LIBRARY_PATH in RPM specfile.
  • When implicit_default and advanced_version_spec configuration are enabled, automatically define a default and latest symbolic version for each module name (at each module depth for deep modules) if those version names does not already exist. (fix issue #210)
  • Once a module is loaded, the automatically defined symbols associated to it are recorded in loaded environment in the MODULES_LMALTNAME environment variable. They are distinguished from the other alternative names applying to the module by a as| prefix, which qualifies their auto symbol type.
  • When an advanced version specifier list contains symbolic version references, fix resolving to honor default version if part of the specified list. (fix issue #334)

Modules 4.4.1 (2020-01-03)

  • Fix error and warning messages relative to dependency management to enclose dependency specification in single quotes to clearly distinguish specification from each other.
  • Skip output of module loading message if module is already loaded.
  • Doc: add demonstration material played at SC19 to promote the new features of Modules.
  • Contrib: add playdemo script to play recorded demonstration cast.
  • Doc: add a web anchor to each modulefile Tcl command, module sub-command and module environment variable documentation.
  • Install: update RPM spec file to enable build on el8.
  • Doc: fix RST syntax for bullet lists in design docs. (fix issue #306)
  • In case module avail query does not match a directory but only its contained elements (for instance module av mod/7 matches mod/7.1 and mod/7.2 but not mod/), fix query processing to correctly return latest or default element in case --latest or --default flags are set.
  • In case a module avail query performed in a no-indepth mode with --latest or --default flags either enabled or disabled, fix query processing to return directory elements if they are part of result.
  • When a module avail query performed in no-indepth mode targets a virtual module, fix result to filter-out the directory holding the virtual module from result.
  • Fix module avail --default queries when modulefile default version does not match query: select latest version from modulefiles matching query unless implicit_default configuration is disabled in which case no default version is returned.
  • Improve highlighting of module avail and whatis search result by coloring module names matching search query expressed with the advanced version specifiers. name@1,3 or name@1:3 queries now highlight name/1 and name/3 strings found in search result.
  • Contrib: add the mlprof script which wraps modulecmd.tcl to collect profiling information on its execution.
  • Contrib: adapt mb script to profile modulecmd.tcl run tests rather bench them when profile argument is passed to the script.
  • Improve overall performances of module names and versions comparison by introducing optimized procedures and caching in memory module search results.

Modules 4.4.0 (2019-11-17)

  • Doc: add Return file basename on module-info name for full path modulefile recipe to cookbook. (fix issue #297)
  • Rework internal handling of configuration options to gather all option definitions in a global array and use the same initialization and retrieval procedure, named getConf, for all these options.
  • Add the setConf, unsetConf and lappendConf procedures to provide unified ways to set the value of configuration option. These procedures should be used in site configuration files to override configuration option value instead of directly setting corresponding option variable as it was done in previous Modules releases.
  • Add the ability to match module specification in a case insensitive manner. Default case sensitiveness behavior is set at ./configure time with the --with-icase option. It could be superseded with the MODULES_ICASE environment variable, that could be set with config module sub-command through the icase option. Command-line switch --icase (-i) enables to supersede defined case sensitiveness configuration. (fix issue #212 with contribution from Eric Deveaud)
  • Introduce the extended default mechanism, to help selecting a module when only the first numbers in its version are specified. Starting portion of the version, part separated from the rest of the version string by a . character, will get matched to the appropriate complete version name. In case multiple versions match partial version specified and only one module should be returned, default version (implicit or explicit) among matches is returned. In case implicit_default option is disabled and no explicit default is found among matches, an error is returned. This mechanism is enabled through a new configuration option named extended_default (which defines MODULES_EXTENDED_DEFAULT environment variable when set). It may be enabled by default in modulecmd.tcl script with option --enable-extended-default passed to the ./configure script.
  • Introduce the advanced module version specifiers mechanism to specify finer constraints on module version. This new feature enables to filter the module selection to a given version list or range by specifying after the module name a version constraint prefixed by the @ character. It leverages the version specifier syntax of the Spack package manager. A single version can be specified with the @version syntax, a list of versions with @version1,version2,..., a greater than or equal to range with @version1: syntax, a less than or equal to range with @:version2 and an in between or equal to range with @version1:version2 syntax. In case implicit_default option is disabled and no explicit default is found among version specifier matches, an error is returned. This mechanism is enabled through a new configuration option named advanced_version_spec (which defines MODULES_ADVANCED_VERSION_SPEC environment variable when set). It may be enabled by default in modulecmd.tcl script with option --enable-advanced-version-spec passed to the ./configure script.
  • Conflict defined with a generic module name or an advanced version specifier may match multiple loaded modules (generally in case multiple loaded modules share same root name). Loaded environment analysis has been fixed to bind conflict to all loaded modules matching it. As a result the Dependent Reload mechanism is not triggered when one loaded module matching conflict is removed if another loaded module still match the conflict.
  • Doc: add Module selection contexts, Insensitive case, Extended default and Advanced module version specifiers design notes.
  • Make MODULESHOME environment variable controllable through the config sub-command with home configuration option. A --with-moduleshome argument is also added to the ./configure script to set specific default value for this option at installation time. (fix issue #292)

Modules 4.3.1 (2019-09-21)

  • Contrib: add mb script to bench Modules versions.
  • Correct modulecmd.tcl script startup to correctly report error in case Tcl extension library fails to load. (fix issue #284)
  • Install: fix typo on CFLAGS definition in lib/Makefile. (fix issue #287 with contribution from Felix Neumärker)
  • Remove useless code in Modules Tcl extension library
  • Make URLs in README correctly rendered in HTML. (contribution from Per Persson)
  • Doc: clarify modulefile evaluation modes in modulefile.4 man page. (fix issue #289)
  • When looking at the closest match among loaded modules when switching module with just a single module argument specified, load the information on the currently set environment to get the alternative names of loaded modules prior to look at closest module match. (fix issue #290)
  • Doc: describe the way to determine the site-specific configuration script location in cookbook recipes implying the installation of such a file. (fix issue #266)
  • Doc: add Log module command recipe to cookbook. (fix issue #283)
  • Doc: add Expose procedures and variables to modulefiles recipe to cookbook.
  • Doc: add Make defined modulepaths persist over sudo recipe to cookbook.
  • Doc: add Ensure user fully qualify the modules they use recipe to cookbook.
  • Introduce the wa_277 configuration option to workaround an issue with Tcsh history mechanism. Default module alias definition for Tcsh hits an issue with shell history mechanism: erroneous history entries are recorded each time the module command is called. When wa_277 option is enabled (which sets the MODULES_WA_277 environment variable to 1), an alternative module alias is defined which fixes the history mechanism issue. However the alternative definition of the module alias weakens shell evaluation of the code produced by modulefiles. Characters with special meaning for Tcsh shell (like { and }) may not be used anymore in shell alias definition elsewhere the evaluation of the code produced by modulefiles will return a syntax error. (fix issue #277)
  • Doc: add Tips for Code Reuse in Modulefiles recipe to cookbook. (contribution from Tom Payerle)
  • Fix the whatis and paths sub-command results for module symbolic versions targeting a directory when implicit_default configuration option is disabled. No error is returned and same result is now obtained whether the symbolic name or its target is used as argument for those two sub-commands. (fix issue #294)
  • Fix the whatis and paths sub-command results for module aliases targeting a directory when implicit_default configuration option is disabled. No error is returned and same result is now obtained whether the alias name or its target is used as argument for those two sub-commands. (fix issue #295)
  • Rework all the ternary operator expressions in modulecmd.tcl that may result in a nan value (whatever the case used to write this string) as the expr Tcl command raises an error when it returns such a value, which breaks Modules as soon as a modulefile, an alias or a symbolic version is named nan. (fix issue #296)

Modules 4.3.0 (2019-07-26)

  • Introduce Vim addon files to highlight the modulefile syntax. Installation of these files, which is enabled by default, is controlled by the --enable-vim-addons and --vimdatadir configure options. (contribution from Felix Neumärker)
  • If modulefile is fully read, cache the content read and the file header computed to avoid another file read if the same modulefile need to be read multiple times.
  • Except for path, paths, list, avail and aliases module commands always fully read a modulefile whether its full content is needed or just its header to verify its validity. Proceed this way to only read file once on commands that first just check modulefile validity then read again valid files to get their full content.
  • Introduce Modules Tcl extension library (written in C) to extend Tcl language in order to provide more optimized I/O commands to read a file or a directory content than native Tcl commands do.
  • Install: add --libdir, --enable-libtclenvmodules, --with-tcl and --with-tclinclude options to configure script to control libtclenvmodules build and installation.
  • When an error is caught during modulecmd.tcl first initialization steps, ensure the error report facility is initialized to render error message.
  • When looking for modulefiles in enabled modulepaths, take .modulerc file found at the root of a modulepath directory into account. Which means these rc files are now evaluated like global rc files and can be used to define module aliases targeting modulefiles stored in the underlying file tree.
  • Correctly get available default (-d) and latest (-L) version whether search pattern is passed with an ending forward slash character or not or if it contains a * wildcard character.
  • Append a forward slash character to any directory result of an avail command to better distinguish these directories from regular files.
  • Introduce the ability to control whether avail command search results should recursively include or not modulefiles from directories matching search query by use of the --indepth and --no-indepth command-line switches or the environment variable MODULES_AVAIL_INDEPTH. Default behavior is set at the ./configure time with the --enable-avail-indepth and --disable-avail-indepth switches. (fix issue #150)
  • Update bash, fish and zsh completion scripts to propose available modulefiles in the no in depth mode.
  • Add the ability to graphically enhance some part of the produced output to improve readability by the use of the --color command-line switch or the MODULES_COLOR environment variable. Both accept the following values: never, auto and always. When color mode is set to auto, output is colored if stderr is attached to a terminal. Default color mode could be controlled at configure time with the --enable-color and the --disable-color option, which respectively correspond to the auto and never color mode.
  • Control the color to apply to each element with the MODULES_COLORS environment variable or the --with-dark-background-colors and --with-light-background-colors configure options. These variable and options take as value a colon-separated list in the same fashion LS_COLORS does. In this list, each element that should be highlighted is associated to a Select Graphic Rendition (SGR) code.
  • Inform Modules of the terminal background color with the MODULES_TERM_BACKGROUND environment variable or the --with-terminal-background configure option, which helps to determine if either the dark or light background colors should be used to color output in case no specific color set is defined with the MODULES_COLORS.
  • Color prefix tag of debug, error, warning, module error and info messages.
  • Highlight the modulefile or collection name when reporting messages for a an action made over this modulefile or collection.
  • Color the modulepaths reported on a use command.
  • Highlight title of separator lines or column name of table header.
  • Color modulepaths, directories, aliases and symbols reported by the avail, aliases, list, whatis and search commands.
  • When color mode is enabled and module aliases are colored, do not associate them a @ tag as the color already distinguish them from regular modulefile.
  • When color mode is enabled and a Select Graphic Rendition (SGR) code is set for the default modulefile symbol, apply this SGR code to the modulefile name instead of associating it the default symbol tag.
  • Highlight matched module search query string among avail, whatis and search command results.
  • Highlight the modulefile and collection full path name on display, help, test and saveshow command reports.
  • Color modulefile Tcl commands set in a modulefile on a display command report.
  • Color module commands set in a collection on a saveshow command report.
  • Re-introduce clear sub-command. (fix issue #203)
  • Leverage --force command-line switch on clear sub-command to skip confirmation dialog. (fix issue #268)
  • Init: improve readability of variable definition operations by writing one definition operation per line rather having multiple commands on a single line like VAR=val; export VAR. (fix issue #225)
  • Add the ability to define a site-specific configuration file with an environment variable: MODULES_SITECONFIG. When set, the script file pointed by the variable is sourced (if readable) after the site-specific configuration file initially defined in modulecmd.tcl. (contribution from Ben Bowers, fix issue #234)
  • Doc: add description in the module.1 man page of MODULERCFILE in the environment section and siteconfig.tcl in the files section.
  • Install: provide at installation time a bare site-specific configuration script in designated etcdir if no pre-existing siteconfig.tcl file is found at designated location.
  • Introduce the config sub-command to get and set modulecmd.tcl options and to report its current state.
  • Contrib: update createmodule.py script to support execution from the cmd shell. (contribution from Jacques Raphanel, fix issue #270)
  • Add the ability to configure when unloading a module and multiple loaded modules match request if firstly loaded module should be chosen or lastly loaded module. Configure option --with-unload-match-order defines this setting which can be superseded with the MODULES_UNLOAD_MATCH_ORDER environment variable. This variable can be set with the option unload_match_order on the config sub-command. By default, lastly loaded module is selected. It is recommended to keep this behavior when the modulefiles used express dependencies between each other.
  • Add the ability to configure whether an implicit default version should be defined for modules with no default version explicitly defined. When enabled, which stays the default behavior, a module version is automatically selected (latest one) when the generic name of the module is passed. When implicit default selection is disabled, the name of modules to evaluate should be fully qualified elsewhere an error is returned. This option is set at ./configure time with the --enable-implicit-default and --disable-implicit-default options. It could be superseded with the MODULES_IMPLICIT_DEFAULT environment variable, that could be set with config module sub-command through the implicit_default option.
  • Install: add to the configure script the --with-locked-configs option to ignore environment variable superseding of Modules configurations defined in modulecmd.tcl script. Lockable configuration option are extra_siteconfig and implicit_default. Currently locked options are reported through the locked_configs option on the config sub-command.
  • Introduce the ability to control the module search match. Search query string should match module name start or any part of module fully qualified name. Default search match behavior is set at ./configure time with the --with-search-match option. It could be superseded with the MODULES_SEARCH_MATCH environment variable, that could be set with config module sub-command through the search_match option. Command-line switches --starts-with (-S) and --contains (-C) for avail module sub-command enable to supersede defined search match configuration.
  • Introduce the ability not to set the shell startup file that ensure module command is defined once shell has been initialized. Setting shell startup file currently means defining ENV and BASH_ENV environment variables to the Modules bourne shell initialization script. ./configure options --enable-set-shell-startup and --disable-set-shell-startup define if shell startup should be set or not by default. It could be superseded with the MODULES_SET_SHELL_STARTUP environment variable, that could be set with config module sub-command through the set_shell_startup option.
  • Cookbook: add the test-modulefiles recipe. (fix issue #182 with contribution from Colin Marquardt)
  • Fix location of global RC file to @etcdir@/rc instead of @prefix@/etc/rc to cope with @etcdir@ specific setup (@etcdir@ defaults to @prefix@/etc).
  • Take into account Modules initialization configurations found in etc directory if they exist rather in init directory. If initrc configuration file is found in etcdir then it is preferred over modulerc file in initdir. Following the same trend, modulespath configuration file is found in etcdir then it is preferred over .modulespath file in initdir.
  • Introduce the ability to install the Modules initialization configuration files in the etcdir rather than in the initdir. A new configure option is introduced for this task: --with-initconf-in. Accepted values for this option are: etcdir or initdir (default).
  • Add the --enable-modulespath configure option, which is an alias for the --enable-dotmodulespath option as .modulespath configuration file is named modulespath when installed in etcdic.
  • Install: update RPM spec file to disable set_shell_startup option by default, set /etc/environment-modules as configuration directory and store Modules initialization configuration files in it.
  • Report an error when a module load or unload evaluation aborts due to the use of the break or exit modulefile commands. This error notification clarifies that module evaluation failed. (fix issue #267)
  • Remove the message block display output for the reload, purge and restore sub-commands to preserve this output style for modulefile evaluation modes (load, unload and switch) and thus clarify understanding.
  • When unloading a module that contains a module load or module switch modulefile command, inhibit the unload performed of the useless requirement when auto_handling mode is disabled if currently performing a purge, reload or restore sub-command. As the unload sequence is determined and managed from these top commands.
  • Add ability to control module command message verbosity with configuration option. Introduced verbosity levels from the least to the most verbose are silent, concise, normal, verbose and debug. This option could be set at ./configure time with --with-verbosity option. It could be superseded with the MODULES_VERBOSITY environment variable, that could be set with config module sub-command through the verbosity option. Silent, verbose and debug verbosity modes can be set at the command-line level respectively with --silent/-s, --verbose/-v and --debug/-D command-line switches. (fix issue #204)
  • When verbosity level is normal or higher, reports every module loads or unloads performed to restore a collection or source a scriptfile, even if there is no specific message to output for these module evaluations. Clarifies what module evaluations have been triggered by these sub-commands.
  • Also honor the CLICOLOR and CLICOLOR_FORCE environment variables to define color mode. (fix issue #279)

Modules 4.2.5 (2019-07-08)

  • Correctly escape ? character in shell alias. (fix issue #275)
  • When resolving the enabled list of modulepaths, ensure resolved path entries are unique. (fix issue #274)
  • Right trim '#' characters from the fetched modulefile magic cookie string to ensure a correct compatibility version comparison. Useful when modulefile first line is equal to #%Module4.2##############.
  • Fix argument parsing for the append-path, prepend-path and remove-path modulefile commands to consider every arguments found after the variable name as variable values and not command option even if argument starts with - character. (fix issue #278)
  • Fix automatic loading of modulefiles when multiple module names are set on a single module load modulefile command. When auto_handling mode was disabled, the load of not loaded modules was not achieved as soon as some modules on this list were already loaded. (fix issue #281)

Modules 4.2.4 (2019-04-26)

  • Better track each module evaluation and the context associated to it in order to report a more accurate information on the additional modules loaded or unloaded when proceeding the main evaluation request. (fix issue #244, #245, #246, #247 and #248)
  • Doc: preserve quotes and dashes when making HTML docs. (fix issue #250 with contribution from Riccardo Coccioli)
  • Fix hanging list sub-command when terminal width is equal to the single column text width to be printed. (contribution from Jesper Dahlberg)
  • During an additional evaluation triggered by an automated module handling mechanism, ensure warning and error messages are reported under the message block of the main evaluation. (fix issue #252)
  • During the unload of a module when the automated module handling mode is disabled, report a warning message for each unload of a useless requirement that fails as done when the automated module handling mode is enabled. (fix issue #253)
  • When multiple modules are listed on a prereq command, drop the output of those modules that fails to load (by the Requirement Load automated mechanism) to only keep the output of the module whose load succeed. (fix issue #254)
  • Fix switch sub-command when the switched-off module cannot be unloaded when other loaded modules depend on it. Whole switch process is failed and no load of the switched-on module is attempted. (fix issue #251)
  • When switching modules, report failure of switched-off module unload or switched-on module load under the message block of the switch action. A failed switched-off module unload is reported as an error, as it aborts the switch evaluation, whereas a failed switched-on module load is reported as a warning. (fix issue #255)
  • When a module requirement is seen missing but the load of this module was attempted, report a more specific error or warning message to let user understand that the load of the requirement was attempted but failed. (fix issue #257)
  • When loading a module, report any missing requirement on the message reporting block corresponding to this module load. This warning or error message comes in addition to the eventual Requirement Load message reported under the message block of the main evaluation. (fix issue #258)
  • When unloading a module which has some dependent module still loaded, produce a more specific error or warning message if an evaluation of these dependent modules has been realized or if the unload of the required module is forced. (fix issue #259)
  • When a conflicting module is seen loaded but the unload of this module was attempted, report a Conflict Unload error or warning message toward the main evaluation message block. (fix issue #261)
  • When loading a module, report any loaded conflict on the message reporting block corresponding to this module load. This warning or error message comes in addition to the eventual Conflict Unload message reported under the message block of the main evaluation. (fix issue #261)
  • Correctly report loading state of conflicting module. (fix issue #262)
  • Adapt warning, error and info messages relative to the Dependent Reload mechanism to distinguish the unload phase from the load (reload) phase of this mechanism. In the automated module handling summary report, unloaded modules via this mechanism are reported in the Unloading dependent list and modules reloaded afterward are reported against the Reloading dependent list. (fix issue #263)
  • When the automated module handling mode is disabled, do not attempt to load a requirement expressed in a modulefile with a module load command, if this requirement is already loaded or loading.
  • Skip load or unload evaluation of a module whose respectively load or unload was already attempted but failed. If this second evaluation attempt occurs within the same main evaluation frame. (fix issue #264)
  • When reloading modules through the Dependent Reload automated mechanism, prevent modules to automatically load of other modules with the module load modulefile command, as it is done for the prereq command. (fix issue #265)
  • Raise an error when an invalid option is set on append-path, prepend-path or remove-path modulefile command. (fix issue #249)
  • Zsh initializes by default the MANPATH environment variable to an empty value when it starts. To preserve manpath system configuration even after addition to this variable by modulefiles, set MANPATH variable to : if found empty. (improve fix for issue #224)
  • Doc: provide a short installation guideline in README file. (fix issue #230)

Modules 4.2.3 (2019-03-23)

  • Add all the module dependency-related internal information to those saved prior a modulefile evaluation in order to correctly restore internal state in case modulefile evaluation fails.
  • Init: in shell initialization scripts, initialize MANPATH if not set with a value that preserves manpath system configuration even after addition of paths to this variable by modulefiles. (fix issue#224)
  • Enable to define an entire path entry to the MODULEPATH variable which corresponds to a variable reference only. (fix issue#223)
  • Cookbook: add the modulefiles-in-git recipe. (contribution from Scott Johnson)
  • When module switch commands are found in modulefiles, track switched-off modulefile as a conflict and switched-to modulefile as a requirement to apply same behaviors than for module load and module unload commands in modulefiles. If module switch has only one argument, do not define a conflict toward switched-off modulefile. CAUTION: it is not recommended to use `module switch` command in modulefiles. (fix issue#229)
  • When unloading a module, revert module switch commands found in modulefile: switched-on module is converted to a module unload, like for module load command. Nothing is done for switched-off module, like for module unload command. (fix issue#226)
  • For default element in a modulefile directory which is a module alias that points to a modulefile, when this modulefile is loaded, it receives as alternative names the eventual module aliases set on the distant directory holding the alias pointing to it. (fix issue#231)
  • When unloading a module that contains module load or module switch commands in its modulefile, select for unload the automatically loaded requirement module which has been loaded prior its dependent. (fix issue#232)
  • Doc: describe Emacs settings useful for adhering to coding conventions in CONTRIBUTING guide. (fix issue #233 with contribution from Ben Bowers)
  • When looking for a loaded or loading dependency requirement, select among the eventual multiple candidates the closest match to the dependent module.
  • During the unload of a module, if the unload of one of its dependent (by the Dependent Unload mechanism) fails, abort the whole unload process. Exception made if the force mode is enabled. In this case failing module stays loaded and the Dependent Unload mechanism continues with next module to unload.
  • During the unload of a module, if the unload of one of its useless requirements (by the Useless Requirement Unload mechanism) fails, keep the requirements of this failing module loaded. Such error is reported as a warning and it does not stop the whole unload process. (fix issue#240)
  • During the load or the unload of a module, if the unload of one of its dependent (by the Dependent Reload mechanism) fails, abort the whole unload or load process. Exception made if the force mode is enabled. In this case failing module stays loaded and Dependent Reload mechanism continues with next module to unload. This failing module is removed from the Dependent Reload list, so it will not take part of the load phrase of the mechanism. (fix issue#239)
  • During the load or the unload of a module, if the load of one of its dependent (by the Dependent Reload mechanism) fails, abort the whole unload or load process. Exception made if the force mode is enabled. In this case failing module stays loaded and Dependent Reload mechanism continues with next module to load. When the mechanism is applied during a switch command, force mode is enabled by default on the load phase. (fix issue#241)
  • When reloading all loaded modules with the reload sub-command, if one reloading module fails to unload or load, abort the whole reload process to preserve environment sanity. (fix issue#237)
  • During the unload of a module when the automated module handling mode is disabled and this module declares its requirements with the module load modulefile command. If the unload of one of its useless requirements (by the Useless Requirement Unload mechanism) fails, whole unload process is not aborted and continue with next module to unload. (fix issue#238)
  • Contrib: add mtreview utility script that analyzes test suite log file to compare actual and expected output of failed test. mt does not output the full test suite logs anymore but only the information produced by mtreview on failed tests.
  • Install: exclude Continuous Integration configurations from dist tarballs.

Modules 4.2.2 (2019-02-17)

  • Correct the Dependent Unload mechanism when it triggers the unload of 2 modules making together a requirement from another module. This module is now also added to the dependent modules to unload.
  • Doc: add a cookbook section in the documentation and port there the 3 pre-existing recipes: inhibit-report-info, top-priority-values and unload-firstly-loaded.
  • Doc: add a CONTRIBUTING guide.
  • Doc: fix a typo on the Python initialization example in module man page.
  • Doc: add a FAQ entry to describe the use of module from Makefile. (with contribution from Robert McLay)
  • Trim any white-space, newline or ; characters at the beginning or end of the function body passed to set-function modulefile command.
  • Init: add recognition of the --auto, --no-auto and --force command-line switches in fish shell completion script.
  • Init: add recognition of the --auto, --no-auto, --force, --paginate and --no-pager command-line switches in zsh shell completion script.
  • When the load of a modulefile is asked but a conflict is registered against this modulefile by an already loaded module, the load evaluation is now performed and the conflict is checked after this evaluation. If the conflict is still there, this evaluation (and the evaluation of its requirements) is rolled back. (fix issue#216)
  • Init: fix _module_not_yet_loaded alias in tcsh completion script to handle situation when noclobber variable is set. Also ensure actual rm command is called and not an alias. (fix issue#219)
  • Fix warning message when the load of a modulefile is forced over a reflexive conflict (message was reported twice).
  • When looking at the dependency of a loaded module, only consider requirement loaded before dependent module (holding a prior position in the loaded module list) as valid. Those loaded after dependent module are considered as an unmet dependency thus they are not taking part in the Dependent Unload, the Useless Requirement Unload and the Dependent Reload mechanisms.

Modules 4.2.1 (2018-11-11)

  • Cookbook: add the inhibit-report-info recipe.
  • Cookbook: port unload-firstly-loaded and top-priority-values recipes to v4.2.
  • Init: fix listing of loaded modules for fish and tcsh shell completions.
  • Init: fix saved collection listing when no collection found for bash, zsh, tcsh and fish shell completions.
  • Adapt system modulefile Tcl command to execute the command passed as argument through shell, like it is performed on compatibility version. (fix issue#205)
  • Correctly filter modulefile search memory cache entries when using a full search result to search later on a specific modulefile.
  • Prefix debug messages by information on the current modulefile or modulerc interpreter if any.
  • Init: fix listing of loaded modules on unload and switch sub-commands for bash shell completion.
  • Refrain module unload modulefile command from unloading a module required by another loading module.
  • Enable is-loaded modulefile Tcl command in modulerc interpretation context, like done on compatibility version. (fix issue#207)
  • Check a required module is not already loading before attempting to load it. Helps to handle cyclic dependencies.
  • Compute loaded modules requirement dependency relations without cycle and consider the module closing the cycle in a constraint violation state to avoid reloading loops on the Dependent Reload mechanism.
  • Safely unset dependency reference when computing dependency relations as some dependencies expressed may target same module.
  • Ensure a loaded module matching multiple entries of a same or prereq will just be considered as one module matching this requirement.
  • Init: quote prompt in csh and tcsh script with :q rather double quotes to accommodate prompts with embedded newlines. (fix issue#209 with contribution from Satya Mishra)
  • Init: skip shell environment alteration if autoinit command fails. (fix issue#208)
  • Reword path-like variable element counter reference handling to simply ignore the counter values not coherent with the content of related path-like variable. (fix issue#206)

Modules 4.2.0 (2018-10-18)

  • Add chdir and puts environment settings to the per-modulefile evaluation saved context. So previous values of these settings are restored in case of evaluation failure.
  • Fix save and restore of x-resource environment settings on the per-modulefile evaluation context.
  • Use the correct warning procedure to report the full reference counter inconsistency message (so this message is fully inhibited during global whatis evaluations).
  • Make append-path, prepend-path, remove-path and unsetenv commands alter env Tcl global array during display, help, test or whatis evaluation modes. Thus an invalid argument passed to these commands will now raise error on these modes. (see Environment variable change through modulefile evaluation context section in MIGRATING document)
  • On whatis mode, append-path, prepend-path, remove-path, setenv and unsetenv commands initialize variables if undefined but do not set them to their accurate value for performance concern.
  • Clear value instead of unsetting it during an unload mode evaluation of setenv or *-path commands to avoid breaking later reference to the variable in modulefile.
  • Make getenv command returns value on help, test or whatis evaluation modes. (fix issue#188)
  • Add an argument to the getenv command to return the value of this argument if the queried variable is undefined.
  • Use a different modulefile interpreter for each evaluation mode.
  • Adapt the procedure called for each modulefile command depending on the evaluation mode to adapt behavior of these commands to the module command currently running.
  • Report calling name and arguments for modulefile commands on display mode. For the commands evaluated during this mode, trigger this report at the end of the evaluation.
  • Inhibit chdir, conflict, module, module-log, module-trace, module-user, module-verbosity, prereq, set-alias, system, unset-alias, x-resource commands on help, test and whatis evaluation modes.
  • Ignore chdir, module, module-trace, module-verbosity, module-user and module-log commands found during modulerc evaluation.
  • Correctly restore an empty string value on sub-interpreter global variables when sanitizing this interpreter between two modulefile/modulerc evaluations.
  • Cache in memory results of a modulefile search to reuse it in case of rerun instead of re-walking the filesystem.
  • Evaluate global rc files once module sub-command is known and registered, so it can be queried during their evaluation.
  • Rename _moduleraw shell function in _module_raw to use a common _module_ prefix for all module-related internal shell functions.
  • Install: add --enable-append-binpath and --enable-append-manpath configure options to append rather prepend the bin or man directory when adding them to the relative environment variable.
  • Doc: clarify documentation for module usage on scripting language like Perl or Python to mention that arguments to the module function should be passed as list and not as a single string.
  • When interpreting a setenv modulefile order during an unload evaluation, variable is still set to be unset in generated shell code but it is set to the value defined on the setenv order in the interpreter context instead of being cleared.
  • Register the conflicts defined by loaded modules in the environment (variable MODULES_LMCONFLICT) and ensure they keep satisfied. (see Modulefile conflict constraints consistency section in MIGRATING document)
  • Register the prereqs defined by loaded modules in the environment (variable MODULES_LMPREREQ) and ensure they keep satisfied. (see Modulefile prereq constraints consistency section in MIGRATING document)
  • Introduce the automated module handling mode, which consists in additional actions triggered when loading or unloading a modulefile to satisfy the dependency constraints it declares. Those actions are when loading a modulefile: the Requirement Load and the Dependent Reload. When unloading a modulefile, Dependent Unload, Useless Requirement Unload and Dependent Reload actions are triggered. (see Automated module handling mode section in MIGRATING document)
  • Track the loaded modules that have been automatically loaded (with environment variable MODULES_LMNOTUASKED) to distinguish them from modules that have been explicitly asked by user. This information helps to determine what module becomes a useless requirement once all its dependent modules are unloaded.
  • Track in saved collections the loaded modules that have been automatically loaded by add of a --notuasked argument to module load collection lines. So this information is restored in loaded environment when collection is restored. This --notuasked argument is ignored outside of a collection restore context.
  • Consider modules loaded from a module source file as explicitly asked by user.
  • Install: add --enable-auto-handling configure option to enable or disable the automatic modulefile handling mechanism.
  • Process list of loaded modules or modules to load one by one during the restore, purge and reload sub-commands whatever the auto handling mode is.
  • Add the ability to control whether the auto_handling mode should be enabled or disabled with an environment variable called MODULES_AUTO_HANDLING or from the command-line with --auto and --no-auto switches. These command-line switches are ignored when called from modulefile.
  • Init: add pager-related command-line options in shell completion scripts.
  • Doc: describe MODULES_LMCONFLICT, MODULES_LMPREREQ and MODULES_LMNOTUASKED in module.1 man page.
  • Add -f and --force command-line switches to by-pass dependency consistency during load, unload or switch sub-commands. (see By-passing module defined constraints section in MIGRATING document)
  • Disallow collection save or loaded modules reload if some loaded modules have some of their dependency constraints unsatisfied.
  • The Dependent Reload action of a load, unload and switch sub-commands excludes modules that have unsatisfied constraints and includes modules whose constraints are satisfied again (when sub-command process solves a conflict for instance).
  • Doc: describe --force, --auto and --no-auto command-line switches and MODULES_AUTO_HANDLING variable in module.1 man page.
  • Ignore directories .SYNC (DesignSync) and .sos (SOS) when walking through modulepath directory content. (contribution from Colin Marquardt)
  • Install: look for make rather gmake on MSYS2.
  • Fix exec() usage in Python module function definition to retrieve the correct return status on Python3.
  • Cookbook: add the top-priority-values and unload-firstly-loaded recipes.
  • Install: add gcc to the build requirements in RPM specfile.
  • Silent any prereq violation warning message when processing Dependent Reload mechanism or purge sub-command.
  • Doc: mention createmodule.sh and createmodule.py scripts in FAQ. (fix issue#189)
  • Register all alternative names of loaded modules in environment with MODULES_LMALTNAME variable. These names correspond to the symbolic versions and aliases resolving to the loaded modules. Helps to consistenly solve conflict or prereq constraints set over these alternative names. (fix issue#143 / see Consistency of module load/unload commands in modulefile section in MIGRATING document)
  • Doc: describe MODULES_LMALTNAME in module.1 man page.
  • Install: add --with-bin-search-path configure option to get in control of the path list used to search the tools required to build and configure Modules. (fix issue#164)
  • Install: add --enable-silent-shell-debug-support configure option to add the ability to control whether or not code to support silent shell debug should be added to the module function and sh-kind initialization scripts. (fix issue#166)
  • Install: add --enable-quarantine-support configure option to add the ability to control whether or not code to support quarantine mechanism should be added to the module function and initialization scripts. (fix issue#167)
  • Check version set in modulefile magic cookie. If modulefile sets a version number greater than modulecmd.tcl script version, this modulefile is not evaluated like when no magic cookie is set at all. (fix issue#171 / see Express Modules compatibility of modulefile with versioned magic cookie section in MIGRATING document)
  • Fix uninitialized variable in procedure producing list of element output. (fix issue#195)
  • Ensure the consistency of module load modulefile command once the modulefile defining it has been loaded by assimilating this command to a prereq command. Thus the defined constraint is recorded in the MODULES_LMPREREQ environment variable. Same approach is used for module unload modulefile command which is assimilated to a conflict command. Thus the defined constraint is recorded in the MODULES_LMCONFLICT environment variable. (see Modulefile alias and symbolic modulefile name consistency section in MIGRATING document)
  • Only look at loaded modules when unloading so unloading an nonexistent modulefile does not produce an error anymore. (fix issue#199)
  • Report error raised from modulefile evaluation as ERROR rather WARNING, like when a conflict constraint is hit. Moreover this kind of evaluation error is now silenced on global evaluation like when proceding avail or search sub-commands.
  • Record messages to report them by block on when processing a load or an unload modulefile evaluation to improve readability on these evaluating modes that may cascade additional actions. (see Improved module message report section in MIGRATING document)
  • Foreground load, unload, switch and restore actions (ie. asked on the command-line) now report a summary of the additional load and unload evaluations that were eventually triggered in the process.
  • Support del and remove aliases for unload sub-command like on compatibility version. (fix issue#200 with contribution from Wenzler)
  • Correctly transmit the arguments along with the command to execute on system modulefile command. (fix issue#201)
  • Contrib: add mt utility script which helps to run just specific part of the test suite.
  • Introduce set-function and unset-function modulefile commands to define shell function on sh-kind and fish shells. (fix issue#193 with contribution from Ben Bowers)

Modules 4.1.4 (2018-08-20)

  • Doc: fix typo on getenv command description in modulefile(4) man page and clarify this command should be preferred over ::env variable to query environment variable value in modulefile.
  • Init: fix bash and zsh completion scripts to enable Extended Regular Expression (ERE) on sed command with -E argument (rather -r) for compatibility with OS X's and BSDs' sed. (fix issue#178)
  • Handle default version sets on an hidden modulefile (were not found previously). (fix issue#177)
  • Init: fix ksh initialization script for ksh88 compatibility. (fix issue#159)
  • Install: use sed command rather grep and cut in configure and Makefile scripts. (fix issue#175 with contribution from Michael Sternberg)
  • Fix typo, tab indentation and pipe opening mode on createmodule.py utility script. (contribution from Jan Synacek)
  • Check ModulesVersion value set from .version rc file to ensure this value refers to a version name in current directory. Report error if a nested value is detected and ignore this value. (fix issue#176)

Modules 4.1.3 (2018-06-18)

  • Make setenv command alter env Tcl global array during help, test or whatis evaluation modes. (fix issue#160)
  • Doc: describe MANPATH variable special treatment on compatibility version in diff_v3_v4 document.
  • Initialize and export _moduleraw SH shell function if stderr is attached to a terminal. Was previously checking stdout. (fix issue#169)
  • For csh shells, quote code generated by modulecmd.tcl to pass it to the eval shell command.
  • Escape special characters when producing code to define shell aliases (fix issue#165)
  • Correct modulefile lookup when a modulefile directory is overwritten by a module alias definition but it contains an empty sub-directory. (fix issue#170)
  • Doc: describe getenv command in modulefile(4) man page.
  • Improve SH shell detection in profile.sh initialization script to use shell variable on bash or zsh to determine current shell name. (fix issue#173)

Modules 4.1.2 (2018-03-31)

  • Add an example global rc file in contrib/etc directory that ensures MODULEPATH is always defined.
  • Check HOME environment variable is defined on savelist and is-saved commands or raise error if not.
  • Fix saving of deep module default version in collection when version pinning is disabled: if foo/bar/version is default version for foo, collection will retain just foo (was retaining foo/bar).
  • Enable to save and restore collections containing full path modulefiles eventually with no modulepath defined.
  • Run puts command not related to stderr or stdout channels in calling modulefile context to correctly get access to the targeted file channel. (fix issue#157)
  • Quote autoinit result for eval interpretation on SH-kind shells to avoid parameter expansion to randomly occur on generated code depending on file or directory names of current working directory. (fix RH bug#1549664)
  • Ignore empty elements found in MODULEPATH, LOADEDMODULES or _LMFILES_ to ensure all elements in these variables are non-empty strings.
  • Raise error if loaded environment is in an inconsistent state when calling commands requiring correlation of information from the LOADEDMODULES and the _LMFILES_ environment variables. Error raised on load, unload, switch, reload, purge, list, save and restore commands. May affect info-loaded or is-loaded commands if module passed as argument to these command is specified as a full path modulefile.
  • Fix list command to process loaded modules information before performing any content output.
  • Install: adapt configure script and Makefiles to support installation on Cygwin system.
  • Detect terminal width on Windows cmd terminal with mode command.
  • Improve Windows cmd shell support: error code returned, echoing text, shell alias creation and removal, working directory change.
  • Raise error when an empty module name is passed to module sub-commands like load, display or unload.
  • Raise error when an empty collection name is passed to module sub-commands like save, saveshow or restore.
  • Raise error when an empty path is passed to module unuse sub-command, like already done on use sub-command.
  • Clear argument list if an empty module command name is passed.
  • Fix module function definition for all shells in autoinit command to correctly handle empty-string parameters or parameters containing white-spaces, quotes, escape characters.
  • Fix module function definition for Python to accept being called with no argument.
  • Fix parameter expansion on module function for all SH-kind shells when quarantine mode is activated.
  • Escape \ character when producing R shell code.

Modules 4.1.1 (2018-02-17)

  • Make separator lines, used on display command result for instance, fit small screen width.
  • Install: give ability to build and install Modules from git repository without documentation if sphinx-build cannot be found.
  • Install: adapt configure script and Makefiles to support installation on FreeBSD, Solaris and OS X systems. (fix issue#147)
  • Rework code generated by autoinit for sh-kind shells to avoid use of local variables as those are defined differently through the sh variants. (also fix issue#147)
  • Init: use a default value on undefined variables in sh-kind scripts to avoid unbound variables in bash -eu mode. (fix issue#151)
  • Correctly detect terminal column number on Solaris.
  • Init: fix csh init script to get compatibility with pure csh shell
  • Sanitize content of MODULEPATH before using it at run-time, to make potential relative paths absolute, remove trailing slashes, etc. (fix issue#152)
  • Check loaded modulefiles still exists before displaying statistics on them during a list action.
  • Use a specific reference counter variable name (MODULES_MODSHARE_<VAR> instead of <VAR>_modshare) for DYLD-specific variables. (fix issue#153)
  • No error raise when updating a DYLD or LD path-like variable on OS X when System Integrity Protection (SIP) is enabled. In this situation, these variables are not exported in subshell context, so they appear undefined.
  • Init: protect arguments passed to the _moduleraw sh function from interfering content of current working directory. (fix issue#154)
  • Install: move hostname RPM requirement to the compat sub-package.
  • Start pager process only if some text has to be printed. (partially fix issue#146)
  • Ignore PAGER environment variable to configure Modules pager to avoid side effects coming from a general pager configuration not compatible with Modules pager handling. (fix issue#146)
  • Do not blank anymore default Modules pager options if default pager is less when the LESS environment variable is defined. (fix issue#146)

Warning

With this bugfix release, changes have been made on the pager setup to avoid side effects coming from the system general pager configuration. As a result PAGER environment variable is now ignored and MODULES_PAGER should be used instead to adapt Modules pager configuration at run-time.

Modules 4.1.0 (2018-01-15)

  • Extend stderr output redirection on sh-kind shells to all terminal-attached shell session, not only interactive shell session.
  • Extend shell code produced by the autoinit command to perform the same environment initialization as done in init shell scripts (default value set for module-specific environment variables, parse or source of configuration files).
  • Make init shell scripts rely on autoinit command to define the module command and setup its default environment.
  • Fix error rendering code for Tcl shell by producing a call to the error procedure.
  • Introduce pager support to handle informational messages, using less command with -eFKRX options by default. Environment variable MODULES_PAGER or PAGER may be used to supersede default pager command and options. --paginate and --no-pager switches enable or disable pager from the command line.
  • Install: add --with-pager and --with-pager-opts configure options to define default pager command and its relative command-line options.
  • Introduce quarantine mechanism to protect module execution against side effect coming from the current environment definition. Variables whose name has been put in MODULES_RUN_QUARANTINE will be emptied or set to the value hold by MODULES_RUNENV_<VAR> in the modulecmd.tcl run-time environment. Quarantine variable original value is then restored within modulecmd.tcl execution context once it has started.
  • Install: add --with-quarantine-vars configure option to define at build time the MODULES_RUN_QUARANTINE and MODULES_RUNENV_<VAR> environment variables set in initialization scripts.
  • Add MODULES_SILENT_SHELL_DEBUG environment variable to disable on sh shell and derivatives any xtrace or verbose debugging property for the duration of either the module command or the module shell initialization script. (fix issue#121)
  • Change error code produced by modulecmd.tcl for the Tcl, Perl, Python, Ruby, CMake and R scripting languages to return a 'false' boolean value in case of error rather raising a fatal exception.
  • Adapt module function definition for Tcl, Perl, Python, Ruby, CMake and R scripting languages to always return a value, result of the modulecmd.tcl run. When modulecmd.tcl run does not produce a specific status, a 'true' boolean value is returned. On CMake, resulting value is returned though a module_result global variable.
  • Spool content sent to the stdout channel with puts command during a modulefile interpretation, to effectively transmit this content to stdout after rendering the environment changes made by this modulefile. (fix issue#113)
  • Introduce append-path, prepend-path, remove-path and is-loaded module sub-commands, based on existing modulefile-specific Tcl commands. (fix issue#116)
  • Introduce is-saved, is-used and is-avail modulefile Tcl commands and module sub-commands to test availability of collection, modulepath or modulefile.
  • Raise error when a call to path or paths module sub-commands is attempted during a modulefile interpretation. Both commands now return text rather print text on scripting languages. An empty string is returned in no match case instead of a false boolean value.
  • Introduce module-info loaded modulefile command and its module sub-command counterpart info-loaded. This new command returns name of the modules currently loaded corresponding to the name passed as argument. (fix issue#3)
  • Fix is-loaded command to correctly handle multiple module names passed as argument (fix issue#138)
  • Support no argument on is-loaded, is-saved and is-used commands to return if anything is respectively loaded, saved or used.
  • Interpret module source command set in modulefile in unload mode when the modulefile itself is interpreted in this mode.
  • Consider a modulefile passed with name starting by ./ or ../ a full path name modulefile, like those starting by /. These kind of names are converted to absolute path names, for instance to register them in loaded modulefile list during a load command.
  • Correlate modulefile passed as full path name (starting by either ./, ../ or /) to already loaded modulefile registered with regular module name (file name without its modulepath prefix) to prevent for instance from loading twice same modulefile. Correlate in the same way regular module name to already loaded full path name modulefile.
  • Introduce MODULES_COLLECTION_PIN_VERSION environment variable to record modulefile version number when saving collections even if version corresponds to the default one. (fix issue#89)
  • Fix location of etc/rc global RC file to @prefix@/etc/rc instead of $MODULESHOME/etc/rc not to depend on MODULESHOME environment variable value.
  • Strengthen argument check for append-path, prepend-path and remove-path modulefile Tcl commands and module sub-commands. Raise error if argument list is not correct.
  • Fix support for the --delim=C argument form on append-path, prepend-path and remove-path commands.
  • Fix path reference counter handling in case path element is an empty string. Distinguish an empty path element from a variable set empty to clear it.
  • Pass multiple path elements separated by delimiter character as one string on append-path, prepend-path and remove-path commands.
  • Accept multiple path element arguments on append-path, prepend-path and remove-path commands.
  • Introduce the --duplicates argument option to append-path and prepend-path commands to add a path element already registered in variable.
  • Introduce the --index argument option to remove-path command to delete a path entry by passing its position index in variable.
  • Provide the ability to setup a site-specific configuration sourced at the start of modulecmd.tcl main procedure. This configuration is a Tcl script named siteconfig.tcl which enables to supersede any Tcl definition made in modulecmd.tcl. Location of this file is controlled at configure time with the --etcdir option.
  • Add the ability to handle paths containing reference to environment variable in MODULEPATH. When these kind of paths are used by module command, the variable references are converted to their corresponding value or to an empty string if they are not defined.
  • Enclose value set to environment variable on Tcl within curly braces rather double quotes to protect special characters in it from interpretation.
  • Correctly parse .modulespath initialization file to handle lines without any # character or to handle files with no content to extract.
  • Re-introduce the --enable-versioning configure option, which appends Modules version to installation prefix and deploy a versions modulepath shared between all versioning enabled Modules installation. A modulefile corresponding to Modules version is added to the shared modulepath and enables to switch from one Modules version to another.
  • Fix removal of CMake generated temporary script file by stripping newline character from script file name.
  • Add MODULES_CMD environment variable to expose path to the currently active module command script. This variable is set at initialization time.
  • Introduce modulecmd wrapper script, installed in binary directory, which executes the active module command.
  • Fix modulefile Tcl interpreter reset when handling list variables. (fix issue#145)
  • Introduce 'module-virtual' modulefile Tcl command to associate a virtual module name to a modulefile. This module can be located with its virtual name and the associated modulefile is the script interpreted when loading, unloading, etc.
  • Resolution of relative paths occurring during a modulefile interpretation to target a modulefile or a modulepath now takes the directory of the currently interpreted modulefile as the current working directory to solve the relative paths.

Modules 4.0.0 (2017-10-16)

Starting with this release, modules-tcl has become Modules. The following changes describe the differences with last modules-tcl release (1.923). To learn about the changes between this release and last Modules 3.2 release, please see the MIGRATING document.

  • Relax constraint on command-line argument position so options and switches can be passed either before or after command name.
  • Report unsupported option warning rather stop on error when compatibility-version specific command-line switches are passed ( --force, --human, --verbose, --silent, --create, --icase, --userlvl).
  • Keep empty module load line in shell configuration files after running the initrm or initclear commands.
  • Always return the value of tcl_platform(osVersion) for uname release
  • Optimize code output, for Perl to only return 1; once for a no-operation situation and for Python to not import os when there is only an error to render.
  • Use value of system command uname -n for uname nodename.
  • Add support for CMake shell
  • Ignore / character used as suffix in modulefile name passed on command line.
  • Rename Perl initialization script in perl.pm and Python in python.py.
  • Add support for Ruby shell (with contribution from Tammo Tjarks)
  • Add support for R shell (with contribution from Roy Storey)
  • When a default is set for a given module name, target modulefile can be referred on as modulename/default in addition to just modulename.
  • Locate symbolic versions on avail command even these symbols are set over a module alias or another symbolic version. In this situation the symbol spread along the resolution path until reaching a modulefile.
  • Define a more standard shebang on modulecmd.tcl script.
  • Determine modulefile corresponding to given module name using the loaded context only on unload situation.
  • Enable to unload mod/dir/subdir/vers when unload of mod or mod/dir asked. Was previously working only if deep module to unload was also the default version for these root names.
  • Make -l/-t switches mutually exclusive. Last switch mentioned on the command-line is honored.
  • Output parsable modulepath header when -l/-t switches are enabled.
  • When searching for a module in a given modulepath directory, if a module alias or a symbolic version matches searched module but the target of this alias or symbol is not found in current modulepath directory, search for this target restarting search from the first modulepath in list to ensure modulepath priority.
  • Solve aliases or symbolic versions looking for all modulepaths on search and paths commands. Was previously solved if their target was found in same modulepath directory.
  • Add support for hidden dot modulefiles. A hidden modulefile does not appear in case of wild search, it is only returned when search is about its exact name.
  • No table header print in --long mode on an avail command if no result are returned.
  • Add blank line between displayed list of elements, for instance between modulepath content on avail command.
  • Improve readability of error messages encountered during modulefile execution by putting Tcl error message first after the Module ERROR prefix.
  • Do not exit immediately when an internal error occurs in currently interpreted modulefile. Consider this interpretation as failed and continue to proceed the other modulefile arguments.
  • When multiple modulefiles are passed on display, help and test commands only output one separator line between 2 interpreted modulefiles.
  • Fix environment settings stack handling issue when restoring stack after a failed attempt to load a modulefile in a modulefile.
  • Failed attempt to load or unload a modulefile within a modulefile now leads to this upper modulefile load or unload failure. Previously upper modulefile were loaded respectively unloaded even if its dependent sub-modulefile failed to load or unload.
  • During a switch command, if the unloading part fails the loading part will not be tried. Unloading part fails if module to unload does not exist or its unload interpretation raise error.
  • Init: use module source rather shell command source to load modulerc system configuration in sh-kind, csh-kind and fish shell init scripts.
  • Install: transform configuration options to bind to an existing compatibility Modules version into option (--enable-compat-version) to build and install this compatibility version along with main version.
  • Init: adapt initialization scripts to handle both main and compatibility version. By default a shell script enables main version and if the environment variable MODULES_USE_COMPAT_VERSION is set to 1, the compatibility version is enabled instead of main version.
  • Install: import from compatibility version and install add.modules and mkroot utility scripts (scripts developed by R.K. Owen).
  • Install: update RPM spec file to handle compatibility version as a compat sub-package.
  • Add completion script for Fish shell (contribution from BEFH).
  • Doc: extend content of diff_v3_v4 to details all noticeable changes between v3.2 and v4.0.
  • Doc: introduce MIGRATING guide to learn the major changes when moving from v3.2 to v4.0.
  • Fix list command when full pathname modulefile is loaded (fix bug#132)
  • Install: handle version number though git tags in scripts, documentation and RPM spec file.
  • Doc: migrate documents from POD format to reStructuredText to benefit from Sphinx documentation framework and Read The Docs publishing capabilities.

Above changes describe the differences with modules-tcl release 1.923. To learn about the changes between Modules 4.0 and last Modules 3.2 release, please see the MIGRATING document.

modules-tcl-1.923 (2017-07-20)

  • Fix aliases command when a global or user RC file is set.
  • Find and solve global or user RC aliases and symbolic versions on search, whatis and paths commands.
  • Do not look at currently loaded modules to resolve the target of a module alias.
  • Rework default and latest versions search on avail command. Correct display when at a given level a sub-directory element is last element in directory among modulefiles. Previously sub-directory was printed but last file among modulefiles was also printed (2 latest versions at the same level). A directory tagged "default" does not appear anymore in default listing result as its content (the default version found in that directory) will be displayed.
  • When an alias is set and overrides name of an existing directory, take this alias into account for default and latest choice and ignore directory content.
  • Bad default set will lead to no result displayed for the corresponding module in case of default avail display.
  • Correct inclusion of aliases in output result when these aliases are not part of the exact same module path than module path of the search.
  • Rewrite existing shell initialization file with initadd, initprepend, initswitch, initrm and initclear commands rather than writing a new file then copying this new file to replace the existing initialization file. In addition only re-writes shell initialization file if its content need to be altered.
  • Raise an error on initadd, initprepend, initswitch, initrm and initclear commands when no module load line are found in shell initialization file.
  • Normalize error messages for the various collection-related commands when collection cannot be accessed.
  • Cleanup existing reference counters of a path list variable when this variable is altered by a setenv or an unsetenv command.
  • Init: do not pollute tab-completion with moduleraw command. (Bert Wesarg)
  • Make use of the same Tcl interp for each modulefile interpretation and use another one for each modulerc (but the same for each modulerc). By doing so we proceed like on C-version where same interpreter is used across modulefile or modulerc interpretation. Huge performance improvement is achieved with this change on commands making intensive use of interp like avail. Interpreter state is reset from one interpretation to another: the initial variable and procedure state is restored before each new interpretation to avoid spread of definitions from one interpretation to another. Also in case of nested interpretation each interpretation level has its own interpreter so a module loaded by another does not influence the interpretation of the module loading it.
  • Improve performance of aliases and symbolic versions resolution by computing these resolution at definition time. As a consequence resolution loop are not registered anymore and produce an error message when spotted not at display time.
  • Reduce number of access system call by trying access to modulefile when reading the content of a modulefile directory rather testing access before trying it.
  • No error raise on empty argument list for load. To cope with initadd behavior that requires at least an empty module load line in startup files. (fix SF bug#88)
  • Fix initadd to handle load line without trailing space. Was previously expecting load directive to be written "module load " to get a match. With fix, module load line will also be matched.
  • Like C-version catch raised error when break or continue are called from outside of a loop to handle them as when they are called from modulefile main body. (fix SF bug#87)
  • Return error on module use command when an empty path string is provided rather ignoring it.
  • Workaround min and max functions and lreverse procedure for correct operations under Tcl version 8.4.
  • Install: add --with-tclsh configure option to give the ability to choose the Tcl interpreter shell to setup in initialization scripts.
  • Handle error raised from the ModulesDisplay, ModulesHelp and ModulesTest procedures in the same way than for the evaluation of the modulefile content. An error occurring during the evaluation of the modulefile content will lead to no evaluation of the display, help and test command specific functions.
  • Remove debug module command
  • Doc: describe path, paths and autoinit module command.
  • Correct use of xrdb tool when not installed in default path.
  • Fix init* module commands to behave more like C-version and document remaining differences in diff_with_c-version.
  • Init: make sh init script closer to POSIX specification to support sh flavors different than Bash or Zsh like Dash.
  • Fix column-mode display for very short width terminal.
  • Install: introduce an install non-regression testsuite which is triggered by the make testinstall command and checks modules-tcl installation is operational.
  • Init: fix modulerc load test on fish init script.
  • Init: fix interactive shell test on sh init script.
  • Install: add --enable-example-modulefiles configure option that install by default some modulefiles provided as example in the system modulefiles directory.
  • Install: when uninstalling, do not remove modulefiles directory if it is not empty.
  • Add completion script for Zsh shell.
  • Add module test command to trigger when called execution of a ModulesTest procedure in target modulefile following same kind of mechanism than module help.

modules-tcl-1.832 (2017-04-29)

  • Fix getenv sub-command to correctly return environment variable value.
  • Clarify in man-pages display of module alias and symbolic version-name on avail command and management of file access issue when locating modulefiles.
  • Distinguish access issue (permission denied) from find issue (cannot locate) when trying to access directly a directory or a modulefile as done on load, display or whatis commands. In addition on this kind of access, not readable .modulerc files are ignored rather producing a missing magic cookie error.
  • When mode is set to unload, module load commands in modulefile are interpreted as module unload commands. To guaranty correct behavior regarding requirements, the module list passed to the load command is reversed to unload the modulefiles in the reverse order than they have been loaded.
  • Correct display command to only report module commands set in modulefile and not those set in the various .modulerc on the path toward this modulefile.
  • Fix bash and tcsh completion scripts to eliminate symbolic version names from avail command result.
  • Improve avail command when a symbolic version-name is passed as argument to return the modulefile target of this symbolic version-name.
  • When looking for an implicit default in a directory, now a module alias is taken into account so it can be returned as the last element in it (highest numerically sorted version).
  • Fix list command to correctly display the default tag along loaded modules when set via a .version file.
  • Fix long output of list command to display the symbolic version-names associated to each loaded module if any.
  • Improve avail command to return alias module when an alias name is passed as argument.
  • On a --default listing, a modulefile does not appear anymore if a directory is set default at the same level. On a --latest listing, a directory does not appear anymore if set default but not the latest.
  • Read modulerc and validate its header in a single open/read/close sequence instead of two in order to reduce to number of IO operations during an avail command.
  • Drastically reduce grid size computation time which removes overhead when displaying module avail results in column-mode.
  • Translate module name to currently interpreted module name when name correspond to the last part this interpreted module only in case of symbolic version-name or alias resolution.
  • Avoid resetting regular path (/usr/bin) or manpath (/usr/share/man) when switching from Tcl to C version in switchml utility.
  • Raise error on x-resource if DISPLAY environment variable is not set.
  • Fix lisp init script which was broken for environment change actions.

modules-tcl-1.775 (2017-03-07)

  • Improve README with examples, requirements, links, etc. Also update INSTALL documentation with details on the new configure/make/make install process.
  • Add display of a release date next to the version number when calling for --help or --version.
  • Update diff_with_c-version document to describe the features of the Tcl-version that are not supported on the C-version. Also state that the diff takes C version 3.2.10 against Tcl version 1.729 as a basis.
  • Introduce switchml tool, a shell function (or alias for csh or tcsh shells) that swap currently enabled Modules version (C or Tcl) by the other version (C or Tcl). Configure option --with-cver-initdir must be defined to enable switchml in initialization script.
  • Define a PATH and MANPATH in shell initialization scripts that point to the defined modules-tcl installation directories.
  • Give ability to generate distribution tarball from the git repository with Makefile dist target.
  • Introduce an installation process for this software following the configure/make/make install fashion. Configure step enables to choose installation paths and init scripts features to activate. Make step mainly translates init scripts with the configuration set. Make install creates target directories and copy files into them.
  • Fix MODULESHOME setup in autoinit command to define it as an absolute path and set it to the upper directory when modulecmd.tcl is located in a bin or a libexec directory.
  • Correct alias and version resolution on avail command which was erroneous in case of a modulefile holding symbols (like default) and targeted by aliases. Avail output was showing the aliases holding the symbols instead of the modulefile.

modules-tcl-1.729 (2017-02-01)

  • Add documentation in module(1) man page on the modulefile collection concept and the relative save, restore, saverm, saveshow and savelist commands.
  • Add document to list the differences of the functionalities that can be found on the C-version of the Modules package compared to the Tcl-version.
  • Improve modulecmd.tcl shebang to only search tclsh once if found in PATH.
  • Add module-info mode check against remove and switch values.
  • Introduce module-info command Modules-specific Tcl command to distinguish complex load or unload commands that cannot be determined with module-info mode alone. For instance a modulefile can now be aware that a switch, a restore or a purge command is currently being run.
  • Enable usage of module-info Modules-specific Tcl command from a modulerc file.
  • Fix module-info specified Modules-specific Tcl command.
  • No exit raise on modulefile or modulerc error during avail, aliases, whatis and search commands to avoid harming results from these global commands if error exists in a few modulefiles.
  • Exit with error code when a critical error is encountered when interpreting a modulefile or a modulerc.
  • Inhibit non-critical error report raised from modulefiles during avail, aliases, whatis and search commands to avoid error flood when parsing all modulefiles or modulercs.
  • Handle multiple lines of module-whatis Modules-specific Tcl commands defined for the same modulefile.
  • Handle multiple arguments passed to the module-whatis Modules-specific Tcl commands. They are joined to get a single line of text.
  • Return error on whatis command if searched modulefile is not found.

modules-tcl-1.704 (2017-01-20)

  • Set path variable counter to 1 for paths without a known reference count (was previously set to 999999999).
  • Introduce envml utility which acts as an application launcher where module commands are instantiated to setup environment before launching the given application.
  • Always register paths provided to be part of MODULEPATH environment variable as absolute paths to get independent from the current working directory.
  • Inhibit next modulefiles interpretation with exit Modules-specific Tcl command only if current mode is load.
  • Add argument to module-info shell and module-info shelltype to test current shell or shelltype value.
  • Fix use of default version-name to not consider it as a module symbol if a modulefile is named default.
  • Fix path variable counters when : character is used in elements of a path-like variable.
  • Update module(1) and modulefile(4) man pages to clear content specific to the C version of Modules and add content specific to or adapt content that behave differently on this Tcl version.
  • Fix TCLSH variable issue in Python init script.

modules-tcl-1.677 (2017-01-04)

  • Make switch command handle a single argument. The modulefile to switch to is the one passed on the command-line and the modulefile to unload is assumed to be the currently loaded module with the same root name as this modulefile specified on the command-line.
  • Make switch command idempotent by always ending up with old unloaded and new loaded, whatever the starting situation is.
  • Fix exit Modules-specific Tcl command.
  • Add refresh command as alias on reload command.
  • Add dummy module-log, module-trace, module-user and module-verbosity Modules-specific Tcl commands to enable support for modulefiles using them.
  • Fix system Modules-specific Tcl command to behave like described on the man page.
  • Fix module list when module loaded with full path
  • Disable g_force property by default to avoid loading a modulefile already loaded. It also avoids path element reference counting to get increased when the same module is asked twice for load.
  • Clarify module-info mode option and set help mode on module help command.
  • Clarify module-info flags and user options.
  • Handle empty or separator path on add-path ad unload-path commands.
  • Delete environment variable targeted by an unsetenv command on unload mode if no value has been provided along. On display mode, print environment variable value if any has been passed to unsetenv command.
  • When setting Tcl variable, enclose value within double quotes.
  • Fix perl quoting style for variable set, escape single quotes rather double quotes.
  • Call unuse command instead of use command on a module unload.
  • Fix continue Modules-specific Tcl command.
  • Add chdir Modules-specific Tcl command.
  • Fix break Modules-specific Tcl command.

modules-tcl-1.655 (2016-11-23)

  • No display of modulepath header if no module found in it.
  • Remove call to module aliases on module avail command, as aliases are now directly included in the avail results.
  • Include module aliases in the displayed result of an avail command. Also display aliases defined in a global or user modulerc file.
  • Exit with error code if error occurred on display or help commands.
  • Fix module-info symbols resolution.
  • Better handling of .modulerc and .version files when searching for a modulefile.
  • Fix module-info version resolution.
  • Fix module-info alias resolution.
  • Register alias and version by the short module name and improve their resolution to avoid loop.
  • Source $MODULERCFILE/modulerc when $MODULERCFILE is dir.
  • Make it so you can do module avail un, wildcard * character implied.

modules-tcl-1.632 (2016-09-06)

  • Raise error if command does not receive the excepted number of arguments.
  • Improve column-mode display to get a denser output on avail command.
  • Standardize the output of Warning, Error, InternalBug and ErrorAndExit messages.
  • Add short option -d for --delim on prepend-path.
  • Introduce collection target concept to distinguish between machines, environments or domains that are incompatible with each other.
  • Introduce saveshow command, to display content of saved collections.
  • Improve save and restore commands to handle collection specified as absolute or relative file path.
  • Introduce saverm command, to delete saved collections.
  • Enable to restore collection with multiple modulefiles specified on the same line.
  • Fix restore command when there is no module to load in collection.
  • Fix restore command when collection fully rewind module paths.
  • Fix restore command to preserve module path order set in collection.
  • Raise error if try to save an empty environment in a collection.

modules-tcl-1.602 (2016-08-13)

  • Add support for Fish shell.
  • Import recent tests added to C-version on 10-use and 50-cmds testsuites.
  • Add short option -d for --delim on append-path and remove-path.
  • Fix load and implement unload x-resource.
  • Fix Python code that was broken or not Python3-compliant. Fixed code is used to define the module command, to render error and to process x-resource.
  • Always dictionary-sort (also called numerical-sort) list of modulefiles or list of collections.
  • Fix bash completion script to be compliant with bash posix mode.

modules-tcl-1.578 (2014-12-24)

  • First release to be described in this NEWS file but it does not mean this is the first version of modules-tcl as this Modules flavor is born in 2002.
  • At this stage, modules-tcl handles a majority of the module commands and modulefile Tcl commands available on C version.

Frequently Asked Questions

Module command

How does the module command work?

The command module is an alias for something like:

sh:

module ()
{
   eval `/some/path/modulecmd sh $*`
}

csh:

eval `/some/path/modulecmd csh !*`

Where the modulecmd outputs valid shell commands to stdout which manipulates the shell's environment. Any text that is meant to be seen by the user must be sent to stderr. For example:

puts stderr "\n\tSome Text to Show\n"
I put the module command in a script and I run the script... it doesn't change my environment?

A child process (script) can not change the parent process environment. A module load in a script only affects the environment for the script itself. The only way you can have a script change the current environment is to source the script which reads it into the current process.

sh:

. somescript

csh:

source somescript
How do I capture the module command output?

This ties in with the very first question. Since the module command is essentially an eval, the visible output to the screen must necessarily be sent to stderr. It becomes a matter on how to capture output from stderr for the various shells. The following examples just show how to spool the output from the avail command to a file. This also works for the various other module commands like list, display, etc. There are also various tricks for piping stderr to another program.

sh:

module avail 2> spoolfile

csh: (overwrite existing file)

module avail >&! spoolfile
How to use the module command from Makefile?

To make use of the module command from a Makefile, the shell initialization script should first be sourced within Makefile rule to define the module function in that context. Environment variable MODULESHOME may help to locate the shell initialization script in a generic way, like done in the following example:

module_list:
   source $$MODULESHOME/init/bash; module list

Modulefiles

I want the modulefile to source some rc script that came with some application

If you want to do this... you can, but that's not what modules is all about. It is possible to have the modulefile output some text to stdout to source some script when loading. However, you lose the advantage of this tool, because you won't be able to unload this environment. If you're tempted to do this... don't.

However, you can craft a modulefile by capturing the environment variables created or changed by the rc script. This is the goal of the createmodule.py and the createmodule.sh utilities provided in the modules distribution:

/usr/share/Modules/bin/createmodule.py somescript

You can also check out https://sourceforge.net/projects/env2/, which can translate the shell file into a modulefile and possibly reduce the amount of work needed to convert and maintain.

How do I specify the default modulefile for some modulefile directory?

Modules usually uses the the highest lexicographically sorted modulefile under the directory, unless there is a .version file in that directory which has a format like the following where "native" is a modulefile (or a sub-directory) in that directory. It's also possible to set the default with a .modulerc file with a module-version command.

#%Module1.0#####################################################################
##
## version file for Perl
##
set ModulesVersion "native"
I don't want a default modulefile for the directory?

Follow the same prescription as setting a default, but give some bogus value, say no_default. The module command will return an error message when no specific version is given.

Build Issues

The configure script complains about Tclx
...
checking for TclX configuration (tclxConfig.sh)... not found
checking for TclX version... using 8.4
checking TCLX_VERSION... 8.4
checking TCLX_LIB_SPEC... TCLX_LIB_SPEC not found, need to use --with-tclx-lib
checking TCLX_INCLUDE_SPEC... TCLX_INCLUDE_SPEC not found, need to use --with-tclx-inc
...

TclX is an optional library that can speed up some operations. You don't need TclX for modules to compile and work, so you can add the --without-tclx option when configuring and it should proceed to completion. In fact, it should have succeeded anyways and just not attempt to use TclX.

Otherwise, you can load the TclX library package for your OS and the configure script should find it. If not then if you know where the tclxConfig.sh file or the library and include files are placed then use the following options:

--with-tclx=<dir>       directory containing TclX configuration
                        (tclxConfig.sh) [[searches]]
--with-tclx-ver=X.Y     TclX version to use [[search]]
--with-tclx-lib=<dir>   directory containing tclx libraries (libtclxX.Y)
                        [[none]]
--with-tclx-inc=<dir>   directory containing tclx include files
                        (tclExtend.h,...) [[none]]

Meta Information

Why does modules use Tcl?

The first versions of the Modules package used shell scripts to do its magic. The original authors then chose to implement the same in C to speed things up and to add features. At the time the only easily embeddable interpreter was Tcl which provided a standard language and the glue. Now that other interpreters are available they could be embedded, but haven't so far. There is also a pure Tcl version available.

How can I help?

We can use help at various levels. The best way to contribute is to send in a patch file (see the FAQ on how to generate a patch file) with whatever fixes. The patch will be reviewed and tested. If you are a regular contributer then you'll likely be invited to become a developer and to have direct source access, and the fame, power, and prestige that all entails.

How do I download the source repository?

Anonymously clone the git repository, view the list of branches, and set to a specific branch:

git clone git://git.code.sf.net/p/modules/git modules-myversion
cd modules-myversion
git branch -a
git checkout modules-3-X-Y
git status
How do I generate a patch file?
If you're starting from a tarball

Unpack the tarball and it should place the sources into a directory named modules-3.X.Y , then rename the directory to modules-3.X.Y-myversion or something like that. Make whatever changes you want, and be sure to test the changes and if you can add tests to identify the bug and the fix... that will endear yourself to the developers.

Once you have the changes in your version, then unpack the original sources from the tarball in a directory side-by-side to the directory with your version, and at that parent level run the following diff command:

diff -u -r -P -N modules-3.X.Y modules-3.X.Y-myversion  > my.patch
If you're starting from the git cloned repository:

From within the git repositories.

git diff > my.patch

Differences between versions 3.2 and 4

This document lists functionality differences between Modules version 3.2 and Modules version 4. Modules version 3.2 is also referred in this document as compatibility version whereas Modules version 4 is also referred as new main version. Modules version 4 is based on what was previously called Modules-Tcl. The goal of this document is to reference the features of the compatibility version that are missing or behave differently on the new main version and the features that can only be found on this new version.

First part the of the document covers the features that are missing or that behave differently on Modules 4 than on compatibility version. The second part of the document covers the features that are specific to the Modules 4 version thus missing on compatibility version. Comparison takes as a basis version 3.2.10 of compatibility version against Modules version 4.0. Any change made past these versions will explicitly mention the release number starting from the difference appears or disappears.

Regarding missing features, this document only lists their name or the command line argument related to them. Please refer to the module and the modulefile man pages of the compatibility or main version to learn the details about these features.

Last but not least, the numerous bugs or limitations spotted on Modules 3.2 across the years have been fixed in Modules 4.0. The list of reported issues that have been solved are available on the project code forge.

Features missing or with different behavior than compatibility version

This section describes the features of the compatibility version that are not supported or that behave differently on Modules 4.

Package Initialization

MODULESBEGINENV environment snapshot functionality is not supported on version 4. Modules collection mechanism should be preferred to save and restore sets of enabled modulepaths and loaded modulefiles. Modules 4 also introduces a system configuration file init/modulerc, located in Modules installation directory. Starting version 4.3, this file could also be etc/initrc. This modulerc/initrc file is sourced by Modules shell initialization scripts and helps to setup the initial environment right after initializing the module command.

Command line switches

--human

--create, -c

--userlvl lvl, -u lvl

These command line switches are not supported on Modules 4. When these options are passed on the command-line, it produces an Unsupported option warning and command is ran with the unsupported switches ignored.

--ter

--lon

--sil

--verb

These intermediate-form command line switches are not supported on Modules 4. Short or long switch name should be used instead.

-h

This command line switch is short version of --help switch on Modules 4 whereas it is short version of --human switch on compatibility version.

--force, -f

This command line switch was not supported starting Modules version 4.0 but reintroduced starting version 4.2 with a different meaning: instead of enabling an active dependency resolution mechanism --force command line switch now enables to by-pass dependency consistency when loading or unloading a modulefile.

--verbose, -v

--silent, -s

These command line switches were not supported starting Modules version 4.0 but reintroduced starting version 4.3. However, reintroduced --silent switch does not redirect stderr channel to /dev/null if stderr is found not to be a tty.

--icase, -i

This command line switch was not supported starting Modules version 4.0 but reintroduced starting version 4.4. When --icase switch is now set it applies to search query string and module specificiation on all sub-commands and modulefile Tcl commands.
Module Sub-Commands

On compatibility version, paths composing the MODULEPATH environment variable may contain reference to environment variable. These variable references are resolved dynamically when MODULEPATH is looked at during module sub-command action like avail. This feature was missing on Modules 4.0 but it has been re-introduced on Modules 4.1.

update

This module sub-commands is not supported on Modules 4.

clear

This command line switch was not supported starting Modules version 4.0 but reintroduced starting version 4.3. It now takes into account the --force command-line switch to skip confirmation dialog.

refresh

On compatibility version, only the shell aliases defined by the currently loaded modulefiles are set again on a refresh command. Whereas on new main version this command is an alias on the reload command that unload then load all the currently loaded modulefiles.

help

Compatibility version redirects output made on stdout in ModulesHelp Tcl procedure to stderr.

During an help sub-command, only the ModulesHelp Tcl procedure of a modulefile is interpreted on compatibility version. Version 4 interprets all the content of the modulefile, then call the ModulesHelp procedure if it exists.

On version 4, ModulesHelp subroutine is not ran if an error occurred during the interpretation of the modulefile main body.

display

On version 4, ModulesDisplay subroutine is not ran if an error occurred during the interpretation of the modulefile main body.

avail

On compatibility version, the same Tcl interpreter is used for the interpretation of all .modulerc or .version files during an avail command but the state of this interpreter is not reset between each interpretation. So some variable and procedure definitions may spread from one interpretation to another on this compatibility version. Modules 4 reuses the same interpreter for all .modulerc or .version interpretation but it is cleaned between each interpretation to protect from definition spread.

In case of --terse or --long mode, all enabled modulepaths will be displayed whether they hold result to display or not. Modules 4 outputs only the modulepaths where matching results are found. Modulepaths with no result to report are discarded from output.

apropos

On Modules 4, string passed as argument is always searched in a case insensitive manner.

load

On Modules 4, the value of an environment variable is set even if the new value is the same as the current value of this variable in environment.

When an already loaded modulefiles is asked for load again, compatibility version will refresh the shell alias definition this modulefile hold if any, whereas Modules 4 will ignore the new load order.

In case of modulefile loading another modulefile, if sub-modulefile load fails calling modulefile will still be loaded on compatibility version whereas Modules 4 will also abort calling modulefile load. Compatibility version behavior could be restored by enclosing module load command and arguments within catch Tcl command.

Starting with version 4.1, content sent to the stdout channel during a modulefile interpretation is spooled to effectively transmit this content to stdout after rendering the environment changes made by this modulefile.

unload

On Modules 4, the value of an environment variable is set even if the new value is the same as the current value of this variable in environment.

Compatibility version enables to load a modulefile by passing on the command-line the name of a module alias or symbolic version pointing to this modulefile. However this module alias or symbolic version name cannot be used to unload the modulefile once loaded. Modules 4 enables to pass a module alias or symbolic version name to unload a loaded modulefile referred by this name.

On versions 4.0 and 4.1, unloading an unexistent modulefile generates an Unable to locate modulefile error. Starting with version 4.2, unloading a module only looks at loaded module list and does not trigger a modulefile search. So starting version 4.2 the same behavior than Modules compatibility version is obtained.

Starting with version 4.1, content sent to the stdout channel during a modulefile interpretation is spooled to effectively transmit this content to stdout after rendering the environment changes made by this modulefile.

When the specified module to unload matches multiple loaded modules, Modules 4 unloads lastly loaded module whereas compatibility version unloads firstly loaded module. A configuration option unload_match_order has been introduced in version 4.3 and it enables to restore the behavior of compatibility version when it is set to returnfirst.

switch

When switching on version 4 an old modulefile by a new one, no error is raised if old modulefile is not currently loaded. In this situation compatibility version throws an error and abort switch action.

When switching on Modules 4 an old modulefile by a new one, this new modulefile does not keep the position that the old modulefile had in the LOADEDMODULES list as done on compatibility version but it is appended to the end of the list. Same goes for PATH-like environment variables: replaced PATH component is appended to the end or prepended to the beginning of the relative PATH-like variable, not appended or prepended relatively to the position hold by the swapped PATH component.

When a modulefile loads another modulefile with a module load order, this sub-module is not unloaded when the top modulefile is swapped-out during a switch command on compatibility version. Version 4 interprets the swapped-out modulefile in unload mode, so the module load order is interpreted as module unload order and sub-module is unloaded.

use

When the modulepath to enable is passed as a relative path, compatibility version will set it using passed relative name whereas Modules 4 will determine the corresponding absolute path and will register it rather passed relative name.

unuse

Modules 4 provides path element counting feature which increases a reference counter each time a given path entry is added to a given environment variable. This feature also applies to the MODULEPATH environment variable. As consequence a modulepath entry element is removed from the modulepath enabled list only if the related element counter is equal to 1. When unusing a modulepath if its reference counter is greater than 1, modulepath is kept enabled and reference counter is decreased by 1.

whatis

On Modules 4, environment variable edition commands (setenv, unsetenv, append-path, prepend-path and remove-path) do no set variable to the defined value on the modulefile evaluation context during a whatis evaluation. Instead environment variables are initialized with an empty value if undefined, to avoid raising error when attempting access to an undefined element during the modulefile evaluation.

initadd

initprepend

initswitch

On version 4 no message is displayed to give details on how list of modulefiles to load has been altered in initialization file.

initrm

No message is displayed on Modules 4 to inform of the modulefiles that have been removed from the loading list in initialization file.

Empty module load line is left on version 4 when last modulefile from a line is asked to be removed. On compatibility version module load null line is set in this case.

initclear

Empty module load lines are left on version 4 whereas module load null lines are set on compatibility version.
Modules Specific Tcl Commands

On version 4 the characters that have a special meaning when rendered in shells are escaped when used in value to set environment variables, shell alias or shell functions. Thus it is possible to set environment elements with value containing these special characters (like backticks). On version 3 these special characters were evaluated by shell when recording the environment changes implied by modulefile evaluation.

append-path

prepend-path

Modules 4 produces an error when adding a bare colon character : as a path element to a path-like variable, as this colon cannot be distinguished from the colon used for path separator.

Modules 4 supports adding or removing empty path element to a path-like variable, whereas compatibility version looses track of this path element when the path-like variable is modified afterward. Empty path element enables to set a leading colon character :, which has a specific meaning on some regular environment variable like MANPATH or LD_LIBRARY_PATH.

When adding a path element to the MANPATH environment variable, Modules 4 is treating this variable like any other whereas a special treatment was applied on compatibility version: a default MANPATH value, set at configure time, was appended in case MANPATH variable was unset.

remove-path

Modules 4 provides path element counting feature which increases a reference counter each time a given path entry is added to a given environment variable. As consequence a path entry element is removed from a path-like variable only if the related element counter is equal to 1. If this counter is greater than 1, path element is kept in variable and reference counter is decreased by 1.

When unloading a modulefile, remove-path command is not applied to environment variable on Modules 4, whereas on compatibility version it is processed the exact same way than when loading modulefile.

exit

On Modules 4 code passed to the exit Modules specific Tcl command will not be thrown to be the module return value.

module-alias

module-version

In case the specified aliased module or the symbolic version introduces a resolution loop with already defined aliases or symbolic versions, this new alias or symbolic version is not registered and an error message is raised. On compatibility version, alias or symbolic version introducing loop are registered as the modulefile resolution is not computed at registration time.

module-info

module-info flags

module-info trace

module-info tracepat

module-info user

These module-info options are related to compatibility version-specific features so they are available on Modules 4 but with a dummy implementation that always returns false or an empty value.

module-info mode

During an unload sub-command, unload is returned instead of remove. However if mode is tested against remove value, true will be returned.

During a switch sub-command, unload then load is returned instead of switch1 then switch2 then switch3. However if mode is tested against switch value, true will be returned.

module-info name

If the module name passed to the command-line has been specified as a full path name, the module-info name used in modulefile returns this file base name on compatibility version whereas it returns on Modules 4+ the full path name as it is identified by this name once loaded.

module-info version

Declared aliases or symbolic versions are not registered anymore if they introduce a resolution loop. As a result module-info version does not return an *undef* string value as it does not face resolution loop situation anymore.

module-info symbols

Declared aliases or symbolic versions are not registered anymore if they introduce a resolution loop. As a consequence symbolic versions introducing loop situation are not part anymore of the module-info symbols returned result as they are not registered.

A symbolic version sets on a module alias will be propagated toward the resolution path to also apply to the relative modulefile if it still correspond to the same module name.

module-log

module-trace

module-user

module-verbosity

These Modules specific Tcl commands are related to compatibility version-specific features so they are available on Modules 4 but with a dummy implementation that always displays a warning message saying the command is not implemented.

module-whatis

When multiple words are passed as argument to module-whatis but they are not enclosed in double-quotes or curly braces they will be displayed as a single line on Modules 4 whereas compatibility version displays them as one line per word.

set-alias

Whereas compatibility version sets a shell function when variables are in use in alias value on Bourne shell derivatives, Modules 4 always defines a shell alias never a shell function.

setenv

On compatibility version the setenv modulefile command accepts any number of argument, but ignores all arguments after the first two. An error is raised on Modules 4 when more than two arguments are passed to the setenv modulefile command.
Locating Modulefiles

On version 4, when a module alias is set and overrides name of an existing directory, this alias is taken into account to locate the default version of this module name and the modulefiles locating in the directory are ignored.

When looking for an implicit default in a modulefile directory, aliases are taken into account in addition to modulefiles and directories to determine the highest numerically sorted element.

Modules 4 will resolve module alias or symbolic version passed to unload command to then remove the loaded modulefile pointed by the mentioned alias or symbolic version.

Modules 4 resolves module alias or symbolic version pointing to a modulefile located in another modulepath.

When locating modulefiles on Modules 4, if a .modulerc, a .version, a directory or a modulefile cannot be read during the search it is simply ignored with no error message produced. Visibility of modulefiles can thus be adapted to the rights the user has been granted. Exception is made when trying to directly access a directory or a modulefile. In this case, the access issue is returned as an error message. Access issue is also returned when a direct access is made to a module alias or a symbolic version targeting an unreadable modulefile.

Features specific to the new main version

This section describes the features of Modules version 4 that are not supported on the compatibility version. Please refer to the above section for features supported by both versions but behaving differently.

Package Initialization

Compatibility version does not support fish, lisp, tcl and R as code output.

On version 4 and for sh, bash, ksh, zsh and fish shells, text output, like listing from the avail command, is redirected from stderr to stdout after shell command evaluation if shell is in interactive mode. Starting version 4.1, this content redirection occurs if shell session is attached to a terminal.

Starting version 4.5, a new alias or function called ml may be defined at initialization time, to provide a handy frontend to the module command.

Modulecmd startup

Starting with version 4.1, modulecmd.tcl sources upon invocation a site-specific configuration script named siteconfig.tcl. This Tcl script enables to supersede any global variable or procedure definition of modulecmd.tcl.

Command line switches

The following command line switches appeared on new version of Modules are not supported on compatibility version.

Introduced in version New command line switches
4.0 --debug, -D, --default, -d, --latest, -L
4.1 --paginate, --no-pager
4.2 --auto, --no-auto
4.3 --indepth, --no-indepth, --color, --starts-with, -S, --contains, -C
4.5 --json, -j
4.6 --trace, -T, --all, -a, -DD
4.7 -vv, --output, -o, --width, -w
Module Sub-Commands

The following module sub-commands appeared on new version of Modules are not supported on compatibility version.

Introduced in version New module sub-commands
4.0 reload, source, search, save, restore, saverm, saveshow, savelist, path, paths, autoinit, aliases, test
4.1 append-path, prepend-path, remove-path, is-loaded, is-saved is-used, is-avail, info-loaded
4.3 config
4.6 sh-to-mod
4.8 edit, try-load

All module sub-commands will return a non-zero exit code in case of error whereas on compatibility version issues that occurred do not lead to an exit of the module command with a non-zero code.

Starting with version 4.1, module function for all scripting languages, like Perl or Python, always returns a value. In case of error, a false boolean value is returned instead of raising a fatal exception. For module sub-commands returning a text value, the module function will actually return this value. In all other cases a true boolean value is returned.

avail, whatis and apropos

Non-critical errors are not displayed on avail, whatis and apropos sub-commands. Only valid results are returned.

Module aliases are included in the result of these sub-commands. They are displayed in the module path section where they are defined or in a global/user modulerc section for aliases set in user's or global modulerc file. A @ symbol is added in parenthesis next to their name to distinguish them from modulefiles.

Search may be performed with an alias or a symbolic version-name passed as argument.

Arguments to these avail, whatis and apropos commands may use wildcard characters to express glob patterns.

Collections

Modules collections are not supported on compatibility version.

Modules Specific Tcl Commands

The following modulefile Tcl commands appeared on new version of Modules are not supported on compatibility version.

Introduced in version New modulefile Tcl commands
4.0 module-info command, getenv
4.1 module-info loaded, is-saved, is-used, is-avail, module-virtual
4.2 set-function, unset-function
4.6 source-sh, module-hide, module-forbid, module-info usergroups, module-info username
4.7 versioncmp, module-tag, module-info tags
4.8 variant, getvariant

conflict and prereq

Starting with version 4.2, these Modules-specific Tcl commands support being called with a symbolic modulefile or a modulefile alias passed as argument.

module

In case of module load command specifying multiple modulefiles, when mode is set to unload these modulefiles will be unloaded in the reverse order to ensure correct handling of prerequisites.

Starting with version 4.7, the load, unload and switch sub-commands support the --not-req option to avoid recording a prereq or a conflict requirement toward specified modules.

append-path and prepend-path

Starting with version 4.1, both commands handle being called with multiple value arguments and option --duplicates is added.

remove-path

Starting with version 4.1, remove-path handles being called with multiple value arguments and option --index is added.

is-loaded

Starting with version 4.1, is-loaded supports being called with no argument passed. In this case, it returns true if any modulefile is currently loaded, false otherwise.

Starting with version 4.2, is-loaded supports being called with a symbolic modulefile or a modulefile alias passed as argument.

This Modules-specific Tcl command was not enabled for modulerc evaluation starting Modules version 4.0 but it has been reintroduced starting version 4.2.1.

Modules Variables

The following Modules-specific Tcl variables appeared on new version of Modules are not supported on compatibility version.

Introduced in version New Modules-specific Tcl variables
4.7 ModuleTool, ModuleToolVersion
4.8 ModuleVariant

Cookbook

Modules can be used in many ways. The following collection of recipes provides various installation examples that shed lights on how to take advantage of existing Modules features and how to extend the module command to achieve specific needs.

Handling Compiler and other Package Dependencies

When creating a collection of software (applications and libraries) for users to use, there is the problem of ensuring that the user is using the correct builds of everything. Generally, if an user is attempting to compile code making use of the system software collection, you want to ensure that the user is compiling his code with the same compiler that was used to compile the library. This tends to be particularly true of C++ and Fortran code using modules, and parallel codes using MPI libraries.

As a result, in environments supporting multiple compilers, software libraries often end up with multiple installs of libraries and applications of the same version, depending on the compiler and other libraries used to build them. Sometimes there are even additional installs for variants with different threading models, number formats, level of vectorization support, etc. This cookbook describes various strategies for handling the modulefiles to support all these different builds for each package.

For each strategy, we will provide an overview of how it works, and then show how an user might interact with it, usually a similar sequence for each case. We wish to explore how, and how well, each strategy succeeds in handling the multiple builds of the same version of a package, including

  1. Basic dependency handling: seeing how well the strategy supports the loading of the correct build of a package depending on the previously loaded dependencies. And if no appropriate build is available, they should error accordingly.
  2. The module switch command and more advanced dependency handling: how well the strategy supports more advanced cases. E.g. a case wherein several modules are loaded and the user replaces a module upon which other modules currently loaded depend. In general, how well the strategy prevents the user's set of loaded modules from being incompatible.
  3. Visibility into what packages are available. This includes being able to readily see all of the packages installed, seeing what versions of packages are available for a given compiler/MPI/etc combination, and seeing for which compiler/MPI/etc combinations a specific version of a package is available.
  4. How easily the user can navigate the modules for the builds. This includes how well partial modulenames (e.g. omitting version, etc) are handled by the different strategies.

We then try to summarize the strengths, weaknesses, and other attributes of each strategy. We also try to discuss differences in using on older (3.x) and newer (4.x) Environment Modules versions.

In addition to displaying examples for each strategy in this document, we have set up a the test environment as a playground in which you can explore.

Overview of Examples

The examples are a bit more elaborate than in some other cookbooks, so the directory structure under doc/example/compiler-etc-dependencies is similarly more complicated.

Example Software Library

For the purpose of the examples and the playground, we have created a fake example software library, rooted at the subdirectory doc/example/compiler-etc-dependencies/dummy-sw-root beneath where you placed the modules source files. This software tree is intended to represent some of the features you might see in a real software tree, which supports various compiler and MPI libraries, and that has been added to over time, and not always in the most systematic way.

The example software library does not contain any real code; there are dummy scripts for e.g. gcc, mpirun, etc. which just echo then name of the code and what version, compiler, etc. it was supposed to be built for, which is handy to show in the examples that the modulefiles are working as expected. It also shows how such a directory tree might be laid out --- the details of the layout will affect some of the code in the modulefiles, etc. The directory structure can be altered to fit your standards, but would require some minor modification to the modulefiles, etc. Note that there are also a bunch of subdirectories named 1 containing symlinks, these are for the strategy using the Flavours add-on and are discussed in that section.

The software in the example software library consists of:

  • GNU compiler versions 8.2.0 and 9.1.0
  • Intel Parallel Studio suite versions 2018 and 2019 (includes compilers, MPI and MKL)
  • PGI compiler suite versions 18.4 and 19.4
  • OpenMPI version 4.0, built for:
    • gcc/9.1.0
    • intel/2019
    • pgi/19.4
  • OpenMPI version 3.1, built for:
    • gcc versions 8.2.0 and 9.1.0
    • intel versions 2018 and 2019
    • pgi versions 18.4 and 19.4
  • mvapich version 2.3.1, built for:
    • gcc/9.1.0
    • intel/2019
    • pgi/19.4
  • mvapich version 2.1, built for:
    • gcc versions 8.2.0 and 9.1.0
    • intel versions 2018 and 2019
    • pgi versions 18.4 and 19.4
  • foo version 2.4, built for:
    • gcc/9.1.0 and openmpi/4.0
    • gcc/9.1.0 and mvapich/2.3.1
    • gcc/9.1.0 and no MPI
    • intel/2019 and openmpi/4.0
    • intel/2019 and mvapich/2.3.1
    • intel/2019 and intelmpi
    • intel/2019 and no MPI
    • pgi/19.4 and openmpi/3.1
    • pgi/19.4 and no MPI
  • foo version 1.1, built for:
    • gcc/8.2.0 and openmpi/3.1
    • gcc/8.2.0 and mvapich/2.1
    • gcc/8.2.0 and no MPI
    • intel/2018 and openmpi/3.1
    • intel/2018 and mvapich/2.1
    • intel/2018 and intelmpi
    • intel/2018 and no MPI
    • pgi/18.4 and openmpi/3.1
    • pgi/18.4 and mvapich/2.1
    • pgi/18.4 and no MPI
  • bar version 5.4, built with:
    • gcc/9.1.0 and supporting avx2
    • gcc/9.1.0 and supporting avx
  • bar version 4.7, built for:
    • gcc/8.2.0 and supporting avx
    • gcc/8.2.0 and supporting sse4.1

I.e., we have 3 families of compiler suites with 2 different versions each. And two MPI families (openmpi and mvapich) with two versions each, with the most recent version only built with the latest compiler version of each family, and the older version built with both versions of each compiler family. In addition, it is assumed that the intel compiler suites include Intel's MPI library built for that compiler. The application foo depends on the compiler and optionally on MPI libraries and has two versions; the newer version mostly has builds for the latest compiler and MPI (for pgi it only supports the latest compiler and older openmpi), and the older version mostly has builds for the older compiler and MPI. The bar application depends on compiler and has variants depending on size of integers used in the API.

We also assume that the gcc/8.2.0 compiler is the system default; i.e. it is the compiler provided by default by the Linux distro used by the system, and therefore might potentially be available to users without loading any modules.

More directories under doc/example/compiler-etc-dependencies

The modulefiles for the different strategies do not play well with each other, in part because we use the same names for many of the modules between strategies. So in addition to the dummy-sw-root subdirectory, each strategy has its own modulepath tree subdirectory underneath doc/example/compiler-etc-dependencies. There are some minor differences between the modulefiles for the Modulerc-based Strategy depending on whether Environment Modules 3.x or 4.x is being used, so we actually have two trees for that case (modulerc3 and modulerc4).

As there are a fair number of modulefiles, we make use of various tricks in the cookbook Tips for Code Reuse in Modulefiles to minimize the amount of repeated code. In general, the actual modulefiles are small "stubfiles", setting one or a few Tcl variables, and then sourcing a common tcl file which does all the real work. Symlinks are used where possible to avoid duplicating files. The Modulerc-based Strategy also uses some complicated .modulerc files; these are fairly generic and to avoid redundancy are symlinked into the appropriate places in modulepath tree from the modrc_common directory.

We also in some cases use Tcl procedures; for the sake of the examples these our sourced in the files as needed, but if one were to use the strategies needing such in production it would be better to follow the suggestions in Expose procedures and variables to modulefiles and place the required procedures in site config script. The various tcl procedures are placed in the tcllib sub-directory, outside of the modulepaths. These are actually broken up into multiple files for the purpose of this cookbook (so that smaller chunks of code can be looked at in this document).

The example-sessions subdirectory contains various shell scripts used for the usage examples for each strategy (shell scripts are used because there are some slight variations required between the strategies), as well as the outputs of running such scripts. Subdirectories exist for each strategy, and beneath them for each of the two Environment Modules versions (3.2.10 and 4.3.1) used; for brevity not all of them are shown in this document, especially as the 3.x and 4.x differences are often small. In the example outputs, the Environment Modules version and the strategy being employed is indicated in the shell prompt (e.g. mod3-flavours or mod4 (modulerc)).

Using the playground environments

Although we strive to provide a decent discussion in this cookbook, you are encouraged to try things out in the playground in order to get a better feel for things.

Because we use some modulefile names (e.g. gcc, intel, pgi, openmpi, etc) that likely are present on your system as well, it is recommended that if you wish to explore the playground environment that you spawn a new shell, do a "module purge", and then set your MODULEPATH environment variable appropriately for the specific strategy.

The Flavours_strategy, as will be discussed, requires some modifications to your Environment Modules installation. It is recommended that you you make a copy or new installation (for Flavours, a 3.x install works best), and then spawn a new shell and initialize the new Flavours install in that first. Flavours code is not provided with this cookbook.

Some of the modulefiles, etc. require knowledge of where they were installed. To avoid requiring you to update lines in numerous files, we require you to set the environment variable MOD_GIT_ROOTDIR to location where the modules git working directory was cloned. E.g., if you issued the command git clone https://github.com/cea-hpc/modules.git ~/modules.test you should set MOD_GIT_ROOTDIR to ~/modules.test. Please ensure it is exported (use setenv in csh and related shells, or export in Bourne derived shells like bash). This is just a hack to make the examples work better; if you opt to use one of these strategies in production, you will want to hard code some relevant paths; the comments in the modulefiles will describe what needs to be done.

Some more detail on setting up the playground is given at the start of the Examples section for each strategy.

Flavours Strategy

The Flavours strategy uses the Flavours extension to Environment Modules. Unlike the other strategies discussed, this requires the separate download and installation of an extension to Environment Modules.

Installation and Implementation

More details can be found at the website for this extension, but to install this you basically just need to:

  1. Clone the git repo somewhere (git clone https://git.code.sf.net/p/flavours/code flavours-code)
  2. Rename the standard Environment Modules modulecmd file (in the bin subdirectory under the installation root) to modulecmd.wrapped. (It is recommended that you do this in a copy of your production installation, or better yet, in a new install of the 3.x Environment Modules (as Flavours has been developped for Modules 3.x))
  3. Copy the modulecmd.wrapper file from Flavours to the bin subdirectory above. Make sure the modulecmd.wrapper file is executable.
  4. Symlink modulecmd.wrapper to modulecmd
  5. Edit modulecmd.wrapper where indicated to give fully qualified path to modulecmd.wrapped
  6. Copy the flavours.tcl and pkgIndex.tcl files to some (possibly new) directory under the modules installation roor, and set TCLLIBPATH to that directory (you probably will want to add that to the various modules init scripts)

The module command invokes modulecmd, which in this case is results in the Flavours wrapper bash script modulecmd.wrapper being invoked. This calls the renamed standard modulecmd.wrapped command. This wrapper command catches and processes certain output from the modulefile evaluation intended for its consumption.

The modulefiles themselves make use of various commands in the Tcl module flavours. Many of these are just flavours variants of standard modulefile commands, e.g. flavours prepend-path versus prepend-path. Some important flavours commands:

  • package require flavours: This loads the Tcl package flavours, and should occur near the top of your modulefile
  • flavours init: This initializes the flavours package, and should be the first of the flavours commands issued. Typically call right after the package load.
  • flavours prereq: Like the standard prereq command, this declares a prerequisite. But it also does quite a bit more, as is discussed further below.
  • flavours root: This is used to set the root for where the package is actually installed. This is used when generating the flavours path
  • flavours revision: seem intended to allow for changes in the path format in future versions of flavours. It is used in constructing the final path to the package.
  • flavours conflict: This is similar to the standard conflict command, but enhanced to recognize the flavours prereqs above.
  • flavours commit: This should be called after the root, revision, and prereq subcommands of flavours are called, and before any of the path subcommands. It seems to be responsible for taking all those values to above and constructing the path to the package.
  • flavours path: This returns a string with the path to the specific build of the package.
  • flavours prepend-path, flavour append-path: These work much like the standard prepend-path and append-path, except that the value being prepended/appended to the environment variable has the path (as returned by flavours path) prepended to it with the appropriate directory separator. E.g., to add to the PATH variable the bin subdirectory of the root directory where the specific build was installed, use flavours prepend-path PATH bin
  • flavours cleanup: This should be called after all flavours subcommands are finished and before exiting the script to ensure proper cleanup. Among other things, it ensures that any packages that depend on this package will get reloaded if this package is switched out.

The flavours prereq command accepts the new -class parameter, allowing it to require a class of packages; e.g. one could use -class compiler to indicate that it has a prereq on a compiler (any of the modules gnu, intel, or pgi). The allowable classes, and the package basenames that are in each class, is defined in flavours.tcl in the Tcl associative array _class. The ones shipped by default are

  • compiler: consisting of gnu, intel, and pgi
  • mpi: consisting of openmpi, mvapich2, mvapich, intelmpi
  • linalg: consisting of mkl, atlas, acml, netlib

You will likely want to adjust these if you go with flavours in production.

The flavours prereq command also accepts the parameter -optional, which declares optional prerequisites. Although it sounds a little oxymoronic, this comes into play with the secondary purpose of the command in declaring the components of the path, as discussed below. If a prereq is not optional, the modulefile will complain if nothing satisfying the prereq has been module loaded previously. If the prereq is optional, the modulefile will not complain if it was not loaded, but will use the prereq in constructing the path to the build of the package if it was loaded.

The flavours prereq command also defines the components which will comprise the final path to the directory containing the specific build of the package. The order of the prereq commands controls the order of the components in the path.

The modulefile will check that all non-optional flavours prereq commands are satisfied, and then construct a path to the installation root for this build of the package using the packages satisfying the prereqs. The resultant path is composed of:

  • the value from flavours root
  • directory separator (/)
  • the value from flavours revision
  • directory separator (/)
  • a prefix created by concatenating the package names satisfying the prereqs, in order. The package name and version will be separated by a hypen (-), as will the different components.

So if flavours root was set to /local/software/foo/1.7, revision to 1, and the package had prereqs compiler and mpi, and gnu/9.1.0 and openmpi/4.0 were loaded, the resulting path would be /local/software/foo/1.7/1/gnu-9.1.0-openmpi-4.0.

The modulefile actually will test for the existence of that directory, and if not found will return an error to the that the package was not built for that combination of prereqs. You either need to install your packages using the above directory schema, or create symlinks linking that scheme to where you actually install the packages.

Examples

We now look at the example modulefiles for flavours. To use the examples, you must

  1. Have Flavours extension installed. NOTE these examples will NOT work without the Flavours installed.
  2. Set (and export) MOD_GIT_ROOTDIR to where you git-cloned the modules source
  3. Do a module purge, and then set your MODULEPATH to $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/flavours

We start with the module avail command:

[mod4-flavours]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/flavours -
bar/4.7  foo/2.4    intel/2018        mvapich/2.1    openmpi/4.0  simd/avx
bar/5.4  gnu/8.2.0  intel/2019        mvapich/2.3.1  pgi/18.4     simd/avx2
foo/1.1  gnu/9.1.0  intelmpi/default  openmpi/3.1    pgi/19.4     simd/sse4.1

We note that we only see the package names and versions; e.g. foo/2.4, without any mention of the compilers and MPI libraries for which it is built. This terser stype was an intentional design goal of the authors. Also of note are the intelmpi and simd packages. The Flavours approach relies on seeing what modules have been loaded previously in order to determine what 'flavor' of the requested package should be loaded. To support the different builds of bar which depend on the CPU vectorization commands supported, we need to add a "dummy" package simd. The module definition is quite trivial; a simple stub file like

#%Module
# Modulefile for CPU vectorization support

set simd avx
set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/common

and the main content in the common file:

# Common stuff for "simd" modulefiles
# Using "flavours" strategy
#
# This file expects the following Tcl variable to have been previously defined:
#	simd: The level of simd support, eg. avx, avx2, sse4.1

# Initialise "flavours"
package require flavours
flavours init

proc ModulesHelp { } {
   global simd
   puts stderr "
This is a dummy modulefile to indicate that the CPU vectorization
should be set to '$simd' where possible.

"
}

module-whatis "CPU Vectorization support level: $simd"

# Even in production, this modulefile would not do anything
conflict simd

# Reload any modules with this as a prerequisite
flavours cleanup

Basically it just declares a help procedure and whatis text. This way, an user can load the appropriate simd module to control which variant of bar they will get. The only interesting aspect is that near the beginning of the file we do a package require flavours and flavours init, and add a flavours cleanup near the bottom. The lines at the beginning instruct Tcl command to load the Flavours package, and then initialize the package. The flavours cleanup is required so that if the simd module is switched out, any modulefiles that depend on it get reloaded.

In our example, we assumed that the Intel MPI libraries are automatically set up properly if one were to load the intel module, and we assumed the Intel MPI libraries were not supported for either the GNU or PGI compilers. However, we also wished to allow for foo to be used without any MPI support. So we need a way to distinguish if someone wants to use an Intel compiler build of foo without MPI or with the Intel MPI libraries. Our choice for this example was to require one to explicitly module load intelmpi if one wished to use the Intel MPI variant --- we do not bother with a real version number because assuming the version is determined by the version of intel (the Intel Parallel Studio version). So the intelmpi modulefile is similar to the simd modulefiles, a dummy modulefile. Again, it includes the flavours init and flavours cleanup wrapping to ensure proper reloading of dependent modules should it be switched out.

If you were to support Intel MPI for non-intel compilers, you could create your intelmpi modulefiles as usual, and then add a default or intel "dummy" version to use the version that is part of the intel Parallel Studio. Or you could separate the intelmpi bits from the intel modulefile so both non-intel and intel compilers need to explicitly module load intelmpi.

The modulefiles for the various compilers are all pretty much standard, except for the same three flavours lines as the simd modulefile: package require flavours, flavours init, and flavours cleanup. These are required to ensure dependent modulefiles get reloaded if the compiler is switched out. We also note that the modulefile for the GNU Compiler Collection is referred to as gnu, not gcc (this is due to how the compiler class is defined in flavours.tcl, and we did not bother to change that for the purposes of this cookbook).

With the openmpi and mvapich MPI libraries, things start to get interesting. These all should setup the environment for a different build depending on the compiler loaded. The real work is done in the common tcl file, as shown below:

# Common stuff for "openmpi" modulefiles
# Using "flavours" strategy
#
# Expects the following Tcl variables to have been previously set:
#	version: version of openmpi

# Common parts of modulefile for openmpi

# Initialise "flavours"
package require flavours
flavours init

proc ModulesHelp { } {
   global version
   puts stderr "
openmpi: Test dummy version of OpenMPI $version

For testing packages depending on compilers/MPI

"
}

module-whatis "Dummy openmpi $version"

# Construct flavour name
flavours prereq -class compiler
flavours conflict openmpi

# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root

flavours root     $swroot/openmpi/$version
flavours revision 1
flavours commit

# Set environment variables
setenv MPI_DIR [flavours path]

# Prepend to environment variables (paths relative to
# the directory containing the flavour)
flavours prepend-path PATH            bin
flavours prepend-path LIBRARY_PATH    lib
flavours prepend-path LD_LIBRARY_PATH lib
flavours prepend-path CPATH           include

# Reload any modules with this as a prerequisite
flavours cleanup

Like the previous cases, the file starts with the Tcl command to load the package, followed by the flavours init command.

The flavours prereq command states that this package requires a compiler to have been previously loaded, and that the path to the specific build to use will depend on that. We note the use of the -class parameter; the exact definition of the compiler class is in the compiler field of the Tcl associative hash _class defined in flavours.tcl.

The flavours root sets the root directory of where the builds for this package is installed. We use the MOD_GIT_ROOTDIR environment variable for convenience in this example, but in production you would generally hardcode a path. The result of all the directives is that the build will be found in a path named after the compiler (since in this case there is only one flavour prereq); e.g. for gcc version 9.1.0, we expect to find the build in $swroot/openmpi/4.0/1/gnu-9.1.0. If you do not use that naming convention for your installation directories, you can use symlinks to fake it.

The flavours path command in the setenv MPI_DIR statement sets MPI_DIR to the aforementioned build path. The flavours prepend-path commands prepend to the environment variable specified by the first argument the result of prepending the flavours path to their second argument. E.g., the first such, assuming openmpi version 4.0 was requested and gnu/9.1.0 loaded, would be basically the same as a standard Modules command:

prepend-path PATH $swroot/openmpi/4.0/1/gnu-9.1.0/bin

The following shows how this would appear to the user:

[mod4-flavours]$ module purge
[mod4-flavours]$ module load pgi/19.4
[mod4-flavours]$ module load openmpi/4.0
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/4.0
[mod4-flavours]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4-flavours]$ module unload openmpi
[mod4-flavours]$ module switch pgi intel/2019
[mod4-flavours]$ module load openmpi/4.0
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0
[mod4-flavours]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4-flavours]$ module unload openmpi
[mod4-flavours]$ module switch intel gnu/9.1.0
[mod4-flavours]$ module load openmpi/4.0
[mod4-flavours]$ mpirun
mpirun (openmpi/4.0, gcc/9.1.0)
[mod4-flavours]$ module unload openmpi
[mod4-flavours]$ module switch gnu gnu/8.2.0
[mod4-flavours]$ module load openmpi/4.0
openmpi/4.0 - no flavour compatible with modules 'gnu/8.2.0'

Loading openmpi/4.0
  ERROR: Module evaluation aborted
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) gnu/8.2.0

Here we note that once a compiler is loaded, the PATH and the other environment variables are set appropriately to point to the bin dir for the particular build of openmpi/4.0, as evidenced by the output of our dummy mpirun command. At the end, we attempt to load openmpi/4.0 for gnu/8.2.0, and receive an error because our dummy SW library does not contain a matching build. This is determined from the flavours path; if the path does not exist (in this example $swroot/openmpi/4.0/1/gnu-8.2.0) it will abort in this fashion.

In the above, we have explicitly unloaded openmpi, switched the compilers, and then reloaded openmpi. A nice feature of Flavours is that it can handle the switching out of compilers or other modulefiles which other modulefiles depend on, as:

[mod3-flavours]$ module purge
[mod3-flavours]$ module load pgi/19.4
[mod3-flavours]$ module load openmpi
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
  1) pgi/19.4      2) openmpi/4.0
[mod3-flavours]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod3-flavours]$ module switch pgi intel/2019
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
  1) intel/2019    2) openmpi/4.0
[mod3-flavours]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod3-flavours]$ module switch intel intel/2018
openmpi/4.0 - no flavour compatible with modules 'intel/2018'
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
  1) intel/2019    2) openmpi/4.0
[mod3-flavours]$ mpirun
mpirun (openmpi/4.0, intel/2019)

Note that when we switched between the pgi and intel compilers above, Flavours automatically "unloaded" and "reloaded" the openmpi module. This happens in the flavours cleanup portion of the compiler modulefiles, and is due to openmpi declaring a flavours prereq on the compiler class.

Note

The above behavior with switch was done with version 3.2.10 of Environment Modules; it does not appear to work with 4.3.1.

Note that when we further tried to replace version 2019 of the intel compiler with the 2018 version, the module switch of the compilers failed because openmpi/4.0 was not built with intel/2018. Since the user never explicitly requested version 4.0 of openmpi (it was defaulted in the initial load as the latest version of openmpi available for pgi/19.4), it would have been nicer had the attempted reload of openmpi allowed it t:o default to the 3.1 version (as the latest version available for intel/2018). Nevertheless, it behaved well in this situation; the module switch failed with a reasonable error message and the resulting set of modules was still consistent.

We also note that if we attempt to load openmpi without having previously loading a compiler, we will get an error:

[mod4-flavours]$ module purge
[mod4-flavours]$ module load openmpi/3.1
openmpi/3.1 depends on one of the module(s) 'gnu intel pgi'

Loading openmpi/3.1
  ERROR: Module evaluation aborted
[mod4-flavours]$ module list
No Modulefiles Currently Loaded.
[mod4-flavours]$ module purge
[mod4-flavours]$ module load gnu/8.2.0
[mod4-flavours]$ module load openmpi
openmpi/4.0 - no flavour compatible with modules 'gnu/8.2.0'

Loading openmpi/4.0
  ERROR: Module evaluation aborted
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) gnu/8.2.0

In particular, there is no support for a "default" compiler; if e.g. you wished to make the distribution supplied gcc the default compiler, you will need to have the initializations scripts automatically do a module load of that compiler (possibly a dummy modulefile like simd/intelmpi if the compiler is already in the user's path) in your user's start up dot files or similar. We also note that there is no additional intelligence in the version defaulting --- in the last example, we have gnu/8.2.0 loaded and if we try to load openmpi without specifying a version, it defaults to version 4.0 as that is the latest version of openmpi without regard for the fact that there is no build of openmpi version 4.0 for gnu/8.2.0 (but there is such for openmpi/3.1).

The situation for foo is more complicated, as it depends both on the compiler and optionally on the MPI library. But with Flavours, the modulefile is only slightly more complicated, e.g. for the common file is:

# Common stuff for "foo" modulefiles
# Using "flavours" strategy
#
# Expects the following Tcl variables to have been previously set:
#	version: version of foo

# Initialise "flavours"
package require flavours
flavours init

proc ModulesHelp { } {
   global version
   puts stderr "
foo: Test dummy version of foo $version

For testing packages depending on compilers/MPI

"
}

module-whatis "Dummy foo $version"

# Construct flavour name
flavours prereq -class compiler
flavours prereq -optional -class mpi
flavours conflict foo

# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root

flavours root     $swroot/foo/$version
flavours revision 1
flavours commit

# Set environment variables
setenv FOO_DIR [flavours path]

# Prepend to environment variables (paths relative to
# the directory containing the flavour)
flavours prepend-path PATH            bin
flavours prepend-path LIBRARY_PATH    lib
flavours prepend-path LD_LIBRARY_PATH lib
flavours prepend-path CPATH           include

# Reload any modules with this as a prerequisite
flavours cleanup

Basically, the main difference is the addition of the line flavours prereq -optional -class mpi. This instructs Flavours that there is an additional, optional prereq. The order of the prereq lines matter, as that controls the resultant flavors path. With the current configuration, assuming gnu/9.1.0 and openmpi/4.0 were loaded, the path would become $swroot/foo/2.4/1/gnu-9.1.0-openmpi-4.0. If the order were reversed, the openmpi-4.0 would precede the gnu-9.1.0. Because the MPI requirement is optional, if gnu/9.1.0 was loaded and no MPI library loaded, the path would evaluate to $swroot/foo/2.4/1/gnu-9.1.0.

We show how it works below:

[mod4-flavours]$ module purge
[mod4-flavours]$ module load pgi/19.4
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (pgi/19.4, nompi)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module load openmpi/3.1
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/3.1   3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (pgi/19.4, openmpi/3.1)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module unload openmpi
[mod4-flavours]$ module switch pgi intel/2019
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (intel/2019, nompi)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module load intelmpi
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) intelmpi/default   3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (intel/2019, intelmpi)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module switch intelmpi mvapich/2.3.1
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) mvapich/2.3.1   3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (intel/2019, mvapich/2.3.1)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module switch mvapich openmpi/4.0
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0   3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (intel/2019, openmpi/4.0)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module unload openmpi
[mod4-flavours]$ module switch intel/2019 gnu/9.1.0
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) gnu/9.1.0   2) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module load mvapich/2.3.1
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) gnu/9.1.0   2) mvapich/2.3.1   3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (gcc/9.1.0, mvapich/2.3.1)
[mod4-flavours]$ module unload foo
[mod4-flavours]$ module switch mvapich openmpi/4.0
[mod4-flavours]$ module load foo/2.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) gnu/9.1.0   2) openmpi/4.0   3) foo/2.4
[mod4-flavours]$ foo
foo 2.4 (gcc/9.1.0, openmpi/4.0)

So basically, if the user loads a compiler, the the environment variables (PATH, etc) are set up for the correct build of foo. If no MPI library was loaded, a version of foo built without MPI will be loaded, otherwise, a version of foo built with the loaded MPI library will be loaded. This is shown by the output of the foo command. Note also how we use the dummy intelmpi package to indicate a desire for the intelmpi enabled version.

The 3.x version of Environment Modules supports using the switch command on either the compiler or MPI library, and will result in reloading of foo and the MPI library.

[mod3-flavours]$ module purge
[mod3-flavours]$ module load pgi/18.4
[mod3-flavours]$ module load openmpi/3.1
[mod3-flavours]$ module load foo/1.1
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
  1) pgi/18.4      2) openmpi/3.1   3) foo/1.1
[mod3-flavours]$ foo
foo 1.1 (pgi/18.4, openmpi/3.1)
[mod3-flavours]$ module switch pgi intel/2018
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
  1) intel/2018    2) openmpi/3.1   3) foo/1.1
[mod3-flavours]$ foo
foo 1.1 (intel/2018, openmpi/3.1)
[mod3-flavours]$ mpirun
mpirun (openmpi/3.1, intel/2018)
[mod3-flavours]$ module purge
[mod3-flavours]$ module load intel/2019
[mod3-flavours]$ module load foo
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
  1) intel/2019   2) foo/2.4
[mod3-flavours]$ foo
foo 2.4 (intel/2019, nompi)
[mod3-flavours]$ module load openmpi
[mod3-flavours]$ module list
Currently Loaded Modulefiles:
  1) intel/2019    2) foo/2.4       3) openmpi/4.0
[mod3-flavours]$ foo
foo 2.4 (intel/2019, openmpi/4.0)

In particular note the final case, wherein we load intel/2019 then foo, and get the version of foo built without MPI. When we subsequently load openmpi, foo is reloaded to be the openmpi version (this is because the hooks to reload foo are in the flavours cleanup part of the openmpi modulefile, and foo declared its optional dependency on MPI). Also, we don't bother showing it, but if you were to attempt to load foo without at least a compiler loaded, it would display an error.

Our final example for flavours is the bar command. Here in addition to the compiler dependency, we have versions for different SIMD vectorization supported. Again, the difference in the modulefile is small, e.g.

# Common stuff for "bar" modulefiles
# Using "flavours" strategy
#
# Expects the following Tcl variables to have been previously set:
#	version: version of bar

# Initialise "flavours"
package require flavours
flavours init

proc ModulesHelp { } {
   global version
   puts stderr "
bar: Test dummy version of bar $version

For testing packages depending on compilers/MPI

"
}

module-whatis "Dummy bar $version"

# Construct flavour name
flavours prereq -class compiler
flavours prereq simd
flavours conflict bar

# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root

flavours root     $swroot/bar/$version
flavours revision 1
flavours commit

# Set environment variables
setenv BAR_DIR [flavours path]

# Prepend to environment variables (paths relative to
# the directory containing the flavour)
flavours prepend-path PATH            bin
flavours prepend-path LIBRARY_PATH    lib
flavours prepend-path LD_LIBRARY_PATH lib
flavours prepend-path CPATH           include

# Reload any modules with this as a prerequisite
flavours cleanup

Basically, the optional flavours prereq on the mpi class from the foo package is replaced by a (mandatory) flavours prereq on the simd dummy package. We note that Flavours package knows nothing about our simd dummy package until we add it as a prereq for bar. (This is in contrast to the compiler and mpi classes). Usage would be like:

[mod4-flavours]$ module purge
[mod4-flavours]$ module load gnu/9.1.0
[mod4-flavours]$ module load simd/avx2
[mod4-flavours]$ module load bar/5.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) gnu/9.1.0   2) simd/avx2   3) bar/5.4
[mod4-flavours]$ bar
bar 5.4 (gcc/9.1.0, avx2)
[mod4-flavours]$ module unload bar
[mod4-flavours]$ module switch simd simd/avx
[mod4-flavours]$ module load bar/5.4
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) gnu/9.1.0   2) simd/avx   3) bar/5.4
[mod4-flavours]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4-flavours]$ module unload bar
[mod4-flavours]$ module switch simd simd/sse4.1
[mod4-flavours]$ module load bar/5.4
bar/5.4 - no flavour compatible with modules 'gnu/9.1.0 simd/sse4.1'

Loading bar/5.4
  ERROR: Module evaluation aborted
[mod4-flavours]$ module list
Currently Loaded Modulefiles:
 1) gnu/9.1.0   2) simd/sse4.1

Here we note that as both the compiler and simd prereqs are non-optional, it complains unless both have been previously loaded. When both have been loaded, the PATH and other environment variables are set appropriately for the requested build; and if it does not exist and error is produced.

Summary of Flavours
  • It is an external extension to Environment Modules, requiring additional installation steps.
  • The git repository appears to have been last updated in 2013; although which means that it has not been updated for Environment Modules 4.x, a simple experimentation indicates that it still works, with the exception of automatic reloading of a module if any of the modules it depends on are switched. However, the Flavours extension does not appear to be actively supported.
  • The Flavours package (using Environment Modules 3.x) fully supports the module switch syntax, with the switching out of a dependency (e.g. a compiler) causing the reload of all modulefiles depending on it. (however the test of this feature failed when using Environment Modules 4.x.)
  • The syntax for modulefiles is elegant, and one can easily extend the basic compiler dependency modulefile to add additional dependencies. Even for packages/dummy packages that the Flavours extension knows nothing about (e.g. simd in the above example).
  • The Flavours package will provide a shorter module avail output, only e.g. giving package name and version and not listing a separate modulefile for each combination of package, version, compiler+version, MPI library+version, etc.
  • The Flavours package will fail with an error message if user tries to load a package which was not built for the values of the dependency packages loaded.
  • The Flavours package will fail to load with an error message if any dependent package is not already loaded. In particular, it will not attempt to default these. So if such defaults are desired, you will need to have initialization scripts automatically load the appropriate modules.
  • The Flavours package does not include any mechanism for more intelligent defaulting. I.e., if an user requests to load a package without specifying the version desired, the version will be defaulted to the latest version (or whatever the .modulerc file specifies) without regard for which versions support the versions of the compiler and other prereq-ed packages the user has loaded. While one could write custom .modulerc files for such, Flavours does not provide any tools for simplifying such.
Homebrewed Flavors Strategy

Although the Flavours extension described above has an elegance about it, one can achieve much of the same functionality in modulefiles using standard Environment Modules and Tcl commands. This can be facilitated by the definition of some useful Tcl procedures. For lack of a better name, we will refer to this strategy as Homebrewed flavors.

Implementation

This strategy just makes use of standard Environment Modules and Tcl procedures to query what modules of a given type are loaded and to construct the path to the software package accordingly. To avoid needless (and error prone) repetition of code, we collect these into several Tcl procedures of our own. Ideally, these should be placed in a site configuration Tcl file and exposed to modulefiles as explained in the cookbook Expose procedures and variables to modulefiles. However, to avoid the need for that in these examples, we instead have placed them into a file and use the MOD_GIT_ROOTDIR to locate and source that file in the relevant modulefiles. (Actually, we have a single tcl file that is sourced both for this and some other strategies, and it sources several files so that we can break up the discussion of the the Tcl procedures. All of that is just for the purposes of this cookbook; normally you just put the procedures you need in the one site config file).

We discuss the various Tcl procedures here, as they are what provide most of the functionality. We start with the routines for generic loaded modules:

#--------------------------------------------------------------------
# GetLoadedModules
#
# Returns a tcl list of all modules loaded.  From $ENV{LOADEDMODULES}
proc GetLoadedModules { } {
   global env
   #Handle case if no modules loaded
   if { [info exists env(LOADEDMODULES)] == 0 } {
      #No modules loaded, return empty list
      return [ list ]
   }
   set loadedenv $env(LOADEDMODULES)
   set loaded [ split $loadedenv : ]
   return $loaded
}

#--------------------------------------------------------------------
# GetTagOfModuleLoaded(pkg)
#
# Looks for a loaded module matching ^$pkg/, and returns the tag matched.
# Returns {} if no tags matched (i.e. module pkg not loaded)
proc GetTagOfModuleLoaded { mymodule } {
   set loadedlist [ GetLoadedModules ]
   set regex "^$mymodule/"
   set fndidx [ lsearch -regex $loadedlist $regex ]
   if {  $fndidx == -1 }  { return {} }
   set found [ lindex $loadedlist $fndidx ]
   return $found
}

This defines the two Tcl procedures:

  • GetLoadedModules : this returns the list of loaded modules, from the LOADEDMODULES
  • GetTagOfModuleLoaded : this takes as argument the base name of a package, and returns the first full spec for the matching package, or an empty string if no matching package found.

The Tcl procedure GetTagOfModuleLoaded can be used to find out what version of a given package is loaded, and is enough for many packages. However, for compilers, and similar, a bit more is needed. For compilers:

#--------------------------------------------------------------------
# GetDefaultCompiler:
#
# Returns the default compiler, gcc/8.2.0
proc GetDefaultCompiler { } {
   return "gcc/8.2.0"
}

#--------------------------------------------------------------------
# RequireCompiler:
#
# Does a module load of specified compiler $mycomp.
# Includes special handling if $mycomp is the default compiler
proc RequireCompiler { mycomp } {
   # If your module tree is set up so that there is no module for the
   # default compiler (because e.g. it is available w/out loading a module
   # anyway), you can uncomment the following block which will cause
   # RequireCompiler to do nothing if mycomp is the default compiler
   #set defComp [GetDefaultCompiler]
   #if { $mycomp eq $defComp } {
      #return
   #}
   module load $mycomp
}

#--------------------------------------------------------------------
# GetKnownCompilerFamilies:
#
# Returns a list of recognized compiler family names
#E.g. gcc, intel, pgi
proc GetKnownCompilerFamilies { } {
   set cfamilies {gcc intel pgi}
   return $cfamilies
}

#--------------------------------------------------------------------
# GetLoadedCompiler:
#
# Returns the string for the compiler we are using (i.e. was previously
# module loaded).  E.g., gcc/8.2.0
# If no compiler was previously loaded, then if the optional parameter
# $pathDefault is set, it will look for a compiler family and
# version in the last components to the path to the current modulefile or
# .modulerc, and if found, uses that.
# If still no path found, it will return the value of [GetDefaultCompiler]
# if $useDefault is set.
# Otherwise, returns empty string.
#
# Takes the following arguments:
#	pathDefault: boolean, default false.  If set, attempt to determine
#		the compiler from the full path to the modulefile if
#		no compiler was loaded.
#	useDefault: boolean, default false.  If set, return the value of
#		GetDefaultCompiler if no compiler is loaded or found from
#		path (if $pathDefault).
#	loadIt: boolean, default false.  If set and a compiler
#		was defaulted from path of GetDefaultCompiler, we will
#		module load that compiler.
#		Ignored unless either pathDefault or useDefault is set
#	requireIt: boolean, default false.  If set, we will prereq the
#		compiler before returning.
proc GetLoadedCompiler {{pathDefault 0} { useDefault 0}
   {loadIt 0 } { requireIt 0 } } {
   global ModulesCurrentModulefile
   set ctag {}
   set cfams [ GetKnownCompilerFamilies ]
   foreach cfam $cfams {
      if { [ is-loaded $cfam ] } {
         set ctag [ GetTagOfModuleLoaded $cfam  ]
         if { $requireIt } { prereq $ctag }
         return $ctag
      }
   }

   # No loaded compiler found, try to default from path to modulefile?
   if { $pathDefault} {
      set moduledir [file dirname $ModulesCurrentModulefile ]
      set cversion [file tail $moduledir]
      set tmppath [file dirname $moduledir]
      set cfamily [file tail $tmppath]

      if { [lsearch $cfams $cfamily] > -1 } {
         # We matched a known compiler family in our path
         set ctag "$cfamily/$cversion"
         if { $loadIt } { RequireCompiler $ctag }
         if { $requireIt } { prereq $ctag }
         return $ctag
      }
   }

   # Still no compiler, default to GetDefaultCompiler>
   if { $useDefault } {
      set ctag [ GetDefaultCompiler ]
      if { $loadIt } { RequireCompiler $ctag }
      if { $requireIt } { prereq $ctag }
      return $ctag
   }

   #Nothing found, and not defaulting
   return $ctag
}

We defined four procedures above:

  • GetDefaultCompiler : this simply returns the name of our default compiler, which for this example is gcc/8.2.0
  • RequireCompiler : this simply does a module load on the specified compiler. It is kept as a separate procedure just in case you wish to intercept and prevent the loading of the default compiler (e.g. because no modulefile exists for it). In our example, there is a modulefile for it and so it is just a wrapper for module load.
  • GetKnownCompilerFamilies : this simply returns a Tcl list of known compiler families.
  • GetLoadedCompiler: this is the procedure that does the main work, and is described in detail below.

The GetLoadedCompiler procedure basically checks if any packages matching the names in GetKnownCompilerFamilies have been previously loaded. If so, it returns the modulefile specification for the first one found, and returns. If not, if pathDefault is set and there is a recognized compiler name and version in the last two components of the module specification, it will return that compiler. Otherwise, if the optional flag useDefault is set, it will return the value from GetDefaultCompiler. If all else fails, returns the empty string.

If the optional parameter loadIt is set, if a compiler was defaulted (i.e. not returned because it was already loaded), the procedure will call RequireCompiler to module load it.

If the optional parameter requireIt is set, we invoke prereq on the compiler found before returning.

A similar set of procedures exist for the MPI libraries, namely:

#--------------------------------------------------------------------
# RequireMPI:
#
# Does a module load of specified MPI library $mympi
# Includes special handling if $mympi is nompi or intelmpi (or one of its aliases)
# If $mympi is nompi, nothing is loaded.
# If the optional parameter noLoadIntel is set (default false), and if
# $mympi is intelmpi (or intel or impi), then we do not load module if
# the loaded compiler is intel (as we assume that provides intel MPI as well)
proc RequireMPI { mympi {noLoadIntel 0} } {
   # We do not do anything if mympi is nompi
   if { $mympi eq {nompi} } { return }
   if { $noLoadIntel } {
      # Get the basename of requested MPI library
      set mympiSplit [ split $mympi / ]
      set mympiBase [ lindex $mympiSplit 0 ]
      # Check if we requested an intel MPI
      set intelList "intelmpi impi intel intelmpi-mt impi-mt intel-mt"
      if { [lsearch $intelList $mympiBase ] > -1 } {
         # We requested intelmpi in some form
         # Check if an intel compiler was loaded
         set curComp [ GetLoadedCompiler 1]
         if { $curComp ne {} } {
            set curCompSplit [ split $curComp / ]
            set curCompBase [ lindex $curCompSplit 0 ]
            if { $curCompBase eq {intel} } {
               # $noLoadIntel is set, requested MPI is intel MPI, and intel compiler loaded
               return
            }
         }
      }
   }
   module load $mympi
}

#--------------------------------------------------------------------
# GetKnownMpiFamilies:
#
# Returns a list of recognized MPI library family names
#E.g. gcc, intel, pgi
proc GetKnownMpiFamilies { } {
   set mfamilies {openmpi mpavich intelmpi}
   return $mfamilies
}

#--------------------------------------------------------------------
# GetLoadedMPI:
#
# Returns the string for the MPI library we are using (i.e. was previously
# module loaded), or empty string if nothing loaded (from is-loaded command)
# E.g., intel/2013.1.117
# Takes an optional argument,
#	useIntel: boolean, default false.  If set, returns 'intelmpi' if
#		no MPI library is loaded but intel compiler is loaded
#	forceIt: boolean, default false.  If set, prereq MPI lib before returning.
#	requireIt: boolean, default false.  If set, prereq the MPI library
proc GetLoadedMPI { { useIntel 0} {forceIt 0} {requireIt 0} } {
   set mtag {}
   foreach mfam [ GetKnownMpiFamilies ] {
      if { [ is-loaded $mfam ] } {
         set mtag [ GetTagOfModuleLoaded $mfam  ]
         if { $requireIt } { prereq $ctag }
         return $mtag
      }
   }
   # No loaded compiler found, should we check for Intel compiler and return intelmpi?
   if { $useIntel } {
      #Yes
      set ctag [ GetCompilerLoaded ]
      set cSplit [ split $ctag / ]
      set cBase [ lindex $cSplit 0 ]
      if { $cBase eq intel } { return intelmpi }
   }
   return $mtag
}

The three procedures here are analogues of the compiler versions:

  • RequireMPI : this basically does a module load of the specified MPI library. It has some added logic so that it will not do a module load if the MPI library is nompi. Also, if the optional parameter noLoadIntel is set, if the MPI library is intelmpi (or a variant of that name) and the loaded compiler in intel, we assume that no additional module needs to be loaded. For this strategy, we want to load intelmpi modules, because, just like in the Flavours Strategy, we need to provide dummy intelmpi modules to allow one to request the use of the Intel MPI library.
  • GetKnownMpiFamilies : this returns a list of known MPI library family names. Used in GetLoadedMPI
  • GetLoadedMPI : This is the analogue of GetLoadedCompiler. If an MPI library is loaded, it will return the name of that module. If the optional requireIt flag is set, it will do a prereq on the MPI library before returning. The first optional argument, useIntel, indicates whether this module should return intelmpi if no MPI library is loaded but an Intel compiler is loaded.

The modulefiles for the compilers are basically standard; unlike the Flavours Strategy there is nothing special needed in these. Likewise for the dummy simd and intelmpi modules (the latter is only this basic because we assume intelmpi is only available if an Intel compiler is loaded. If one allowed for intelmpi with other compilers, it would more closely resemble the other MPI libraries).

We also define some Tcl procedures for generating warning and error messages, namely

#--------------------------------------------------------------------
# PrintIfLoading:
#
# Prints supplied text to stderr but only if in "load" mode
proc PrintIfLoading { args } {
   if [ module-info mode load ] {
      set tmp [ join $args ]
      puts stderr "$tmp"
   }
}

#--------------------------------------------------------------------
# PrintLoadInfo:
#
# Prints supplied text to stderr as informational message, but only
# if actually trying to load the module.
proc PrintLoadInfo { args } {
   set tmp [ join $args ]
   PrintIfLoading "
\[INFO\] $tmp
"
}

#--------------------------------------------------------------------
# PrintLoadWarning:
#
# Prints supplied text to stderr as warning message, but only
# if actually trying to load the module.
proc PrintLoadWarning { args } {
   set tmp [ join $args ]
   PrintIfLoading "
WARNING:
$tmp

"
}

#--------------------------------------------------------------------
# PrintLoadError:
#
# Like PrintLoadWarning, but as error message and does a "break"
proc PrintLoadError { args } {
   set tmp [ join $args ]
   PrintIfLoading "
**** ERROR *****:
$tmp

"
   if [ module-info mode load ] {
      break
   }
}

These procedures:

  • PrintIfLoading: will print supplied text to stderr only when in load mode
  • PrintLoadInfo: will print supplied text as informational text, but only when trying to load a module
  • PrintLoadWarning: will print supplied text as warning text, but only when trying to load a module
  • PrintLoadError: will print supplied text as error text and abort, but only when trying to load a module

The gist of this is that we might wish to print errors if an user tries to load an incompatible modulefile, but do not wish to print errors if they are merely doing a help, display, or whatis command.

The interesting bit begins with the openmpi and mvapich modulefiles. These both depend on the compiler, we show the main part of the openmpi modulefile below:

# Common modulefile for openmpi
# Using "homebrewed flavors" strategy
# Expects the following variables to have been
# previously defined:
#	version: version of openmpi

# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root

# Also get location of and load common procedures
# This is a hack for the cookbook examples, in production
# one should either
# 1) declare the procedures in a site config file (preferred)
# 2) hardcode the path to $tcllibdir and common_utilities.tcl
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

proc ModulesHelp { } {
   global version
   puts stderr "
openmpi: Test dummy version of OpenMPI $version

For testing packages depending on compilers/MPI

"
}

module-whatis "Dummy openmpi $version"

# Figure out what compiler we have loaded
# Optional args ensure will default to and load/prereq default compiler
# if no compiler currently loaded
set ctag [ GetLoadedCompiler 1 1 1 ]

# Compute the installation prefix
set pkgroot $swroot/openmpi
set vroot $pkgroot/$version
set prefix $vroot/$ctag

# Make sure there is a build for this openmpi version/compiler
# I.e. that the prefix exists
if ![ file exists $prefix ] {
   # Not built for this compiler, alert user and abort
   PrintLoadError "
openmpi/$version does not appear to be built for compiler $ctag

Please select a different openmpi version or different compiler.
"
}

# We need to prereq the compiler to allow autohandling to work
prereq $ctag

# Set environment variables
setenv MPI_DIR $prefix

set bindir $prefix/bin
set libdir $prefix/lib
set incdir $prefix/include

prepend-path PATH            $bindir
prepend-path LIBRARY_PATH    $libdir
prepend-path LD_LIBRARY_PATH $libdir
prepend-path CPATH           $incdir

We begin by sourcing the common_utilities file which defined the previously described Tcl procedures. Normally it is recommended that you put those procedures in a site config Tcl script and expose them to the modulefiles using the techniques described in the cookbook Expose procedures and variables to modulefiles. Even if you opt against that and decide to source a Tcl file, it is recommended to hard code the path.

The next interesting bit comes when we set the local Tcl variable ctag by calling the GetLoadedCompiler procedure. We allow the procedure to default the compiler, and because we have a default compiler defined we should always get a value. (If no default compiler was defined, one would have to handle the error if no compiler was loaded/defaulted.) We then use the value of ctag to set the path to the build of the package. To ensure that the package is built for this compiler, we do a quick check that the package installation path exists.

The modulefile for foo is a bit more complex:

# Common stuff for "foo" modulefiles
# Using "homebrewed flavors" strategy
#
# This file expects the following Tcl variables to have been
# previously set:
#	version: version of foo

# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root

# Also get location of and load common procedures
# This is a hack for the cookbook examples, in production
# one should either
# 1) declare the procedures in a site config file (preferred)
# 2) hardcode the path to $tcllibdir and common_utilities.tcl
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

proc ModulesHelp { } {
   global version
   puts stderr "
foo: Test dummy version of foo $version

For testing packages depending on compilers/MPI

"
}

module-whatis "Dummy foo $version"

# Figure out what compiler we have loaded
# Will default to and load default compiler if none loaded
set ctag [ GetLoadedCompiler 1 1 1 ]

# Figure out what MPI we have loaded
# We do NOT want to default to intelmpi if intel compiler loaded
set mtag [ GetLoadedMPI ]
# Set mtag it nompi if no MPI library found loaded
if { $mtag eq {} } { set mtag nompi }
# Set mtag to intelmpi if intelmpi/default found loaded
if { $mtag eq {intelmpi/default} } { set mtag intelmpi }

# Compute the installation prefix
set pkgroot $swroot/foo
set vroot $pkgroot/$version
set prefix $vroot/$ctag/$mtag

# Make sure there is a build for this foo version/compiler/MPI library
# I.e. that the prefix exists
if ![ file exists $prefix ] {
   # Not built for this compiler/MPI, alert user and abort
   PrintLoadError "
foo/$version does not appear to be built for compiler $ctag and MPI $mtag

Please select a different openmpi version or different compiler/MPI library combination.
"
}

# We need to prereq the compiler to allow autohandling to work
prereq $ctag
# and the MPI library if used
if { $mtag ne {nompi} } {
   # We currently require one to load intelmpi to use Intel MPI with
   # intel compilers. So we prereq on intelmpi as well
   # But if not, we could place the prereq below in an if-then block
   # so is only executed if mtag != intelmpi
   prereq $mtag
}

# Set environment variables
setenv FOO_DIR $prefix

set bindir $prefix/bin
set libdir $prefix/lib
set incdir $prefix/include

prepend-path PATH            $bindir
prepend-path LIBRARY_PATH    $libdir
prepend-path LD_LIBRARY_PATH $libdir
prepend-path CPATH           $incdir

conflict foo

The main difference between this modulefile, depending on both compiler and optionally MPI, and the openmpi modilefile above, is that in addition to detecting which compiler is loaded, we call GetLoadedMPI to determine the MPI library which was loaded, and use both of them in constructing the prefix to the installed foo.

Examples

We now look at the example modulefiles for the Homebrewed flavors strategy. To use the examples, you must #. Set (and export) MOD_GIT_ROOTDIR to where you git-cloned the modules source #. Do a module purge, and then set your MODULEPATH to:

$MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/homebrewed

The Homebrewed flavors strategy behaves much like the Flavours Strategy in practice. The module avail command,

[mod4 (homebrewed)]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/homebrewed -
bar/4.7  foo/2.4    intel/2018        mvapich/2.1    openmpi/4.0  simd/avx
bar/5.4  gcc/8.2.0  intel/2019        mvapich/2.3.1  pgi/18.4     simd/avx2
foo/1.1  gcc/9.1.0  intelmpi/default  openmpi/3.1    pgi/19.4     simd/sse4.1

looks basically the same, showing the a concise listing of packages and versions without information on the compilers and MPI libraries they were built with.

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load pgi/19.4
[mod4 (homebrewed)]$ module load openmpi/4.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/4.0
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (homebrewed)]$ module unload openmpi
[mod4 (homebrewed)]$ module switch --auto pgi intel/2019
[mod4 (homebrewed)]$ module load openmpi/4.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4 (homebrewed)]$ module unload openmpi
[mod4 (homebrewed)]$ module switch --auto intel gcc/9.1.0
[mod4 (homebrewed)]$ module load openmpi/4.0
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, gcc/9.1.0)
[mod4 (homebrewed)]$ module unload openmpi
[mod4 (homebrewed)]$ module switch --auto gcc gcc/8.2.0
[mod4 (homebrewed)]$ module load openmpi/4.0

**** ERROR *****:
openmpi/4.0 does not appear to be built for compiler gcc/8.2.0
Please select a different openmpi version or different compiler.


Loading openmpi/4.0
  ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0

Again, once a compiler is loaded, loading openmpi will set the PATH, etc. for the correct build of openmpi, as evidenced by the output of the dummy mpirun command. Notice that the module list command only shows the version of openmpi loaded, and contains no information about what compiler it was built with (you just have to assume it matches the loaded compiler). And again we note that if one attempts to load a version of openmpi (e.g. 4.0) that was not built for the specified compiler (e.g. gcc/8.2.0), an error is generated.

Unlike in Flavours Strategy, we did not put any code in the modulefiles to cause dependent modulefiles to be reloaded if a module they depend on gets switched out. However, starting with Environment Modules 4.2, a feature called Automated module handling was added. Without this feature, attempting to switch out a module upon which other modules depended could be problematic, as evidenced in this sequence below (using Environment Modules 3.2.10 and so without automated module handling mode):

[mod3 (homebrewed)]$ module purge
[mod3 (homebrewed)]$ module load pgi/19.4
[mod3 (homebrewed)]$ module load openmpi
[mod3 (homebrewed)]$ module list
Currently Loaded Modulefiles:
  1) pgi/19.4      2) openmpi/4.0
[mod3 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod3 (homebrewed)]$ module switch pgi intel/2019
[mod3 (homebrewed)]$ module list
Currently Loaded Modulefiles:
  1) intel/2019    2) openmpi/4.0
[mod3 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)

Here we note that we were able to switch out the pgi compiler for the intel compiler, but the openmpi module was not reloaded and the environment is still set for openmpi compiled with the pgi compiler, and that this inconsistency is not readily determined from the module list command.

Environment Modules 4.x, even with automated module handling disabled, is better --- in a command sequence as above the module switch from pgi to intel would fail due to the prereq module. However, with the Automated module handling enabled (this feature is disabled by default but could be enabled on a per-command basis with the --auto flag), things work much better, as evidenced below:

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load pgi/19.4
[mod4 (homebrewed)]$ module load openmpi
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/4.0
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (homebrewed)]$ module switch --auto pgi intel/2019
Switching from pgi/19.4 to intel/2019
  Unloading dependent: openmpi/4.0
  Reloading dependent: openmpi/4.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4 (homebrewed)]$ module switch --auto intel intel/2018

**** ERROR *****:
openmpi/4.0 does not appear to be built for compiler intel/2018
Please select a different openmpi version or different compiler.


Loading openmpi/4.0
  ERROR: Module evaluation aborted

Switching from intel/2019 to intel/2018
  WARNING: Reload of dependent openmpi/4.0 failed
  Unloading dependent: openmpi/4.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2018
[mod4 (homebrewed)]$ mpirun
mpirun: command not found

When we switch out the pgi compiler for the intel/2019 compiler, the openmpi module is automatically unloaded before the compiler switch and reload afterwards, so we end up with the correct build of openmpi for the newly loaded intel/2019 compiler. If one then switches out intel/2019 replacing it with intel/2018, the openmpi module is first unloaded, the compilers are switched, and as there is no openmpi/4.0 build for intel/2018, a warning is given and the openmpi module is left unloaded. Since the user never specifically requested version 4.0 of openmpi (it was defaulted in the original module load of openmpi as that was the latest version available for pgi/19.4), it would have been nicer if on the switch of intel compiler versions the reload only attempted a module load openmpi instead of module load openmpi/4.0, but nevertheless this well behaved. The openmpi module is dropped with a warning and the user has a consistent set of modules loaded.

We note that the modulefile is able to default the compiler, so when we attempt to load openmpi without having previously loaded a compiler, as in

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load openmpi/3.1
Loading openmpi/3.1
  Loading requirement: gcc/8.2.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) openmpi/3.1
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/3.1, gcc/8.2.0)
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load gcc/8.2.0
[mod4 (homebrewed)]$ module load openmpi

**** ERROR *****:
openmpi/4.0 does not appear to be built for compiler gcc/8.2.0
Please select a different openmpi version or different compiler.


Loading openmpi/4.0
  ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0

it will default to the default compiler, gcc/8.2.0. Note however, that if one does not specify version 3.1 of openmpi, it will still default to 4.0 and fail to load as there is no build of openmpi/4.0 for gcc/8.2.0.

The situation is similar for foo:

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load pgi/19.4
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (pgi/19.4, nompi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module load openmpi/3.1
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/3.1   3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (pgi/19.4, openmpi/3.1)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module unload openmpi
[mod4 (homebrewed)]$ module switch --auto pgi intel/2019
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module load intelmpi
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) intelmpi/default   3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, intelmpi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module switch --auto intelmpi mvapich/2.3.1
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) mvapich/2.3.1   3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module switch --auto mvapich openmpi/4.0
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0   3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, openmpi/4.0)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module unload openmpi
[mod4 (homebrewed)]$ module switch --auto intel/2019 gcc/9.1.0
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module load mvapich/2.3.1
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) mvapich/2.3.1   3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4 (homebrewed)]$ module unload foo
[mod4 (homebrewed)]$ module switch --auto mvapich openmpi/4.0
[mod4 (homebrewed)]$ module load foo/2.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) openmpi/4.0   3) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (gcc/9.1.0, openmpi/4.0)

Again, one can load a compiler without an MPI library to get the non-MPI version of foo, or a compiler and MPI library to get the MPI version. The dummy intelmpi modulefile is used to allow one to indicate that the Intel MPI library is desired. The automated module handling mode can again allow the switch functionality work properly, as in

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load pgi/18.4
[mod4 (homebrewed)]$ module load openmpi/3.1
[mod4 (homebrewed)]$ module load foo/1.1
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) pgi/18.4   2) openmpi/3.1   3) foo/1.1
[mod4 (homebrewed)]$ foo
foo 1.1 (pgi/18.4, openmpi/3.1)
[mod4 (homebrewed)]$ module switch --auto pgi intel/2018
Switching from pgi/18.4 to intel/2018
  Unloading dependent: foo/1.1 openmpi/3.1
  Reloading dependent: openmpi/3.1 foo/1.1
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2018   2) openmpi/3.1   3) foo/1.1
[mod4 (homebrewed)]$ foo
foo 1.1 (intel/2018, openmpi/3.1)
[mod4 (homebrewed)]$ mpirun
mpirun (openmpi/3.1, intel/2018)
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load intel/2019
[mod4 (homebrewed)]$ module load foo
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) foo/2.4
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (homebrewed)]$ module load openmpi
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) foo/2.4   3) openmpi/4.0
[mod4 (homebrewed)]$ foo
foo 2.4 (intel/2019, nompi)

Here we note a deficiency in the switch support as compared to Flavours Strategy. In the last example after loading intel/2019 and foo, we have the non-MPI build of foo as expected. However, upon subsequently loading the openmpi module, we still have the non-MPI version of foo loaded, as evidenced by the output of the dummy foo command. I.e., the foo package was not automatically reloaded, as there was no prereq in the foo modulefile on an MPI library (as in the non-MPI build there is no MPI library to prereq). Also note that module list does not really inform one of this fact.

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load foo

**** ERROR *****:
foo/2.4 does not appear to be built for compiler gcc/8.2.0 and MPI nompi
Please select a different openmpi version or different compiler/MPI library combination.


Loading foo/2.4
  ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
No Modulefiles Currently Loaded.
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load foo/1.1
Loading foo/1.1
  Loading requirement: gcc/8.2.0
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) foo/1.1
[mod4 (homebrewed)]$ foo
foo 1.1 (gcc/8.2.0, nompi)
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load pgi/18.4
[mod4 (homebrewed)]$ module load foo

**** ERROR *****:
foo/2.4 does not appear to be built for compiler pgi/18.4 and MPI nompi
Please select a different openmpi version or different compiler/MPI library combination.


Loading foo/2.4
  ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) pgi/18.4

Above, we see once more that the compiler can be defaulted, but that the defaulting mechanism is not smart enough to default the version of foo based on the compiler loaded (or defaulted to).

The situation with bar is basically the same; with a compiler and simd module loaded, the environment for the appropriate build of bar is loaded when you module load bar.

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load gcc/9.1.0
[mod4 (homebrewed)]$ module load simd/avx2
[mod4 (homebrewed)]$ module load bar/5.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) simd/avx2   3) bar/5.4
[mod4 (homebrewed)]$ bar
bar 5.4 (gcc/9.1.0, avx2)
[mod4 (homebrewed)]$ module unload bar
[mod4 (homebrewed)]$ module switch --auto simd simd/avx
[mod4 (homebrewed)]$ module load bar/5.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) simd/avx   3) bar/5.4
[mod4 (homebrewed)]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4 (homebrewed)]$ module unload bar
[mod4 (homebrewed)]$ module switch --auto simd simd/sse4.1
[mod4 (homebrewed)]$ module load bar/5.4

**** ERROR *****:
foo/5.4 does not appear to be built for compiler gcc/9.1.0 and simd/sse4.1
Please select a different openmpi version or different compiler/simd combination.


Loading bar/5.4
  ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) simd/sse4.1

And an error is generated if there is no build for that combination of compiler and simd. The automatic handling of modules again allows the switch command to work as expected:

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load gcc/9.1.0
[mod4 (homebrewed)]$ module load simd/avx
[mod4 (homebrewed)]$ module load bar/5.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) simd/avx   3) bar/5.4
[mod4 (homebrewed)]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4 (homebrewed)]$ module switch --auto simd simd/avx2
Switching from simd/avx to simd/avx2
  Unloading dependent: bar/5.4
  Reloading dependent: bar/5.4
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) simd/avx2   3) bar/5.4
[mod4 (homebrewed)]$ bar
bar 5.4 (gcc/9.1.0, avx2)

and both the simd level and compiler can be defaulted, but one still has to choose a version of bar which supports the defaults.

[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load bar

[INFO] Setting simd to simd/sse4.1

**** ERROR *****:
foo/5.4 does not appear to be built for compiler gcc/8.2.0 and simd/sse4.1
Please select a different openmpi version or different compiler/simd combination.


Loading bar/5.4
  ERROR: Module evaluation aborted
[mod4 (homebrewed)]$ module list
No Modulefiles Currently Loaded.
[mod4 (homebrewed)]$ module purge
[mod4 (homebrewed)]$ module load bar/4.7

[INFO] Setting simd to simd/sse4.1

Loading bar/4.7
  Loading requirement: gcc/8.2.0 simd/sse4.1
[mod4 (homebrewed)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) simd/sse4.1   3) bar/4.7
[mod4 (homebrewed)]$ bar
bar 4.7 (gcc/8.2.0, sse4.1)
Summary of Homebrewed flavors strategy
  • The Automated module handling feature (introduced in Environment Modules 4.2) allows for the switching out of a dependency (e.g. a compiler) to cause the reload of all modulefiles depending on it. Without the automated module handling (as is default for 4.x, and the only option for 3.x), the switching of a compiler only changes the compiler and leaves the modulefiles that depend on the compiler unchanged.
  • The various Tcl procedures make it somewhat easy to determine which compiler, MPI, etc. modules have been loaded and set the paths appropriately. Not as elegant or easy to use as Flavours Strategy, but not difficult
  • Like the Flavours package, the module avail output is concise, only e.g. giving package name and version rather than listing a separate modulefile for each combination of package, version, compiler+version, MPI library+version, etc.
  • Modules will fail with an error message if user tries to load a package which was not built for the values of the dependency packages loaded.
  • With Automated module handling mode, the dependencies of a module asked for load could automatically be loaded. But it requires that these dependencies are clearly specified with mention of the versions for which a build is available.
  • It does not include a mechanism for more intelligent defaulting. I.e., if an user requests to load a package without specifying the version desired, the version will be defaulted to the latest version (or whatever the .modulerc file specifies) without regard for which versions support the versions of the compiler and other prereq-ed packages the user has loaded.
  • Note that future releases of Environment Modules will introduce additional mechanisms to the Automated module handling mode, which will improve the user experience on such Homebrewed flavors setup.
Modulerc-based Strategy

The previous two strategies used additional code in the modulefile to determine which compiler, etc. was loaded and adjust the values for PATH, etc. accordingly. The Modulerc-based strategy instead uses .modulerc files to direct the module command to the proper modulefile depending on what compiler, etc. was previously loaded. Because of this, there are a number of differences in behavior and what is seen by the user, most notably many, many more modulefiles. Whether this is good or bad is a matter of taste.

Implementation

Whereas the Homebrewed Flavors Strategy had the modulefile invoke a Tcl procedure to determine which, if any, version of a module like a compiler was loaded and then adjust paths, the Modulerc-based strategy instead uses the same Tcl procedures to default the modulefile which will be loaded. This implies that there is a distinct modulefile for every build of the package, and an immediate consequence is that this strategy has many more modulefiles than the others. We make use of the techniques in the cookbook Tips for Code Reuse in Modulefiles to reduce the total amount of code; the actual modulefiles for each build are typically small stubfiles defining a couple of Tcl variables and then sourcing a common script (unique to each package) which does all the real work.

The modules will be named with components for the different dependencies, so the one for openmpi version 4.0 built with gcc version 9.1.0 would be openmpi/4.0/gcc/9.1.0; similarly the module for foo version 1.1 built for pgi version 18.4 and mvapich 2.1 would be foo/1.1/pgi/18.4/mvapich/2.1.

The .modulerc files themselves are not trivial, but these can generally be written in a generic fashion, usable by multiple packages, and can just be symlinked to the appropriate locations. We define five such files which can be linked as .modulerc:

Two of these files can be linked in the module tree at various places as .modulerc for defaulting the compiler. One to default to the family portion of the compiler (e.g. gcc, intel, or pgi), and one for the version. For the family portion of the compiler, we have the file modulerc.select_compiler_family as below:

#%Module
# Choose compiler family
#
# This check if any compiler was previously "module loaded", or if no compiler
# previously loaded, will use the default compiler.  Either way, it will check
# if there is a subdirectory matching the compiler family name, and if so
# will default to that.
#
# If no subdirectory matching compiler family name is found, we just return
# without defaulting.  The modulecmd will default based on its internal rules,
# and it is up to the resulting module file to either load an appropriate
# compiler if no compiler is loaded or to abort with appropriate error
# messages if it wants an incompatible compiler.
#
# Usage:
# In most cases, can simply symlink .modulerc to this file
#
# In more complicated cases, .modulerc can source this file, and can
# then test the variable _did_default, which will be true if we set
# a default for modules for the next level, or false otherwise (in which
# case your .modulerc can set one)

# Source some required Tcl procedures here.  Hack for cookbook
# making use of environment variable for location.
# In production, this should ideally be in a site config file.
# At minimum, hardcode the path
set rootdir $::env(MOD_GIT_ROOTDIR)
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

set moduledir [file dirname $ModulesCurrentModulefile]
set baseComponent [ file tail $moduledir ]

# Get the currently loaded compiler, or default comp.  Don't module load/prereq it
set fullCompilerTag [ GetLoadedCompiler 1 1 ]
set tmpCompTag [ GetPackageFamilyVersion $fullCompilerTag ]
set compilerFamily [ lindex $tmpCompTag 0 ]
set compFamList "$compilerFamily"

# Allow either gnu or gcc for GCC compilers
if { $compilerFamily eq {gcc} || $compilerFamily  eq {gnu} } {
   set compFamList "gnu gcc"
}

# _did_default will be true if we actually default something
# Useful in case a .modulerc sources us, and wants to set a default if we
# did not
set _did_default false

if { $compilerFamily ne {} } {
   # Default to the family of currently loaded compiler
   set firstChild [ FirstChildModuleInList $compFamList ]
   if { $firstChild ne {} } {
      # Compiler family found, so default to it
      module-version $baseComponent/$firstChild default
      set _did_default true
   }
}

The file starts by sourcing a set of useful Tcl procedures. For the purpose of the example for this cookbook this is done based on the MOD_GIT_ROOTDIR environment variable. If you were to use this in production, it is recommended that the Tcl procedures be placed in a site configuration script and exposed to modulefiles via the techniques described in the cookbook Expose procedures and variables to modulefiles. At the minimum, it is recommended to hardcode the path to the common_utilities.tcl file.

The modulerc script then determines the directory it is in using the Tcl variable ModulesCurrentModulefile. It then uses the GetLoadedCompiler Tcl procedure (which was discussed in the section on the Homebrewed Flavors Strategy. We then parse the resulting module name into family and version pieces (we will discuss the procedure GetPackageFamilyVersion later; for now suffice to say it takes a module name and returns a Tcl list with family and version).

We do some trickery to support the use of either gcc or gnu as family names for the GNU compiler suite, and then see if there is a directory or modulefile underneath the directory containing the .modulerc file, and if so defaults to it. For this purpose, we use the Tcl procedure FirstChildModuleInList (defined in common_utilities.tcl). This and a related procedure are defined as:

#--------------------------------------------------------------------
# FirstChildModuleInList
#
# Given a list of names of modulefile components, returns the
# first one found in the directory of the current modulefile
# (typically a .modulerc file).  Returns empty string if none
# of them were found.
# NOTE: we do NOT check for module magic signature or even that symlink
# targets exist.
proc FirstChildModuleInList { modlist } {
   global ModulesCurrentModulefile
   # Get directory for current modulefile
   set moduledir [ file dirname $ModulesCurrentModulefile ]
   # See if any names in $modlist in moduledir
   foreach mcomp $modlist {
      if [ file exists $moduledir/$mcomp ] { return $mcomp }
   }
   # Nothing found
   return
}

#--------------------------------------------------------------------
# ChildModuleExists
#
# Takes a module path component, and returns true if that path component
# exists beneath the current level, i.e. in the directory from which
# this .modulerc file was called.
# NOTE: we do NOT check for module magic signature or even that symlink
# targets exist.
proc ChildModuleExists { pathcomponent } {
   global ModulesCurrentModulefile
   # Get directory for current modulefile
   set moduledir [ file dirname $ModulesCurrentModulefile ]
   # See if file exists in moduledir
   return [ file exists $moduledir/$pathcomponent ]
}

and basically make use of the Tcl variable ModulesCurrentModulefile. They are limited, however, in that they will not work properly if the module is split across multiple modulepaths. That is, ChildModuleExists and FirstChildModuleInList will only detect children in the same directory as the modulerc file they are called from; e.g., if two paths in the MODULEPATH both have directories for openmpi, one with an intel subdirectory and one with a pgi subdirectory and the modulerc invoking ChildModuleExists, the ChildModuleExists procedure will not see the intel directory.

The modulerc.select_compiler_version file is similar,

#%Module
# Choose compiler version
#
# This check if any compiler was previously "module loaded", or if no compiler
# previously loaded, will use the default compiler.  Either way, it will check
# if there is a subdirectory matching the compiler family name, and if so
# will default to that.
#
# If no subdirectory matching compiler family name is found, we just return
# without defaulting.  The modulecmd will default based on its internal rules,
# and it is up to the resulting module file to either load an appropriate
# compiler if no compiler is loaded or to abort with appropriate error
# messages if it wants an incompatible compiler.
#
# Usage:
# In most cases, can simply symlink .modulerc to this file
#
# In more complicated cases, .modulerc can source this file, and can
# then test the variable _did_default, which will be true if we set
# a default for modules for the next level, or false otherwise (in which
# case your .modulerc can set one)

# Source some required Tcl procedures here.  Hack for cookbook
# making use of environment variable for location.
# In production, this should ideally be in a site config file.
# At minimum, hardcode the path
set rootdir $::env(MOD_GIT_ROOTDIR)
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

set moduledir [file dirname $ModulesCurrentModulefile]
set parentdir [ file tail $moduledir ]

# _did_default will be true if we actually default something
# Useful in case a .modulerc sources us, and wants to set a default if we
# did not
set _did_default false

# Get the currently loaded compiler or default, but do not load/prereq it
set compilerTag [ GetLoadedCompiler 1 1 ]

if { $compilerTag ne {} } {
   # Compiler loaded or defaulted
   # Make sure it matches our parent dir
   set tmpCompTag [ GetPackageFamilyVersion $compilerTag ]
   set compilerFamily [ lindex $tmpCompTag 0 ]
   set compilerVersion [ lindex $tmpCompTag 1 ]

   # Convert any gnus to gccs
   set tmp1 $compilerFamily
   if { $tmp1 eq {gnu} } { set tmp1 gcc }
   set tmp2 $parentdir
   if { $tmp2 eq {gnu} } { set tmp2 gcc }

   if { $tmp1 eq $tmp2 } {
      # If reach here, our parent dir agrees with
      # the family of our loaded/defaulted compiler
      if { $compilerVersion ne {} } {
         # We have a version, does a modulefile/dir exist for it
         if [ ChildModuleExists $compilerVersion ] {
            # There is a modulefile/dir for this compiler version, default to it
            module-version $compilerFamily/$compilerVersion default
            set _did_default true
         }
      }
   }
}

Again, we source the common_utitilies.tcl file and use ModulesCurrentModulefile to get the directory in which the .modulerc script resides. It then uses GetLoadedCompiler to determine what if any compiler was loaded. If one was, we split into family and version, and ensure that the directory containing the .modulerc file matches the family name. If so, it checks if there is a directory or modulefile in that directory matching the version name, and if so defaults to it.

For both of the .modulerc scripts we examined, we note that in case of errors, etc., the script just exits without defaulting. One might be tempted to have the .modulerc script actually return an error in such cases (perhaps using module-info mode to only have it output during load operations). This, however, runs into issues:

  • For Environment Modules 3.x: for some reason, when modulecmd processes .modulerc scripts, module-info mode load always returns true. Because of this, users would get spurious errors when doing a module avail if the .modulerc scripts error-ed, because they all get processed regardless of what compiler is loaded.
  • For versions 4.x: the modulecmd process tends to process all .modulerc files underneath the package root during a load, which will similarly generate spurious errors if the .modulerc scripts throw errors.

Because of this, if an user explicitly requests a modulefile that conflicts with the loaded compiler (e.g. the user does a module load pgi/19.4 followed by a module load openmpi/4.0/intel/2019), the modulefile needs to detect this and error appropriately. A similar situation can happen if one of the .modulerc scripts fail to default, presumably because there is no appropriate build for the requested package; the modulecmd will default to something, but likely not what is wanted. To facilitate this, we define the Tcl procedure LoadedCompilerMatches:

#--------------------------------------------------------------------
# LoadedCompilerMatches:
#
# Takes the tag of the desired compiler $wanted (in form of FAMILY/VERSION)
# Checks if any compiler is loaded, and will throw error if one is
# loaded and it does not match $wanted.
# If compilers match, it will prereq the compiler if $requireIt set
# If no compiler is loaded, will load if optional boolean parameter
# $loadIt is set (default unset), and prereq if $requireIt is set, and
# then return.
# Option parameters:
# requireIt: boolean, default false. If set, require $wanted
# loadIt: boolean, default false.  If set, load $wanted if no compiler
#	already loaded
# modTag: string, defaults go [module-info specified].  Used in error messages
proc LoadedCompilerMatches { wanted {requireIt 0} { loadIt 0 } {modTag {} } } {
   # Default modTag
   if { $modTag eq {} } { set modTag [ module-info specified ] }
   # If no compiler given in $wanted, just return
   if { $wanted eq {} } { return }

   # Get loaded compiler (w/out defaults)
   set loaded [ GetLoadedCompiler 0  0 ]
   # If no compiler is loaded, then require it if asked, and return
   if { $loaded eq {} } {
      if { $loadIt } {
         RequireCompiler $wanted
         if { $requireIt } { prereq $wanted }
      }
      return
   }

   # Have a loaded compiler, split into family and version
   set tmpLoaded [ GetPackageFamilyVersion $loaded ]
   set loadedFam [ lindex $tmpLoaded 0 ]
   set loadedVer [ lindex $tmpLoaded 1 ]
   # Always use 'gcc' not 'gnu'
   if { $loadedFam eq {gnu} } { set loadedFam gcc }

   # Split wanted into family and version
   set tmpWanted [ split $wanted / ]
   set wantedLen [ llength $tmpWanted ]
   #Nothing to do if no components to wanted
   if { $wantedLen < 1 } { return }

   set wantedFam [ lindex $tmpWanted 0 ]
   # Always use 'gcc' not 'gnu'
   if { $wantedFam eq {gnu} } { set wantedFam gcc }

   # Ensure families match
   if { $loadedFam ne $wantedFam } {
      PrintLoadError "Compiler Mismatch
Package $modTag does not appear to be built for currently
loaded compiler $loaded."
   }

   # OK, families match
   # If no version specified in $wanted, we are basically done
   if { $wantedLen < 2 } {
      if { $requireIt } { prereq $wanted }
      return
   }

   # Ensure versions match
   set wantedVer [ lindex $tmpWanted 1 ]
   if { $loadedVer eq $wantedVer } {
      if { $requireIt } { prereq $wanted }
      return
   }
   # Versions don't match
   PrintLoadError "Compiler Mismatch
Package $modTag does not appear to be built for currently
loaded compiler $loaded."
}

The procedure takes a string for the family and version of the compiler that the modulefile expects, and ensures that if a compiler is loaded, they match. If they match, the procedure just returns, and if not, it spits out an error indicating a compiler mismatch. It also takes optional boolean flags: requireIt to determine if the procedure should prereq the compiler before returning, and loadIt to determine if, when no compiler was previously loaded, whether the procedure should load it (using RequireCompiler so it can properly handle the default compiler specially if needed). The requireIt flag should generally be set for Environment Modules 4.x (as that will allow the automatic handling of modules to properly recognize dependencies, even though as we will see that does not work too well for this strategy as yet), and should not be set for versions 3.x (as the loading of modules with loadIt does not occur until after the prereq, causing module loads to fail).

The resulting modulefile for something depending only on the compiler, using mvapich as an example, then would look like:

# Common modulefile for mvapich
# Using "modulerc based" strategy
# Expects the following variables to have been
# previously defined:
#	version: version of mvapich
#	compilerTag: the compiler to use

# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root

# Also get location of and load common procedures
# This is a hack for the cookbook examples, in production
# one should either
# 1) declare the procedures in a site config file (preferred)
# 2) hardcode the path to $tcllibdir and common_utilities.tcl
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

proc ModulesHelp { } {
   global version compilerTag
   puts stderr "
mvapich: Test dummy version of mvapich $version

mvapich version: $version
Compiler: $compilerTag

For testing packages depending on compilers/MPI

"
}

module-whatis "Dummy mvapich $version (built for $compilerTag)"

# Make sure loadedCompiler matches what we want.  And
# load compiler if no compiler loaded
LoadedCompilerMatches $compilerTag 1 1
# For Env Modules 3.x, set requireIt to 0
#LoadedCompilerMatches $compilerTag 0 1

# Compute the installation prefix
set pkgroot $swroot/mvapich
set vroot $pkgroot/$version
set prefix $vroot/$compilerTag

# Set environment variables
setenv MPI_DIR $prefix

set bindir $prefix/bin
set libdir $prefix/lib
set incdir $prefix/include

prepend-path PATH            $bindir
prepend-path LIBRARY_PATH    $libdir
prepend-path LD_LIBRARY_PATH $libdir
prepend-path CPATH           $incdir

Basically, the modulefile knows what compiler it wants (in the above example, that is set by the stubfile and passed as the Tcl variable compilerTag into the common script above), and then calls the LoadedCompilerMatches procedure above to ensure the loaded compiler matches what the modulefile wants (and to load it if no compiler is loaded).

So the mvapich directory in the MODULEPATH consists of subdirectories for each version of mvapich supported (e.g. 2.1 and 2.3.1). In each of the version subdirectories, we symlink modulerc.select_compiler_family as .modulerc, and create an additional layer of subdirectories, one for each compiler family for which the version of mvapich is built (e.g. gcc, intel, pgi). In each compiler family directory under each mvapich version, we symlink modulerc.select_compiler_version as .modulerc, and create a stub modulefile named for the compiler version. The stub modulefile then defines some Tcl variables for the version of mvapich and the compiler family/version, and sources the common file above. E.g., for mvapich/2.3.1/intel/2019, the stubfile would look like

#%Module
# Dummy modulefile for mvapich

set version 2.3.1
set compilerTag intel/2019
set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/../../common

So the mvapich directory in MODULEPATH would have a structure like

mvapich
├── 2.1
│   ├── gcc
│   │   ├── 8.2.0
│   │   ├── 9.1.0
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── intel
│   │   ├── 2018
│   │   ├── 2019
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── .modulerc -> symlink to modulerc.select_compiler_family
│   └── pgi
│       ├── 18.4
│       ├── 19.4
│       └── .modulerc -> symlink to modulerc.select_compiler_version
├── 2.3.1
│   ├── gcc
│   │   ├── 9.1.0
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── intel
│   │   ├── 2019
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── .modulerc -> symlink to modulerc.select_compiler_family
│   └── pgi
│       ├── 19.4
│       └── .modulerc -> symlink to modulerc.select_compiler_version
└── common

With this directory structure, an user can load a compiler and then module load a specific version of mvapich, and if a build of that version of mvapich exists for that compiler, the environment will be set up properly, and otherwise an error reported. However, if the user module loads mvapich without specifying a version, it will simply default the version to the latest version of mvapich available, regardless of whether there is a build of that version of mvapich for the loaded compiler. Ideally, we would like it so that if one issues a module load of a package without specifying a version, it should load the latest version available that is compatible with any loaded compilers or other modules.

A simple change can enable that. We add a symlink to modulerc.select_compiler_family as .modulerc directly under the mvapich directory, and create an additional subdirectory under the mvapich directory for each compiler family. We symlink modulerc.select_compiler_version as .modulerc under each compiler family subdirectory, along with a subdirectory for each version of that compiler family that a version of mvapich is built for. We then symlink the various stub modulefiles under the mvapich/VERSION into the corresponding mvapich/COMPILER_FAMILY/COMPILER_VERSION directory, with the symlink's name being the mvapich version number. So the mvapich directory tree will now look like:

mvapich
├── 2.1
│   ├── gcc
│   │   ├── 8.2.0
│   │   ├── 9.1.0
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── intel
│   │   ├── 2018
│   │   ├── 2019
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── .modulerc -> symlink to modulerc.select_compiler_family
│   └── pgi
│       ├── 18.4
│       ├── 19.4
│       └── .modulerc -> symlink to modulerc.select_compiler_version
├── 2.3.1
│   ├── gcc
│   │   ├── 9.1.0
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── intel
│   │   ├── 2019
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── .modulerc -> symlink to modulerc.select_compiler_family
│   └── pgi
│       ├── 19.4
│       └── .modulerc -> symlink to modulerc.select_compiler_version
├── common
├── gcc
│   ├── 8.2.0
│   │   └── 2.1 -> symlink to ../../2.1/gcc/8.2.0
│   ├── 9.1.0
│   │   ├── 2.1 -> symlink to ../../2.1/gcc/9.1.0
│   │   └── 2.3.1 -> symlink to ../../2.3.1/gcc/9.1.0
│   └── .modulerc -> symlink to modulerc.select_compiler_version
├── intel
│   ├── 2018
│   │   └── 2.1 -> symlink to ../../2.1/intel/2018
│   ├── 2019
│   │   ├── 2.1 -> symlink to ../../2.1/intel/2019
│   │   └── 2.3.1 -> symlink to ../../2.3.1/intel/2019
│   └── .modulerc -> symlink to modulerc.select_compiler_version
├── .modulerc -> symlink to modulerc.select_compiler_family
└── pgi
    ├── 18.4
    │   └── 2.1 -> symlink to ../../2.1/pgi/18.4
    ├── 19.4
    │   ├── 2.1 -> symlink to ../../2.1/pgi/19.4
    │   └── 2.3.1 -> symlink to ../../2.3.1/pgi/19.4
    └── .modulerc -> symlink to modulerc.select_compiler_version

When one attempts to module load mvapich without specifying a version, the .modulerc file will default to the family of the loaded compiler (or the default compiler if none loaded). It then descends into the corresponding compiler family directory, where the .modulerc there will default to the version of the loaded or defaulted compiler. After descending into the compiler version directory, there is no .modulerc, so the standard modulecmd defaulting mechanism will select the latest version.

We now have two ways to refer to the same build of mvapich, namely one using the traditional form

PACKAGE/PKGVERSION/COMPILER_FAMILY/COMPILER_VERSION
(e.g. mvapich/2.3.1/pgi/19.4)

and a new one giving the compiler family and version immediately after package name, with the version of the package coming last

PACKAGE/COMPILER_FAMILY/COMPILER_VERSION/PKGVERSION
(e.g. mvapich/pgi/19.4/2.3.1)

These both refer to the same build of the package (e.g. version 2.3.1 of mvapich, built for version 19.4 of the PGI compiler suite). Indeed, through the judicious use of symlinks, these actually refer to the modulefile on disk, but with two distinct names depending on the path used to get to it. In general, we allow for modules to be named with multiple dependency packages and/or flags if doing so would make it easier for a user to default to the correct package.

This is the primary reason for the procedure GetPackageFamilyVersion --- it splits a module name into components and takes the first component as the family name. It then examines the second component, and if it matches the name of a known package (like a compiler family), it uses the last component as the version. Otherwise, the second component is assumed to be the version. The code is as follows:

#--------------------------------------------------------------------
# GetPackageFamilyVersion
#
# Given the fully qualified spec for a modulefile, returns a list
# with the package family name and version.
#
# E.g., for foo/1.1/gcc/8.2.0/openmpi/3.1 would return {foo 1.1}
# But also handles stuff like foo/gcc/8.2.0/openmpi/3.1/1.1 (returning
# again {foo 1.1})
proc GetPackageFamilyVersion { tag } {
   # Return empty list if given empty tag
   if { $tag eq {} } { return {} }

   # Split tag into components
   set components [ split $tag / ]
   set compLen [ llength $components ]
   if { $compLen < 1 } { return {} }

   # Family should always be first
   set family [ lindex $components 0 ]
   if { $compLen < 2 } { return $family }

   # First guess, version immediately follows family
   set version [ lindex $components 1 ]
   # Check if it matches known non-version stuff following family
   # Compilers, MPIs, other things branch on
   set nonVers { gnu gcc pgi intel openmpi ompi intelmpi impi mvapich
      mvapich2 avx avx2 sse4.1 sse3 python perl hdf hdf4 hdf5 }
   set tmp [ lsearch $nonVers $version ]
   if { $tmp > -1 } {
      # The component immediately following family was NOT a version
      # Use last component as version
      set version [ lindex $components end ]
   }
   return  "$family $version"
}

Similarly, there are two corresponding .modulerc scripts for defaulting the MPI library: one to default the family (e.g. openmpi, mvapich, intelmpi) and one to default the version. The family version, shown below:

#%Module
# Choose MPI family
#
# This check if any MPI lib was previously "module loaded"
#
# If no MPI library was previously module loaded (and intel compiler suite
# was not loaded, see below) and nompiFlag is set (default is set), then
# if a "nompi" subdirectory exists, we will default to that.
# If nompiFlag is not set, then we just return w/out setting any default
# (allowing modulecmd to default on its own.  It is the responsibility
# and the resulting modulefile to lado/require the MPI lib it needs or err
# as appropriate).
#
# If an MPI library was loaded, then we check for a subdirectory named after
# the family of the loaded MPI library, and if such exists, default to that.
# If it does not exist, we return w/out defaulting, allowing modulecmd to
# default according to its own rules.  The resulting modulefile is responsible
# for aborting/complaining about the MPI library mismatch.
#
# Special handling is needed for intel MPI, as it does not get explicitly
# loaded if the Intel compiler suite is being used.  So if no MPI library
# is loaded, we check to see if an Intel compiler was loaded.  If so, we
# check for an "intelmpi" or similar subdirectory, and if found default to
# that.  If not found, we proceed as above for the no MPI library case
# (defaulting to "nompi" if found and nompiFlag set, or just returning w/out
# defaulting otherwise).
#
# Usage:
# In most cases, can simply symlink .modulerc to this file
#
# In more complicated cases, .modulerc can source this file, and can
# then test the variable _did_default, which will be true if we set
# a default for modules for the next level, or false otherwise (in which
# case your .modulerc can set one)

# Source some required Tcl procedures here.  Hack for cookbook
# making use of environment variable for location.
# In production, this should ideally be in a site config file.
# At minimum, hardcode the path
set rootdir $::env(MOD_GIT_ROOTDIR)
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

#Default nompiFlag to set
if ![info exists nompiFlag] { set nompiFlag true }

# _did_default will be true if we actually default something
# Useful in case a .modulerc sources us, and wants to set a default if we
# did not
set _did_default false

set moduledir [file dirname $ModulesCurrentModulefile]
set baseComponent [ file tail $moduledir ]

# Get the currently loaded MPI library
set fullMpiTag [ GetLoadedMPI 0 ]
set tmpMpiTag [ GetPackageFamilyVersion $fullMpiTag ]
set mpiFamily [ lindex $tmpMpiTag 0 ]

if { $mpiFamily eq {} } {
   # No MPI module loaded
   # What compiler is loaded (default from path if needed, but not GetDefaultCompiler)
   set fullComp [ GetLoadedCompiler 1 0 ]
   set tmpComp [ GetPackageFamilyVersion $fullComp ]
   set compFam [ lindex $tmpComp 0 ]
   if { $compFam eq {intel} } {
      # OK, no MPI explicitly loaded, but using Intel compiler
      # So set mpiFamily to first found in list below
      set mpiList {intelmpi impi intel intelmpi-mt impi-mt intel-mp}
      set mpiFamily [ FirstChildModuleInList $mpiList ]
   }
}

if { $mpiFamily eq {} } {
   # No MPI lib was explicitly loaded, and if intel compiler was loaded,
   # did not find a subdir matching intelmpi or one of its variants
   # Looks for a "nompi" dir if $nompiFlag is set
   if { $nompiFlag } {
      if [ ChildModuleExists nompi ] {
         # nompi subdir exists, default to it
         module-version $baseComponent/nompi default
         set _did_default true
      }
   }
} else {
   # Either MPI lib was explicitly loaded, or implied intelmpi
   # Default to the subdir named after MPI family, if it exists
   if [ ChildModuleExists $mpiFamily ] {
      module-version $baseComponent/$mpiFamily default
      set _did_default true
   }
}

is similar to the corresponding compiler version, but contains additional logic to handle the intelmpi case. We treat intelmpi specially, because in our example here we assume the environment for Intel MPI is setup properly when the user loads the intel module. If no MPI library is explicitly loaded, but the intel module is loaded, than we look for a subdirectory named intelmpi (or one of its aliases) and default to it if it exists. Otherwise, if no MPI module is loaded, it looks for and will default to a directory or modulefile named nompi if it exists.

The modulerc.select_mpi_version script is also similar to its compiler counterpart,

#%Module
# Choose MPI version
#
# This check if any MPI lib was previously "module loaded"
#
# If an MPI library was loaded, and the family of loaded MPI lib matches
# the name of the parent dir for this .modulerc, and there is a subdir
# of that directory matching the MPI version, then we default to that
# subdir.
#
# If no MPI library was previously module loaded, or the loaded MPI family
# does not match the name of the parent directory, or no subdir matching
# the MPI version is found, then we just return w/out defaulting.
# The modulecmd will then default according to its own rules, and it is
# up to the resulting modulefile to either load the appropriate MPI library
# or abort/complain about an MPI mismatch as appropriate.
#
# NOTE: no special handling is needed for intelmpi, as we assume that if
# intelmpi is selected because the intel compiler was loaded, then we
# are to use the intelmpi that shipped with the compiler suite, and so
# there is no needed for additional versioning of the intelmpi beneath
# the intelmpi subdir.  If intelmpi is being used with a non-intel compiler,
# then it is assumed intelmpi was explicitly loaded.
#
# Usage:
# In most cases, can simply symlink .modulerc to this file
#
# In more complicated cases, .modulerc can source this file, and can
# then test the variable _did_default, which will be true if we set
# a default for modules for the next level, or false otherwise (in which
# case your .modulerc can set one)

# Source some required Tcl procedures here.  Hack for cookbook
# making use of environment variable for location.
# In production, this should ideally be in a site config file.
# At minimum, hardcode the path
set rootdir $::env(MOD_GIT_ROOTDIR)
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

# _did_default will be true if we actually default something
set _did_default false

set moduledir [file dirname $ModulesCurrentModulefile]
set parentDir [ file tail $moduledir ]

# Get the currently loaded compiler, w/out defaulting
set fullMpiTag [ GetLoadedMPI 0 ]

if { $fullMpiTag ne {} } {
   # We have an MPI module loaded
   set tmpMpiTag [ GetPackageFamilyVersion $fullMpiTag ]
   set mpiFamily [ lindex $tmpMpiTag 0 ]
   set mpiVersion [ lindex $tmpMpiTag 1 ]

   # Canonicalize intelmpi variants in parentDir and mpiFamily for comparison
   set intelList "intelmpi impi intel intelmpi-mt impi-mt intel-mt"
   set tmpMpiFamily $mpiFamily
   if { [lsearch $intelList $tmpMpiFamily] > -1 } {
	set tmpMpiFamily intelmpi
   }
   set tmpParentDir $parentDir
   if { [lsearch $intelList $tmpParentDir] > -1 } {
	set tmpParentDir intelmpi
   }

   if { $tmpParentDir eq $tmpMpiFamily } {
      # The loaded MPI lib family name matches parentDir
      # So see if have subdir matching version, and if so, default it
      if { $mpiVersion ne {} } {
         if [ ChildModuleExists $mpiVersion ] {
            module-version $mpiFamily/$mpiVersion default
            set _did_default true
         }
      }
   }
}

It checks if any MPI library was explicitly loaded, and if so it checks if the family of the loaded MPI module matches the name of the parent directory of this .modulerc file. If so, it checks for the existence of a subdirectory matching the version, and if found it defaults to it. There is no special handling needed for the nompi case, as since there is no version attached to the MPI library in the nompi case, this .modulerc is not present there. Similarly, for the case of Intel MPI when the Intel compiler suite is loaded, we expect the Intel MPI version to be that from the Intel compiler suite, so no version or this .modulerc is needed.

As the MPI modules depend themselves on the compiler, it is assumed that any package depending on the MPI libraries also depend on the compiler, and so will have a structure like the one described below for foo

foo
├── 1.1
│   ├── gcc
│   │   ├── 8.2.0
│   │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│   │   │   ├── mvapich
│   │   │   │   ├── 2.1
│   │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   │   ├── nompi
│   │   │   └── openmpi
│   │   │       ├── 3.1
│   │   │       └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── intel
│   │   ├── 2018
│   │   │   ├── intelmpi
│   │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│   │   │   ├── mvapich
│   │   │   │   ├── 2.1
│   │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   │   ├── nompi
│   │   │   └── openmpi
│   │   │       ├── 3.1
│   │   │       └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── .modulerc -> symlink to modulerc.select_compiler_family
│   └── pgi
│       ├── 18.4
│       │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│       │   ├── mvapich
│       │   │   ├── 2.1
│       │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│       │   ├── nompi
│       │   └── openmpi
│       │       ├── 3.1
│       │       └── .modulerc -> symlink to modulerc.select_mpi_version
│       └── .modulerc -> symlink to modulerc.select_compiler_version
├── 2.4
│   ├── gcc
│   │   ├── 9.1.0
│   │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│   │   │   ├── mvapich
│   │   │   │   ├── 2.3.1
│   │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   │   ├── nompi
│   │   │   └── openmpi
│   │   │       ├── 4.0
│   │   │       └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── intel
│   │   ├── 2019
│   │   │   ├── intelmpi
│   │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│   │   │   ├── mvapich
│   │   │   │   ├── 2.3.1
│   │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   │   ├── nompi
│   │   │   └── openmpi
│   │   │       ├── 4.0
│   │   │       └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   ├── .modulerc -> symlink to modulerc.select_compiler_family
│   └── pgi
│       ├── 19.4
│       │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│       │   ├── nompi
│       │   └── openmpi
│       │       ├── 3.1
│       │       └── .modulerc -> symlink to modulerc.select_mpi_version
│       └── .modulerc -> symlink to modulerc.select_compiler_version
├── common
├── gcc
│   ├── 8.2.0
│   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│   │   ├── mvapich
│   │   │   ├── 2.1
│   │   │   │   └── 1.1 -> ../../../../1.1/gcc/8.2.0/mvapich/2.1
│   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   ├── nompi
│   │   │   └── 1.1 -> ../../../1.1/gcc/8.2.0/nompi
│   │   └── openmpi
│   │       ├── 3.1
│   │       │   └── 1.1 -> ../../../../1.1/gcc/8.2.0/openmpi/3.1
│   │       └── .modulerc -> symlink to modulerc.select_mpi_version
│   ├── 9.1.0
│   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│   │   ├── mvapich
│   │   │   ├── 2.3.1
│   │   │   │   └── 2.4 -> ../../../../2.4/gcc/9.1.0/mvapich/2.3.1
│   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   ├── nompi
│   │   │   └── 2.4 -> ../../../2.4/gcc/9.1.0/nompi
│   │   └── openmpi
│   │       ├── 4.0
│   │       │   └── 2.4 -> ../../../../2.4/gcc/9.1.0/openmpi/4.0
│   │       └── .modulerc -> symlink to modulerc.select_mpi_version
│   └── .modulerc -> symlink to modulerc.select_compiler_version
├── intel
│   ├── 2018
│   │   ├── intelmpi
│   │   │   └── 1.1 -> ../../../1.1/intel/2018/intelmpi
│   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│   │   ├── mvapich
│   │   │   ├── 2.1
│   │   │   │   └── 1.1 -> ../../../../1.1/intel/2018/mvapich/2.1
│   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   ├── nompi
│   │   │   └── 1.1 -> ../../../1.1/intel/2018/nompi
│   │   └── openmpi
│   │       ├── 3.1
│   │       │   └── 1.1 -> ../../../../1.1/intel/2018/openmpi/3.1
│   │       └── .modulerc -> symlink to modulerc.select_mpi_version
│   ├── 2019
│   │   ├── intelmpi
│   │   │   └── 2.4 -> ../../../2.4/intel/2019/intelmpi
│   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
│   │   ├── mvapich
│   │   │   ├── 2.3.1
│   │   │   │   └── 2.4 -> ../../../../2.4/intel/2019/mvapich/2.3.1
│   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
│   │   ├── nompi
│   │   │   └── 2.4 -> ../../../2.4/intel/2019/nompi
│   │   └── openmpi
│   │       ├── 4.0
│   │       │   └── 2.4 -> ../../../../2.4/intel/2019/openmpi/4.0
│   │       └── .modulerc -> symlink to modulerc.select_mpi_version
│   └── .modulerc -> symlink to modulerc.select_compiler_version
├── .modulerc -> symlink to modulerc.select_compiler_family
└── pgi
    ├── 18.4
    │   ├── .modulerc -> symlink to modulerc.select_mpi_family
    │   ├── mvapich
    │   │   ├── 2.1
    │   │   │   └── 1.1 -> ../../../../1.1/pgi/18.4/mvapich/2.1
    │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
    │   ├── nompi
    │   │   └── 1.1 -> ../../../1.1/pgi/18.4/nompi
    │   └── openmpi
    │       ├── 3.1
    │       │   └── 1.1 -> ../../../../1.1/pgi/18.4/openmpi/3.1
    │       └── .modulerc -> symlink to modulerc.select_mpi_version
    ├── 19.4
    │   ├── .modulerc -> symlink to modulerc.select_mpi_family
    │   ├── nompi
    │   │   └── 2.4 -> ../../../2.4/pgi/19.4/nompi
    │   └── openmpi
    │       ├── 3.1
    │       │   └── 2.4 -> ../../../../2.4/pgi/19.4/openmpi/3.1
    │       └── .modulerc -> symlink to modulerc.select_mpi_version
    └── .modulerc -> symlink to modulerc.select_compiler_version

Although the directory tree above is somewhat lengthy, it is similar to the case for a module depending only on the compiler, but expanded to handle MPI dependencies as well. There is a subdirectory under foo for each version of foo. Each of these subdirectories have further subdirectories for the compiler families for which that version of foo was built, and under the subdirectories for each compiler family are subdirectories for each version of that compiler family for which foo was built. Underneath these are stub modulefiles for nompi (and for intel compilers, intelmpi) builds of the package, and subdirectories for each MPI family, containing stub modulefiles for each version of the MPI family the package was built for.

Additionally, there are subdirectories directly under foo for each compiler family the package was built for, with each having subdirectories for the version of the compiler. Beneath each of those are one or two more layers of subdirectories for the MPI family and version (the version layer is omitted in the nompi or intelmpi cases), underneath which are symlinks to the corresponding stub modulefile from the foo/FOOVERSION path.

By placing the appropriate .modulerc files (actually, symlinks to the correct modulerc file), when the user enters a partial modulename the modulecmd will descend this directory tree based on the previously loaded compiler and MPI library to get to correct build of the package, if it exists. If no MPI library was previously loaded, it will search for an intelmpi build if an intel compiler was loaded, or a nompi build if no intelmpi build found or a non-intel compiler was used. If only MPI builds were made, it will default (using standard module defaulting rules) to one of the MPI builds, and the appropriate MPI library will be loaded when the foo module gets loaded (via the LoadedMPIMatches Tcl procedure described below). If a compiler and MPI library were loaded but no build for that combination exists, again a modulefile will be defaulted to (using standard module defaulting rules), but an error will be generated by the one of the LoadedCompilerMatches or LoadedMPIMatches calls in the foo modulefile (see below).

The common code of the modulefile is fairly standard, as shown below

# Common stuff for "foo" modulefiles
# Using "modulerc based" strategy
#
# This file expects the following Tcl variables to have been
# previously set:
#	version: version of foo
#	compilerTag: compiler was built for
#	mpiTag: mpi was built for

# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root

# Also get location of and load common procedures
# This is a hack for the cookbook examples, in production
# one should either
# 1) declare the procedures in a site config file (preferred)
# 2) hardcode the path to $tcllibdir and common_utilities.tcl
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

proc ModulesHelp { } {
   global version compilerTag mpiTag
   puts stderr "
foo: Test dummy version of foo $version

For testing packages depending on compilers/MPI

Foo: $version
Compiler: $compilerTag
MPI: $mpiTag

"
}

module-whatis "Dummy foo $version ($compilerTag, $mpiTag)"

# Make sure loaded compiler and MPI match what we want.
# And load them if not already loaded
LoadedCompilerMatches $compilerTag 1 1
# But don't load intelmpi if compiler is intel
LoadedMpiMatches $mpiTag 1 1 1
# For Env Modules 3.x, set requireIt to false
#LoadedCompilerMatches $compilerTag 0 1
#LoadedMpiMatches $mpiTag 0 1 1

# Compute the installation prefix
set pkgroot $swroot/foo
set vroot $pkgroot/$version
set prefix $vroot/$compilerTag/$mpiTag

# Set environment variables
setenv FOO_DIR $prefix

set bindir $prefix/bin
set libdir $prefix/lib
set incdir $prefix/include

prepend-path PATH            $bindir
prepend-path LIBRARY_PATH    $libdir
prepend-path LD_LIBRARY_PATH $libdir
prepend-path CPATH           $incdir

conflict foo

The main difference from a standard modulefile is the inclusion of the invocations of LoadedCompilerMatches and LoadedMpiMatches. This is needed to ensure that the compiler and MPI for the modulefile being processed are compatible with any which are loaded. The LoadedMpiMatches procedure is defined by:

#--------------------------------------------------------------------
# LoadedMpiMatches:
#
# Takes the tag of the desired MPI library $wanted (in form of FAMILY/VERSION)
# Checks if any MPI is loaded, and will throw error if one is
# loaded and it does not match $wanted.
# If MPIs match, it will prereq the MPI lib if $requireIt set
# If no MPI is loaded, will load if optional boolean parameter
# $loadIt is set (default unset), and prereq if $requireIt is set, and
# then return.
# Option parameters:
# requireIt: boolean, default false. If set, will prereq $wanted
# loadIt: boolean, default false. If set, load $wanted
#	if no MPI is already loaded
# noLoadIntel: boolean, passed to RequireMPI.  If set,
#	will not attempt to load intelmpi variants if compiler is intel
# forceNoMpi: boolean, default false.  If set and $wanted is 'nompi',
#  	insist that no MPI module is loaded
# modTag: string, defaults go [module-info specified].  Used in error messages
proc LoadedMpiMatches { wanted {requireIt 0} { loadIt 0 } { noLoadIntel 0 }
   { forceNoMpi 0 } {modTag {} } } {
   # Default modTag
   if { $modTag eq {} } { set modTag [ module-info specified ] }
   # If no MPI given in $wanted, just return
   if { $wanted eq {} } { return }

   # Get loaded MPI, no defaulting to intelmpi
   set loaded [ GetLoadedMPI 0 ]

   # If wanted is nompi, no need to do anything unless forceNoMpi set
   if { $wanted eq {nompi} } {
      if { $forceNoMpi } {
         # Complain if any MPI loaded
         if { $loaded ne {} } {
            PrintLoadError "MPI Mismatch
You have an MPI library loaded ($loaded), but package $modTag
really insists that no MPI library be loaded.

Please unload the MPI library and try again."
         }
      }
      return
   }

   # If no MPI is loaded, then load it if $loadIt (this is valid even
   # in edge cases of nompi or intelmpi), prereq it if $requireIt,
   # abd return
   if { $loaded eq {} } {
      if { $loadIt } {
         RequireMPI $wanted $noLoadIntel
         if { $requireIt } { prereq $wanted }
      }
      return
   }

   # Have a loaded MPI, split into family and version
   set tmpLoaded [ GetPackageFamilyVersion $loaded ]
   set loadedFam [ lindex $tmpLoaded 0 ]
   set loadedVer [ lindex $tmpLoaded 1 ]

   # Split wanted into family and version
   set tmpWanted [ split $wanted / ]
   set wantedLen [ llength $tmpWanted ]
   #Nothing to do if no components to wanted
   if { $wantedLen < 1 } { return }
   set wantedFam [ lindex $tmpWanted 0 ]

   # Ensure families match
   if { $loadedFam ne $wantedFam } {
      PrintLoadError "MPI Mismatch

Package $modTag does not appear to be built for currently
loaded MPI library $loaded."
   }

   # OK, families match
   # If no version specified in $wanted, we are basically done
   if { $wantedLen < 2 } {
      if { $requireIt } { prereq $wanted }
      return
   }

   # Ensure versions match
   set wantedVer [ lindex $tmpWanted 1 ]
   if { $loadedVer eq $wantedVer } {
      if { $requireIt } { prereq $wanted }
      return
   }
   # Versions don't match
   PrintLoadError "MPI Mismatch

Package $modTag does not appear to be built for currently
loaded MPI library $loaded."
}

Basically, it determines what if any MPI library was previously module loaded. If one was loaded, it ensures the family and version match what was expected, and if not exit with an error. The use of the GetPackageFamilyVersion procedure is important, because as discussed previously packages can be represented by more than one name (e.g. the modulefile for version 4.0 of openmpi for gcc/9.1.0 could be named openmpi/4.0/gcc/9.1.0 or openmpi/gcc/9.1.0/4.0) and we need to ensure either is recognized.

Although we only showed cases for compilers and MPI libraries, the techniques above can be adapted to other cases as well. For simple cases, where everything falls under a single package name, one can use GetTagOfModuleLoaded to determine the version of the package loaded in the .modulerc and in the modulefile to ensure the version matches.

For bar, we added variants on CPU vectorization support. Rather than add a simd modulefile, we just add variants to the bar modules. So the bar modulefile tree would look like:

bar
├── 4.7
│   ├── gcc
│   │   ├── 8.2.0
│   │   │   ├── avx
│   │   │   ├── .modulerc -> symlink to modulerc.default_lowest_simd
│   │   │   └── sse4.1
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   └── .modulerc -> symlink to modulerc.select_compiler_family
├── 5.4
│   ├── gcc
│   │   ├── 9.1.0
│   │   │   ├── avx
│   │   │   └── avx2
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   └── .modulerc -> symlink to modulerc.select_compiler_family
├── avx
│   ├── gcc
│   │   ├── 8.2.0
│   │   │   └── 4.7 -> ../../../4.7/gcc/8.2.0/avx
│   │   ├── 9.1.0
│   │   │   └── 5.4 -> ../../../5.4/gcc/9.1.0/avx
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   └── .modulerc -> symlink to modulerc.select_compiler_family
├── avx2
│   ├── gcc
│   │   ├── 9.1.0
│   │   │   └── 5.4 -> ../../../5.4/gcc/9.1.0/avx2
│   │   └── .modulerc -> symlink to modulerc.select_compiler_version
│   └── .modulerc -> symlink to modulerc.select_compiler_family
├── common
├── gcc
│   ├── 8.2.0
│   │   ├── avx
│   │   │   └── 4.7 -> ../../../4.7/gcc/8.2.0/avx
│   │   ├── .modulerc -> symlink to modulerc.default_lowest_simd
│   │   └── sse4.1
│   │       └── 4.7 -> ../../../4.7/gcc/8.2.0/sse4.1
│   ├── 9.1.0
│   │   ├── avx
│   │   │   └── 5.4 -> ../../../5.4/gcc/9.1.0/avx
│   │   ├── avx2
│   │   │   └── 5.4 -> ../../../5.4/gcc/9.1.0/avx2
│   │   └── .modulerc -> symlink to modulerc.default_lowest_simd
│   └── .modulerc -> symlink to modulerc.select_compiler_version
├── .modulerc -> symlink to modulerc.select_compiler_family
└── sse4.1
    ├── gcc
    │   ├── 8.2.0
    │   │   └── 4.7 -> ../../../4.7/gcc/8.2.0/sse4.1
    │   └── .modulerc -> symlink to modulerc.select_compiler_version
    └── .modulerc -> symlink to modulerc.select_compiler_family

Here we added yet another alternate set of module names. So the module for bar version 5.4 built with gcc version 9.1.0 and avx2 support can be called:

  • bar/5.4/gcc/9.1.0/avx2
  • bar/gcc/9.1.0/avx2/5.4
  • bar/avx2/gcc/9.1.0/5.4

Due to this multiplicity of names, if an user does a module load bar, the .modulerc immediately under bar will cause modulecmd to default along the path for the loaded (or defaulted) compiler, and then the modulerc.default_lowest_simd will default to the lowest simd level. If the user module loads bar/VERSION, it will find the build for that version of bar with the appropriate compiler (and again, default to the lowest simd level). And if the user module loads bar/avx or another simd level, then it will load the latest version of bar built for the loaded compiler and simd specified.

The modulerc.default_lowest_simd script looks like:

#%Module
# Default to lowest SIMD support available
#
# Defaults to first of sse4.1, avx, or avx2 found
#
# Usage:
# In most cases, can simply symlink .modulerc to this file
#
# In more complicated cases, .modulerc can source this file, and can
# then test the variable _did_default, which will be true if we set
# a default for modules for the next level, or false otherwise (in which
# case your .modulerc can set one)

# Source some required Tcl procedures here.  Hack for cookbook
# making use of environment variable for location.
# In production, this should ideally be in a site config file.
# At minimum, hardcode the path
set rootdir $::env(MOD_GIT_ROOTDIR)
set tcllibdir $rootdir/doc/example/compiler-etc-dependencies/tcllib
source $tcllibdir/common_utilities.tcl

set moduledir [file dirname $ModulesCurrentModulefile]
set baseComponent [ file tail $moduledir ]

# _did_default will be true if we actually default something
# Useful in case a .modulerc sources us, and wants to set a default if we
# did not
set _did_default false

set simdList {sse4.1 avx avx2}
set firstChild [ FirstChildModuleInList $simdList ]
if { $firstChild ne {} } {
   module-version $baseComponent/$firstChild default
   set _did_default true
}

It will default to the lowest SIMD level, but could easily be adapted to do something else.

Examples

We now look at the example modulefiles for the Modulerc-based strategy. As noted previously, the best choice of whether to set the requireIt flag to LoadedCompilerMatches and LoadedMPIMatches (instructing them to to a prereq on the requesting compiler/MPI module) depends on whether one is using a 3.x or 4.x version of Environment Modules. Due to this, we provide two modulefile trees, one for 3.x and one for 4.x; they are basicaly identical except for that matter. This leads to slightly different instructions on how to use the examples, depending on which version of Environment Modules is being used, namely:

  1. Set (and export) MOD_GIT_ROOTDIR to where you git-cloned the modules source
  2. Do a module purge
  3. If using Environment Modules 3.x, set your MODULEPATH to $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc3
  4. If using Environment Modules 4.x, set your MODULEPATH to $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4

As with the previous cases, we start with a module avail command, and here we see the first big difference:

[mod4 (modulerc)]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4 -
bar/4.7/gcc/(default)                 foo/intel/2019/intelmpi/2.4
bar/4.7/gcc/8.2.0/(default)           foo/intel/2019/mvapich/2.3.1/2.4
bar/4.7/gcc/8.2.0/avx                 foo/intel/2019/nompi/2.4
bar/4.7/gcc/8.2.0/sse4.1(default)     foo/intel/2019/openmpi/4.0/2.4
bar/5.4/gcc/(default)                 foo/pgi/18.4/mvapich/2.1/1.1
bar/5.4/gcc/9.1.0/avx                 foo/pgi/18.4/nompi/(default)
bar/5.4/gcc/9.1.0/avx2                foo/pgi/18.4/nompi/1.1
bar/avx/gcc/(default)                 foo/pgi/18.4/openmpi/3.1/1.1
bar/avx/gcc/8.2.0/(default)           foo/pgi/19.4/nompi/(default)
bar/avx/gcc/8.2.0/4.7                 foo/pgi/19.4/nompi/2.4
bar/avx/gcc/9.1.0/5.4                 foo/pgi/19.4/openmpi/3.1/2.4
bar/avx2/gcc/(default)                gcc/8.2.0
bar/avx2/gcc/9.1.0/5.4                gcc/9.1.0
bar/gcc/(default)                     intel/2018
bar/gcc/8.2.0/(default)               intel/2019
bar/gcc/8.2.0/avx/4.7                 mvapich/2.1/gcc/(default)
bar/gcc/8.2.0/sse4.1/(default)        mvapich/2.1/gcc/8.2.0(default)
bar/gcc/8.2.0/sse4.1/4.7              mvapich/2.1/gcc/9.1.0
bar/gcc/9.1.0/avx/(default)           mvapich/2.1/intel/2018
bar/gcc/9.1.0/avx/5.4                 mvapich/2.1/intel/2019
bar/gcc/9.1.0/avx2/5.4                mvapich/2.1/pgi/18.4
bar/sse4.1/gcc/(default)              mvapich/2.1/pgi/19.4
bar/sse4.1/gcc/8.2.0/(default)        mvapich/2.3.1/gcc/(default)
bar/sse4.1/gcc/8.2.0/4.7              mvapich/2.3.1/gcc/9.1.0
foo/1.1/gcc/(default)                 mvapich/2.3.1/intel/2019
foo/1.1/gcc/8.2.0/(default)           mvapich/2.3.1/pgi/19.4
foo/1.1/gcc/8.2.0/mvapich/2.1         mvapich/gcc/(default)
foo/1.1/gcc/8.2.0/nompi(default)      mvapich/gcc/8.2.0/(default)
foo/1.1/gcc/8.2.0/openmpi/3.1         mvapich/gcc/8.2.0/2.1
foo/1.1/intel/2018/intelmpi(default)  mvapich/gcc/9.1.0/2.1
foo/1.1/intel/2018/mvapich/2.1        mvapich/gcc/9.1.0/2.3.1
foo/1.1/intel/2018/nompi              mvapich/intel/2018/2.1
foo/1.1/intel/2018/openmpi/3.1        mvapich/intel/2019/2.1
foo/1.1/pgi/18.4/mvapich/2.1          mvapich/intel/2019/2.3.1
foo/1.1/pgi/18.4/nompi(default)       mvapich/pgi/18.4/2.1
foo/1.1/pgi/18.4/openmpi/3.1          mvapich/pgi/19.4/2.1
foo/2.4/gcc/(default)                 mvapich/pgi/19.4/2.3.1
foo/2.4/gcc/9.1.0/mvapich/2.3.1       openmpi/3.1/gcc/(default)
foo/2.4/gcc/9.1.0/nompi(default)      openmpi/3.1/gcc/8.2.0(default)
foo/2.4/gcc/9.1.0/openmpi/4.0         openmpi/3.1/gcc/9.1.0
foo/2.4/intel/2019/intelmpi(default)  openmpi/3.1/intel/2018
foo/2.4/intel/2019/mvapich/2.3.1      openmpi/3.1/intel/2019
foo/2.4/intel/2019/nompi              openmpi/3.1/pgi/18.4
foo/2.4/intel/2019/openmpi/4.0        openmpi/3.1/pgi/19.4
foo/2.4/pgi/19.4/nompi(default)       openmpi/4.0/gcc/(default)
foo/2.4/pgi/19.4/openmpi/3.1          openmpi/4.0/gcc/9.1.0
foo/gcc/(default)                     openmpi/4.0/intel/2019
foo/gcc/8.2.0/(default)               openmpi/4.0/pgi/19.4
foo/gcc/8.2.0/mvapich/2.1/1.1         openmpi/gcc/(default)
foo/gcc/8.2.0/nompi/(default)         openmpi/gcc/8.2.0/(default)
foo/gcc/8.2.0/nompi/1.1               openmpi/gcc/8.2.0/3.1
foo/gcc/8.2.0/openmpi/3.1/1.1         openmpi/gcc/9.1.0/3.1
foo/gcc/9.1.0/mvapich/2.3.1/2.4       openmpi/gcc/9.1.0/4.0
foo/gcc/9.1.0/nompi/(default)         openmpi/intel/2018/3.1
foo/gcc/9.1.0/nompi/2.4               openmpi/intel/2019/3.1
foo/gcc/9.1.0/openmpi/4.0/2.4         openmpi/intel/2019/4.0
foo/intel/2018/intelmpi/(default)     openmpi/pgi/18.4/3.1
foo/intel/2018/intelmpi/1.1           openmpi/pgi/19.4/3.1
foo/intel/2018/mvapich/2.1/1.1        openmpi/pgi/19.4/4.0
foo/intel/2018/nompi/1.1              pgi/18.4
foo/intel/2018/openmpi/3.1/1.1        pgi/19.4
foo/intel/2019/intelmpi/(default)
[mod4 (modulerc)]$ module avail mvapich
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4 -
mvapich/2.1/gcc/(default)       mvapich/gcc/(default)
mvapich/2.1/gcc/8.2.0(default)  mvapich/gcc/8.2.0/(default)
mvapich/2.1/gcc/9.1.0           mvapich/gcc/8.2.0/2.1
mvapich/2.1/intel/2018          mvapich/gcc/9.1.0/2.1
mvapich/2.1/intel/2019          mvapich/gcc/9.1.0/2.3.1
mvapich/2.1/pgi/18.4            mvapich/intel/2018/2.1
mvapich/2.1/pgi/19.4            mvapich/intel/2019/2.1
mvapich/2.3.1/gcc/(default)     mvapich/intel/2019/2.3.1
mvapich/2.3.1/gcc/9.1.0         mvapich/pgi/18.4/2.1
mvapich/2.3.1/intel/2019        mvapich/pgi/19.4/2.1
mvapich/2.3.1/pgi/19.4          mvapich/pgi/19.4/2.3.1

Unlike the previous cases wherein only package names and versions were shown (because a single modulefile handled all the builds for that package and version), here we see a listing for every build of a package and version. Indeed, we not only see one such listing, but multiple listings per build in many cases (e.g. openmpi/3.1/intel/2018 and openmpi/intel/2018/3.1).

While admittedly the output of a module avail command without specifying any package is rather overwhelming, when a package is specified the output tends to be more reasonable, informing one of which builds of the package are available. This strategy deliberately opts for the presentation of more rather than less information.

The standard functionality of selecting the correct build of a package based on the loaded compiler, e.g.

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load pgi/19.4
[mod4 (modulerc)]$ module load openmpi/4.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/4.0/pgi/19.4(default)
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (modulerc)]$ module unload openmpi
[mod4 (modulerc)]$ module switch --auto pgi intel/2019
[mod4 (modulerc)]$ module load openmpi/4.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0/intel/2019(default)
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4 (modulerc)]$ module unload openmpi
[mod4 (modulerc)]$ module switch --auto intel gcc/9.1.0
[mod4 (modulerc)]$ module load openmpi/4.0
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/4.0, gcc/9.1.0)
[mod4 (modulerc)]$ module unload openmpi
[mod4 (modulerc)]$ module switch --auto gcc gcc/8.2.0
[mod4 (modulerc)]$ module load openmpi/4.0

**** ERROR *****:
Compiler Mismatch
Package openmpi/4.0 does not appear to be built for currently
loaded compiler gcc/8.2.0.


Loading openmpi/4.0/gcc/9.1.0
  ERROR: Module evaluation aborted
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0

works as expected (and fails with the expected error when one attempts to load a module not compatible with the loaded compiler). The only significant difference between the previously examined strategies is that the module list command provides information about what variant of each package is loaded.

The module switch command, however, does not work as well as one would like. While it indeeds switches the specified module, it does not successfully reload the modules which depend on the replaced module, even with the Automated module handling feature enabled. As currently implemented, the automated module handling feature attempts to reload dependent modules using the fully qualified module name, and as in this strategy the fully qualified modulename includes the information about the module that was switched out (e.g. pgi/19.4 in the example below), it will be incompatible with the replacement module (e.g. intel/2019 in example below). It is hoped a future version of modulecmd will allow for reloading based on the name specified when the module was loaded. But as things currently stand, the automatic module handling will throw an error attempting to reload the depend module, resulting in the the dependent modules being unloaded.

Without automated module handling mode (i.e. for older Environment Modules or without the --auto flag), the dependent modules remain loaded and there is inconsistency in the loaded modules. But at least module list clearly shows such.

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load pgi/19.4
[mod4 (modulerc)]$ module load openmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/pgi/19.4/4.0
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (modulerc)]$ module switch --auto pgi intel/2019

**** ERROR *****:
Compiler Mismatch
Package openmpi/pgi/19.4/4.0 does not appear to be built for currently
loaded compiler intel/2019.


Loading openmpi/pgi/19.4/4.0
  ERROR: Module evaluation aborted

Switching from pgi/19.4 to intel/2019
  WARNING: Reload of dependent openmpi/pgi/19.4/4.0 failed
  Unloading dependent: openmpi/pgi/19.4/4.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019
[mod4 (modulerc)]$ mpirun
mpirun: command not found

The defaulting of modules is more successful, however, as seen below:

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load openmpi/3.1
Loading openmpi/3.1/gcc/8.2.0
  Loading requirement: gcc/8.2.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) openmpi/3.1/gcc/8.2.0(default)
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/3.1, gcc/8.2.0)
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load gcc/8.2.0
[mod4 (modulerc)]$ module load openmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) openmpi/gcc/8.2.0/3.1
[mod4 (modulerc)]$ mpirun
mpirun (openmpi/3.1, gcc/8.2.0)

Here it not only defaults to the default compiler, but if one tries to load openmpi without specifying a version, it will find and load the latest version compatible with the loaded compiler.

The situation is similar for foo:

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load pgi/19.4
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) foo/2.4/pgi/19.4/nompi(default)
[mod4 (modulerc)]$ foo
foo 2.4 (pgi/19.4, nompi)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module load openmpi/3.1
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4                        3) foo/2.4/pgi/19.4/openmpi/3.1(default)
 2) openmpi/3.1/pgi/19.4(default)
[mod4 (modulerc)]$ foo
foo 2.4 (pgi/19.4, openmpi/3.1)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module unload openmpi
[mod4 (modulerc)]$ module switch --auto pgi intel/2019
[mod4 (modulerc)]$ module load foo/2.4
Loading foo/2.4/intel/2019/intelmpi
  ERROR: foo/2.4/intel/2019/intelmpi cannot be loaded due to missing prereq.
    HINT: the following module must be loaded first: intelmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019
[mod4 (modulerc)]$ foo
foo: command not found
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module load foo/2.4/intel/2019/nompi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) foo/2.4/intel/2019/nompi
[mod4 (modulerc)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module switch --auto intelmpi mvapich/2.3.1
[mod4 (modulerc)]$ module load foo/2.4
Loading foo/2.4/intel/2019/intelmpi
  ERROR: foo/2.4/intel/2019/intelmpi cannot be loaded due to missing prereq.
    HINT: the following module must be loaded first: intelmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) mvapich/2.3.1/intel/2019(default)
[mod4 (modulerc)]$ foo
foo: command not found
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module switch --auto mvapich openmpi/4.0
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019
 2) openmpi/4.0/intel/2019(default)
 3) foo/2.4/intel/2019/openmpi/4.0(default)
[mod4 (modulerc)]$ foo
foo 2.4 (intel/2019, openmpi/4.0)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module unload openmpi
[mod4 (modulerc)]$ module switch --auto intel/2019 gcc/9.1.0
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) foo/2.4/gcc/9.1.0/nompi(default)
[mod4 (modulerc)]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module load mvapich/2.3.1
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0                          3) foo/2.4/gcc/9.1.0/nompi(default)
 2) mvapich/2.3.1/gcc/9.1.0(default)
[mod4 (modulerc)]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4 (modulerc)]$ module unload foo
[mod4 (modulerc)]$ module switch --auto mvapich openmpi/4.0
[mod4 (modulerc)]$ module load foo/2.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0                        3) foo/2.4/gcc/9.1.0/openmpi/4.0(default)
 2) openmpi/4.0/gcc/9.1.0(default)
[mod4 (modulerc)]$ foo
foo 2.4 (gcc/9.1.0, openmpi/4.0)

As expected, the correct version of foo is loaded depending on the previously loaded compiler and MPI libraries. Note however that there is no dummy module for Intel MPI, and that when the intel compiler is loaded but no MPI library is explicitly loaded, module load foo/2.4 gets the Intel MPI build. Such only occurs if there is an Intel MPI build in the tree of modulefiles; otherwise a nompi build will be loaded if present. If both versions are present, as in this case, one needs to give the explicit foo/2.4/intel/2019/nompi specification.

Again, the module switch command only succeeds in switching the specified module, and does not successfully reload the modules depending on the switched module. With automated module handling mode, there will be errors reported and the dependent modules will be unloaded; without it the dependent modules will not get unloaded, and there will be inconsistent dependencies (but at least module list will show such).

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load pgi/18.4
[mod4 (modulerc)]$ module load openmpi/3.1
[mod4 (modulerc)]$ module load foo/1.1
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) pgi/18.4                        3) foo/1.1/pgi/18.4/openmpi/3.1(default)
 2) openmpi/3.1/pgi/18.4(default)
[mod4 (modulerc)]$ foo
foo 1.1 (pgi/18.4, openmpi/3.1)
[mod4 (modulerc)]$ module switch --auto pgi intel/2018

**** ERROR *****:
Compiler Mismatch
Package openmpi/3.1/pgi/18.4 does not appear to be built for currently
loaded compiler intel/2018.


Loading openmpi/3.1/pgi/18.4
  ERROR: Module evaluation aborted

**** ERROR *****:
Compiler Mismatch
Package foo/1.1/pgi/18.4/openmpi/3.1 does not appear to be built for currently
loaded compiler intel/2018.


Loading foo/1.1/pgi/18.4/openmpi/3.1
  ERROR: Module evaluation aborted

Switching from pgi/18.4 to intel/2018
  WARNING: Reload of dependent openmpi/3.1/pgi/18.4 failed
  WARNING: Reload of dependent foo/1.1/pgi/18.4/openmpi/3.1 failed
  Unloading dependent: foo/1.1/pgi/18.4/openmpi/3.1 openmpi/3.1/pgi/18.4
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2018
[mod4 (modulerc)]$ foo
foo: command not found
[mod4 (modulerc)]$ mpirun
mpirun: command not found
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load intel/2019
[mod4 (modulerc)]$ module load foo
Loading foo/intel/2019/intelmpi/2.4
  ERROR: foo/intel/2019/intelmpi/2.4 cannot be loaded due to missing prereq.
    HINT: the following module must be loaded first: intelmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019
[mod4 (modulerc)]$ foo
foo: command not found
[mod4 (modulerc)]$ module load openmpi
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/intel/2019/4.0
[mod4 (modulerc)]$ foo
foo: command not found

The defaulting of modules works relatively well, as shown below:

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load foo
Loading foo/gcc/8.2.0/nompi/1.1
  Loading requirement: gcc/8.2.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) foo/gcc/8.2.0/nompi/1.1
[mod4 (modulerc)]$ foo
foo 1.1 (gcc/8.2.0, nompi)
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load foo/1.1
Loading foo/1.1/gcc/8.2.0/nompi
  Loading requirement: gcc/8.2.0
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) foo/1.1/gcc/8.2.0/nompi(default)
[mod4 (modulerc)]$ foo
foo 1.1 (gcc/8.2.0, nompi)
[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load pgi/18.4
[mod4 (modulerc)]$ module load foo
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) pgi/18.4   2) foo/pgi/18.4/nompi/1.1
[mod4 (modulerc)]$ foo
foo 1.1 (pgi/18.4, nompi)

If one attempts to load foo without specifying a version or having previously loaded a compiler module, the compiler will be defaulted (to gcc/8.2.0 compiler, as that is what we declared via the GetDefaultCompiler Tcl procedure), and the latest version of foo compatible with that compiler (and no MPI) will be loaded.

The situation with bar is similar. We do not have a dummy simd module, so the builds with different CPU vectorization support are specified by appending /avx, etc. to the bar package name. As with previous strategies, if one attempts to load a simd variant which was not built for the compiler loaded, an error will occur.

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module load gcc/9.1.0
[mod4 (modulerc)]$ module load bar/avx2
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) bar/avx2/gcc/9.1.0/5.4
[mod4 (modulerc)]$ bar
bar 5.4 (gcc/9.1.0, avx2)
[mod4 (modulerc)]$ module unload bar
[mod4 (modulerc)]$ module load bar/avx
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) bar/avx/gcc/9.1.0/5.4
[mod4 (modulerc)]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4 (modulerc)]$ module unload bar
[mod4 (modulerc)]$ module load bar/sse4.1

**** ERROR *****:
Compiler Mismatch
Package bar/sse4.1 does not appear to be built for currently
loaded compiler gcc/9.1.0.


Loading bar/sse4.1/gcc/8.2.0/4.7
  ERROR: Module evaluation aborted
[mod4 (modulerc)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0
Summary of Modulerc-based strategy
  • The modulefiles are fairly standard. The only extra logic needed is to ensure that any loaded compiler/MPI or other dependency matches what the modulefile expects, and that can be handled fairly easily with the addition of one or two calls to Tcl procedures.
  • The .modulerc files do the work of routing partial specifications to the correct modulefile, and follow a fairly standard pattern. For the most part, these are generic/independent of the package, and typically can be written once and symlinked to the appropriate places in the directory tree.
  • This strategy involves the use of many more modulefiles than the previously examined strategies, having at least one modulefile per build, and in some cases multiple per build. The output of the module avail command without any module specified is rather overwhelming, although when a module is specified, module avail will provide information about what builds are available.
  • Because each build has a specific module, the module list command shows exactly what builds of the various modules are loaded.
  • The module switch command does not work very well. In particular, when one switches out a module on which other modules depend, the dependent modules are not successfully reloaded. Even with the Automated module handling feature (introduced in 4.2), the dependent modules are unloaded but do not get reloaded properly since currently this feature attempts to reload based on the fully qualified name of the loaded module (which includes a dependency on the module switched out), not the specification used when the module was loaded.
  • In general, when the user tries to load a module based on a partially specified modulename, the .modulerc files handle it well, and the latest version of the module consistent with the specification and previously loaded modules will be used. If no compiler was previously loaded, will use the default compiler, or the latest compiler if nothing matches the default compiler.
  • Modules will fail with an error message if user tries to load a package which was not built for the values of the dependency packages loaded.
Modulepath-based Strategy

This strategy makes use of the ability of modules to support multiple directories in the MODULEPATH environment variable. Everytime a module is loaded on which other modules might depend, a new path is added to MODULEPATH containing the modulefiles which depend on the newly added module.

This is basically similar to the strategy that is used in the hierarchical modulefile approach of Lmod.

Implementation

The Homebrewed flavors and Modulerc-based strategies were based on modifying the modulefiles of modules which depended on other modules. The Modulepath-based strategy, in contrast, is based on modifying the modulefiles for modules which other modules might depend on.

To begin with, we set MODULEPATH to a Core directory containing modulefiles for modules which do not depend on any other modules. The modulefiles for compilers will typically be put in here. For this example, we opt not to use a dummy simd module (for reasons explained later), but instead add avx, avx2, sse4.1 variants to the bar modules, but otherwise they might belong here. Other modules which do not depend on compilers, etc. could be place in here as well, e.g. applications which do not expose libraries, etc. So the directory structure would look like:

Core
├── gcc
│   ├── 8.2.0
│   ├── 9.1.0
│   └── common
├── intel
│   ├── 2018
│   ├── 2019
│   └── common
└── pgi
    ├── 18.4
    ├── 19.4
    └── common

A typical common file for the gcc compiler would be something like

# Example modulefiles for compiler-etc-dependency cookbook
# For "modulepath based" strategyy
#
# Common stuff for gcc
#
# Expects version to have been previously set

proc ModulesHelp { } {
   global version

   puts stderr "
This is the dummy GNU compiler suite modulefile for the cookbook
	Handling Compiler and other Package Dependencies

It does not actually do anything

gcc version: $version

"
}

module-whatis "Dummy Gnu $version for cookbook"

# Find the software root.  In production, you should
# hardcode to your real software root
set gitroot $::env(MOD_GIT_ROOTDIR)
set swroot $gitroot/doc/example/compiler-etc-dependencies/dummy-sw-root

set pkgroot $swroot/gcc
set vroot $pkgroot/$version
set bindir $vroot/bin

prepend-path PATH $bindir

# don't load multiple versions of this module (or other compilers)
conflict gcc
conflict gnu
conflict pgi
conflict intel

# Add the proper modulepath
# In production this should be hard-coded, but using $gitroot for cookbook
set modpathroot $gitroot/doc/example/compiler-etc-dependencies/modulepath
set newmodpath $modpathroot/Compiler/gcc/$version
module use $newmodpath

The most interesting aspect is the module use at the end. We add to MODULEPATH a directory under Compiler for this specific compiler and version. Other compilers/versions would add their own specific path to MODULEPATH.

In each of these compiler specific directories, modulefile directory trees exist for the supported MPI libraries and bar. We also include a foo directory containing the non-MPI variants of foo. It would look like:

Compiler
├── gcc
│   ├── 8.2.0
│   │   ├── bar
│   │   │   ├── 4.7
│   │   │   │   ├── avx
│   │   │   │   ├── .modulerc -> symlink to modulerc.default_lowest_simd
│   │   │   │   └── sse4.1
│   │   │   └── common -> symlink to bar/common
│   │   ├── foo
│   │   │   ├── 1.1
│   │   │   └── common -> symlink to foo/common
│   │   ├── mvapich
│   │   │   ├── 2.1
│   │   │   └── common -> symlink to mvapich/common
│   │   └── openmpi
│   │       ├── 3.1
│   │       └── common -> symlink to openmpi/common
│   └── 9.1.0
│       ├── bar
│       │   ├── 5.4
│       │   │   ├── avx
│       │   │   ├── avx2
│       │   │   └── .modulerc -> symlink to modulerc.default_lowest_simd
│       │   └── common -> symlink to bar/common
│       ├── foo
│       │   ├── 2.4
│       │   └── common -> symlink to foo/common
│       ├── mvapich
│       │   ├── 2.1
│       │   ├── 2.3.1
│       │   └── common -> symlink to mvapich/common
│       └── openmpi
│           ├── 3.1
│           ├── 4.0
│           └── common -> symlink to openmpi/common
├── intel
│   ├── 2018
│   │   └── ... modules depending on compiler=intel/2018
│   └── 2019
│       └── ... modules depending on compiler=intel/2019
└── pgi
    ├── 18.4
    │   └── ... modules depending on compiler=pgi/18.4
    └── 19.4
        └── ... modules depending on compiler=pgi/18.4

For brevity, we only show the directory structure for the modules depending on the two versions of gcc in detail; the other compilers will have similar structures.

Basically, there is a separate modulepath for each compiler, containing only the modulefiles depending on that specific build of the compiler (and not also depending on something else, like MPI library). This certainly enforces the consistency of loaded modules; one could not load a specific version of gcc (say gcc/9.1.0) and an incompatible version of foo (e.g. foo/1.1), because all of the foo modulefiles are in compiler specific module trees and there is no foo/1.1 in the gcc/9.1.0 moduletree. Conflict statements in the compiler modulefiles will prevent one from loading multiple compilers, thereby preventing multiple compiler specific modulepaths (unless the user explicitly does a module use or similar, and there is only so far one can go in preventing users from shooting themselves in the foot).

The modulefiles for foo and bar are fairly straightforward; we have a common file which does the heavy lifting (and since this can be made independent of version and compiler, this is actually a symlink to a package specific common file in a common directory external to the modulepath trees). We then add some small stubfiles which set variables for the specific build information, which mostly are determined by their position in the tree structure (e.g. the stubfiles under Compiler/gcc/9.1.0 are all going to set compilerTag to gcc/9.1.0, etc), and then invoke the common file.

The modulefiles for openmpi are largely similar. E.g., for gcc/9.1.0 we have a small stubfile like the following (for version 4.0)

#%Module
# Dummy modulefile for openmpi

set version 4.0
set compilerTag gcc/9.1.0
set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/common

which defines the version and compilerTag variables for the OpenMPI version and compiler version, and then invokes the common script

# Common modulefile for openmpi
# Using "modulepath" strategy
# Expects the following variables to have been
# previously defined:
#	version: version of openmpi
#	compilerTag: compiler being used with

# Declare the path where the packages are installed
# The reference to the environment variable is a hack
# for this example; normally one would hard code a path
set rootdir $::env(MOD_GIT_ROOTDIR)
set swroot $rootdir/doc/example/compiler-etc-dependencies/dummy-sw-root

proc ModulesHelp { } {
   global version compilerTag
   puts stderr "
openmpi: Test dummy version of OpenMPI $version

For testing packages depending on compilers/MPI
Version: $version
Compiler: $compilerTag

"
}

module-whatis "Dummy openmpi $version (for $compilerTag)"

# Find the software root.  In production, you should
# hardcode to your real software root
set gitroot $::env(MOD_GIT_ROOTDIR)
set swroot $gitroot/doc/example/compiler-etc-dependencies/dummy-sw-root

# Compute the installation prefix
set pkgroot $swroot/openmpi
set vroot $pkgroot/$version
set prefix $vroot/$compilerTag

# We need to prereq the compiler to allow autohandling to work
prereq $compilerTag

# Set environment variables
setenv MPI_DIR $prefix

set bindir $prefix/bin
set libdir $prefix/lib
set incdir $prefix/include

prepend-path PATH            $bindir
prepend-path LIBRARY_PATH    $libdir
prepend-path LD_LIBRARY_PATH $libdir
prepend-path CPATH           $incdir

# Add the proper modulepath
# In production this should be hard-coded, but using $gitroot for cookbook
set modpathroot $gitroot/doc/example/compiler-etc-dependencies/modulepath
set newmodpath $modpathroot/CompilerMPI/$compilerTag/openmpi/$version
module use $newmodpath

which does the usual stuff (define a help function, whatis string, and sets assorted environment variables like PATH, LIBRARY_PATH, etc. for using OpenMPI), and then adds another directory to the MODULEPATH. This new directory, under CompilerMPI is for modulefiles which depend on the compiler AND the MPI library. (It is assumed that all modulefiles depending on MPI libraries also depend on the compiler). This new branch in the modulepath would have a structure like

CompilerMPI
├── gcc
│   ├── 8.2.0
│   │   ├── mvapich
│   │   │   └── 2.1
│   │   │       └── foo
│   │   │           ├── 1.1
│   │   │           └── common -> symlink to foo/common
│   │   └── openmpi
│   │       └── 3.1
│   │           └── foo
│   │               ├── 1.1
│   │               └── common -> symlink to foo/common
│   └── 9.1.0
│       ├── mvapich
│       │   └── 2.3.1
│       │       └── ... modules depending on gcc/9.1.0 and mvapich/2.3.1
│       └── openmpi
│           └── 4.0
│               └── ... modules depending on gcc/9.1.0 and openmpi/4.0
├── intel
│   ├── 2018
│   │   ├── intelmpi
│   │   │   └── default
│   │   │       └── ... modules depending on intel/2018 and its included MPI lib
│   │   ├── mvapich
│   │   │   └── 2.1
│   │   │       └── ... modules depending on intel/2018 and mvapich/2.1
│   │   └── openmpi
│   │       └── 3.1
│   │           └── ... modules depending on intel/2018 and openmpi/3.1
│   └── 2019
│       ├── intelmpi
│       │   └── default
│       │       └── ... modules depending on intel/2019 and its included MPI lib
│       ├── mvapich
│       │   └── 2.3.1
│       │       └── ... modules depending on intel/2019 and mvapich/2.3.1
│       └── openmpi
│           └── 4.0
│               └── ... modules depending on intel/2019 and openmpi/4.0
└── pgi
    ├── 18.4
    │   ├── mvapich
    │   │   └── 2.1
    │   │       └── ... modules depending on pgi/18.4 and mvapich/2.1
    │   └── openmpi
    │       └── 3.1
    │           └── ... modules depending on pgi/18.4 and openmpi/3.1
    └── 19.4
        └── openmpi
            ├── 3.1
            │   └── ... modules depending on pgi/19.4 and openmpi/3.1
            └── 4.0
                └── ... modules depending on pgi/19.4 and openmpi/3.1

Basically, there is a separate modulepath for each combination of compiler and MPI library, listing all modulefiles depending on that compiler and MPI library. The modulefiles underneath each directory simply do what is needed to load that version of the package built with the specified compiler and MPI library.

This process could be continued further, as needed. E.g., if you had packages which depended on NetCDF, and you had multiple builds of NetCDF for a given compiler/MPI combination, you could add another modulepath tree, e.g. CompilerMPINetCDF, branching on each compiler, MPI, and NetCDF triplet. But things can also get unwieldy if carried too far.

One could also add modulepath trees for disjoint features like SIMD level, but this turns out to be a bit tricky. E.g., if you had a simd branch as well as a compiler branch, with both simd and compiler appearing in Core, things are fine for the Simd and Compiler trees. However, if there were to be modules depending on both, e.g. a CompilerSimd branch, then because compiler and simd are disjoint and could be loaded in either order, the modulefiles for both would need to handle adding the CompilerSimd branch depending on whether the other was previously loaded.

Examples

We now look at the example usage for the Modulepath-based strategy. To use these examples, you must:

  1. Set (and export) MOD_GIT_ROOTDIR to where you git-cloned the modules source
  2. Do a module purge
  3. Set your MODULEPATH to $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Core

Note that we set the MODULEPATH to the Core subdirectory; the Core branch is for those modulefiles that do not depend on compiler or MPI library, and that should be available from the start.

As with the previous cases, we start with a module avail command, and here we see the first big difference:

[mod4 (modulepath)]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Core -
gcc/8.2.0  gcc/9.1.0  intel/2018  intel/2019  pgi/18.4  pgi/19.4
[mod4 (modulepath)]$ module load gcc/9.1.0
[mod4 (modulepath)]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Compiler/gcc/9.1.0 -
bar/5.4/avx(default)  foo/2.4      mvapich/2.3.1  openmpi/4.0
bar/5.4/avx2          mvapich/2.1  openmpi/3.1

- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Core -
gcc/8.2.0  gcc/9.1.0  intel/2018  intel/2019  pgi/18.4  pgi/19.4
[mod4 (modulepath)]$ module load openmpi/4.0
[mod4 (modulepath)]$ module avail
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/CompilerMPI/gcc/9.1.0/openmpi/4.0 -
foo/2.4

- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Compiler/gcc/9.1.0 -
bar/5.4/avx(default)  foo/2.4      mvapich/2.3.1  openmpi/4.0
bar/5.4/avx2          mvapich/2.1  openmpi/3.1

- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Core -
gcc/8.2.0  gcc/9.1.0  intel/2018  intel/2019  pgi/18.4  pgi/19.4

When we first do a module avail, we only see the modulefiles for the compilers. The MPI libraries, foo, bar, etc. all depend on at least the compiler, and so will not become "available" until a compiler (and possibly MPI library as well) is loaded. In a production environment there would likely be other modulefiles available in Core (i.e. an application which is only used as a standalone application and does not provide an API/libraries to link against would likely be placed in Core), but in our simple example, only the compilers appear in Core. As we load a compiler and then an MPI library, additional modulefiles appear available. In a production environment, one might wish to set things up so that a compiler (and maybe MPI library) modulefile is automatically loaded when the user logs in.

The downside of this is that it becomes difficult for an user to know what software is available, at least via the module command. I.e., if application foobar is only built for a single compiler/MPI combination, it will not show up in module avail unless that specific compiler/MPI combination were previously loaded. Lmod adds a module spider command which allows the user to list all packages installed for any compiler/MPI combination, and can be used to also find out which compiler/MPI combinations are needed to access a specific modulefile, but Environment Modules does not have a comparable function at this time. If you were to use this strategy in a production environment, you would likely need to generate lists of available packages (and their compiler/MPI/etc dependencies) in a web page or other documentation area which can be frequently updated.

The standard functionality of selecting the correct build of a package based on the loaded compiler, e.g.

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load pgi/19.4
[mod4 (modulepath)]$ module load openmpi/4.0
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/4.0
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (modulepath)]$ module unload openmpi
[mod4 (modulepath)]$ module switch --auto pgi intel/2019
[mod4 (modulepath)]$ module load openmpi/4.0
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4 (modulepath)]$ module unload openmpi
[mod4 (modulepath)]$ module switch --auto intel gcc/9.1.0
[mod4 (modulepath)]$ module load openmpi/4.0
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/4.0, gcc/9.1.0)
[mod4 (modulepath)]$ module unload openmpi
[mod4 (modulepath)]$ module switch --auto gcc gcc/8.2.0
[mod4 (modulepath)]$ module load openmpi/4.0
ERROR: Unable to locate a modulefile for 'openmpi/4.0'
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0

works as expected. As shown in the last command above, when requesting a module not built for the loaded compiler (e.g. openmpi/4.0 when gcc/8.2.0 is loaded), the module command simply returns Unable to locate a modulefile as there is no corresponding modulefile in the current list of MODULEPATHS.

The functionality of the module switch command depends on version and/or settings of Environment Modules. For version 3.x, or 4.x with the Automated module handling feature disabled, it does not work well, modules which depend on the switched out module are not reloaded, leading to inconsistent environments. But for 4.x versions with the automated module handling feature enabled (as indicated by the --auto after the switch, although that can be made the default), it works as expected. Modules depending on the module being switched out get unloaded and reloaded if possible, as shown below:

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load pgi/19.4
[mod4 (modulepath)]$ module load openmpi
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/4.0
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/4.0, pgi/19.4)
[mod4 (modulepath)]$ module switch --auto pgi intel/2019
Switching from pgi/19.4 to intel/2019
  Unloading dependent: openmpi/4.0
  Reloading dependent: openmpi/4.0
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/4.0, intel/2019)
[mod4 (modulepath)]$ module switch --auto intel intel/2018
Switching from intel/2019 to intel/2018
  ERROR: Unable to locate a modulefile for 'openmpi/4.0'
  WARNING: Reload of dependent openmpi/4.0 failed
  Unloading dependent: openmpi/4.0
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2018
[mod4 (modulepath)]$ mpirun
mpirun: command not found

The modulefiles themselves basically have no support for defaulting a compiler; the modulefiles for the various MPI libraries are simply not even available until a modulefile for a compiler is loaded, as seen below:

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load openmpi/3.1
ERROR: Unable to locate a modulefile for 'openmpi/3.1'
[mod4 (modulepath)]$ module list
No Modulefiles Currently Loaded.
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load gcc/8.2.0
[mod4 (modulepath)]$ module load openmpi
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) openmpi/3.1
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/3.1, gcc/8.2.0)

However, this could be somewhat mitigated by having the modulefile for the default compiler automatically loaded for the user upon login (e.g. in their dot files).

The situation is similar for foo:

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load pgi/19.4
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (pgi/19.4, nompi)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module load openmpi/3.1
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) pgi/19.4   2) openmpi/3.1   3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (pgi/19.4, openmpi/3.1)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module unload openmpi
[mod4 (modulepath)]$ module switch --auto pgi intel/2019
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module load intelmpi
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) intelmpi/default   3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, intelmpi)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module switch --auto intelmpi mvapich/2.3.1
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) mvapich/2.3.1   3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, mvapich/2.3.1)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module switch --auto mvapich openmpi/4.0
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) openmpi/4.0   3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, openmpi/4.0)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module unload openmpi
[mod4 (modulepath)]$ module switch --auto intel/2019 gcc/9.1.0
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (gcc/9.1.0, nompi)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module load mvapich/2.3.1
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) mvapich/2.3.1   3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (gcc/9.1.0, mvapich/2.3.1)
[mod4 (modulepath)]$ module unload foo
[mod4 (modulepath)]$ module switch --auto mvapich openmpi/4.0
[mod4 (modulepath)]$ module load foo/2.4
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) openmpi/4.0   3) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (gcc/9.1.0, openmpi/4.0)

As expected, the correct version of foo is loaded depending on the previously loaded compiler and MPI libraries. Here again we use an intelmpi modulefile to indicate when we wish to use Intel MPI libraries, even though they are enabled by the intel module. The intelmpi modulefile basically just adds the modulepath for the intel-compiler and intelmpi dependent modules.

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load pgi/18.4
[mod4 (modulepath)]$ module load openmpi/3.1
[mod4 (modulepath)]$ module load foo/1.1
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) pgi/18.4   2) openmpi/3.1   3) foo/1.1
[mod4 (modulepath)]$ foo
foo 1.1 (pgi/18.4, openmpi/3.1)
[mod4 (modulepath)]$ module switch --auto pgi intel/2018
Switching from pgi/18.4 to intel/2018
  Unloading dependent: foo/1.1 openmpi/3.1
  Reloading dependent: openmpi/3.1 foo/1.1
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2018   2) openmpi/3.1   3) foo/1.1
[mod4 (modulepath)]$ foo
foo 1.1 (intel/2018, openmpi/3.1)
[mod4 (modulepath)]$ mpirun
mpirun (openmpi/3.1, intel/2018)
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load intel/2019
[mod4 (modulepath)]$ module load foo
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) foo/2.4
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, nompi)
[mod4 (modulepath)]$ module load openmpi
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) intel/2019   2) foo/2.4   3) openmpi/4.0
[mod4 (modulepath)]$ foo
foo 2.4 (intel/2019, nompi)

Note again the same deficiency in the switch report as the Homebrewed Flavors Strategy had; in the last case above wherein we loaded foo with intel/2019 loaded but no MPI module. As expected, a non-MPI build of foo was loaded. However, when an openmpi module is subsequently loaded, foo does not get reloaded and we still have the non-MPI build of foo, as evidenced by the output of the foo command. And that this fact is not obvious from the module list output.

The behavior when defaulting is nicer. Without any compiler or MPI libraries Here we note a deficiency in the switch support as compared to Flavours. In the last example after loading intel/2019 and foo, we have the non-MPI build of foo as expected. However, upon subsequently loading the openmpi module, we still have the non-MPI version of foo loaded, as evidenced by the output of the dummy foo command. I.e., the foo package was not automatically reloaded, as there was no prereq in the foo modulefile on an MPI library (as in the non-MPI build there is no MPI library to prereq). Also note that module list does not really inform one of this fact.

The ability to default partial modulenames in this strategy is mixed. Without any compiler loaded, most modulefiles are not even visible/available. However, once a compiler is loaded, trying to load a module without specifying the version will end up loading the latest version compatible with the loaded compiler (as no incompatible versions are visible) as seen below:

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load foo
ERROR: Unable to locate a modulefile for 'foo'
[mod4 (modulepath)]$ module load gcc/8.2.0
[mod4 (modulepath)]$ module load foo
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) foo/1.1
[mod4 (modulepath)]$ foo
foo 1.1 (gcc/8.2.0, nompi)
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load gcc/8.2.0
[mod4 (modulepath)]$ module load openmpi
[mod4 (modulepath)]$ module load foo
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) openmpi/3.1   3) foo/1.1
[mod4 (modulepath)]$ foo
foo 1.1 (gcc/8.2.0, openmpi/3.1)
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load foo/1.1
ERROR: Unable to locate a modulefile for 'foo/1.1'
[mod4 (modulepath)]$ module list
No Modulefiles Currently Loaded.
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load pgi/18.4
[mod4 (modulepath)]$ module load foo
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) pgi/18.4   2) foo/1.1
[mod4 (modulepath)]$ foo
foo 1.1 (pgi/18.4, nompi)

This is better than in the Flavours or Homebrewed flavors strategies. If one were to use this strategy in production, it is recommended to have a default compiler module (and maybe even an MPI library, etc) automatically loaded when the user logs in, thereby allowing for a reasonable defaulting ability (and more reasonable module avail output).

The situation with bar is similar. We do not have a dummy simd module, so the builds with different CPU vectorization support are specified by appending /avx, etc. to the bar package name. As with previous strategies, if one attempts to load a simd variant which was not built for the compiler loaded, an error will occur.

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load gcc/9.1.0
[mod4 (modulepath)]$ module load bar/5.4/avx2
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) bar/5.4/avx2
[mod4 (modulepath)]$ bar
bar 5.4 (gcc/9.1.0, avx2)
[mod4 (modulepath)]$ module unload bar
[mod4 (modulepath)]$ module load bar/5.4/avx
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) bar/5.4/avx(default)
[mod4 (modulepath)]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4 (modulepath)]$ module unload bar
[mod4 (modulepath)]$ module load bar/5.4/sse4.1
ERROR: Unable to locate a modulefile for 'bar/5.4/sse4.1'
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0

Defaulting is handled well, as shown

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load gcc/9.1.0
[mod4 (modulepath)]$ module load bar
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/9.1.0   2) bar/5.4/avx(default)
[mod4 (modulepath)]$ bar
bar 5.4 (gcc/9.1.0, avx)
[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load gcc/8.2.0
[mod4 (modulepath)]$ module load bar/4.7
[mod4 (modulepath)]$ module list
Currently Loaded Modulefiles:
 1) gcc/8.2.0   2) bar/4.7/sse4.1(default)
[mod4 (modulepath)]$ bar
bar 4.7 (gcc/8.2.0, sse4.1)

In particular, one case specify bar/avx2 or bar/avx or bar/sse4.1 and the latest version of bar consistent with the specification and any previously loaded compiler (or default compiler) will be loaded

Summary of Modulepath-based strategy
  • It is quite difficult for an user to get an inconsistent environment, at least when automated module handling mode is used. The modulepaths are managed so that only modulefiles consistent with the previously loaded compiler/MPI library are visible.
  • With automated module handling mode, the module switch command is very nicely supported.
  • The module command does not lend itself to readily seeing all the packages which are installed on the system, nor seeing for which compiler/MPI combinations a certain package is built. The module avail command just does not show any modules not compatible with the currently loaded compiler and/or MPI libraries. Lmod needed to add a module spider command to address this, but no such functionality currently exists in Environment Modules. If one were to use this in production, you would need to provide something similar to the Lmod spider subcommand, or at least provide frequently updated web pages or similar with this information.
  • This strategy involves the use of many more modulepaths than the previously examined strategies, having at least one modulepath per compiler installed, and typically also one for each compiler/MPI library combination. If you add additional layers of dependency, things get even more complicated, which can be limiting. It is especially problematic if you have two disjoint dependencies -- i.e. neither dependencies is dependent on the other; in this case you might load one or the other, or both (in either order). If there are modulefiles dependent on both of these dependencies, the last one loaded (and only the last one) needs to handle adding the appropriate modulepath for the modulefiles depending on both.
  • In general, a module will fail to load with an error message if any dependent module is not already loaded. The error will actually state that the module cannot be found, as the modulepath containing it will not be added to the MODULEPATH until the modules depended on are loaded. So if you need to default a compiler, etc., you will need to have the user's initialization scripts automatically load the appropriate modulefile upon login.
  • However, what we have been referring to as "more intelligent defaulting" is effectively supported. I.e., if a user requests to load a package without specifying the version desired, the version will be defaulted to the latest version compatible with the currently loaded compiler and other dependencies. This is because the incompatible versions are not visible in the module tree.
Comparison of Strategies

All of the strategies discussed above have their peculiar strengths and weaknesses. The decision of which strategy to use will depend on how these strengths and weaknesses impact your design goals. It is advised to play around in the sandbox environments a bit, as actual use tends to help make clear which downsides you are willing to accept and which you are not.

With the exception of the Flavours, which works best with Environment Modules version 3.x, all of the other strategies work as well or better on the newer 4.x versions of Environment Modules. This difference is most visible in the discussion of features around the module switch command. So in the following discussions we will assume Environment Modules version 3.x for the Flavours strategies, as that is the version it works best with. For all other strategies, we assume Environment Modules version 4.x, with automated module handling mode enabled, as these strategies work best in that scenario.

The Homebrewed flavors, Modulerc-based, and Modulepath-based strategies all require a significant amount of "assembly" in order to get them working for a production environment; the example scripts and procedures provided here should help significantly, but should not be considered as a polished product. You will likely need to customize and extend for your environment. The Flavours Strategy, on the other hand, has been packaged to present as a finished product and so requires less "assembly" to get working, but does not appear to be actively maintained, so the reduced up-front work might be neutralized by the costs of self support.

Basic Dependency Handling

All of the strategies discussed support a basic level of dependency handling. If a user attempts to load a package, they get the build of the package appropriate for the previously loaded compiler or other dependencies, or, if no appropriate build is found, an error message indicating such. E.g, if the user does:

module load gcc/8.2.0
module load foo/1.1

their environment will be set up to run foo version 1.1 built with gcc version 8.2.0.

All of the strategies discussed meet this criterion, with both 3.x and 4.x versions of Environment Modules.

Advanced Dependency Handling (e.g. the module switch subcommand)

Things are more complicated when we allow for the modules upon which other loaded modules might depend to be changed. This generally involves the module switch command.

While all of the strategies handle well the easy case, when we switch a module upon which no other loaded modules depend, the trick comes when one or more currently loaded modules depend on the module being switched out. E.g., if we assume that module foo depends on the previously loaded compiler, then ideally, we would like for a sequence like:

module load intel/2019
module load foo
module switch intel pgi/18.4

to have the same effective outcome as if the user issued the commands:

module load pgi/18.4
module load foo

I.e., switching out the compiler will cause foo to be reloaded with the new compiler dependency. And ideally in this example we would have pgi/18.4 loaded along with the latest version of foo compatible with that compiler.

This is handled reasonably well with the Flavours (using Environment Modules 3.x) and the Homebrewed flavors and Modulepath-based (both using Environment Modules 4.x) strategies. For the Homebrewed flavors and Modulepath-based strategies, this relies on the Automated module handling feature (As of version 4, this is disabled by default. It can be enabled in modulecmd.tcl, or by running module config auto_handling 1, or by adding the --auto flag to the modules command.)

For these strategies, swapping out a compiler or other module upon which a loaded module depends will cause the the module with a dependency to be unloaded, the module being switched being swapped out, and the module that had a dependency being reloaded.

Our example sequence above actually exposes a subtle issue. For the Flavours and Homebrewed flavors strategies, the sequence of loading pgi/18.4 followed by foo (using our example software library) would fail, as foo would be defaulted to version 2.4 and as there was no build of foo/2.4 for pgi/18.4, the module load foo would fail. And indeed, the switch sequence above would not be completely successful under either strategy: for the Homebrewed Flavors Strategy, the compiler would be swapped out and the reload of foo would fail with an error (so effectively equivalent to the second sequence). For the Flavours Strategy, it detects that it would be unable to reload foo, and the switch command fails (i.e. the compiler remains intel/2019).

For the Modulepath-based Strategy, the second sequence (loading pgi/18.4 then foo) would actually work, as the modulepath exposed by loading pgi/18.4 only has versions of foo that are compatible with pgi/18.4. But the switch command does not work as well as would be desired. This is because that as currently implemented, when the automated module handling code does the reload of the module that was unloaded due to dependencies, it attempts to reload based on the fully qualified name of the module that was loaded, and not based on the (possibly partial) name specified when the module was originally loaded. So, after the module load intel/2019, the module load foo will result in foo/2.4 being loaded. When we switch out intel/2019 for pgi/18.4, foo gets unloaded, but after the compiler swap, it tries to reload foo/2.4. As there is no build of foo/2.4 for pgi/18.4 (and thus no modulefile in the current modulepath), the module is not found and is unable to be loaded. As in the Homebrewed flavors case, the compiler is switched but foo is left unloaded (with an appropriate warning to that effect).

This same issue prevents the Automated module handling feature as currently implemented from being very useful with the Modulerc-based. Basically, whenever you have a module loaded that depends on another module, switching out the module depended upon will fail to reload the other module. This is true even in less pathological cases, like:

module load intel/2019
module load foo
module switch intel pgi/19.4

which succeeds under the other three strategies (as foo/2.4 is built for pgi/19.4). This is because when foo is initially loaded in the above scenario, the actual modulename loaded is foo/intel/2019/2.4. With the switch command, foo gets unloaded, the compilers are switched, and the automated modules handling code tries to reload foo/intel/2019/2.4. That modulename clearly depends on intel/2019, not pgi/19.4, and so will fail to load (with an error message). The net result is that the compiler gets switched, but any modules depending on the compiler (or whatever module being switched) get unloaded (with a warning to that effect). It is hoped that a future version of the Automated module handling feature will have an option to reload the dependent modules using the modulename they were loaded with; this should allow for the switch command to work well in the Modulerc-based Strategy, as well as make edges case (like the pgi/18.4 example discussed above) work better with the Modulepath-based Strategy.

There is another edge case which only the Flavours Strategy handles well. Consider a scenario like:

module purge
module load intel/2019
module load foo
foo
# this should be foo/2.4 built for intel/2019 without MPI
module load openmpi
foo
# What build is this, with or without MPI?

With the Flavours Strategy, the final foo will be built for intel/2019 with openmpi support; i.e. Flavours supports the concept of optional prerequisites, and foo has an optional prerequisite on the MPI libraries, so when openmpi is loaded, any previously loaded modules which depend on it (through a regular or optional prerequisite) get reloaded.

All of the other strategies rely on automated module handling mode for automatic reloads, and in this case no reload of foo will be initiated as foo does not have MPI libraries listed as a prerequisite (otherwise a no MPI version could not be loaded). This is especially problematic for the Homebrewed flavors and Modulepath-based strategies, since it is not easily to tell from a module list or similar command which version of foo the environment is set for (for the Modulerc-based Strategy, the full modulename from module list will indicate that).

However, despite some potential issues with some edge cases, the Flavours, Homebrewed flavors, and Modulepath-based strategies all handle this challenge well. The Modulerc-based approach has a poor showing (although at least the modules depending on the module switched out will be unloaded, so the set of loaded modules is consistent).

Note also that this topic shows the most dependence on the version of Environment Modules. For the Flavours Strategy, the switching of a module upon which other modules depended does not work (the wrapper script returned weird errors). The other strategies all rely on the Automated module handling feature to enable the unload and reload of modules which depend on the module being switched out. Thus for 3.x versions of Environment Modules, the switch of a compiler, etc. will only result in the compiler being replaced, and any modules which depend on the compiler will not be reloaded. This means they will still be pointing to versions built for the old compiler, leading to an inconsistent set of modules being loaded. This is particularly bad in the Homebrewed flavors case, as a module load will not even inform one of that fact. Even if automated module handling mode is disabled on version 4.x of Environment Modules situation is better as loaded environment consistency is assured: the switch of a compiler will be denied since a dependent module is detected loaded.

Visibility into what packages are available

Another set of criteria to weigh has to do with visibility into the available packages. We are interested in

  1. Viewing all of the packages installed on a system
  2. Determining what versions of a specific package are available for a given compiler/MPI/etc combination.
  3. Seeing what compiler/MPI/etc combinations a specific version of a package
  4. For packages that we have currently loaded, determining which compiler/MPI/etc they were built for.

In terms of seeing all of packages that are available, the Flavours and Homebrewed flavors strategies are probably the best. The module avail command will list all modulefiles available, which will typically just be a list of packagename/version for each installed version. On a production system with many packages installed, even this can be a bit overwhelming to the user.

The module avail command will also list all modulefiles for every installed package in the Modulerc-based Strategy, but here every build of every version of every package will have a distinct modulefile. I.e., you will not just have foo/1.1 and foo/2.4 listings, but foo/1.1/gcc/8.2.0/nompi, foo/1.1/gcc/8.2.0/mvapich/2.1, foo/1.1/gcc/8.2.0/openmpi/3.1, foo/1.1/intel/2018/nompi, foo/1.1/intel/2018/intelmpi, etc. modulefiles. Actually, the situation is even worse than that, as in this strategy there are often multiple modulefiles for the same build in order to ease navigation of the module tree, e.g. we could have foo/1.1/gcc/8.2.0/nompi and foo/gcc/8.2.0/nompi/1.1 representing the same build of the same package. Whereas an unqualified module avail in the Flavours or Homebrewed flavors strategies can be a bit overwhelming in a large environment, with the Modulerc-based Strategy it can become practically unusable.

While an unqualified module avail in the Flavours, Homebrewed flavors, and especially Modulerc-based strategies can inundate the user with modulenames, the Modulepath-based Strategy has the opposite problem. With the Modulepath-based Strategy strategy, the modulefiles are split across multiple, often multually incompatible, modulepaths, so the module avail command will never return a list of all modulefiles installed, only those available given the previously loaded compiler/MPI libraries/etc. E.g., if a package foobar is only installed for a particular compiler/MPI combination, it will not appear in any module avail listing unless that particular compiler and MPI were previously loaded.

While the number of modulefiles listed in an unqualified module avail command in the Modulerc-based Strategy is unwieldy, if one adds a partial package specification argument, the number of modulefiles returned is greatly reduced. This smaller list can provide information about which compilers/MPI/etc. are compatible with a specific version of a package. For example, to see the compilers for which foo/2.4 is built, we can do something like:

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module avail foo/2.4
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4 -
foo/2.4/gcc/(default)                 foo/2.4/intel/2019/mvapich/2.3.1
foo/2.4/gcc/9.1.0/mvapich/2.3.1       foo/2.4/intel/2019/nompi
foo/2.4/gcc/9.1.0/nompi(default)      foo/2.4/intel/2019/openmpi/4.0
foo/2.4/gcc/9.1.0/openmpi/4.0         foo/2.4/pgi/19.4/nompi(default)
foo/2.4/intel/2019/intelmpi(default)  foo/2.4/pgi/19.4/openmpi/3.1

Similary, to see the builds of foo using gcc compilers, one can do something like:

[mod4 (modulerc)]$ module purge
[mod4 (modulerc)]$ module avail foo/gcc
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4 -
foo/gcc/(default)              foo/gcc/8.2.0/openmpi/3.1/1.1
foo/gcc/8.2.0/(default)        foo/gcc/9.1.0/mvapich/2.3.1/2.4
foo/gcc/8.2.0/mvapich/2.1/1.1  foo/gcc/9.1.0/nompi/(default)
foo/gcc/8.2.0/nompi/(default)  foo/gcc/9.1.0/nompi/2.4
foo/gcc/8.2.0/nompi/1.1        foo/gcc/9.1.0/openmpi/4.0/2.4

The other strategies do not readily provide that information. The next best case is the Modulepath-based Strategy, because here at least a module avail will tell you what versions of a package are compatible with the currently loaded compiler/MPI/etc, e.g.:

[mod4 (modulepath)]$ module purge
[mod4 (modulepath)]$ module load intel/2019
[mod4 (modulepath)]$ module avail foo/2.4
- $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Compiler/intel/2019 -
foo/2.4

But the Flavours and Homebrewed flavors strategies do not readily show what versions of packages are built for which compilers, etc. You would need to load a compiler/MPI/etc combination, and then try loading a particular version of a package.

Lastly, with the Modulerc-based Strategy, the module list command explicitly shows information about the compiler/MPI/etc used to build the loaded modules, as that is part of the full modulename. This information is not explicitly visible in the other three strategies, but that is usually not a problem. As long as the various user commands result in a set of modules wherein every module depending on either a compiler or MPI library, etc. is built for the currently loaded compiler/MPI library/etc., that information is redundant.

The only cases where this is potentially an issue are the sort of edge cases described in the previous section, e.g. if one were to do something like:

module purge
module load intel/2019
module load foo
foo
# this should be foo/2.4 built for intel/2019 without MPI
module load openmpi
foo
# What build is this, with or without MPI?

In the above example, for the Flavours, Homebrewed flavors, and Modulepath-based strategies a module list at the end would simply list something like:

Currently Loaded Modulefiles:
1) intel/2019   2) foo/2.4   3) openmpi/4.0

For the Flavours Strategy, foo would be built with MPI support, but for the other two, foo would still be the non-MPI build, which is not readily apparent from the above output (although possibly could be inferred by the ordering of the modules).

Ease of Navigating the Module Tree

The final criterion to be discussed is the ease of navigating the module tree. Ideally, we would like it so that when an user gives a partial modulename the system resolves in a sensible manner. Typically a partial modulename will be a package name without a version, in which case ideally the latest version of the package (compatible with previously loaded compiler, etc), should be loaded. And indeed, for a simple case like:

module purge
module load intel
module load foo

all the strategies do just that. The intel module will default to intel/2019, and foo to version 2.4 built for intel/2019. However, with even a slight modification things begin to diverge, e.g.:

module purge
module load intel/2018
module load foo

In this second case, the Flavours and Homebrewed flavors strategies still default the version of foo to version 2.4, despite there being no build of foo/2.4 for intel/2018. That is because they do not do intelligent defaulting, and will simply default the version of foo to the highest version they see even though it is not compatible with the loaded compiler. Thus the load of foo will fail for Flavours and Homebrewed flavors strategies. However, for the Modulerc-based and Modulepath-based strategies, the module load of foo will successfully load foo version 1.1 built for intel/2018. For the Modulepath-based Strategy, this is because the only foo modules currently visible are compatible with the loaded intel/2018 compiler, and it simply defaults to the highest version seen. For the Modulerc-based Strategy, the .modulerc file directly under foo directs it down the subdirectory for the intel compiler family, which has a .modulerc defaulting to the 2018 subdirectory (as the currently loaded compiler version), which has a .modulerc defaulting to the nompi subdirectory (the MPI family as no MPI library was loaded). At this point, there is not another .modulerc file, so the latest version will be defaulted to, and foo/intel/2018/nompi/1.1 module will be loaded.

The situation is analogous when there are chained dependencies (e.g. compiler and MPI libraries); the Flavours and Homebrewed flavors strategies will default to the latest version of the package they know about, regardless of whether it is compatible with the loaded environment. If it is compatible, it will be loaded and all is well, if not the load command will fail with an error message. Not horrible; the set of loaded modules is still consistent, but not terribly user-friendly either.

For the Modulerc-based and Modulepath-based strategies, the latest build compatible with the various dependencies will be loaded (or if no build is compatible, the load will fail with an appropriate error).

For the Flavours, Homebrewed flavors, and Modulepath-based strategies, typically modulenames only consist of the package name and version components, so generally only partial modulenames seen are packages without a version. The Modulerc-based Strategy has much longer modulenames, and therefore a greater variety of partial modulenames, which can allow for interesting possibilities. For example, one could do module load foo/pgi, which (assuming no compiler was previously loaded), would load the latest version of foo built with the latest version of the pgi compiler. The module command will start looking for a modulefile in the foo/pgi subdirectory of the foo module structure, where the .modulerc file will attempt to default to the version of pgi which was loaded, but as no compiler was loaded, the modulerc file will terminate without setting a default, and the modulecmd will default to the latest version, and descend into that directory (19.4). Here, the .modulerc will attempt to default to the MPI family, but since none is loaded (as that would require a compiler to have been loaded), it defaults to the nompi directory, and then to the latest version in the nompi subdirectory. So the module foo/pgi/19.4/nompi/2.4 will get loaded, which will also cause pgi/19.4 to get loaded. Should we have issued that module load after having loaded a non-pgi compiler, the same module file would get loaded, but it would detect an incompatible previously loaded compiler and abort with an error. Should a pgi compiler have been previously loaded, we would have gotten the latest version of foo built for that compiler (and possibly with any previously loaded MPI library).

The above is supported because we create a PACKAGE/COMPILER_FAMILY/COMPILER_VERSION/.../PKGVERSION tree in the package's modulefile directory for packages depending on the compiler. Possibly with additional components to the modulename for MPI libraries, etc. These are created to allow for the more basic module load foo case, but because the module path is descended component by component, it works if you jump in in the middle as well (and will fail if you try something stupid like :

module load intel/2019
module load foo/pgi

But a more interesting case is when there are somewhat disjoint options on the builds. Consider the case of the bar package from the examples, which in addition to the compiler and builds supporting different SIMD levels. For the Flavours and Homebrewed flavors strategies, we handled this with a pseudo-simd package and modules. This worked, and allowed for one to select a build that matched both compiler and SIMD level, but you likely need to manually select a bar version compatible with both of these, as these strategies will not do intelligent defaulting of versions.

For the Modulepath-based Strategy, we added the SIMD label as a final component to the modulename, but the defaulting was somewhat simple (e.g. alway default to lowest SIMD).

With the Modulerc-based Strategy, we added an additional naming scheme for the bar modulefiles, so they can start with bar/BARVERSION, or bar/COMPILER_FAMILY, or bar/SIMD_LEVEL. This not only makes it easier to view which versions of bar for which compilers are built with support for a given SIMD level (e.g. module avail bar/avx2), but also one can do something like:

module load pgi/19.4
module load bar/avx

to load the latest version of bar built for pgi/19.4 with AVX support. While there is overhead associated with adding a new naming scheme like that (you add another set of modulefiles to bar and a new subdirectory tree), this can be done on a per package basis and when done judiciously it can be convenient for the users.

Conclusions

We have presented four strategies for dealing with modulefiles for packages with multiple builds depending on compiler, MPI, and/or other factors. All four strategies can deal with the basic requirement of loading of the correct build of a package depending on the previously loaded dependencies, or failing with a reasonable error message if no such build is available. They all have their own strengths and weaknesses. This document tried to present these strategies objectively and has been made to help you to evaluate how to handle this issue at your site.

Ensure user fully qualify the modules they use

When managing a vast software catalog accessible by multiple users, it is often desired that these users understand exactly what are the software they enable in their environment. This is especially important when new software versions are introduced in the catalog.

On a regular Modules installation, if multiple versions of a given module are available, latest version is selected when user load this software module by specifying its generic name (module load software). So when a new version is installed for software, users automatically end up with this new version enabled in their environment.

Depending on software, a precise version control may be required to avoid issues. In this case the implicit default version selection done by Modules is not desired. This recipe describe the way to handle this kind of situation.

Implementation

Starting version 4.3 of Modules, a new option called implicit_default is introduced. When enabled (which means configuration option is set to 1) if a module is specified by its generic name an implicit default is automatically defined (latest version) if no explicit default is defined. When implicit_default configuration option is disabled (when set to 0), no implicit default is computed and an error is returned to user in case module has been specified by its generic name but no default version has been set.

So to ensure user fully qualify the modules they use, this implicit_default configuration option should be disabled. It could be achieved by setting the option in the initrc file installed in the configuration directory (or in the modulerc file installed in the initialization script directory if this location is preferred).

initrc
#%Module

module config implicit_default 0

It may be desired to lock this option, to ensure users do not alter it afterward. The locked_configs configuration option fills this need. By setting this option and the implicit_default in a site-specific configuration script, users will not be able to change the implicit_default behavior you configure.

siteconfig.tcl
# disable implicit_default option
setConf implicit_default 0

# forbid `implicit_default` config option superseding
lappendConf locked_configs implicit_default

Compatible with Modules v4.3+

Installation

Create site-specific configuration directory if it does not exist yet:

$ mkdir /etc/environment-modules

Then copy there the initialization RC file of this recipe:

$ cp example/ensure-user-qualify-modules/initrc /etc/environment-modules/

Or if you want to enforce the implicit_default disablement setup for users, copy there the site-specific configuration script of this recipe:

$ cp example/ensure-user-qualify-modules/siteconfig.tcl /etc/environment-modules/

If you currently use Modules version 4.3, copy the configuration script specific to this version:

$ cp example/ensure-user-qualify-modules/siteconfig.tcl-4.3 /etc/environment-modules/siteconfig.tcl

Note

Defined location for the site-specific configuration script may vary from one installation to another. To determine the expected location for this file on your setup, check the value of the siteconfig option:

$ module config siteconfig

Once file is installed, you could verify option value is correct:

$ module config implicit_default
Modules Release 4.3.0 (2019-07-26)

- Config. name ---------.- Value (set by if default overridden) ---------------
implicit_default          0 (locked)
Usage example

Enable the modulepath where the example modulefiles are located:

$ module use example/ensure-user-qualify-modules/modulefiles

Attempt to load a module, which does not defined a default version, by its generic name:

$ module load -v softa
ERROR: No default version defined for 'softa'

Load succeed if module is fully qualified:

$ module load -v softa/1
Loading softa/1

For modules which define explicitly a default version, they can still be loaded by their generic name:

$ module load -v softb
Loading softb/1

Expose procedures and variables to modulefiles

Same piece of code may be relevant for multiple modulefiles in your setup. Sharing code (procedures and variables) is preferred to redefining it in each modulefile. The following recipe describes an efficient way to define Tcl procedures and variables that will be then available from the modulefile evaluation context.

Implementation

To expose site-specific procedures and variables across all modulefiles during their evaluation a site-specific configuration script is proposed.

This script exposes registered procedures to the Tcl interpreters that evaluate either modulerc or modulefile scripts.

By using the env global array, which holds environment variables, variables could also be exposed to those Tcl interpreters. Variables exposed this way, will be found set within modulefile or modulerc evaluation context but will not be exported to the environment changes the modulecmd.tcl script produces.

siteconfig.tcl
# append site-specific procedures referenced in the g_siteProcsToExposeToItrp
# variable defined below to the list of procedures exposed in the modulefile
# and modulerc evaluation interpreters
proc addSiteProcsToItrpAliasList {itrpaliasvname keyname args} {
   # ensure this proc is only called once at itrpaliasvname initialization
   trace remove variable $itrpaliasvname write addSiteProcsToItrpAliasList
   upvar #0 $itrpaliasvname itrpaliasv
   # register each site-specific procedure
   foreach procname $::g_siteProcsToExposeToItrp {
      if {$keyname ne {}} {
         set itrpaliasv($procname) $procname
      } else {
         lappend itrpaliasv $procname $procname
      }
   }
}
trace add variable ::g_modfileBaseAliases write addSiteProcsToItrpAliasList
trace add variable ::g_modrcAliases write addSiteProcsToItrpAliasList


# Define here site-specific procedures that should be exposed to modulefile
# and modulerc interpreter contexts
# *Beware* not to override an existing procedure of modulecmd.tcl script
proc mysiteproc {} {
   return sitevalue
}

# list all site-specific procedures to expose to modulefile and modulerc
# interpreter contexts
set g_siteProcsToExposeToItrp [list mysiteproc]

# Define here site-specific variables that should be exposed to modulefile
# and modulerc interpreter contexts. Use environment variable env array to
# transmit these variables
# *Beware* not to override an existing environment variable
set env(mysitevar) sitevarvalue

Compatible with Modules v4.2+

Installation

Create site-specific configuration directory if it does not exist yet:

$ mkdir /etc/environment-modules

Then copy there the site-specific configuration script of this recipe:

$ cp example/expose-procs-vars-to-modulefiles/siteconfig.tcl /etc/environment-modules/

Note

Defined location for the site-specific configuration script may vary from one installation to another. To determine the expected location for this file on your setup, check the value of the siteconfig option on Modules version 4.3 or above:

$ module config siteconfig

On older version of Modules, check the modulecmd.tcl script:

$ grep '^set g_siteconfig ' $MODULES_CMD

The configuration script proposed should then be adapted to your needs:

  • define your own procedures
  • register them into the g_siteProcsToExposeToItrp list variable to expose them to the modulefile and modulerc evaluation contexts
  • define your own variables using the env array variable
Usage example

Enable the modulepath where the example modulefiles are located:

$ module use example/expose-procs-vars-to-modulefiles/modulefiles

Display one of the example modulefiles that makes use of the site-specific procedure and variable defined in the siteconfig.tcl script:

$ module show foo
-------------------------------------------------------------------
.../modulefiles/foo/1:

setenv          FOO1 sitevalue
setenv          FOO2 sitevarvalue
-------------------------------------------------------------------

Load one example modulefile and check that the environment variable it defines, which rely on the site-specific procedure and variable, are well set:

$ module load bar
$ echo $BAR1
sitevalue
$ echo $BAR2
sitevarvalue

Hide and forbid modules

module is often used to give access to hundreds of different software to a large variety of users, each of them provided in several versions. In such situation users ends up facing thousands of modulefiles which could be confusing.

On the other side, site's staff needs to decommission older software version to ensure for instance that only the best matches for their hardware resources remain. Such work needs specific care to avoid breaking the computing workflow of users.

This recipe provides examples on how to hide or/and forbid modulefiles to better cope with software life cycles and help both user and staff to get their bearings in large environments.

Implementation

The module-hide and module-forbid modulefile commands block respectively the visibility or the load evaluation of the modulefiles they target. These two commands have a variety of options to inhibit their action or display specific messages. Combining both commands enables to hide and forbid specified modulefiles.

Compatible with Modules v4.6+

Note

Modules v4.7+ is required to use the --hidden-loaded option of module-hide command.

Usage examples

The use cases below describe situations where hiding or/and forbidding use of modulefiles can be helpful.

The following set of modules will be used in the examples of this recipe. Each use case will progressively add module-hide and module-forbid statements in modulepath's .modulerc file to improve adapt module visibility or forbid access to them.

$ module avail
-- ../example/hide-and-forbid-modules/modulefiles --
appA/1.0  bioappA/1.0  chemappA/1.0  deplibA/1.0
appA/2.0  bioappA/2.0  chemappA/2.0  deplibA/2.0
appB/1.0  bioappB/1.0  chemappB/1.0  deplibB/1.0
appB/2.0  bioappB/2.0  chemappB/2.0  deplibB/2.0
Limiting view to the useful software only

With a large number of available modules it is interesting to reduce user's visibility to only report the modules of interest.

Some modules may require other to be loaded as dependency. This is the case for appA and appB which depend on deplibA or deplibB. These two dependency libraries are not interesting by themselves so they could be hidden by default but they should stay findable for appA or appB modules to load their dependency.

module-hide --soft could be used to address this need. Designated module will be hidden unless searched. Following lines are added to modulepath's .modulerc file

# hide modules only loaded as dependency
module-hide --soft deplibA
module-hide --soft --hidden-loaded deplibB

As a result deplibA and deplibB are not returned anymore on a global avail sub-command:

$ module avail
-- ../example/hide-and-forbid-modules/modulefiles --
appA/1.0  appB/2.0     bioappB/1.0   chemappA/2.0
appA/2.0  bioappA/1.0  bioappB/2.0   chemappB/1.0
appB/1.0  bioappA/2.0  chemappA/1.0  chemappB/2.0

Yet these modules are still found when loading the modules of the application requiring them:

$ module load appA
Loading appA/2.0
  Loading requirement: deplibA/2.0

In some cases it is desirable to hide such dependency modules also when they are loaded. This could be achieved by using the --hidden-loaded option of the module-hide command. Hidden loaded modules do not appear by default on list sub-command output unless --all is set. Queries like is-loaded still detect such modules as loaded even if hidden. In addition loading or unloading informational messages related to these modules are not reported unless a verbosity mode higher than verbose is configured.

$ module load appB
$ module list
Currently Loaded Modulefiles:
 1) appB/2.0
$ module list --all
Currently Loaded Modulefiles:
 1) deplibB/2.0   2) appB/2.0
$ module is-loaded deplibB
$ echo $?
0

Going further, among scientific applications some are only useful for a given scientific field. A site may provide many software covering many scientific fields but a user may only be concerned by one of these fields. In our example software are provided for biology users (bioappA and bioappB) and other software for chemistry users (chemappA and chemappB).

Say every biology users are part of a bio Unix group and every chemistry users are member of a chem Unix group. Available software visibility could be improved by only reporting the bio software to the bio users and the chem software to the chem users.

# hide modules not from user's scientific field
module-hide --soft --not-group bio bioappA bioappB
module-hide --soft --not-group chem chemappA chemappB

With the above statements put in modulepath's .modulerc file, the bio software are not seen anymore by chem users:

$ module avail
-- ../example/hide-and-forbid-modules/modulefiles --
appA/1.0  appB/1.0  chemappA/1.0  chemappB/1.0
appA/2.0  appB/2.0  chemappA/2.0  chemappB/2.0

However as they are softly hidden, these modules can still be seen if queried or if --all option is used:

$ module avail bioappA
-- ../example/hide-and-forbid-modules/modulefiles --
bioappA/1.0  bioappA/2.0
$ module avail --all
-- ../example/hide-and-forbid-modules/modulefiles --
appA/1.0  bioappA/1.0  chemappA/1.0  deplibA/1.0
appA/2.0  bioappA/2.0  chemappA/2.0  deplibA/2.0
appB/1.0  bioappB/1.0  chemappB/1.0  deplibB/1.0
appB/2.0  bioappB/2.0  chemappB/2.0  deplibB/2.0
Software requiring administrative actions prior usage

Some software may require an administrative step to get allowed to use them, like the signature of a user agreement. Such software should be reported among the available modules but should not be used prior the administrative step achieved.

In our example, the chemappA application requires a user charter to be signed and then sent to the site staff. The access to the chemappA is controlled by the chemappA Unix group: a user needs to be part of this group to access and use the application.

# forbid use of modules unless software user agreement is signed
set msg {User agreement for ChemAppA application must be validated to use it
1. Please connect to https://ChemAppA.example.org
2. Read the Term of Use and sign it
3. Send back this signed agreement to our service desk}
module-forbid --not-group chemappA --message $msg chemappA

With the above module-forbid statement put in modulepath's .modulerc file, the software is still visible but its load is denied unless if the user is part of the chemappA group.

$ module avail chemappA
-- ../example/hide-and-forbid-modules/modulefiles --
chemappA/1.0  chemappA/2.0
$ module load chemappA
ERROR: Access to module 'chemappA/2.0' is denied
  User agreement for ChemAppA application must be validated to use it
  1. Please connect to https://ChemAppA.example.org
  2. Read the Term of Use and sign it
  3. Send back this signed agreement to our service desk

A specific message is provided through the --message option to guide the user to complete the required administrative step. Once group membership is acquired, the module can be seamlessly loaded.

$ id --groups --name
chemappA chem
$ module load -v chemappA
Loading chemappA/2.0

Note

Do not forget to protect the access to the directory where the software is installed to really ensure that only allowed users can use it.

Software limited to particular users

Some applications may be restricted to a limited set of users. For instance because such application should not be disclosed or because it requires a license that is paid only by a few users.

In our example, the appC application works with token-based licenses. The only users that can use this software are those that have paid for a license token. Other users should not access nor even see the availability of this application. Users that have bought a license token are added to the appC Unix group.

# fully hide and forbid modules unless user owns a license token
module-hide --hard --not-group appC appC
set msg {Access is restricted to owners of license token}
module-forbid --not-group appC --message $msg appC

The above statements have been added in modulepath's .modulerc file. The module-hide --hard command is used to completely remove visibility for non-authorized users.

$ module avail
-- ../example/hide-and-forbid-modules/modulefiles --
appA/1.0  appB/1.0  chemappA/1.0  chemappB/1.0
appA/2.0  appB/2.0  chemappA/2.0  chemappB/2.0
$ module avail appC
$ module load appC
ERROR: Unable to locate a modulefile for 'appC'

The module-forbid statement added for appC helps to get a clear error message for the non-authorized users that are aware of the existence of the module name and version (instead of getting a modulefile location error).

$ module load appC/2.0
ERROR: Access to module 'appC/2.0' is denied
  Access is restricted to owners of license token

Alternatively such restrictions on modulefiles can be achieved by adapting file permission mode instead of adding statements in modulepath's .modulerc file.

$ chmod 640 ../example/hide-and-forbid-modules/modulefiles/appC/*
$ chgrp appC ../example/hide-and-forbid-modules/modulefiles/appC/*

However restricting file permission mode does not enable to authorize several Unix groups or users to access those modulefiles or to have specific error messages unlike when module-hide and module-forbid commands are used.

Note

Do not forget to protect the access to the directory where the software is installed to really ensure that only the authorized users can use it.

Software life cycle

When providing a new software version, it may be interesting to have a test phase, for instance during one week, to make this new version checked by some some pilot users prior the general availability.

In our example, we are currently November 13th and chempappB/2.0 has been installed two days ago. A test phase is ongoing and users that are member of the pilot Unix group can already access this software prior its general availability set for November 18th.

# test new version of chemappB prior general availability
module-hide --before 2020-11-18 --not-group pilot chemappB/2.0

With the above statement added to modulepath's .modulerc file, common users will not see the new module until November 18th.

$ date
Fri 13 Nov 2020 02:04:21 PM CET
$ module avail chemappB
-- ../example/hide-and-forbid-modules/modulefiles --
chemappB/1.0

At some point software need to be decommissioned to ensure for instance that buggy or under-optimized versions are not used anymore by users.

In our example, appA/1.0 needs to be removed as a newer version is available for appA and this new version fixes a lot of issues that were encountered with version 1.0. Decommission date for appA/1.0 is planned on November 18th.

# decommission of old version of appA
set nearmsg {appA/1.0 will be decommissioned, please use appA/2.0}
set msg {appA/1.0 is decommissioned, please use appA/2.0}
module-hide --hard --after 2020-11-18 --not-group eol appA/1.0
module-forbid --after 2020-11-18 --not-group eol \
   --nearly-message $nearmsg --message $msg appA/1.0

The above statements added to modulepath's .modulerc file will remove visibility and access to the decommissioned module on November 18th. Only users part of the eol Unix group will still keep an access to the software (which is useful in case some users are unfortunately stuck on this specific version of appA). Specific messages are set to guide users toward selecting another version for appA.

$ module avail appA
-- ../example/hide-and-forbid-modules/modulefiles --
appA/1.0  appA/2.0
$ module load appA/1.0
Loading appA/1.0
  WARNING: Access to module will be denied starting '2020-11-18'
    appA/1.0 will be decommissioned, please use appA/2.0
  Loading requirement: deplibA/1.0

Prior decommission date, module will still be visible and loadable. However when loading the module a warning message will appear when the expiry date will be close. The nearly_forbidden_days module configuration defines starting when such warning message should be reported prior decommission date (14 days by default).

$ module config nearly_forbidden_days
Modules Release 4.6.0 (2020-09-16)

- Config. name ---------.- Value (set by if default overridden) ---------------
nearly_forbidden_days     14

Once the decommissioned date is over, appA/1.0 has disappeared unless for users member of the eol Unix group.

$ date
Thu 19 Nov 2020 02:46:27 PM CET
$ module avail appA
-- ../example/hide-and-forbid-modules/modulefiles --
appA/2.0
$ module load appA/1.0
ERROR: Access to module 'appA/1.0' is denied
  appA/1.0 is decommissioned, please use appA/2.0

Note

Do not forget to protect the access to the directory where the software is installed prior its general availability and after its decommissioned date.

Inhibit output of informative messages

Since Modules v4.2, additional module load or unload triggered by the load or the unload of a modulefile are reported to the user to help understand what happened automatically. These informative messages may not be desired sometimes and here is a proposed way to inhibit them.

Implementation

Starting version v4.3, a verbosity configuration option is introduced to increase or decrease the variety of the messages produced by the module command. To inhibit the output of the info-level messages, the concise verbosity level should be selected:

$ module config verbosity concise

For v4.2 versions, a site-specific configuration script is proposed to inhibit the output of the info-level messages.

siteconfig.tcl
set g_inhibit_inforeport 1 ;# Non-critical info reporting disabled if == 1

# override 'reportInfo' procedure to inhibit messages if g_inhibit_inforeport
# is set to 1
proc reportInfo {message {title INFO}} {
   if {!$::g_inhibit_inforeport} {
      # use reportError for conveniance but there is no error here
      reportError $message 0 $title 0
   }
}

Compatible with Modules v4.2

Installation (only for version older than v4.3)

Create site-specific configuration directory if it does not exist yet:

$ mkdir /etc/environment-modules

Then copy there the site-specific configuration script of this recipe:

$ cp example/inhibit-report-info/siteconfig.tcl /etc/environment-modules/

Note

Defined location for the site-specific configuration script may vary from one installation to another. To determine the expected location for this file on your setup, check the modulecmd.tcl script:

$ grep '^set g_siteconfig ' $MODULES_CMD
Usage example

With a bare bar modulefile:

bar
#%Module

And a foo modulefile that pre-requires bar:

foo
#%Module
prereq bar

Enable the modulepath where the example modulefiles are located:

$ module use example/inhibit-report-info/modulefiles

Load foo with auto handling mode enabled. The info-level message inhibition should let foo load quiet:

$ module load --auto foo
$

Log module command

It is sometimes desired to better understand the module usage on the system we manage. Especially to determine what are the modulefiles most used and what are those never used that could be removed. This recipe describes a way to track the module usage by logging each request made.

Implementation

Logging module commands is implemented by the use of a site-specific configuration that supersedes the definition of the module command to execute the logger command, in order to send a message to the system log, prior processing the module command itself. The log message sent describes the module command called and who called it.

siteconfig.tcl
# override 'module' procedure to log each call made by user
rename ::module ::__module
proc module {command args} {
   if {[depthState modulename] == 0} {
      exec logger -t module "[get-env USER]: $command [join $args]"
   }
   return [eval __module "{$command}" $args]
}

Compatible with Modules v4.2+

Installation

Create site-specific configuration directory if it does not exist yet:

$ mkdir /etc/environment-modules

Then copy there the site-specific configuration script of this recipe:

$ cp example/log-module-commands/siteconfig.tcl /etc/environment-modules/

Note

Defined location for the site-specific configuration script may vary from one installation to another. To determine the expected location for this file on your setup, check the value of the siteconfig option on Modules version 4.3 or above:

$ module config siteconfig

On older version of Modules, check the modulecmd.tcl script:

$ grep '^set g_siteconfig ' $MODULES_CMD
Usage example

Loading a bare modulefile:

$ module load example

A log entry can then be retrieved from system log files:

$ journalctl -q -t module -n 1
Sep 12 20:24:01 hostname module[9925]: username: load example

Return file basename on module-info name for full path modulefile

When module name is specified as a full pathname, the module-info name command used in modulefile was returning the file basename on Modules compatibility version. Starting version 4 of Modules, the full pathname is returned when module is specified this way as once loaded this module is identified by its full pathname. This recipe describes a way to get back the behavior of Modules compatibility version for the module-info name modulefile command.

Implementation

Return file basename on module-info name for modules specified as full path modulefile is implemented by the use of a site-specific configuration that supersedes the definition of the module-info name command to return modulefile basename instead of full pathname.

siteconfig.tcl
# override 'module-info' procedure to return file basename for 'name' action
# when modulefile is specified as a full path file
rename ::module-info ::__module-info
proc module-info {what {more {}}} {
   if {$what eq {name}} {
      set name [currentModuleName]
      if {[isModuleFullPath $name]} {
         return [file tail $name]
      } else {
         return $name
      }
   } else {
      return [__module-info $what $more]
   }
}

Compatible with Modules v4.2+

Installation

Create site-specific configuration directory if it does not exist yet:

$ mkdir /etc/environment-modules

Then copy there the site-specific configuration script of this recipe:

$ cp example/module-info-name-return-basename/siteconfig.tcl /etc/environment-modules/

Note

Defined location for the site-specific configuration script may vary from one installation to another. To determine the expected location for this file on your setup, check the value of the siteconfig option on Modules version 4.3 or above:

$ module config siteconfig

On older version of Modules, check the modulecmd.tcl script:

$ grep '^set g_siteconfig ' $MODULES_CMD
Usage example

With an info/name modulefile that sets an environment variable with the result of the module-info name modulefile command:

info/name
#%Module
setenv MODNAME [module-info name]

Load info/name by its full pathname then check value of the environment variable set:

$ module load ./example/module-info-name-return-basename/modulefiles/info/name
$ echo $MODNAME
name

Version control your Modulefiles using Git

The Modules concept is a great way to manage versions of tools on a common server, but managing the Modulefiles themselves is error-prone when not controlled using a SCM like Git.

Goals
  • To be able to create, edit, and test Modulefiles without the risk of breaking other users.
  • To be able to track changes to the system-wide Modulefiles using Git.
  • To enable testing of new tool versions (with their associated Modulefiles) before making the new version generally available (since the new Modulefile will not be public until pushed).
Assumptions

Details are tweakable via TCL variables

  • You have Environment Modules version 4.1 or later installed on your system, and Git version 2.4 or later.

  • You have a Unix user named modules that exists only for managing the Modulefiles, and controlling updates to them.

  • The path /home/modules/modulefiles is in your modulerc file:

    module use --append /home/modules/modulefiles

  • Other Unix users (not named modules) use the modules from /home/modules/modulefiles.

Principles of Operation

First we create a git repo for the Modulefiles.

Then we install a Modulefile named localmodules that, when loaded, switches MODULEPATH to a locally-created git clone of the Modulefiles. When unloaded, it switches MODULEPATH back to the default.

After this, any time a user wants to edit the Modulefiles, he works in his local git repo. After editing, testing, and commiting to the local git repo, git push updates the main repository, which (assuming the user knows the password for user modules) automatically updates /home/modules/modulefiles.

Implementation
localmodules
#%Module4.1
##
## When loaded, use a local git repo for our Modulefiles,
## instead of the system-wide location that is typically used.
##
## This is useful for editing and testing Modulefiles.
##
## Tune to your system:
## $specialuser, $localmoddir, $globalmodfetch, $globalmodpush
##
## Author: Scott Johnson <scottjohnsoninsf@gmail.com>
##
## Originally developed at:
## github.com/scottj97/environment-modules-in-git


eval set [array get env HOME]

# Special Unix username that owns the main Modulefiles repository
set specialuser modules

# Where each user's local copy should go:
set localmoddir $HOME/modulefiles

# Where `git fetch` can find the main repository.
# This must be a filesystem path, since it will also be used by Modules:
set globalmodfetch /home/modules/modulefiles

# Where `git push` should write to the main repository:
set globalmodpush ssh://modules@localhost/home/modules/modulefiles



proc ModulesHelp {} {
   global localmoddir
   puts stderr "localmodules: switch to local Git repo for Modulefiles\n"
   puts stderr {This is useful for editing and testing Modulefiles.}
   puts stderr {Usage:}
   puts stderr {   $ module load localmodules}
   puts stderr "Then edit and test modulefiles in $localmoddir"
   puts stderr {When complete, git commit and push, then}
   puts stderr {   $ module unload localmodules}
}

module-whatis   {switch MODULEPATH to local git repo}

# Runs `system` and dies if error code is nonzero
proc safeSystem cmd {
   set retcode [system $cmd]
   if {[expr {$retcode != 0}]} {
      error "`$cmd` returned non-zero exit code: $retcode"
   }
}

# Make sure $localmoddir is what we expect, so we don't clobber
# anybody's work if they happen to have something unexpected here
proc ensureProperLocalmoddir {} {
   global localmoddir
   global globalmodfetch
   global globalmodpush

   # Make sure it's a directory
   if {![file isdirectory $localmoddir]} {
      error "a file named $localmoddir already exists, and\
         I don't want to clobber it"
   }

   # Make sure it has the expected .git remote setup
   if {![file isdirectory [file join $localmoddir .git]]} {
      error "expected git repo inside $localmoddir, found none"
   }
   safeSystem "if \[ `git -C $localmoddir remote get-url origin` !=\
      \"$globalmodfetch\" ]; then exit 1; fi"
   safeSystem "if \[ `git -C $localmoddir remote get-url --push origin` !=\
      \"$globalmodpush\" ]; then exit 1; fi"
}

# No $localmoddir exists, so run `git clone` to create
proc createLocalmoddir {} {
   global localmoddir
   global globalmodfetch
   global globalmodpush

   safeSystem "git clone $globalmodfetch $localmoddir"
   safeSystem "git -C $localmoddir remote set-url --push origin $globalmodpush"
}

proc checkRepoStatus {} {
   global localmoddir
   safeSystem "git -C $localmoddir remote update"
   safeSystem "git -C $localmoddir status"
}

# Special modules user not allowed to load or unload this.
# Why? Because it would defeat the purpose of tracking changes, if the
# changes were committed by this anonymous special user.
eval set [array get env USER]
if {$USER == $specialuser} {
   error "special user $specialuser should not load this module"
}

# create directory if necessary
if [module-info mode load] {
   if {![file exists $localmoddir]} {
      createLocalmoddir
   }
   ensureProperLocalmoddir
}

## These two work for load, but not for unload
## (the $globalmodfetch doesn't get put back in):
##   remove-path MODULEPATH $globalmodfetch
##   append-path MODULEPATH $localmoddir

## These two work for load, but not for unload
## (the $globalmodfetch doesn't get put back in):
##   module unuse $globalmodfetch
##   module use --append $localmoddir

## This method assumes that $MODULES_REPO is part
## of MODULEPATH, e.g. in your modulerc:
##   setenv MODULES_REPO /home/modules/modulefiles
##   module use /\$MODULES_REPO


if [module-info mode load] {
   setenv MODULES_REPO $localmoddir
}
if [module-info mode unload] {
   # unsetenv gets converted to setenv since we're unloading
   unsetenv MODULES_REPO $globalmodfetch
}

if [module-info mode load] {
   puts stderr "\nSwitched to local modulefiles in $localmoddir"
   puts stderr {When editing and testing complete, git commit and push, then:}
   puts stderr "   $ module unload localmodules\n"
   checkRepoStatus
}
Installation

First convert the existing Modulefiles into a git repo at /home/modules/modulefiles, as user modules:

cd /home/modules/modulefiles
git init
git add .
git commit -m 'Initial checkin of existing Modulefiles'
# Enable updates when receiving pushes:
git config --local receive.denyCurrentBranch updateInstead

Edit your global $MODULESHOME/init/modulerc file to use the new env var MODULES_REPO instead of hard-coding the path (requires Modules 4.1). Your modulerc should have:

setenv MODULES_REPO /home/modules/modulefiles
module use /\$MODULES_REPO

(The extra slash required before $MODULES_REPO is a bug to be fixed in 4.2.3.)

Copy the localmodules file from the Modules source tree to your repo:

cd /home/modules/modulefiles
curl --output localmodules https://raw.githubusercontent.com/cea-hpc/modules/master/doc/example/modulefiles-in-git/modulefiles/localmodules

Edit paths in the top of localmodules, if your installation differs from the assumptions, then:

git add localmodules
git commit -m 'Add localmodules from github.com/cea-hpc/modules'
Usage example

As a regular user (i.e. anyone but user modules):

module load localmodules
cd $HOME/modulefiles

Edit, test, then:

git commit -am 'Make some edits'
git push
module unload localmodules

After a successful push and unload, it is safe to delete your local $HOME/modulefiles directory if you wish.

Make defined modulepaths persist over sudo

When running a command as another user with sudo, current user environment is most of the time flushed for security concerns. As a result, if one would like to use the modulecmd.tcl script in such context, an error is returned as modulecmd.tcl does not find modulepath defined (MODULEPATH variable is not set). Following recipe describes how to ensure the default modulepaths are set every time the modulecmd.tcl script is run.

Implementation

Every time the modulecmd.tcl script is run, it evaluates during its start-up phase a global RC file following the same kind of evaluation than for modulefiles.

To ensure modulepaths are always defined, a check could be added in this global RC file to verify at least one modulepath is set (thanks to the is-used Tcl modulefile command). If no modulepath is found set, the .modulespath configuration file, which contains the default modulepaths, can be parsed to enable on the fly the default modulepaths (with module use Tcl modulefile command).

Global RC file
#%Module

# ensure MODULEPATH is always defined, use content of .modulespath config file
# to initialize it if not defined
if {![is-used] && [file readable /usr/share/Modules/init/.modulespath]} {
    set fid [open /usr/share/Modules/init/.modulespath r]
    set fdata [split [read $fid] "\n"]
    close $fid
    foreach fline $fdata {
        if {[regexp "^\\s*(.*?)\\s*(#.*|)\$" $fline match patharg] == 1
            && $patharg ne {}} {
            eval module use --append [split $patharg :]
        }
    }
}

Compatible with Modules v4.1+

Installation

Copy the global RC file of this recipe in the configuration directory:

$ cp example/modulepaths-persist-over-sudo/rc /etc/environment-modules/

The location of the .modulespath file defined in the proposed RC script should be adapted to reflect the location where this file is installed on your setup.

Usage example

Without the proposed RC file installed, MODULEPATH environment variable is lost through the sudo call:

$ sudo $MODULES_CMD bash avail >/dev/null
ERROR: No module path defined

Once RC file is installed, flushed MODULEPATH is restored on the fly based on .modulespath configuration file:

$ sudo $MODULES_CMD bash use >/dev/null
Search path for module files (in search order):
  /usr/share/Modules/modulefiles
  /etc/modulefiles
  /usr/share/modulefiles

Thus available modulefiles are found again:

$ sudo $MODULES_CMD bash avail >/dev/null
--------------- /usr/share/Modules/modulefiles ---------------
dot  module-git  module-info  modules  null  use.own

Use new features without breaking old module command

When working on large infrastructure, sometimes the module command is not deployed with the same version everywhere. You may have for instance some old cluster that still uses Modules version 3.2 and a newer supercomputer where the lastest version of Modules 4 is available. In such situation it may however be desired to share the same modulefile catalog.

People providing software that build software and generate the modulefiles to access them fall in a similar situation: a large variety of Modules version may be in use by the end-users of their build product.

This recipe describes how to use features from the latest version of Modules 4 in modulefiles without breaking the use of these modulefiles from older Modules version.

Implementation

The Tcl language provides some introspection mechanisms that help to know what Tcl procedures are available. In the modulefile or modulerc evaluation context, it helps to determine what Tcl modulefile commands are known. So by crafting a conditional test using such mechanism (with Tcl info commands command) it is possible to test if a new feature is available prior using it.

.modulerc
#%Module

# use module-hide only if available
if {[info commands module-hide] eq {module-hide}} {
   module-hide --after 2020-10-01 foo/1.1
}
# use module-forbid only if available
if {[info commands module-forbid] eq {module-forbid}} {
   module-forbid --after 2020-10-01 foo/1.1
}

Compatible with Modules v3.2+

Starting version 4.7 of Modules, two new Tcl variables are introduced in the modulefile and modulerc evaluation context: ModuleTool and ModuleToolVersion. These two variables help to determine respectively what is the module implementation running and what is its version. With this knowledge it is possible to adapt modulefile and modulerc code to cope with a behavior changing over module versions or with different behaviors between different module implementation. The Tcl modulefile command versioncmp has been added along to help comparing software version number (e.g. 4.10 is newer than 4.7).

Starting its version 8.4.8, the Lmod project also supports the ModuleTool and ModuleToolVersion variables and the versioncmp modulefile command. It enables having modulefiles compatible with both module implementations without restricting yourself from using the advanced features from both projects.

Compatible with Modules v4.7+

Usage example

For this recipe, a foo module is available in version 1.1 and 1.2. Version 1.1 is outdated and it has been decided to hide and forbid it starting October 2020. For that, the new module-hide and module-forbid modulefile commands introduced in Modules 4.6 are used.

Enable the modulepath where the example modulefiles are located:

$ module use example/new-features-without-breaking-old-module/modulefiles

By using the conditional test code in the modulepath rc file (see Implementation section above), the two new modulefile commands are not used. So older versions of Modules still in use do not benefit from the dynamic hiding and forbidding features however no error are obtained on these setups:

$ module -V | grep ^VERSION=
VERSION=3.2.13
$ module avail -t foo
/path/to/example/new-features-without-breaking-old-module/modulefiles:
foo/1.1
foo/1.2

Yet the dynamic hiding and forbidding features are enabled for setup using Modules 4.6 or newer version:

$ module -V
Modules Release 4.6.0 (2020-09-16)
$ module avail -t foo
/path/to/example/new-features-without-breaking-old-module/modulefiles:
foo/1.2

Now take a look at bar module which provides a version for each Unix group the current user is member of. User group membership can be retrieved with the usergroups sub-command of module-info starting Modules version 4.6. With older version of Modules, the external command groups has to be used to get this information. By using the ModuleTool and ModuleToolVersion Tcl variables it will be possible to determine if usergroups sub-command is available on module-info.

bar/.modulerc
#%Module

if {[info exists ModuleTool] && $ModuleTool eq {Modules}
   && [versioncmp $ModuleToolVersion 4.6] >= 0} {
   set grouplist [module-info usergroups]
} else {
   set grouplist [exec groups]
}

foreach grp $grouplist {
    module-virtual bar/$grp ./.common
}

Querying available bar module versions should match the list of groups of current user:

$ groups
grp1 grp2
$ module avail -t bar
/path/to/example/new-features-without-breaking-old-module/modulefiles:
bar/grp1
bar/grp2

Note

As the new Tcl variables are introduced in Modules 4.7, the use of the new uergroups sub-command will only be triggered starting Modules 4.7.

Source shell script in modulefile

When working with large software suite providing a shell script for their enablement in user environment, it is usually desired to also provide access to these software through module. However these software enablement may be complex and it may be wise to keep using the shell script provided by software editor rather crafting a modulefile from scratch.

This recipe describes how to make modulefiles for such software by using the enablement shell script provided with them.

Implementation

Modules version 4.6 introduces a new sub-command named sh-to-mod and a new modulefile command named source-sh. The sh-to-mod outputs as a modulefile content the environment changes done by the evaluation of a shell script passed as argument. On the other hand, the source-sh modulefile command sources environment changes done by the evaluation of a shell script passed as argument.

Both new features relies on the same mechanism that starts a designated shell to:

  • get current environment state (environment variables, shell aliases, shell functions and current working directory)
  • source designated shell script with defined arguments
  • get resulting environment state

Once done, environment prior and after script source are compared to determine the corresponding environment changes and translate those changes into modulefile commands (setenv, prepend-path, set-alias, set-function, ...).

sh-to-mod outputs these resulting modulefile commands. This output can be redirected into a file to create a modulefile. source-sh on the other hand sources the resulting modulefile commands to evaluate them as if they were written in the modulefile calling source-sh.

sh-to-mod and source-sh support the following shells: sh, dash, csh, tcsh, bash, ksh, ksh93, zsh and fish.

Compatible with Modules v4.6+

Basic usage example

For this recipe, a dummy software named foo is used as example. foo is installed in version 1.2 in example/source-script-in-modulefile/foo-1.2 directory and it provides a foo-setup.sh script to activate itself in user environment:

foo-setup.sh
#!/bin/bash
export FOOENV="$1"
export PATH=$(dirname $BASH_SOURCE)/bin:$PATH                                                             
alias foo='foobin -q -l'

First line of foo-setup.sh script helps to identify which shell needs to be used to evaluate it: bash.

sh-to-mod may be used to get this script translated as a modulefile:

$ module sh-to-mod bash example/source-script-in-modulefile/foo-1.2/foo-setup.sh arg1
#%Module
prepend-path    PATH example/source-script-in-modulefile/foo-1.2/bin
set-alias       foo {foobin -q -l}
setenv          FOOENV arg1

Output could be redirected into a foo/1.2 file and make it the modulefile to enable software foo:

$ mkdir -p modulefiles/foo
$ module sh-to-mod bash example/source-script-in-modulefile/foo-1.2/foo-setup.sh arg1 >modulefiles/foo/1.2
$ module use ./modulefiles
$ module show foo
-------------------------------------------------------------------
modulefiles/foo/1.2:

prepend-path    PATH example/source-script-in-modulefile/foo-1.2/bin
set-alias       foo {foobin -q -l}
setenv          FOOENV arg1
-------------------------------------------------------------------

Instead of transforming shell script in modulefile, a modulefile using source-sh modulefile command to evaluate shell script at modulefile evaluation time may be written:

modulefiles/foo/1.2
#%Module4.6
source-sh bash example/source-script-in-modulefile/foo-1.2/foo-setup.sh arg1

When displaying a modulefile using source-sh modulefile command, modulefile commands resulting from source-sh evaluation are reported:

$ module show foo/1.2
-------------------------------------------------------------------
.../modulefiles/foo/1.2:

prepend-path    PATH example/source-script-in-modulefile/foo-1.2/bin
set-alias       foo {foobin -q -l}
setenv          FOOENV arg1
-------------------------------------------------------------------

Loading this foo/1.2 module will enable access to software foo:

$ module load foo/1.2
$ alias foo
alias foo='foobin -q -l'
$ foo
foo, version 1.2

Unloading foo/1.2 module will properly revert these environment settings:

$ module unload foo/1.2
$ alias foo
bash: alias: foo: not found
$ foobin
bash: foobin: command not found

As conclusion, these new features enable to leverage the setup scripts that are provided along with software to make them reachable from the module environment.

Usage with shell-specific scripts

When the initialization script provided by software only defines environment variables, this script could be used to setup the user environment through the use of source-sh in a modulefile whatever the shell ran by user, as the module command will accurately translate script changes into the language of the running shell.

For instance the foo/1.2 module, that uses the source-sh modulefile command over the foo-setup.sh bash script, could also be used when running the tcsh or fish shell:

$ echo $version
tcsh 6.22.03 (Astron) 2020-11-18 (x86_64-unknown-linux) ...
$ module show foo/1.2
-------------------------------------------------------------------
.../modulefiles/foo/1.2:

prepend-path    PATH example/source-script-in-modulefile/foo-1.2/bin
set-alias       foo {foobin -q -l}
setenv          FOOENV arg1
-------------------------------------------------------------------
$ module load foo/1.2
$ foo
foo, version 1.2

Software may sometimes provide a specific script for each shell they support as they do not perform their initialization the same way on every shell. Quite often a shell function is defined for sh shells whereas an alias is setup for csh shells (as such shells do not support shell function).

Dummy software bar is used to demonstrate this situation. bar is installed in version 2.1 in example/source-script-in-modulefile/bar-2.1 directory and it provides a bar-setup.sh and a bar-setup.csh scripts to activate itself in user environment, depending on the shell kind used.

bar-setup.sh
#!/bin/bash
export PATH=$(dirname $BASH_SOURCE)/bin:$PATH
bar() {
    barbin -q -l
}
bar-setup.csh
#!/bin/tcsh
setenv PATH "example/source-script-in-modulefile/bar-2.1/bin:$PATH"
alias bar 'barbin -q -l'

To accurately initialize environment for bar software, the bar module needs to call the .sh script if user is currently running a shell from the sh family, or to call the .csh script if user runs a csh-kind shell.

modulefiles/bar/2.1
#%Module4.6
set scriptpath example/source-script-in-modulefile/bar-2.1
switch -- [module-info shelltype] {
    sh {
        source-sh bash $scriptpath/bar-setup.sh
    }
    csh {
        source-sh tcsh $scriptpath/bar-setup.csh
    }
}

This way the bar shell function is initialized when loading module from a user environment running a sh shell:

$ echo $BASH_VERSION
5.1.0(1)-release
$ module use example/source-script-in-modulefile/modulefiles
$ module show bar
-------------------------------------------------------------------
.../modulefiles/bar/2.1:

prepend-path    PATH example/source-script-in-modulefile/bar-2.1/bin
set-function    bar {
    barbin -q -l}
-------------------------------------------------------------------
$ module load bar
$ type bar
bar is a function
bar ()
{
    barbin -q -l
}
$ bar
bar, version 2.1

Whereas the bar shell alias is setup on csh shell environment:

$ echo $version
tcsh 6.22.03 (Astron) 2020-11-18 (x86_64-unknown-linux) ...
$ module use example/source-script-in-modulefile/modulefiles
$ module show bar
-------------------------------------------------------------------
.../modulefiles/bar/2.1:

prepend-path    PATH example/source-script-in-modulefile/bar-2.1/bin
set-alias       bar {barbin -q -l}
-------------------------------------------------------------------
$ module load bar
$ alias bar
barbin -q -l
$ bar
bar, version 2.1

Sticky modules

When providing a configurable environment to users, site's staff may require that some part of this environment remain loaded whatever the user does.

Such feature is for instance useful when every details of the user environment are configured through the use of modulefiles. Even the core setup that is usually configured through the /etc/profile.d initialization scripts. But by using modulefiles for core initialization, end users can fully see how things are setup on their environment by using module display. When the environment core setup is achieved by loading a specific modulefile, it is important that such module remains loaded to keep this initial setup on whatever the module actions the user performs over its environment.

This recipe describes how to keep modulefiles loaded by forbidding their unload. Such unloadable modules are called sticky modules.

Implementation

Sticky modules are simply modules that cannot be unloaded once loaded. Such behavior could be achieved by basically breaking the modulefile evaluation when attempting to unload the sticky module:

if {[module-info mode unload]} {
    break
}

Using the break Tcl command to stop the modulefile evaluation does not require to install a recent version of Modules to get a basic sticky mechanism.

To get a smoother sticky mechanism with two different level of stickyness, allowing to reload environment or to swap a sticky module by another version of the same module name, the sticky and super-sticky module tags have been introduced in Modules v4.7.

A modulefile is declared sticky by applying it the sticky tag with the module-tag modulefile command. Such sticky module cannot be unloaded, unless if the unload action is forced or if the module reloads after being unloaded.

Modulefile can also be defined super-sticky by applying the corresponding module tag. Super-sticky module cannot be unloaded even if the unload action is forced. It can only be unloaded if the module reloads afterward.

In case the stickyness applies to the generic module name (and does not target a specific module version or version-set), one version of the sticky or super-sticky module can be swapped by another version of this same module.

Compatible with Modules v4.7+

Usage examples

For this recipe, a core module acts as the initial setup of user's environment. This module must not be unloaded otherwise user's environment may be considered broken.

So this core module is tagged super-sticky with the module-tag modulefile command in core/.modulerc file:

module-tag super-sticky core

Once module got loaded, it cannot be unloaded even if these unload actions are forced.

$ module list
Currently Loaded Modulefiles:
 1) core/1.0

Key:
super-sticky
$ module unload core
Unloading core/1.0
  ERROR: Unload of super-sticky module 'core/1.0' skipped
$ module purge -f
Unloading core/1.0
  ERROR: Unload of super-sticky module 'core/1.0' skipped
$ module list
Currently Loaded Modulefiles:
 1) core/1.0

Key:
super-sticky

However it is still possible to change version of this super-sticky module.

$ module switch core/2.0
$ module list
Currently Loaded Modulefiles:
 1) core/2.0

Key:
super-sticky

In this recipe environment, the compiler module provides several flavors: compA and compB. Site's staff have decided that user's environment should always have a compiler module loaded by default.

So the compiler module is set sticky with the module-tag modulefile command in compiler/.modulerc file:

module-tag sticky compiler

As stickyness is defined over the generic compiler name, users can switch between available compiler flavors:

$ module list
Currently Loaded Modulefiles:
 1) core/2.0   2) compiler/compB/2.1

Key:
super-sticky  sticky
$ module switch compiler/compA
$ module list
Currently Loaded Modulefiles:
 1) core/2.0   2) compiler/compA/1.2

Key:
super-sticky  sticky

Unload attempt fails by default:

$ module unload compiler
Unloading compiler/compA/1.2
  ERROR: Unload of sticky module 'compiler/compA/1.2' skipped

However if a user really wants to get rid of the compiler module, the unload action can be forced:

$ module unload -f compiler
Unloading compiler/compA/1.2
  WARNING: Unload of sticky module 'compiler/compA/1.2' forced
$ module list
Currently Loaded Modulefiles:
 1) core/2.0

Key:
super-sticky

Last but not least, the sticky modules should get loaded when the user's shell session initializes. So the core and compiler modules should be defined for load in the initialization RC file /etc/environment-modules/initrc:

#%Module
module use --append .../example/sticky-modules/modulefiles
module load core
module load compiler/compB

Testing Modulefiles

The following is an example for a ModulesTest subroutine of a Modulefile and its output. It checks whether the TESTDIR is a directory, checks that it can enter it, and whether a file TESTFILE can successfully be created there.

This code gets executed when you use the module test modulefile command.

Code
test_dir_and_file
#%Module4.0 # -*- mode: tcl; -*-

proc ModulesTest { } {
    set retcode 1  ;# default: 1 meaning PASS

    puts stderr "Running ModulesTest for directory existence..."
    if { [file isdirectory [getenv TESTDIR]] } {
        puts stderr "Is a directory: [getenv TESTDIR]"
    } else {
        puts stderr "ERROR: Is not a directory: [getenv TESTDIR]"
        set retcode 0
    }
    puts stderr "Running ModulesTest for directory existence...done"

    puts stderr "Running ModulesTest for directory permissions..."
    set cmd { cd [getenv TESTDIR] }
    if { [catch $cmd errmsg] } {
        puts stderr "ERROR: Was not able to enter directory [getenv TESTDIR]: ${errmsg}"
        set retcode 0
    } else {
        puts stderr "Was able to enter directory [getenv TESTDIR]"
    }
    puts stderr "Running ModulesTest for directory permissions...done"

    puts stderr "Running ModulesTest for file creation..."
    set cmd { open [getenv TESTFILE] w }
    if { [catch $cmd errmsg] } {
        puts stderr "ERROR: Was not able to create file [getenv TESTFILE]: ${errmsg}"
        set retcode 0
    } else {
        puts stderr "Was able to create file [getenv TESTFILE]"
    }
    puts stderr "Running ModulesTest for file creation...done"

    return $retcode
}

setenv TESTDIR  /tmp/[getenv USER]/testdir
setenv TESTFILE [getenv TESTDIR]/testfile
Usage example

Enable the modulepath where the example modulefiles are located:

$ module use example/test-modulefiles/modulefiles

Run the test both with the test directory not existing and existing:

$ module test test_dir_and_file
-------------------------------------------------------------------
Module Specific Test for .../modulefiles/test_dir_and_file:

Running ModulesTest for directory existence...
ERROR: Is not a directory: /tmp/testuser/testdir
Running ModulesTest for directory existence...done
Running ModulesTest for directory permissions...
ERROR: Was not able to enter directory /tmp/testuser/testdir: couldn't change working directory to "/tmp/testuser/testdir": no such file or directory
Running ModulesTest for directory permissions...done
Running ModulesTest for file creation...
ERROR: Was not able to create file /tmp/testuser/testdir/testfile: couldn't open "/tmp/testuser/testdir/testfile": no such file or directory
Running ModulesTest for file creation...done
Test result: FAIL
-------------------------------------------------------------------
$ mkdir /tmp/$USER/testdir
$ module test test_dir_and_file
-------------------------------------------------------------------
Module Specific Test for .../modulefiles/test_dir_and_file:

Running ModulesTest for directory existence...
Is a directory: /tmp/testuser/testdir
Running ModulesTest for directory existence...done
Running ModulesTest for directory permissions...
Was able to enter directory /tmp/testuser/testdir
Running ModulesTest for directory permissions...done
Running ModulesTest for file creation...
Was able to create file /tmp/testuser/testdir/testfile
Running ModulesTest for file creation...done
Test result: PASS
-------------------------------------------------------------------

Tips for Code Reuse in Modulefiles

Although Modules allow one to manage many packages and versions of packages on a system, managing the modulefiles themselves can become error prone, especially since the differences between modulefiles for different versions of the same package are often rather small. This can lead to issues with the consistency of the module definitions.

E.g., let us say you have a package foo, version 1.0 with a corresponding modulefile. A bit later, foo version 2.0 comes out, and you add a modulefile for that, likely copying the 1.0 version and editing a few lines. Then a bug is discovered in the version 1.0 modulefile, e.g. an environment variable that would be useful in some cases was omitted, or a typo in the help function. The version 1.0 modulefile is fixed, but did you remember to fix the 2.0 modulefile which, since it was created by copying the version 1.0 modulefile, likely has the same error?

This page includes some tips to reduce the amount of redundant code in your modulefiles (i.e. increase the amount of code reuse) so that they are easier to manage and more consistent.

A Simple Example

We return to our "foo" example. A fairly traditional set up would have a "foo" directory somewhere under your MODULEPATH with modulefiles 1.0 and 2.0 underneath it. Each of 1.0 and 2.0 will likely be largely the same, having basically the same help text, whatis, and variables set, with only some changes to some text and paths to reflect the version difference. Example code for this and other cases discussed in this document can be found in the "doc/example/tips-for-code-reuse" subdirectory of this package. The modulefiles for this simple example start in the "foo" subdirectory. For example, the version 1.0 modulefile might look something like

#%Module1.0
##
## foo

proc ModulesHelp { } {
   puts stderr "
FooA: A simple example of modulefile code reuse
Version 1.0

blah, blah, blah
"
}
module-whatis "foo version 1.0"

conflict foo
prepend-path PATH /software/foo/1.0/bin
prepend-path MANPATH /software/foo/1.0/share/man
prepend-path LD_LIBRARY_PATH /software/foo/1.0/lib

Although the above is simple enough, there is still much redundancy between the version 1.0 and 2.0 files, and we do not bother showing the file for 2.0 because it is so similar (only 5 chars change between the 1.0 and 2.0 versions, basically all the "1.0" strings become "2.0"). The command "module avail foo" shows versions 1.0 and 2.0, and loading either will set the environmental variables PATH, MANPATH, and LD_LIBRARY_PATH appropriately, conflict with itself (so only one version of foo can be loaded at a time), and define the help and whatis texts.

To refactor this to increase code reuse, we copy one of the original modulefiles to "common", and change all the references to the version number to a "$version" variable. To avoid confusion with the previous version, we change the package name from "foo" to "fooA", and the new versions can be found in the "fooA" subdirectory of "doc/example/tips-for-code-reuse". After adding some comments, etc, the common module becomes

#Common modulefile for fooA
#Expects the following variables to have been set
#   version: the version of fooA


proc ModulesHelp { } {
   global version
   puts stderr "
FooA: A simple example of modulefile code reuse
Version $version

This is a simple example of code reuse in modulefiles.
We have a couple versions of fooA, and the only differences
in what the modulefiles for the different fooA versions do
is that some paths include the version number.

"
}
module-whatis "fooA version $version"
    
conflict fooA
set rootdir /software/fooA/$version
prepend-path PATH $rootdir/bin
prepend-path MANPATH $rootdir/share/man
prepend-path LD_LIBRARY_PATH $rootdir/lib

We then create a pair of small stub files to replace "1.0" and "2.0". These simply just set the version variable appropriately, and then "source" the "common" file. For "1.0", the file would look like

#%Module1.0
set version 1.0

set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/common

A similar file would be created for "2.0", just changing the version number in the "set version" line. The "set moduledir" line sets the Tcl variable moduledir to contain the name of the directory in which the final modulefile (i.e. "1.0") is located, so we can find the "common" file.

The common file then handles all the work of defining the PATH, MANPATH, and LD_LIBRARY_PATH environmental variables, using the value of the Tcl variable "version" passed to it by the stub file. It also handles defining the help procedure and the whatis text, and the conflict with itself.

Note that the "common" file does NOT start with the Modules magic "#%Module1.0"; this will keep "common" from showing up in "module avail fooA". The "1.0" and "2.0" files do need to start with the magic "#%Module1.0" tag so the module command will "see" them. Because of this, the "module avail fooA" command will just show the 1.0 and 2.0 versions as expected.

The files for this example are in the "doc/example/tips-for-code-reuse/fooA" directory.

A Simple Example, revisited

The "fooA" case above is fairly common, and we can actually improve upon what we did above. To avoid confusion, we will repeat using the package name "fooB", and the files for this example will be in the "doc/example/tips-for-code-reuse/fooB" directory.

Using introspection in the modulefiles, we can get the version number of fooB from the name of the modulefile. So we add code to the top of "common" to default the version variable from the modulename.

We then copy the "1.0" stub modulefile to ".generic", and remove the line which sets the version. Because is starts with a leading ".", the generic modulefile will not be displayed in "module avail" and such. Since this generic modulefile is now version independent, we can replace "1.0" and "2.0" with symlinks to the .generic file.

The new common file looks like

#Common modulefile for fooB
#Expects the following variables to have been set
#   version: the version of fooB, defaults to last component of module tag

if [info exists version]==0 {
   set version [file tail [module-info version [module-info name] ] ]
}

proc ModulesHelp { } {
   global version
   puts stderr "
FooB: A simple example of modulefile code reuse, revisited
Version $version

This our second visit to simple example of code reuse in modulefiles.
Because the all of the differences between the different fooB versions
modulefiles is contained in the version number, we replace the stub
modulefiles for the two versions with symlinks to a generic version
file, and infer the version from the tag given to the module command.

The generic modulefile, .generic, does not appear in 'module avail' 
commands because of the leading period (.).  Indeed, this generic modulefile
does is not even 'bar' specific, and typically we actually put the file
in an utilities directory outside the MODULEPATH and symlink to it from
multiple packages.

"
}
module-whatis "fooB version $version"
    
conflict fooB
set rootdir /software/fooB/$version
prepend-path PATH $rootdir/bin
prepend-path MANPATH $rootdir/share/man
prepend-path LD_LIBRARY_PATH $rootdir/lib

The .generic stub file looks like

#%Module######################
##
## generic modulefile, 
##
## This just sources the file common in the same directory
##
## Usage:
## Just symlink this file to .generic in the directory with common, and
## then make symlink to .generic for all versions of the app

set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/common

Again, the common file handles setting the environmental variables PATH, MANPATH, and LD_LIBRARY_PATH, using the Tcl variable version which was defaulted from modulename. This works even in the case where the user does not specify the full module path, and the modulecmd defaults the version (e.g. if the user types "module load fooB" without specifying the version of fooB.) The common file also handles defining the help procedure and whatis text. When the command "module avail fooB" is issued, the common file does not get listed (because it does not start with the magic "#%Module1.0" tag), nor does the .generic file get listed (because it starts with a period (".")), but the two 1.0 and 2.0 symlinks do get listed, just as one wants.

This replaces a file for each version of fooB with one common file doing all the work, one generic file, and a bunch of symlinks. And the .generic file is not even specific to fooB. Indeed, at our site, we put the generic modulefile in an "utilities" directory outside of the MODULEPATH, and just symlink it to ".generic" in each application directory, and then symlink the versions to this symlinked ".generic". So we end up with one "common" file for each application, one shared generic modulefile, and a bunch of symlinks.

A More Complicated Example

Although the simple cases like the "foo" example above are not uncommon, many packages are more complicated. We now consider the fictitious "bar" package (example modulefiles in "example/tips-for-code-reuse/bar" directory). This is admittedly a contrived example, but it displays some of the ways in which we can modify the above simple cases to handle the more complicated needs of some packages. Our "bar" package has the follow characteristics:

  • We have two versions of "bar" built, "1.0" and "2.0"
  • For each bar version, we have builds for three different threading models: nothreads, pthreads, and openmp.
  • We expect users to use commands specifying both the bar version and the threading model, e.g. "module load fooB/1.0/pthreads" or "module load fooB/2.0/openmp"
  • The path to the installation directories include both the version number of "bar" and the threading model
  • The package requires the environmental variable BAR_LICENSE_FILE to be set appropriately
  • All builds of bar version 1.0 use one license file which is different than that used by bar version2.0
  • Bar 2.0 nothreads and openmp builds use the same license file, with a completely different name than that used by bar version 1.0.
  • The bar 2.0 pthreads build uses its very own license file with a very different filename than those used by bar version 1.0 or the other threading models of version 2.0.
  • The "nothreads" builds have a prerequisite on "fooB", with bar 1.0 wanting fooB version 1.1, and bar 2.0 wanting fooB version 3.2

Even with the above exceptions, there are still more similarities than differences between the various modulefile definitions. Like in the "fooA" case, we will have a "common" script that does almost all the work, and stub files for each of the variants, but in this case the stub files are expected to define some more variables. Our "common" file, which can be found in the "example/tips-for-code-reuse/bar" directory, looks like name "fooB", and the files for this example will be in the

#Common modulefile for bar
#Expects the following variables to have been set
#   version: the version of bar, 
#      defaults to last component of module tag
#   threadingmodel: one of 'nothreads', 'openmp' or 'pthreads'
#   licensefile: or will default based on version

if [info exists version]==0 {
   set version [file tail [module-info version [module-info name] ] ]
}

#Default to nothreads
if [info exists threadingmodel]==0 {
   set threadingmodel nothreads
}

#Default licensefile based on version
if [info exists licensefile]==0  {
   if [string equal $version 1.0] {
      set licensefile /somepath/to/version1/licenseFile.lic
   } else {
      set licensefile /a/completely/different/licenseFile.lic
   }
}

proc ModulesHelp { } {
   global version threadingmodel licensefile

   set threadstr $threadingmodel
   if [ string equal $threadingmodel nothreads ] {
      set threadstr {no threading support}
   }
   puts stderr "
Bar: Not so simple modulefile example
Version $version
Threading Model: $threadstr

This is a more complicated example of code reuse in modulefiles.
We have multiple versions of bar on the system, and each version
comes with multiple variants for different threading support modules.
Plus we assume that the env var BAR_LICENSE_FILE needs to be set
differently depending on the bar version.

Using license file: $licensefile

"
}
module-whatis "bar version $version with threading $threadingmodel"
    
conflict bar
set rootdir /software/bar/$version/$threadingmodel
prepend-path PATH $rootdir/bin
prepend-path MANPATH $rootdir/share/man
prepend-path LD_LIBRARY_PATH $rootdir/lib
setenv BAR_LICENSE_FILE $licensefile

Here we have extended the parameters which are allowed to be passed in from the build specific "stub" files; in addition to the version of "bar", the stub files are expected to have set the variables "threadingmodel" and optionally "licensefile". If "licensefile" was not set, the common script will default it based on the version of "bar". The rest of the "common" file is similar to the "fooA" case, except that we use both "version" and "threadingmodel" in setting the paths, and now define BAR_LICENSE_FILE based on "licensefile". Again, almost all the work is done in the common file, including setting the relevant environmental variables, defining the help procedure and whatis text, and preventing multiple versions of bar being loaded at the same time.

We now create stubfiles for each of the three threading models underneath directories for each bar version. These can all be found under the "example/tips-for-code-reuse/bar" directory. The one for bar 1.0 and openmp looks like

#%Module1.0
set version 1.0
set threadingmodel openmp

set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/../common

The pthreads version is almost identical, just setting threadingmodel to "pthreads" instead of "openmp". The "openmp" build for bar version 2.0 is also very similar, differing only in value "version" is set to. In all three cases, these stubfiles are basically the same as those used in the "fooA" example; we set a couple of Tcl variables (version and threadingmodel), and then source the common file which does all the work. The licensefile variable is left unset so the common file will default it correctly based on the bar version.

The stub file for the pthreads build for bar version 2.0 is as below:

#%Module1.0
set version 2.0
set threadingmodel pthreads

set licensefile /special/license/file/for/bar/2.0/with/pthreads/license.lic

set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/../common

This version is similar to the previous three, with the addition of explicitly setting the "licensefile" variable. This value will be used in the common file to set BAR_LICENSE_FILE, rather than the default which would have been set in the common file had no value been provided.

Finally, the two "nothreads" build also have a slightly different stub file than the other builds. The one for bar version 2.0 is shown below:

#%Module1.0
set version 2.0
set threadingmodel nothreads

prereq fooB/3.2

set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/../common

Here we add a "prereq fooB/3.2" to require that the correct version of fooB was loaded. The bar version 1.0 stubfile is almost the same, just changing the setting of the version variable from 2.0 to 1.0, and changing the version of fooB needed as a prerequisite.

So instead of having six somewhat lengthy modulefiles (~40 lines each), one for each combination of bar version and threading model, we put almost all the work into a single, slightly longer (55 lines) common file, with six small (8 lines or less) stub files which mostly just set variables to define the behavior of the common file.

When customization is needed for a specific build, we showed two ways of accommodating such. First, for simple, more-or-less "one-off" cases, you can add some code to the appropriate "stubfiles" to handle the added complexity, like was done with the "prereq fooB" for the "nothreads" builds in the examples.

Alternatively, one can add some additional logic to the common script to accommodate the complexity, like was done with the license file differences between builds. This is the recommended way to handle complexity which is likely to recur in multiple places. Generally, this is done by:

  • adding a parameter to be passed from the stub file to the common file and setting appropriately in the stub files (e.g. licensefile in bar/2.0/pthreads)
  • adding logic to the common file to default it appropriately (e.g. licensefile for the other builds)

The above was only a simple example of what could be done, but it shows how modulefiles can leverage the fact that the modulefiles are evaluated in a full programming language to increase the amount of code re-use, which should reduce errors, improve consistency, and in general make things more manageable over the long run.

The tips given above should work with both 3.x and 4.x versions of the Tcl based environmental packages. They have been explicitly tested on versions 3.2.10 and 4.3.0.

Top priority environment variable values

Multiple modulefiles may alter the same environment variable. It may be wanted that among those modulefiles one should have the priority over the others and the value it sets should persist even if another modulefile loaded after attempts to alter the variable.

When using the setenv modulefile command, a top priority value should persist over later setenv tries. On append-path modulefile command, this top priority value should stay last position and for prepend-path modulefile command, value should stay first position.

Implementation

Top priority values are implemented by the use of a site-specific configuration that supersedes the definition of the setenv, append-path and prepend-path commands to introduce a --top argument. This argument enables a value set with this flag on to hold top priority and thus cannot be altered unless by another top priority value.

For append-path command, --top ensure value will stay at last position in path-like variable. Same goes for prepend-path where --top ensure first position in path-like variable.

siteconfig.tcl
# override 'setenv' procedure to add a '--top' optional argument
rename ::setenv ::__setenv
proc setenv {args} {
   set topPriority 0
   set errArgMsg "wrong # args: should be \"setenv ?--top? var val\""
   switch -- [llength $args] {
      {3} {
         if {[lindex $args 0] eq "--top"} {
            set topPriority 1
         } else {
            error $errArgMsg
         }
         set var [lindex $args 1]
         set val [lindex $args 2]
      }
      {2} {
         set var [lindex $args 0]
         set val [lindex $args 1]
      }
      default {
         error $errArgMsg
      }
   }

   if {$topPriority} {
      # define an helper variable to know a top-priority value has been set
      if {[currentState mode] ne "display"} {
         __setenv MODULES_PRIORITY_$var $val
      }
      __setenv $var $val
   # set non-priority value only if no top priority value already set
   } elseif {![info exists ::env(MODULES_PRIORITY_$var)]} {
      __setenv $var $val
   }
}

# override 'setenv-un' procedure to interpret the '--top' optional argument
# when setenv is evaluated on an unload mode
rename ::setenv-un ::__setenv-un
proc setenv-un {args} {
   set topPriority 0
   set errArgMsg "wrong # args: should be \"setenv-un ?--top? var val\""
   switch -- [llength $args] {
      {3} {
         if {[lindex $args 0] eq "--top"} {
            set topPriority 1
         } else {
            error $errArgMsg
         }
         set var [lindex $args 1]
         set val [lindex $args 2]
      }
      {2} {
         set var [lindex $args 0]
         set val [lindex $args 1]
      }
      default {
         error $errArgMsg
      }
   }

   if {$topPriority} {
      # define an helper variable to know a top-priority value has been set
      if {[currentState mode] ne "display"} {
         __setenv-un MODULES_PRIORITY_$var $val
      }
      __setenv-un $var $val
   # set non-priority value only if no top priority value already set
   } elseif {![info exists ::env(MODULES_PRIORITY_$var)]} {
      __setenv-un $var $val
   }
}

# override 'add-path' procedure to add a '--top' optional argument, which
# will benefit to the 'append-path' and 'prepend-path' modulefile commands
rename ::add-path ::__add-path
proc add-path {pos args} {
   set keep_top_priority 0
   set arglist [lsearch -all -inline -not -exact $args "--top"]
   lassign [eval parsePathCommandArgs "add-path" $arglist] separator\
      allow_dup idx_val var path_list

   # top priority asked
   if {[llength $arglist] != [llength $args]} {
      # record this top priority value in an helper variable
      __setenv MODULES_PRIORITY_${pos}_$var $path_list
   } elseif {[info exists ::env(MODULES_PRIORITY_${pos}_$var)]} {
      set keep_top_priority 1
   }

   # ensure top-priority value keeps first or last position by unloading it
   # priority new value addition, then restoring it
   if {$keep_top_priority} {
      eval __unload-path $var $::env(MODULES_PRIORITY_${pos}_$var)
   }

   eval __add-path $pos $arglist

   if {$keep_top_priority} {
      eval __add-path $pos $var $::env(MODULES_PRIORITY_${pos}_$var)
   }
}

rename ::unload-path ::__unload-path
proc unload-path {args} {
   set arglist [lsearch -all -inline -not -exact $args "--top"]
   lassign [eval parsePathCommandArgs "unload-path" $arglist] separator\
      allow_dup idx_val var path_list

   if {[llength $arglist] != [llength $args]} {
      # wipe priority helper variable when unloading top priority value
      switch -- [lindex [info level -1] 0] {
         {append-path}  { set pos "append" }
         {prepend-path} { set pos "prepend" }
      }
      if {[info exists pos]} {
         __setenv MODULES_PRIORITY_${pos}_$var $path_list
      }
   }

   eval __unload-path $arglist
}

Compatible with Modules v4.2

Installation

Create site-specific configuration directory if it does not exist yet:

$ mkdir /etc/environment-modules

Then copy there the site-specific configuration script of this recipe:

$ cp example/top-priority-values/siteconfig.tcl /etc/environment-modules/

Note

Defined location for the site-specific configuration script may vary from one installation to another. To determine the expected location for this file on your setup, check the value of the siteconfig option on Modules version 4.3 or above:

$ module config siteconfig

On older version of Modules, check the modulecmd.tcl script:

$ grep '^set g_siteconfig ' $MODULES_CMD
Usage example

With a bar/1 modulefile that sets environment variables in a regular way:

bar/1
#%Module
# define a regular value for variable
setenv TESTVAR value

# prepend a regular value to a path-like variable
prepend-path TESTPATH prevalue

# append a regular value to a path-like variable
append-path TESTPATH postvalue

And a foo/1 modulefile that sets the same variables than bar/1 but with the --top priority flag:

foo/1
#%Module
# define a value for variable holding top priority (not overwritten by non-top priority value)
setenv --top TESTVAR topvalue

# prepend a value to a path-like variable, value that stays first position
prepend-path --top TESTPATH topprevalue

# append a value to a path-like variable, value that stays last position
append-path --top TESTPATH toppostvalue

Enable the modulepath where the example modulefiles are located:

$ module use example/top-priority-values/modulefiles

Load foo/1 then bar/1 modulefiles and check value of the environment variable set:

$ module load foo/1 bar/1
$ echo $TESTVAR
topvalue
$ echo $TESTPATH
topprevalue:prevalue:postvalue:toppostvalue

Unload firstly loaded module matching name

Since Modules v4, unloading a given module name unloads lastly loaded module matching this given name. On Modules v3 the module selected for the unload was the firstly loaded not the lastly loaded. This recipe gives a way to restore the behavior of the v3 version.

Implementation

Starting version v4.3, an unload_match_order configuration option is introduced to control whether firstly loaded module or lastly loaded module should be selected for the unload. To select firstly loaded module:

$ module config unload_match_order returnfirst

For older v4 versions, a site-specific configuration script is proposed to select firstly loaded module matching name rather lastly loaded.

siteconfig.tcl
# override 'getLoadedMatchingName' procedure to set behavior argument to the
# 'returnfirst' value by default
rename ::getLoadedMatchingName ::__getLoadedMatchingName
proc getLoadedMatchingName {name {behavior returnfirst} {loading 0}} {
   return [__getLoadedMatchingName $name $behavior $loading]
}

Compatible with Modules v4.2

Installation (only for version older than v4.3)

Create site-specific configuration directory if it does not exist yet:

$ mkdir /etc/environment-modules

Then copy there the site-specific configuration script of this recipe:

$ cp example/unload-firstly-loaded/siteconfig.tcl /etc/environment-modules/

Note

Defined location for the site-specific configuration script may vary from one installation to another. To determine the expected location for this file on your setup, check the modulecmd.tcl script:

$ grep '^set g_siteconfig ' $MODULES_CMD
Usage example

With a bare foo/1 modulefile:

foo/1
#%Module

And a bare foo/2 modulefile:

foo/2
#%Module

Enable the modulepath where the example modulefiles are located:

$ module use example/unload-firstly-loaded/modulefiles

Load both foo modulefiles then attempt to unload foo name:

$ module load foo/1 foo/2
$ module list
Currently Loaded Modulefiles:
 1) foo/1   2) foo/2
$ module unload foo
$ module list
Currently Loaded Modulefiles:
 1) foo/2

ml

SYNOPSIS

ml [switches] [sub-command [sub-command-args]]

ml [switches] [[-]modulefile...]

DESCRIPTION

ml is a user interface to the Modules package. The Modules package provides for the dynamic modification of the user's environment via modulefiles.

ml acts as a shortcut command to the module command thus it supports all the command line switches and module sub-commands that are supported by module.

ml also provides handy shortcuts to list currently loaded modulefiles, when no argument is provided to ml; to load modulefiles, when modulefile names are passed right after ml command name; to unload modulefiles, when modulefile names prefixed by a minus sign (-) are passed right after ml command name.

Multiple modulefiles to load and to unload can be specified on a single ml command line. In this situation unloads are treated first in specified order, then loads are processed also in specified order (see EXAMPLES section below). If an error occurs among either modulefile loads or unloads, command line processing is stopped and every unloads and loads performed are rolled back.

ml command line is parsed first to match every known command line switches or module sub-commands. So to load a modulefile via the ml shortcut syntax, modulefile name should not equal a module sub-command name. Moreover to unload a modulefile via the ml shortcut syntax, modulefile name should not equal a command line switch short name.

See the DESCRIPTION section in module for the list of supported command line switches and module sub-commands.

EXAMPLES

Loading modulefile foo then look at currently loaded modulefiles:

$ ml foo
$ ml
Currently Loaded Modulefiles:
 1) foo

Unloading modulefile foo then list modulefiles still loaded:

$ ml -foo
$ ml
No Modulefiles Currently Loaded.

Mixing load and unload of modulefiles in a single command. All specified unloads are processed first then loads are performed:

$ ml -v -foo bar -baz qux
Unloading foo
Unloading baz
Loading bar
Loading qux

EXIT STATUS

The ml command exits with 0 if its execution succeed. Elsewhere 1 is returned.

ENVIRONMENT

See the ENVIRONMENT section in module for the list of supported environment variables.

SEE ALSO

module, modulefile

module

SYNOPSIS

module [switches] [sub-command [sub-command-args]]

DESCRIPTION

module is a user interface to the Modules package. The Modules package provides for the dynamic modification of the user's environment via modulefiles.

Each modulefile contains the information needed to configure the shell for an application. Once the Modules package is initialized, the environment can be modified on a per-module basis using the module command which interprets modulefiles. Typically modulefiles instruct the module command to alter or set shell environment variables such as PATH, MANPATH, etc. Modulefiles may be shared by many users on a system and users may have their own set to supplement or replace the shared modulefiles.

The modulefiles are added to and removed from the current environment by the user. The environment changes contained in a modulefile can be summarized through the module command as well. If no arguments are given, a summary of the module usage and sub-commands are shown.

The action for the module command to take is described by the sub-command and its associated arguments.

Package Initialization

The Modules package and the module command are initialized when a shell-specific initialization script is sourced into the shell. The script creates the module command as either an alias or function and creates Modules environment variables.

The module alias or function executes the modulecmd.tcl program located in /usr/share/Modules/libexec and has the shell evaluate the command's output. The first argument to modulecmd.tcl specifies the type of shell.

The initialization scripts are kept in /usr/share/Modules/init/<shell> where <shell> is the name of the sourcing shell. For example, a C Shell user sources the /usr/share/Modules/init/csh script. The sh, csh, tcsh, bash, ksh, zsh and fish shells are supported by modulecmd.tcl. In addition, python, perl, ruby, tcl, cmake, r and lisp "shells" are supported which writes the environment changes to stdout as python, perl, ruby, tcl, lisp, r or cmake code.

Initialization may also be performed by calling the autoinit sub-command of the modulecmd.tcl program. Evaluation into the shell of the result of this command defines the module alias or function.

A ml alias or function may also be defined at initialization time if enabled (see MODULES_ML section). ml is a handy frontend leveraging all module command capabilities with less character typed. See ml for detailed information.

Examples of initialization

C Shell initialization (and derivatives):

source /usr/share/Modules/init/csh
module load modulefile modulefile ...

Bourne Shell (sh) (and derivatives):

. /usr/share/Modules/init/sh
module load modulefile modulefile ...

Perl:

require "/usr/share/Modules/init/perl.pm";
&module('load', 'modulefile', 'modulefile', '...');

Python:

import os
exec(open('/usr/share/Modules/init/python.py').read())
module('load', 'modulefile', 'modulefile', '...')

Bourne Shell (sh) (and derivatives) with autoinit sub-command:

eval "`/usr/share/Modules/libexec/modulecmd.tcl sh autoinit`"
Modulecmd startup

Upon invocation modulecmd.tcl sources a site-specific configuration script if it exists. The location for this script is /etc/environment-modules/siteconfig.tcl. An additional siteconfig script may be specified with the MODULES_SITECONFIG environment variable, if allowed by modulecmd.tcl configuration, and will be loaded if it exists after /etc/environment-modules/siteconfig.tcl. Siteconfig is a Tcl script that enables to supersede any global variable or procedure definition of modulecmd.tcl.

Afterward, modulecmd.tcl sources rc files which contain global, user and modulefile specific setups. These files are interpreted as modulefiles. See modulefile for detailed information.

Upon invocation of modulecmd.tcl module run-command files are sourced in the following order:

  1. Global RC file as specified by MODULERCFILE variable or /etc/environment-modules/rc. If MODULERCFILE points to a directory, the modulerc file in this directory is used as global RC file.
  2. User specific module RC file $HOME/.modulerc
  3. All .modulerc and .version files found during modulefile seeking.
Command line switches

The module command accepts command line switches as its first parameter. These may be used to control output format of all information displayed and the module behavior in case of locating and interpreting modulefiles.

All switches may be entered either in short or long notation. The following switches are accepted:

--all, -a

Include hidden modules in search performed with avail, aliases, list, search or whatis sub-commands. Hard-hidden modules are not affected by this option.

New in version 4.6.

--auto

On load, unload and switch sub-commands, enable automated module handling mode. See also MODULES_AUTO_HANDLING section.

New in version 4.2.

--color=<WHEN>

Colorize the output. WHEN defaults to always or can be never or auto. See also MODULES_COLOR section.

New in version 4.3.

--contains, -C

On avail sub-command, return modules whose fully qualified name contains search query string.

New in version 4.3.

--debug, -D, -DD

Debug mode. Causes module to print debugging messages about its progress. Multiple -D options increase the debug verbosity. The maximum is 2.

New in version 4.0.

Changed in version 4.6: Option form -DD added

--default, -d

On avail sub-command, display only the default version of each module name. Default version is the explicitly set default version or also the implicit default version if the configuration option implicit_default is enabled (see Locating Modulefiles section in the modulefile man page for further details on implicit default version).

New in version 4.0.

--force, -f

On load, unload and switch sub-commands, by-pass any unsatisfied modulefile constraint corresponding to the declared prereq and conflict. Which means for instance that a modulefile will be loaded even if it comes in conflict with another loaded modulefile or that a modulefile will be unloaded even if it is required as a prereq by another modulefile.

On clear sub-command, skip the confirmation dialog and proceed.

New in version 4.3: --force/-f support was dropped on version 4.0 but reintroduced starting version 4.2 with a different meaning: instead of enabling an active dependency resolution mechanism --force command line switch now enables to by-pass dependency consistency when loading or unloading a modulefile.

--help, -h

Give some helpful usage information, and terminates the command.

--icase, -i

Match module specification arguments in a case insensitive manner.

New in version 4.4: --icase/-i support was dropped on version 4.0 but reintroduced starting version 4.4. When set, it now applies to search query string and module specificiation on all sub-commands and modulefile Tcl commands.

--indepth

On avail sub-command, include in search results the matching modulefiles and directories and recursively the modulefiles and directories contained in these matching directories.

New in version 4.3.

--json, -j

Display avail, list, savelist, whatis and search output in JSON format.

New in version 4.5.

--latest, -L

On avail sub-command, display only the highest numerically sorted version of each module name (see Locating Modulefiles section in the modulefile man page).

New in version 4.0.

--long, -l

Display avail, list and savelist output in long format.

--no-auto

On load, unload and switch sub-commands, disable automated module handling mode. See also MODULES_AUTO_HANDLING section.

New in version 4.2.

--no-indepth

On avail sub-command, limit search results to the matching modulefiles and directories found at the depth level expressed by the search query. Thus modulefiles contained in directories part of the result are excluded.

New in version 4.3.

--no-pager

Do not pipe message output into a pager.

New in version 4.1.

--output=LIST, -o LIST

Define the content to report in addition to module names. This option is supported by avail and list sub-commands on their regular or terse output modes. Accepted values are a LIST of elements to report separated by colon character (:). The order of the elements in LIST does not matter.

Accepted elements in LIST for avail sub-command are: modulepath, alias, dirwsym, sym, tag and key.

Accepted elements in LIST for list sub-command are: header, idx, variant, sym, tag and key.

The order of the elements in LIST does not matter. Module names are the only content reported when LIST is set to an empty value.

See also MODULES_AVAIL_OUTPUT and MODULES_LIST_OUTPUT.

New in version 4.7.

Changed in version 4.8: Element variant added for list sub-command

--paginate

Pipe all message output into less (or if set, to the command referred in MODULES_PAGER variable) if error output stream is a terminal. See also MODULES_PAGER section.

New in version 4.1.

--silent, -s

Turn off error, warning and informational messages. module command output result is not affected by silent mode.

New in version 4.3: --silent/-s support was dropped on version 4.0 but reintroduced starting version 4.3.

--starts-with, -S

On avail sub-command, return modules whose name starts with search query string.

New in version 4.3.

--terse, -t

Display avail, list and savelist output in short format.

--trace, -T

Trace mode. Report details on module searches, resolutions, selections and evaluations in addition to printing verbose messages.

New in version 4.6.

--verbose, -v, -vv

Enable verbose messages during module command execution. Multiple -v options increase the verbosity level. The maximum is 2.

New in version 4.3: --verbose/-v support was dropped on version 4.0 but reintroduced starting version 4.3.

Changed in version 4.7: Option form -vv added

--version, -V

Lists the current version of the module command. The command then terminates without further processing.

--width=COLS, -w COLS

Set the width of the output to COLS columns. See also MODULES_TERM_WIDTH section.

New in version 4.7.

Module Sub-Commands
add [--auto|--no-auto] [-f] modulefile...

See load.

aliases [-a]

List all available symbolic version-names and aliases in the current MODULEPATH. All directories in the MODULEPATH are recursively searched in the same manner than for the avail sub-command. Only the symbolic version-names and aliases found in the search are displayed.

New in version 4.0.

Changed in version 4.6: Option --all/-a added

append-path [-d C|--delim C|--delim=C] [--duplicates] variable value...

Append value to environment variable. The variable is a colon, or delimiter, separated list. See append-path in the modulefile man page for further explanation.

New in version 4.1.

apropos [-a] [-j] string

See search.

avail [-d|-L] [-t|-l|-j] [-a] [-o LIST] [-S|-C] [--indepth|--no-indepth] [path...]

List all available modulefiles in the current MODULEPATH. All directories in the MODULEPATH are recursively searched for files containing the modulefile magic cookie. If an argument is given, then each directory in the MODULEPATH is searched for modulefiles whose pathname, symbolic version-name or alias match the argument. Argument may contain wildcard characters. Multiple versions of an application can be supported by creating a subdirectory for the application containing modulefiles for each version.

Symbolic version-names and aliases found in the search are displayed in the result of this sub-command. Symbolic version-names are displayed next to the modulefile they are assigned to within parenthesis. Aliases are listed in the MODULEPATH section where they have been defined. To distinguish aliases from modulefiles a @ symbol is added within parenthesis next to their name. Aliases defined through a global or user specific module RC file are listed under the global/user modulerc section.

When colored output is enabled and a specific graphical rendition is defined for module default version, the default symbol is omitted and instead the defined graphical rendition is applied to the relative modulefile. When colored output is enabled and a specific graphical rendition is defined for module alias, the @ symbol is omitted. The defined graphical rendition applies to the module alias name. See MODULES_COLOR and MODULES_COLORS sections for details on colored output.

Module tags applying to the available modulefiles returned by the avail sub-command are reported along the module name they are associated to (see Module tags section).

A Key section is added at the end of the output in case some elements are reported in parentheses or chevrons along module name or if some graphical rendition is made over some outputed elements. This Key section gives hints on the meaning of such elements.

The parameter path may also refer to a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

Changed in version 4.0: Options --default/-d, --latest/-L added

Changed in version 4.3: Options --starts-with/-S, --contains/-C, --indepth, --no-indepth added

Changed in version 4.5: Option --json/-j added

Changed in version 4.6: Option --all/-a added

Changed in version 4.7: Key section added at end of output

Changed in version 4.7: Option --output/-o added, compatible with regular and terse output modes

clear [-f]

Force the Modules package to believe that no modules are currently loaded. A confirmation is requested if command-line switch -f (or --force) is not passed. Typed confirmation should equal to yes or y in order to proceed.

New in version 4.3: clear support was dropped on version 4.0 but reintroduced starting version 4.3.

config [--dump-state|name [value]|--reset name]

Gets or sets modulecmd.tcl options. Reports the currently set value of passed option name or all existing options if no name passed. If a name and a value are provided, the value of option name is set to value. If command-line switch --reset is passed in addition to a name, overridden value for option name is cleared.

When a reported option value differs from default value a mention is added to indicate whether the overridden value is coming from a command-line switch (cmd-line) or from an environment variable (env-var). When a reported option value is locked and cannot be altered a (locked) mention is added.

If no value is currently set for an option name, the mention <undef> is reported.

When command-line switch --dump-state is passed, current modulecmd.tcl state and Modules-related environment variables are reported in addition to currently set modulecmd.tcl options.

Existing option names are:

advanced_version_spec

Advanced module version specification to finely select modulefiles. Defines environment variable MODULES_ADVANCED_VERSION_SPEC when set.

New in version 4.4.

auto_handling

Automated module handling mode. Defines MODULES_AUTO_HANDLING.

avail_indepth

avail sub-command in depth search mode. Defines MODULES_AVAIL_INDEPTH.

avail_output

Content to report in addition to module names on avail sub-command regular output mode. Defines MODULES_AVAIL_OUTPUT.

New in version 4.7.

avail_terse_output

Content to report in addition to module names on avail sub-command terse output mode. Defines MODULES_AVAIL_TERSE_OUTPUT.

New in version 4.7.

collection_pin_version

Register exact modulefile version in collection. Defines MODULES_COLLECTION_PIN_VERSION.

collection_target

Collection target which is valid for current system. Defines MODULES_COLLECTION_TARGET.

color

Colored output mode. Defines MODULES_COLOR.

colors

Chosen colors to highlight output items. Defines MODULES_COLORS.

contact

Modulefile contact address. Defines MODULECONTACT.

extended_default

Allow partial module version specification. Defines MODULES_EXTENDED_DEFAULT.

New in version 4.4.

editor

Text editor command to open modulefile with through edit sub-command. Defines MODULES_EDITOR.

New in version 4.8.

extra_siteconfig

Additional site-specific configuration script location. Defines MODULES_SITECONFIG.

home

Location of Modules package main directory. Defines MODULESHOME.

New in version 4.4.

icase

Enable case insensitive match. Defines MODULES_ICASE.

New in version 4.4.

ignored_dirs

Directories ignored when looking for modulefiles.

The value of this option cannot be altered.

implicit_default

Set an implicit default version for modules. Defines MODULES_IMPLICIT_DEFAULT.

implicit_requirement

Implicitly define a requirement onto modules specified on module commands in modulefile. Defines MODULES_IMPLICIT_REQUIREMENT.

New in version 4.7.

list_output

Content to report in addition to module names on list sub-command regular output mode. Defines MODULES_LIST_OUTPUT.

New in version 4.7.

list_terse_output

Content to report in addition to module names on list sub-command terse output mode. Defines MODULES_LIST_TERSE_OUTPUT.

New in version 4.7.

locked_configs

Configuration options that cannot be superseded. All options referred in locked_configs value are locked, thus their value cannot be altered.

The value of this option cannot be altered.

mcookie_version_check

Defines if the version set in the Modules magic cookie used in modulefile should be checked against the version of modulecmd.tcl to determine if the modulefile could be evaluated or not. Defines MODULES_MCOOKIE_VERSION_CHECK.

New in version 4.7.

ml

Define ml command at initialization time. Defines MODULES_ML.

New in version 4.5.

nearly_forbidden_days

Set the number of days a module should be considered nearly forbidden prior reaching its expiry date. Defines MODULES_NEARLY_FORBIDDEN_DAYS.

New in version 4.6.

pager

Text viewer to paginate message output. Defines MODULES_PAGER.

rcfile

Global run-command file location. Defines MODULERCFILE.

run_quarantine

Environment variables to indirectly pass to modulecmd.tcl. Defines MODULES_RUN_QUARANTINE.

silent_shell_debug

Disablement of shell debugging property for the module command. Defines MODULES_SILENT_SHELL_DEBUG.

search_match

Module search match style. Defines MODULES_SEARCH_MATCH.

set_shell_startup

Ensure module command definition by setting shell startup file. Defines MODULES_SET_SHELL_STARTUP.

shells_with_ksh_fpath

Ensure module command is defined in ksh when it is started as a sub-shell from the listed shells. Defines MODULES_SHELLS_WITH_KSH_FPATH.

New in version 4.7.

siteconfig

Primary site-specific configuration script location.

The value of this option cannot be altered.

tag_abbrev

Abbreviations to use to report module tags. Defines MODULES_TAG_ABBREV.

New in version 4.7.

tag_color_name

Tags whose name should be colored instead of module name. Defines MODULES_TAG_COLOR_NAME.

New in version 4.7.

tcl_ext_lib

Modules Tcl extension library location.

The value of this option cannot be altered.

term_background

Terminal background color kind. Defines MODULES_TERM_BACKGROUND.

term_width

Set the width of the output. Defines MODULES_TERM_WIDTH.

New in version 4.7.

unload_match_order

Unload firstly loaded or lastly loaded module matching request. Defines MODULES_UNLOAD_MATCH_ORDER.

variant_shortcut

Shortcut characters that could be used to specify or report module variants. Defines MODULES_VARIANT_SHORTCUT.

New in version 4.8.

verbosity

Module command verbosity level. Defines MODULES_VERBOSITY.

wa_277

Workaround for Tcsh history issue. Defines MODULES_WA_277.

New in version 4.3.

display modulefile...

Display information about one or more modulefiles. The display sub-command will list the full path of the modulefile and the environment changes the modulefile will make if loaded. (Note: It will not display any environment changes found within conditional statements.)

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

edit modulefile

Open modulefile for edition with text editor command designated by the editor configuration option.

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

New in version 4.8.

help [modulefile...]

Print the usage of each sub-command. If an argument is given, print the Module-specific help information for the modulefile.

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

info-loaded modulefile

Returns the names of currently loaded modules matching passed modulefile. Returns an empty string if passed modulefile does not match any loaded modules. See module-info loaded in the modulefile man page for further explanation.

New in version 4.1.

initadd modulefile...

Add modulefile to the shell's initialization file in the user's home directory. The startup files checked (in order) are:

C Shell

.modules, .cshrc, .csh_variables and .login

TENEX C Shell

.modules, .tcshrc, .cshrc, .csh_variables and .login

Bourne and Korn Shells

.modules, .profile

GNU Bourne Again Shell

.modules, .bash_profile, .bash_login, .profile and .bashrc

Z Shell

.modules, .zshrc, .zshenv and .zlogin

Friendly Interactive Shell

.modules, .config/fish/config.fish

If a module load line is found in any of these files, the modulefiles are appended to any existing list of modulefiles. The module load line must be located in at least one of the files listed above for any of the init sub-commands to work properly. If the module load line is found in multiple shell initialization files, all of the lines are changed.

initclear

Clear all of the modulefiles from the shell's initialization files.

initlist

List all of the modulefiles loaded from the shell's initialization file.

initprepend modulefile...

Does the same as initadd but prepends the given modules to the beginning of the list.

initrm modulefile...

Remove modulefile from the shell's initialization files.

initswitch modulefile1 modulefile2

Switch modulefile1 with modulefile2 in the shell's initialization files.

is-avail modulefile...

Returns a true value if any of the listed modulefiles exists in enabled MODULEPATH. Returns a false value otherwise. See is-avail in the modulefile man page for further explanation.

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

New in version 4.1.

is-loaded [modulefile...]

Returns a true value if any of the listed modulefiles has been loaded or if any modulefile is loaded in case no argument is provided. Returns a false value otherwise. See is-loaded in the modulefile man page for further explanation.

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

New in version 4.1.

is-saved [collection...]

Returns a true value if any of the listed collections exists or if any collection exists in case no argument is provided. Returns a false value otherwise. See is-saved in the modulefile man page for further explanation.

New in version 4.1.

is-used [directory...]

Returns a true value if any of the listed directories has been enabled in MODULEPATH or if any directory is enabled in case no argument is provided. Returns a false value otherwise. See is-used in the modulefile man page for further explanation.

New in version 4.1.

keyword [-a] [-j] string

See search.

list [-a] [-o LIST] [-t|-l|-j]

List loaded modules.

Module tags applying to the loaded modules are reported along the module name they are associated to (see Module tags section).

Module variants selected on the loaded modules are reported along the module name they belong to (see Module variants section).

A Key section is added at the end of the output in case some elements are reported in parentheses or chevrons along module name or if some graphical rendition is made over some outputed elements. This Key section gives hints on the meaning of such elements.

Changed in version 4.5: Option --json/-j added

Changed in version 4.7: Option --all/-a added

Changed in version 4.7: Key section added at end of output

Changed in version 4.7: Option --output/-o added, compatible with regular and terse output modes.

Changed in version 4.8: Report if enabled the variants selected on loaded modules

load [--auto|--no-auto] [-f] modulefile...

Load modulefile into the shell environment.

Once loaded, the loaded module tag is associated to the loaded module. If module has been automatically loaded by another module, the auto-loaded tag is associated instead (see Module tags section).

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

Changed in version 4.2: Options --auto, --no-auto, --force/-f added

path modulefile

Print path to modulefile.

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

New in version 4.0.

paths modulefile

Print path of available modulefiles matching argument.

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

New in version 4.0.

prepend-path [-d C|--delim C|--delim=C] [--duplicates] variable value...

Prepend value to environment variable. The variable is a colon, or delimiter, separated list. See prepend-path in the modulefile man page for further explanation.

New in version 4.1.

purge [-f]

Unload all loaded modulefiles.

When the --force option is set, also unload modulefiles that are depended by unloadable modules.

Changed in version 4.7: Option --force/-f added

refresh

See reload.

reload

Unload then load all loaded modulefiles.

No unload then load is performed and an error is returned if the loaded modulefiles have unsatisfied constraint corresponding to the prereq and conflict they declare.

New in version 4.0.

remove-path [-d C|--delim C|--delim=C] [--index] variable value...

Remove value from the colon, or delimiter, separated list in environment variable. See remove-path in the modulefile man page for further explanation.

New in version 4.1.

restore [collection]

Restore the environment state as defined in collection. If collection name is not specified, then it is assumed to be the default collection. If collection is a fully qualified path, it is restored from this location rather than from a file under the user's collection directory. If MODULES_COLLECTION_TARGET is set, a suffix equivalent to the value of this variable is appended to the collection file name to restore.

When restoring a collection, the currently set MODULEPATH directory list and the currently loaded modulefiles are unused and unloaded then used and loaded to exactly match the MODULEPATH and loaded modulefiles lists saved in this collection file. The order of the paths and modulefiles set in collection is preserved when restoring. It means that currently loaded modules are unloaded to get the same LOADEDMODULES root than collection and currently used module paths are unused to get the same MODULEPATH root. Then missing module paths are used and missing modulefiles are loaded.

If a module, without a default version explicitly defined, is recorded in a collection by its bare name: loading this module when restoring the collection will fail if the configuration option implicit_default is disabled.

New in version 4.0.

rm [--auto|--no-auto] [-f] modulefile...

See unload.

save [collection]

Record the currently set MODULEPATH directory list and the currently loaded modulefiles in a collection file under the user's collection directory $HOME/.module. If collection name is not specified, then it is assumed to be the default collection. If collection is a fully qualified path, it is saved at this location rather than under the user's collection directory.

If MODULES_COLLECTION_TARGET is set, a suffix equivalent to the value of this variable will be appended to the collection file name.

By default, if a loaded modulefile corresponds to the explicitly defined default module version, the bare module name is recorded. If the configuration option implicit_default is enabled, the bare module name is also recorded for the implicit default module version. If MODULES_COLLECTION_PIN_VERSION is set to 1, module version is always recorded even if it is the default version.

No collection is recorded and an error is returned if the loaded modulefiles have unsatisfied constraint corresponding to the prereq and conflict they declare.

New in version 4.0.

savelist [-t|-l|-j]

List collections that are currently saved under the user's collection directory. If MODULES_COLLECTION_TARGET is set, only collections matching the target suffix will be displayed.

New in version 4.0.

Changed in version 4.5: Option --json/-j added

saverm [collection]

Delete the collection file under the user's collection directory. If collection name is not specified, then it is assumed to be the default collection. If MODULES_COLLECTION_TARGET is set, a suffix equivalent to the value of this variable will be appended to the collection file name.

New in version 4.0.

saveshow [collection]

Display the content of collection. If collection name is not specified, then it is assumed to be the default collection. If collection is a fully qualified path, this location is displayed rather than a collection file under the user's collection directory. If MODULES_COLLECTION_TARGET is set, a suffix equivalent to the value of this variable will be appended to the collection file name.

New in version 4.0.

Seeks through the module-whatis information of all modulefiles for the specified string. All module-whatis information matching the string in a case insensitive manner will be displayed. string may contain wildcard characters.

New in version 4.0: Prior version 4.0 module-whatis information search was performed with apropos or keyword sub-commands.

Changed in version 4.5: Option --json/-j added

Changed in version 4.6: Option --all/-a added

sh-to-mod shell script [arg...]

Evaluate with shell the designated script with defined arguments to find out the environment changes it does. Environment prior and after script evaluation are compared to determine these changes. They are translated into modulefile commands to output the modulefile content equivalent to the evaluation of shell script.

Changes on environment variables, shell aliases, shell functions and current working directory are tracked.

Shell could be specified as a command name or a fully qualified pathname. The following shells are supported: sh, dash, csh, tcsh, bash, ksh, ksh93, zsh and fish.

New in version 4.6.

show modulefile...

See display.

source scriptfile...

Execute scriptfile into the shell environment. scriptfile must be written with modulefile syntax and specified with a fully qualified path. Once executed scriptfile is not marked loaded in shell environment which differ from load sub-command.

New in version 4.0.

swap [--auto|--no-auto] [-f] [modulefile1] modulefile2

See switch.

switch [--auto|--no-auto] [-f] [modulefile1] modulefile2

Switch loaded modulefile1 with modulefile2. If modulefile1 is not specified, then it is assumed to be the currently loaded module with the same root name as modulefile2.

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

Changed in version 4.2: Options --auto, --no-auto, --force/-f added

try-add [--auto|--no-auto] [-f] modulefile...

See try-load.

New in version 4.8.

test modulefile...

Execute and display results of the Module-specific tests for the modulefile.

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

New in version 4.0.

try-load [--auto|--no-auto] [-f] modulefile...

Like load sub-command, load modulefile into the shell environment, but do not complain if modulefile cannot be found. If modulefile is found but its evaluation fails, error is still reported.

Once loaded, the loaded module tag is associated to the loaded module. If module has been automatically loaded by another module, the auto-loaded tag is associated instead (see Module tags section).

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

New in version 4.8.

unload [--auto|--no-auto] [-f] modulefile...

Remove modulefile from the shell environment.

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

Changed in version 4.2: Options --auto, --no-auto, --force/-f added

unuse directory...

Remove one or more directories from the MODULEPATH environment variable if reference counter of these directories is equal to 1 or unknown.

Reference counter of directory in MODULEPATH denotes the number of times directory has been enabled. When attempting to remove directory from MODULEPATH, reference counter variable MODULEPATH_modshare is checked and directory is removed only if its relative counter is equal to 1 or not defined. Otherwise directory is kept and reference counter is decreased by 1.

use [-a|--append] directory...

Prepend one or more directories to the MODULEPATH environment variable. The --append flag will append the directory to MODULEPATH.

Reference counter environment variable MODULEPATH_modshare is also set to increase the number of times directory has been added to MODULEPATH.

whatis [-a] [-j] [modulefile...]

Display the information set up by the module-whatis commands inside the specified modulefiles. These specified modulefiles may be expressed using wildcard characters. If no modulefile is specified, all module-whatis lines will be shown.

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

Changed in version 4.5: Option --json/-j added

Changed in version 4.6: Option --all/-a added

Modulefiles

modulefiles are written in the Tool Command Language (Tcl) and are interpreted by modulecmd.tcl. modulefiles can use conditional statements. Thus the effect a modulefile will have on the environment may change depending upon the current state of the environment.

Environment variables are unset when unloading a modulefile. Thus, it is possible to load a modulefile and then unload it without having the environment variables return to their prior state.

Advanced module version specifiers

When the advanced module version specifiers mechanism is enabled (see MODULES_ADVANCED_VERSION_SPEC), the specification of modulefile passed on Modules sub-commands changes. After the module name a version constraint and variants may be added.

Version specifiers

After the module name a version constraint prefixed by the @ character may be added. It could be directly appended to the module name or separated from it with a space character.

Constraints can be expressed to refine the selection of module version to:

  • a single version with the @version syntax, for instance foo@1.2.3 syntax will select module foo/1.2.3
  • a list of versions with the @version1,version2,... syntax, for instance foo@1.2.3,1.10 will match modules foo/1.2.3 and foo/1.10
  • a range of versions with the @version1:, @:version2 and @version1:version2 syntaxes, for instance foo@1.2: will select all versions of module foo greater than or equal to 1.2, foo@:1.3 will select all versions less than or equal to 1.3 and foo@1.2:1.3 matches all versions between 1.2 and 1.3 including 1.2 and 1.3 versions

Advanced specification of single version or list of versions may benefit from the activation of the extended default mechanism (see MODULES_EXTENDED_DEFAULT) to use an abbreviated notation like @1 to refer to more precise version numbers like 1.2.3. Range of versions on its side natively handles abbreviated versions.

In order to be specified in a range of versions or compared to a range of versions, the version major element should corresponds to a number. For instance 10a, 1.2.3, 1.foo are versions valid for range comparison whereas default or foo.2 versions are invalid for range comparison.

Range of versions can be specified in version list, for instance foo@:1.2,1.4:1.6,1.8:. Such specification helps to exclude specific versions, like versions 1.3 and 1.7 in previous example.

If the implicit default mechanism is also enabled (see MODULES_IMPLICIT_DEFAULT), a default and latest symbolic versions are automatically defined for each module name (also at each directory level for deep modulefiles). These automatic version symbols are defined unless a symbolic version, alias, or regular module version already exists for these default or latest version names. Using the mod@latest (or mod/latest) syntax ensures highest available version will be selected.

The symbolic version loaded may be used over loaded module name to designate the loaded version of the module with associated selected variants. This version symbol should be specified using the @ prefix notation (e.g., foo@loaded). An error is returned if no version of designated module is currently loaded.

Variants

After the module name, variants can be specified. Module variants are alternative evaluation of the same modulefile. A variant is specified by associating a value to its name. This specification is then transmitted to the evaluating modulefile which instantiates the variant in the ModuleVariant array variable when reaching the variant modulefile command declaring this variant.

Variant can be specified with the name=value syntax where name is the declared variant name and value, the value this variant is set to when evaluating the modulefile.

Boolean variants can be specified with the +name syntax to set this variant on and with the -name or ~name syntaxes to set this variant off. The -name syntax is not supported on ml command as the minus sign already means to unload designated module. The ~name and +name syntaxes could also be defined appended to another specification word (e.g., the module name, version or another variant specification), whereas -name syntax must be the start of a new specification word.

Boolean variants may also be specified with the name=value syntax. value should be set to 1, true, t, yes, y or on to enable the variant or it should be set to 0, false, f, no, n or off to disable the variant.

Shortcuts may be used to abbreviate variant specification. The variant_shortcut configuration option associates shortcut character to variant name. With a shortcut defined, variant could be specified with the <shortcut>value syntax. For instance if character % is set as a shortcut for variant foo, the %value syntax is equivalent to the foo=value syntax.

Specific characters used in variant specification syntax cannot be used as part of the name of a module. These specific characters are +, ~, = and all characters set as variant shortcut. Exception is made for + character which could be set one or several consecutive times at the end of module name (e.g., name+ or name++).

New in version 4.4.

Changed in version 4.8: Use of version range is allowed in version list

Changed in version 4.8: Support for module variant added

Module tags

Module tags are piece of information that can be associated to individual modulefiles. Tags could be purely informational or may lead to specific behaviors.

Module tags may be inherited from the module state set by a modulefile command or consequence of a module action. The inherited tags are:

  • auto-loaded: module has been automatically loaded by another module
  • forbidden: module has been set forbidden through the use of the module-forbid command and thus this module cannot be loaded.
  • hidden: module has been set hidden through the use of the module-hide command and thus it is not reported by default among the result of an avail sub-command.
  • hidden-loaded: module has been set hidden once loaded through the use of the module-hide --hidden-loaded command thus it is not reported bu default among the result of a list sub-command.
  • loaded: module is currently loaded
  • nearly-forbidden: module will soon be forbidden, which has been set through the use of the module-forbid command. Thus this module will soon not be able to load anymore.

Tags may also be associated to modules by using the module-tag modulefile command. Among tags that could be set this way, some have a special meaning:

  • sticky: module once loaded cannot be unloaded unless forced or reloaded (see Sticky modules section)
  • super-sticky: module once loaded cannot be unloaded unless reloaded, module cannot be unloaded even if forced (see Sticky modules section)

Module tags are reported along the module they are associated to on avail and list sub-command results. Tags could be reported either:

  • along the module name, all tags set within angle brackets, each tag separated from the others with a colon character (e.g., foo/1.2 <tag1:tag2>).
  • graphically rendered over the module name for each tag associated to a Select Graphic Rendition (SGR) code in the color palette (see MODULES_COLORS)

When an abbreviated string is associated to a tag name (see MODULES_TAG_ABBREV), this abbreviation is used to report tag along the module name or the tag is graphically rendered over the module name if a SGR code is associated with tag abbreviation in the color palette. With an abbreviation set, the SGR code associated to the tag full name is ignored thus an SGR code should be associated to the abbreviation to get a graphical rendering of tag. If the abbreviation associated to a tag corresponds to the empty string, tag is not reported.

Graphical rendering is made over the tag name or abbreviation instead of over the module name for each tag name or abbreviation set in the MODULES_TAG_COLOR_NAME environment variable.

When several tags have to be rendered graphically over the same module name, each tag is rendered over a sub-part of the module name. In case more tags need to be rendered than the total number of characters in the module name, the remaining tags are graphically rendered over the tag name instead of over the module name.

When the JSON output mode is enabled (with --json), tags are reported by their name under the tags attribute. Tag abbreviation and color rendering do not apply on JSON output.

Module tags cannot be used in search query to designate a modulefile.

New in version 4.7.

Sticky modules

Modules are said sticky when they cannot be unloaded (they stick to the loaded environment). Two kind of stickyness can be distinguished:

  • sticky module: cannot be unloaded unless if the unload is forced or if the module is reloaded after being unloaded
  • super-sticky module: cannot be unloaded unless if the module is reloaded after being unloaded; super-sticky modules cannot be unloaded even if the unload is forced.

Modules are designated sticky by associating them the sticky or the super-sticky module tag with the module-tag modulefile command.

When stickyness is defined over the generic module name (and not over a specific module version, a version list or a version range), sticky or super-sticky module can be swapped by another version of module. For instance if the sticky tag is defined over foo module, loaded module foo/1.2 can be swapped by foo/2.0. Such stickyness definition means one version of module should stay loaded whatever version it is.

New in version 4.7.

Module variants

Module variants are alternative evaluation of the same modulefile. A variant is specified by associating a value to its name when designating module. Variant specification relies on the Advanced module version specifiers mechanism.

Once specified, variant's value is transmitted to the evaluating modulefile which instantiates the variant in the ModuleVariant array variable when reaching the variant modulefile command declaring this variant. For instance the module load foo/1.2 bar=value1 command leads to the evaluation of foo/1.2 modulefile with bar=value1 variant specification. When reaching the variant bar value1 value2 value3 command in modulefile during its evaluation, the ModuleVariant(bar) array element is set to the value1 string.

Once variants are instantiated, modulefile's code could check the variant values to adapt the evaluation and define for instance different module requirements or produce different environment variable setup.

Variants are interpreted in contexts where modulefiles are evaluated. Thus the variants specified on module designation are ignored by the avail, whatis, is-avail, path or paths sub-commands.

When modulefile is evaluated a value should be specified for each variant this modulefile declares. When reaching the variant modulefile command declaring a variant, an error is raised if no value is specified for this variant and if no default value is declared. Specified variant value should match a value from the declared accepted value list otherwise an error is raised. Additionally if a variant is specified but does not correspond to a variant declared in modulefile, an error is raised.

Module variants are reported along the module they are associated to on list sub-command results. Variants are reported within curly braces next to module name, each variant definition separated from the others with a colon character (e.g., foo/1.2{variant1=value:+variant2}). Boolean variants are reported with the +name or -name syntaxes. When a shortcut character is defined for a variant (see MODULES_VARIANT_SHORTCUT) it is reported with the <shortcut>value syntax. For instance if % character is defined as a shortcut for variant1: foo/1.2{%value:+variant2}.

When the JSON output mode is enabled (with --json), variants are reported under the variants JSON object as name/value pairs. Values of Boolean variant are set as JSON Boolean. Other values are set as JSON strings. Variant shortcut and color rendering do not apply on JSON output.

New in version 4.8.

Collections

Collections describe a sequence of module use then module load commands that are interpreted by modulecmd.tcl to set the user environment as described by this sequence. When a collection is activated, with the restore sub-command, module paths and loaded modules are unused or unloaded if they are not part or if they are not ordered the same way as in the collection.

Collections are generated by the save sub-command that dumps the current user environment state in terms of module paths and loaded modules. By default collections are saved under the $HOME/.module directory.

Collections may be valid for a given target if they are suffixed. In this case these collections can only be restored if their suffix correspond to the current value of the MODULES_COLLECTION_TARGET environment variable (see the dedicated section of this topic below).

New in version 4.0.

EXIT STATUS

The module command exits with 0 if its execution succeed. Otherwise 1 is returned.

ENVIRONMENT

_LMFILES_

A colon separated list of the full pathname for all loaded modulefiles.

LOADEDMODULES

A colon separated list of all loaded modulefiles.

MODULECONTACT

Email address to contact in case any issue occurs during the interpretation of modulefiles.

New in version 4.0.

MODULEPATH

The path that the module command searches when looking for modulefiles. Typically, it is set to the main modulefiles directory, /usr/share/Modules/modulefiles, by the initialization script. MODULEPATH can be set using module use or by the module initialization script to search group or personal modulefile directories before or after the main modulefile directory.

Path elements registered in the MODULEPATH environment variable may contain reference to environment variables which are converted to their corresponding value by module command each time it looks at the MODULEPATH value. If an environment variable referred in a path element is not defined, its reference is converted to an empty string.

MODULERCFILE

The location of a global run-command file containing modulefile specific setup. See Modulecmd startup section for detailed information.

MODULES_ADVANCED_VERSION_SPEC

If set to 1, enable advanced module version specifiers (see Advanced module version specifiers section). If set to 0, disable advanced module version specifiers.

Advanced module version specifiers enablement is defined in the following order of preference: MODULES_ADVANCED_VERSION_SPEC environment variable then the default set in modulecmd.tcl script configuration. Which means MODULES_ADVANCED_VERSION_SPEC overrides default configuration.

New in version 4.4.

MODULES_AUTO_HANDLING

If set to 1, enable automated module handling mode. If set to 0 disable automated module handling mode. Other values are ignored.

Automated module handling mode consists in additional actions triggered when loading or unloading a modulefile to satisfy the constraints it declares. When loading a modulefile, following actions are triggered:

  • Requirement Load: load of the modulefiles declared as a prereq of the loading modulefile.
  • Dependent Reload: reload of the modulefiles declaring a prereq onto loaded modulefile or declaring a prereq onto a modulefile part of this reloading batch.

When unloading a modulefile, following actions are triggered:

  • Dependent Unload: unload of the modulefiles declaring a non-optional prereq onto unloaded modulefile or declaring a non-optional prereq onto a modulefile part of this unloading batch. A prereq modulefile is considered optional if the prereq definition order is made of multiple modulefiles and at least one alternative modulefile is loaded.
  • Useless Requirement Unload: unload of the prereq modulefiles that have been automatically loaded for either the unloaded modulefile, an unloaded dependent modulefile or a modulefile part of this useless requirement unloading batch. Modulefiles are added to this unloading batch only if they are not required by any other loaded modulefiles.
  • Dependent Reload: reload of the modulefiles declaring a conflict or an optional prereq onto either the unloaded modulefile, an unloaded dependent or an unloaded useless requirement or declaring a prereq onto a modulefile part of this reloading batch.

In case a loaded modulefile has some of its declared constraints unsatisfied (pre-required modulefile not loaded or conflicting modulefile loaded for instance), this loaded modulefile is excluded from the automatic reload actions described above.

For the specific case of the switch sub-command, where a modulefile is unloaded to then load another modulefile. Dependent modulefiles to Unload are merged into the Dependent modulefiles to Reload that are reloaded after the load of the switched-to modulefile.

Automated module handling mode enablement is defined in the following order of preference: --auto/--no-auto command line switches, then MODULES_AUTO_HANDLING environment variable, then the default set in modulecmd.tcl script configuration. Which means MODULES_AUTO_HANDLING overrides default configuration and --auto/--no-auto command line switches override every other ways to enable or disable this mode.

New in version 4.2.

MODULES_AVAIL_INDEPTH

If set to 1, enable in depth search results for avail sub-command. If set to 0 disable avail sub-command in depth mode. Other values are ignored.

When in depth mode is enabled, modulefiles and directories contained in directories matching search query are also included in search results. When disabled these modulefiles and directories contained in matching directories are excluded.

avail sub-command in depth mode enablement is defined in the following order of preference: --indepth/--no-indepth command line switches, then MODULES_AVAIL_INDEPTH environment variable, then the default set in modulecmd.tcl script configuration. Which means MODULES_AVAIL_INDEPTH overrides default configuration and --indepth/--no-indepth command line switches override every other ways to enable or disable this mode.

New in version 4.3.

MODULES_AVAIL_OUTPUT

A colon separated list of the elements to report in addition to module names on avail sub-command regular output mode.

Accepted elements that can be set in value list are:

  • alias: module aliases.
  • dirwsym: directories associated with symbolic versions.
  • key: legend appended at the end of the output to explain it.
  • modulepath: modulepath names set as header prior the list of available modules found in them.
  • sym: symbolic versions associated with available modules.
  • tag: tags associated with available modules.

The order of the elements in the list does not matter. Module names are the only content reported when LIST is set to an empty value.

In case the modulepath element is missing from value list, the available modules from global/user rc and all enabled modulepaths are reported as a single list.

avail sub-command regular output content is defined in the following order of preference: --output/-o command line switches, then MODULES_AVAIL_OUTPUT environment variable, then the default set in modulecmd.tcl script configuration. Which means MODULES_AVAIL_OUTPUT overrides default configuration and --output/-o command line switches override every other ways to configure regular output content.

New in version 4.7.

MODULES_AVAIL_TERSE_OUTPUT

A colon separated list of the elements to report in addition to module names on avail sub-command terse output mode.

See MODULES_AVAIL_OUTPUT to get the accepted elements that can be set in value list.

The order of the elements in the list does not matter. Module names are the only content reported when LIST is set to an empty value.

avail sub-command terse output content is defined in the following order of preference: --output/-o command line switches, then MODULES_AVAIL_TERSE_OUTPUT environment variable, then the default set in modulecmd.tcl script configuration. Which means MODULES_AVAIL_TERSE_OUTPUT overrides default configuration and --output/-o command line switches override every other ways to configure terse output content.

New in version 4.7.

MODULES_CMD

The location of the active module command script.

New in version 4.1.

MODULES_COLLECTION_PIN_VERSION

If set to 1, register exact version number of modulefiles when saving a collection. Otherwise modulefile version number is omitted if it corresponds to the explicitly set default version and also to the implicit default when the configuration option implicit_default is enabled.

New in version 4.1.

MODULES_COLLECTION_TARGET

The collection target that determines what collections are valid thus reachable on the current system.

Collection directory may sometimes be shared on multiple machines which may use different modules setup. For instance modules users may access with the same HOME directory multiple systems using different OS versions. When it happens a collection made on machine 1 may be erroneous on machine 2.

When a target is set, only the collections made for that target are available to the restore, savelist, saveshow and saverm sub-commands. Saving a collection registers the target footprint by suffixing the collection filename with .$MODULES_COLLECTION_TARGET. The collection target is not involved when collection is specified as file path on the saveshow, restore and save sub-commands.

For example, the MODULES_COLLECTION_TARGET variable may be set with results from commands like lsb_release, hostname, dnsdomainname, etc.

New in version 4.0.

MODULES_COLOR

Defines if output should be colored or not. Accepted values are never, auto and always.

When color mode is set to auto, output is colored only if the standard error output channel is attached to a terminal.

Colored output enablement is defined in the following order of preference: --color command line switch, then MODULES_COLOR environment variable, then NO_COLOR, CLICOLOR and CLICOLOR_FORCE environment variables, then the default set in modulecmd.tcl script configuration. Which means MODULES_COLOR overrides default configuration and the NO_COLOR and CLICOLOR/CLICOLOR_FORCE variables. --color command line switch overrides every other ways to enable or disable this mode.

NO_COLOR, CLICOLOR and CLICOLOR_FORCE environment variables are also honored to define color mode. The never mode is set if NO_COLOR is defined (regardless of its value) or if CLICOLOR equals to 0. If CLICOLOR is set to another value, it corresponds to the auto mode. The always mode is set if CLICOLOR_FORCE is set to a value different than 0. NO_COLOR variable prevails over CLICOLOR and CLICOLOR_FORCE. Color mode set with these three variables is superseded by mode set with MODULES_COLOR environment variable.

New in version 4.3.

MODULES_COLORS

Specifies the colors and other attributes used to highlight various parts of the output. Its value is a colon-separated list of output items associated to a Select Graphic Rendition (SGR) code. It follows the same syntax than LS_COLORS.

Output items are designated by keys. Items able to be colorized are: highlighted element (hi), debug information (db), trace information (tr), tag separator (se); Error (er), warning (wa), module error (me) and info (in) message prefixes; Modulepath (mp), directory (di), module alias (al), module variant (va), module symbolic version (sy), module default version (de) and modulefile command (cm).

Module tags can also be colorized. The key to set in the color palette to get a graphical rendering of a tag is the tag name or the tag abbreviation if one is defined for tag. The SGR code applied to a tag name is ignored if an abbreviation is set for this tag thus the SGR code should be defined for this abbreviation to get a graphical rendering. Each basic tag has by default a key set in the color palette, based on its abbreviated string: auto-loaded (aL), forbidden (F), hidden and hidden-loaded (H), loaded (L), nearly-forbidden (nF), sticky (S) and super-sticky (sS).

See the Select Graphic Rendition (SGR) section in the documentation of the text terminal that is used for permitted values and their meaning as character attributes. These substring values are integers in decimal representation and can be concatenated with semicolons. Modules takes care of assembling the result into a complete SGR sequence (\33[...m). Common values to concatenate include 1 for bold, 4 for underline, 30 to 37 for foreground colors and 90 to 97 for 16-color mode foreground colors. See also https://en.wikipedia.org/wiki/ANSI_escape_code#SGR_(Select_Graphic_Rendition)_parameters for a complete SGR code reference.

No graphical rendition will be applied to an output item that could normally be colored but which is not defined in the color set. Thus if MODULES_COLORS is defined empty, no output will be colored at all.

The color set is defined for Modules in the following order of preference: MODULES_COLORS environment variable, then the default set in modulecmd.tcl script configuration. Which means MODULES_COLORS overrides default configuration.

New in version 4.3.

Changed in version 4.6: Output item for trace information (tr) added

Changed in version 4.7: Output items for module tags auto-loaded (aL), forbidden (F), hidden and hidden-loaded (H), loaded (L), nearly-forbidden (nF), sticky (S) and super-sticky (sS) added

Changed in version 4.8: Output item for module variant (va) added

MODULES_EDITOR

Text editor command name or path for use to open modulefile through the edit sub-command.

Editor command is defined for Modules in the following order of preference: MODULES_EDITOR, or VISUAL or EDITOR environment variables, then the default set in modulecmd.tcl script configuration. Which means MODULES_EDITOR overrides VISUAL or EDITOR environment variables and default configuration.

New in version 4.8.

MODULES_EXTENDED_DEFAULT

If set to 1, a specified module version is matched against starting portion of existing module versions, where portion is a substring separated from the rest of the version string by a . character. For example specified modules mod/1 and mod/1.2 will match existing modulefile mod/1.2.3.

In case multiple modulefiles match the specified module version and a single module has to be selected, the explicitly set default version is returned if it is part of matching modulefiles. Otherwise the implicit default among matching modulefiles is returned if defined (see MODULES_IMPLICIT_DEFAULT section)

This environment variable supersedes the value of the configuration option extended_default set in modulecmd.tcl script.

New in version 4.4.

MODULES_ICASE

When module specification are passed as argument to module sub-commands or modulefile Tcl commands, defines the case sensitiveness to apply to match them. When MODULES_ICASE is set to never, a case sensitive match is applied in any cases. When set to search, a case insensitive match is applied to the avail, whatis and paths sub-commands. When set to always, a case insensitive match is also applied to the other module sub-commands and modulefile Tcl commands for the module specification they receive as argument.

Case sensitiveness behavior is defined in the following order of preference: --icase command line switch, which corresponds to the always mode, then MODULES_ICASE environment variable, then the default set in modulecmd.tcl script configuration. Which means MODULES_ICASE overrides default configuration and --icase command line switch overrides every other ways to set case sensitiveness behavior.

New in version 4.4.

MODULES_IMPLICIT_DEFAULT

Defines (if set to 1) or not (if set to 0) an implicit default version for modules without a default version explicitly defined (see Locating Modulefiles section in the modulefile man page).

Without either an explicit or implicit default version defined a module must be fully qualified (version should be specified in addition to its name) to get:

  • targeted by module load, switch, display, help, test and path sub-commands.
  • restored from a collection, unless already loaded in collection-specified order.
  • automatically loaded by automated module handling mechanisms (see MODULES_AUTO_HANDLING section) when declared as module requirement, with prereq or module load modulefile commands.

An error is returned in the above situations if either no explicit or implicit default version is defined.

This environment variable supersedes the value of the configuration option implicit_default set in modulecmd.tcl script. This environment variable is ignored if implicit_default has been declared locked in locked_configs configuration option.

New in version 4.3.

MODULES_IMPLICIT_REQUIREMENT

Defines (if set to 1) or not (if set to 0) an implicit prereq or conflict requirement onto modules specified respectively on module load or module unload commands in modulefile. When enabled an implicit conflict requirement onto switched-off module and a prereq requirement onto switched-on module are also defined for module switch commands used in modulefile.

This environment variable supersedes the value of the configuration option implicit_requirement set in modulecmd.tcl script. MODULES_IMPLICIT_REQUIREMENT is in turn superseded by the --not-req option that applies to a module command in a modulefile.

New in version 4.7.

MODULES_LIST_OUTPUT

A colon separated list of the elements to report in addition to module names on list sub-command regular output mode.

Accepted elements that can be set in value list are:

  • header: sentence to introduce the list of loaded modules or to state that no modules are loaded currently.
  • idx: index position of each loaded module.
  • key: legend appended at the end of the output to explain it.
  • variant: variant values selected for loaded modules.
  • sym: symbolic versions associated with loaded modules.
  • tag: tags associated with loaded modules.

The order of the elements in the list does not matter. Module names are the only content reported when LIST is set to an empty value.

list sub-command regular output content is defined in the following order of preference: --output/-o command line switches, then MODULES_LIST_OUTPUT environment variable, then the default set in modulecmd.tcl script configuration. Which means MODULES_LIST_OUTPUT overrides default configuration and --output/-o command line switches override every other ways to configure regular output content.

New in version 4.7.

Changed in version 4.8: Element variant added

MODULES_LIST_TERSE_OUTPUT

A colon separated list of the elements to report in addition to module names on list sub-command terse output mode.

See MODULES_LIST_OUTPUT to get the accepted elements that can be set in value list.

The order of the elements in the list does not matter. Module names are the only content reported when LIST is set to an empty value.

list sub-command regular output content is defined in the following order of preference: --output/-o command line switches, then MODULES_LIST_TERSE_OUTPUT environment variable, then the default set in modulecmd.tcl script configuration. Which means MODULES_LIST_TERSE_OUTPUT overrides default configuration and --output/-o command line switches override every other ways to configure regular output content.

New in version 4.7.

MODULES_LMALTNAME

A colon separated list of the alternative names set through module-version and module-alias statements corresponding to all loaded modulefiles. Each element in this list starts by the name of the loaded modulefile followed by all alternative names resolving to it. The loaded modulefile and its alternative names are separated by the ampersand character.

Each alternative name stored in MODULES_LMALTNAME is prefixed by the al| string if it corresponds to a module alias or prefixed by the as| string if it corresponds to an automatic version symbol. These prefixes help to distinguish the different kind of alternative name.

This environment variable is intended for module command internal use to get knowledge of the alternative names matching loaded modulefiles in order to keep environment consistent when conflicts or pre-requirements are set over these alternative designations. It also helps to find a match after modulefiles being loaded when unload, is-loaded or info-loaded actions are run over these names.

Starting version 4.7 of Modules, MODULES_LMALTNAME is also used on list sub-command to report the symbolic versions associated with the loaded modules.

New in version 4.2.

MODULES_LMCONFLICT

A colon separated list of the conflict statements defined by all loaded modulefiles. Each element in this list starts by the name of the loaded modulefile declaring the conflict followed by the name of all modulefiles it declares a conflict with. These loaded modulefiles and conflicting modulefile names are separated by the ampersand character.

This environment variable is intended for module command internal use to get knowledge of the conflicts declared by the loaded modulefiles in order to keep environment consistent when a conflicting module is asked for load afterward.

New in version 4.2.

MODULES_LMNOTUASKED

A colon separated list of all loaded modulefiles that were not explicitly asked for load from the command-line.

This environment variable is intended for module command internal use to distinguish the modulefiles that have been loaded automatically from modulefiles that have been asked by users.

New in version 4.2.

MODULES_LMPREREQ

A colon separated list of the prereq statements defined by all loaded modulefiles. Each element in this list starts by the name of the loaded modulefile declaring the pre-requirement followed by the name of all modulefiles it declares a prereq with. These loaded modulefiles and pre-required modulefile names are separated by the ampersand character. When a prereq statement is composed of multiple modulefiles, these modulefile names are separated by the pipe character.

This environment variable is intended for module command internal use to get knowledge of the pre-requirement declared by the loaded modulefiles in order to keep environment consistent when a pre-required module is asked for unload afterward.

New in version 4.2.

MODULES_LMSOURCESH

A colon separated list of the source-sh statements defined by all loaded modulefiles. Each element in this list starts by the name of the loaded modulefile declaring the environment changes made by the evaluation of source-sh scripts. This name is followed by each source-sh statement call and corresponding result achieved in modulefile. The loaded modulefile name and each source-sh statement description are separated by the ampersand character. The source-sh statement call and each resulting modulefile command (corresponding to the environment changes done by sourced script) are separated by the pipe character.

This environment variable is intended for module command internal use to get knowledge of the modulefile commands applied for each source-sh command when loading the modulefile. In order to reverse these modulefile commands when modulefile is unloaded to undo the environment changes.

New in version 4.6.

MODULES_LMTAG

A colon separated list of the tags corresponding to all loaded modulefiles that have been set through module-tag statements or from other modulefile statements like module-forbid (that may apply the nearly-forbidden tag in specific situation) (see Module tags section). Each element in this list starts by the name of the loaded modulefile followed by all tags applying to it. The loaded modulefile and its tags are separated by the ampersand character.

This environment variable is intended for module command internal use to get knowledge of the tags applying to loaded modulefiles in order to report these tags on subcmd:list sub-command output or to apply specific behavior when unloading modulefile.

New in version 4.7.

MODULES_LMVARIANT

A colon separated list of the variant instantiated through variant statements by all loaded modulefiles (see Module variants section). Each element in this list starts by the name of the loaded modulefile followed by all the variant definitions set during the load of this module. The loaded modulefile and each of its variant definition are separated by the ampersand character. Each variant definition starts with the variant name, followed by the variant value set, then a flag to know if variant is of the Boolean type and last element in this definition is a flag to know if the chosen value is the default one for this variant and if it has been automatically set or not. These four elements composing the variant definition are separated by the pipe character.

This environment variable is intended for module command internal use to get knowledge of the variant value defined by the loaded modulefiles in order to keep environment consistent when requirements are set over a specific variant value or just to report these variant values when listing loaded modules.

New in version 4.8.

MODULES_MCOOKIE_VERSION_CHECK

If set to 1, the version set in the Modules magic cookie in modulefile is checked against the current version of modulecmd.tcl to determine if the modulefile can be evaluated.

New in version 4.7.

MODULES_ML

If set to 1, define ml command when initializing Modules (see Package Initialization section). If set to 0, ml command is not defined.

ml command enablement is defined in the following order of preference: MODULES_ML environment variable then the default set in modulecmd.tcl script configuration. Which means MODULES_ML overrides default configuration.

New in version 4.5.

MODULES_NEARLY_FORBIDDEN_DAYS

Number of days a module is considered nearly forbidden prior reaching its expiry date set by module-forbid modulefile command. When a nearly forbidden module is evaluated a warning message is issued to inform module will soon be forbidden. If set to 0, modules will never be considered nearly forbidden. Accepted values are integers comprised between 0 and 365.

This configuration is defined in the following order of preference: MODULES_NEARLY_FORBIDDEN_DAYS environment variable then the default set in modulecmd.tcl script configuration. Which means MODULES_NEARLY_FORBIDDEN_DAYS overrides default configuration.

New in version 4.6.

MODULES_PAGER

Text viewer for use to paginate message output if error output stream is attached to a terminal. The value of this variable is composed of a pager command name or path eventually followed by command-line options.

Paging command and options are defined for Modules in the following order of preference: MODULES_PAGER environment variable, then the default set in modulecmd.tcl script configuration. Which means MODULES_PAGER overrides default configuration.

If MODULES_PAGER variable is set to an empty string or to the value cat, pager will not be launched.

New in version 4.1.

MODULES_RUN_QUARANTINE

A space separated list of environment variable names that should be passed indirectly to modulecmd.tcl to protect its run-time environment from side-effect coming from their current definition.

Each variable found in MODULES_RUN_QUARANTINE will have its value emptied or set to the value of the corresponding MODULES_RUNENV_<VAR> variable when defining modulecmd.tcl run-time environment.

Original values of these environment variables set in quarantine are passed to modulecmd.tcl via <VAR>_modquar variables.

New in version 4.1.

MODULES_RUNENV_<VAR>

Value to set to environment variable <VAR> for modulecmd.tcl run-time execution if <VAR> is referred in MODULES_RUN_QUARANTINE.

New in version 4.1.

MODULES_SEARCH_MATCH

When searching for modules with avail sub-command, defines the way query string should match against available module names. With starts_with value, returned modules are those whose name begins by search query string. When set to contains, any modules whose fully qualified name contains search query string are returned.

Module search match style is defined in the following order of preference: --starts-with and --contains command line switches, then MODULES_SEARCH_MATCH environment variable, then the default set in modulecmd.tcl script configuration. Which means MODULES_SEARCH_MATCH overrides default configuration and --starts-with/--contains command line switches override every other ways to set search match style.

New in version 4.3.

MODULES_SET_SHELL_STARTUP

If set to 1, defines when module command initializes the shell startup file to ensure that the module command is still defined in sub-shells. Setting shell startup file means defining the ENV and BASH_ENV environment variable to the Modules bourne shell initialization script. If set to 0, shell startup file is not defined.

New in version 4.3.

MODULES_SHELLS_WITH_KSH_FPATH

A list of shell on which the FPATH environment variable should be defined at initialization time to point to the ksh-functions directory where the ksh initialization script for module command is located. It enables for the listed shells to get module function defined when starting ksh as sub-shell from there.

Accepted values are a list of shell among sh, bash, csh, tcsh and fish separated by colon character (:).

New in version 4.7.

MODULES_SILENT_SHELL_DEBUG

If set to 1, disable any xtrace or verbose debugging property set on current shell session for the duration of either the module command or the module shell initialization script. Only applies to Bourne Shell (sh) and its derivatives.

New in version 4.1.

MODULES_SITECONFIG

Location of a site-specific configuration script to source into modulecmd.tcl. See also Modulecmd startup section.

This environment variable is ignored if extra_siteconfig has been declared locked in locked_configs configuration option.

New in version 4.3.

MODULES_TAG_ABBREV

Specifies the abbreviation strings used to report module tags (see Module tags section). Its value is a colon-separated list of module tag names associated to an abbreviation string (e.g. tagname=abbrev).

If a tag is associated to an empty string abbreviation, this tag will not be reported. In case the whole MODULES_TAG_ABBREV environment variable is set to an empty string, tags are reported but not abbreviated.

The tag abbreviation definition set in MODULES_TAG_ABBREV environment variable supersedes the default configuration set in modulecmd.tcl script.

New in version 4.7.

MODULES_TAG_COLOR_NAME

Specifies the tag names or abbreviations whose graphical rendering should be applied over themselves instead of being applied over the name of the module they are attached to. Value of MODULES_TAG_COLOR_NAME is a colon-separated list of module tag names or abbreviation strings (see Module tags section).

When a select graphic rendition is defined for a tag name or a tag abbreviation string, it is applied over the module name associated with the tag and tag name or abbreviation is not displayed. When listed in MODULES_TAG_COLOR_NAME environment variable, a tag name or abbreviation is displayed and select graphic rendition is applied over it.

The definition set in MODULES_TAG_COLOR_NAME environment variable supersedes the default configuration set in modulecmd.tcl script.

New in version 4.7.

MODULES_TERM_BACKGROUND

Inform Modules of the terminal background color to determine if the color set for dark background or the color set for light background should be used to color output in case no specific color set is defined with the MODULES_COLORS variable. Accepted values are dark and light.

New in version 4.3.

MODULES_TERM_WIDTH

Specifies the number of columns of the output. If set to 0, the output width will be the full terminal width, which is automatically detected by the module command. Accepted values are integers comprised between 0 and 1000.

This configuration is defined in the following order of preference: --width or -w command line switches, then MODULES_TERM_WIDTH environment variable, then the default set in modulecmd.tcl script configuration. Which means MODULES_TERM_WIDTH overrides default configuration. --width or -w command line switches override every other configuration.

New in version 4.7.

MODULES_UNLOAD_MATCH_ORDER

When a module unload request matches multiple loaded modules, unload firstly loaded module or lastly loaded module. Accepted values are returnfirst and returnlast.

New in version 4.3.

MODULES_USE_COMPAT_VERSION

If set to 1 prior to Modules package initialization, enable Modules compatibility version (3.2 release branch) rather main version at initialization scripts running time. Modules package compatibility version should be installed along with main version for this environment variable to have any effect.

New in version 4.0.

MODULES_VARIANT_SHORTCUT

Specifies the shortcut characters that could be used to specify and report module variants (see Module variants section). Its value is a colon-separated list of variant names associated to a shortcut character (e.g., variantname=shortcutchar).

A variant shortcut must be of one character length and must avoid characters used for other concerns or in module names (i.e., [-+~/@=a-zA-Z0-9]).

If a shortcut is associated to an empty string or an invalid character, this shortcut definition will be ignored.

The variant shortcut definition set in MODULES_VARIANT_SHORTCUT environment variable supersedes the default configuration set in modulecmd.tcl script.

New in version 4.8.

MODULES_VERBOSITY

Defines the verbosity level of the module command. Available verbosity levels from the least to the most verbose are:

  • silent: turn off error, warning and informational messages but does not affect module command output result.
  • concise: enable error and warning messages but disable informational messages.
  • normal: turn on informational messages, like a report of the additional module evaluations triggered by loading or unloading modules, aborted evaluation issues or a report of each module evaluation occurring during a restore or source sub-commands.
  • verbose: add additional informational messages, like a systematic report of the loading or unloading module evaluations.
  • verbose2: report loading or unloading module evaluations of hidden-loaded modules, report if loading module is already loaded or if unloading module is not loaded.
  • trace: provide details on module searches, resolutions, selections and evaluations.
  • debug: print debugging messages about module command execution.
  • debug2: report modulecmd.tcl procedure calls in addition to printing debug messages.

Module command verbosity is defined in the following order of preference: --silent, --verbose, --debug and --trace command line switches, then MODULES_VERBOSITY environment variable, then the default set in modulecmd.tcl script configuration. Which means MODULES_VERBOSITY overrides default configuration and --silent/--verbose/--debug/--trace command line switches overrides every other ways to set verbosity level.

New in version 4.3.

Changed in version 4.6: Verbosity levels trace and debug2 added

Changed in version 4.7: Verbosity level verbose2 added

MODULES_WA_277

If set to 1 prior to Modules package initialization, enables workaround for Tcsh history issue (see https://github.com/cea-hpc/modules/issues/277). This issue leads to erroneous history entries under Tcsh shell. When workaround is enabled, an alternative module alias is defined which fixes the history mechanism issue. However the alternative definition of the module alias weakens shell evaluation of the code produced by modulefiles. Characters with a special meaning for Tcsh shell (like { and }) may not be used anymore in shell alias definition otherwise the evaluation of the code produced by modulefiles will return a syntax error.

New in version 4.3.

MODULESHOME

The location of the main Modules package file directory containing module command initialization scripts, the executable program modulecmd.tcl, and a directory containing a collection of main modulefiles.

<VAR>_modquar

Value of environment variable <VAR> passed to modulecmd.tcl in order to restore <VAR> to this value once started.

New in version 4.1.

<VAR>_modshare

Reference counter variable for path-like variable <VAR>. A colon separated list containing pairs of elements. A pair is formed by a path element followed its usage counter which represents the number of times this path has been enabled in variable <VAR>. A colon separates the two parts of the pair.

New in version 4.0.

FILES

/usr/share/Modules

The MODULESHOME directory.

/etc/environment-modules/siteconfig.tcl

The site-specific configuration script of modulecmd.tcl. An additional configuration script could be defined using the MODULES_SITECONFIG environment variable.

/etc/environment-modules/rc

The system-wide modules rc file. The location of this file can be changed using the MODULERCFILE environment variable as described above.

$HOME/.modulerc

The user specific modules rc file.

$HOME/.module

The user specific collection directory.

/usr/share/Modules/modulefiles

The directory for system-wide modulefiles. The location of the directory can be changed using the MODULEPATH environment variable as described above.

/usr/share/Modules/libexec/modulecmd.tcl

The modulefile interpreter that gets executed upon each invocation of module.

/usr/share/Modules/init/<shell>

The Modules package initialization file sourced into the user's environment.

SEE ALSO

ml, modulefile

modulefile

DESCRIPTION

modulefiles are written in the Tool Command Language, Tcl(n) and are interpreted by the modulecmd.tcl program via the module user interface. modulefiles can be loaded, unloaded, or switched on-the-fly while the user is working; and can be used to implement site policies regarding the access and use of applications.

A modulefile begins with the magic cookie, #%Module. A version number may be placed after this string. The version number is useful as the modulefile format may change thus it reflects the minimum version of modulecmd.tcl required to interpret the modulefile. If a version number doesn't exist, then modulecmd.tcl will assume the modulefile is compatible. Files without the magic cookie or with a version number greater than the current version of modulecmd.tcl will not be interpreted. If the mcookie_version_check configuration is disabled the version number set is not checked.

Each modulefile contains the changes to a user's environment needed to access an application. Tcl is a simple programming language which permits modulefiles to be arbitrarily complex, depending upon the application's and the modulefile writer's needs. If support for extended tcl (tclX) has been configured for your installation of the Modules package, you may use all the extended commands provided by tclX, too.

A typical modulefile is a simple bit of code that set or add entries to the PATH, MANPATH, or other environment variables. A Modulefile is evaluated against current modulecmd.tcl's mode which leads to specific evaluation results. For instance if the modulefile sets a value to an environment variable, this variable is set when modulefile is loaded and unset when modulefile is unloaded.

Tcl has conditional statements that are evaluated when the modulefile is interpreted. This is very effective for managing path or environment changes due to different OS releases or architectures. The user environment information is encapsulated into a single modulefile kept in a central location. The same modulefile is used by every user on any machine. So, from the user's perspective, starting an application is exactly the same irrespective of the machine or platform they are on.

modulefiles also hide the notion of different types of shells. From the user's perspective, changing the environment for one shell looks exactly the same as changing the environment for another shell. This is useful for new or novice users and eliminates the need for statements such as "if you're using the C Shell do this ..., otherwise if you're using the Bourne shell do this ...". Announcing and accessing new software is uniform and independent of the user's shell. From the modulefile writer's perspective, this means one set of information will take care of every type of shell.

Modules Specific Tcl Commands

The Modules Package uses commands which are extensions to the "standard" Tool Command Language Tcl(n) package. Unless otherwise specified, the Module commands return the empty string. Some commands behave differently when a modulefile is loaded or unloaded. The command descriptions assume the modulefile is being loaded.

append-path [-d C|--delim C|--delim=C] [--duplicates] variable value...

See prepend-path.

break

This is not a Modules-specific command, it's actually part of Tcl, which has been overloaded similar to the continue and exit commands to have the effect of causing the module not to be listed as loaded and not affect other modules being loaded concurrently. All non-environment commands within the module will be performed up to this point and processing will continue on to the next module on the command line. The break command will only have this effect if not used within a Tcl loop though.

An example: Suppose that a full selection of modulefiles are needed for various different architectures, but some of the modulefiles are not needed and the user should be alerted. Having the unnecessary modulefile be a link to the following notavail modulefile will perform the task as required.

#%Module1.0
## notavail modulefile
##
proc ModulesHelp { } {
    puts stderr "This module does nothing but alert the user"
    puts stderr "that the [module-info name] module is not available"
}

module-whatis "Notifies user that module is not available."
set curMod [module-info name]
if { [ module-info mode load ] } {
    puts stderr "Note: '$curMod' is not available for [uname sysname]."
}
break
chdir directory

Set the current working directory to directory.

conflict modulefile...

prereq and conflict control whether or not the modulefile will be loaded. The prereq command lists modulefiles which must have been previously loaded before the current modulefile will be loaded. Similarly, the conflict command lists modulefiles which conflict with the current modulefile. If a list contains more than one modulefile, then each member of the list acts as a Boolean OR operation. Multiple prereq and conflict commands may be used to create a Boolean AND operation. If one of the requirements have not been satisfied, an error is reported and the current modulefile makes no changes to the user's environment.

If an argument for prereq is a directory and any modulefile from the directory has been loaded, then the prerequisite is met. For example, specifying X11 as a prereq means that any version of X11, X11/R4 or X11/R5, must be loaded before proceeding.

If an argument for conflict is a directory and any other modulefile from that directory has been loaded, then a conflict will occur. For example, specifying X11 as a conflict will stop X11/R4 and X11/R5 from being loaded at the same time.

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

continue

This is not a modules specific command but another overloaded Tcl command and is similar to the break or exit commands except the module will be listed as loaded as well as performing any environment or Tcl commands up to this point and then continuing on to the next module on the command line. The continue command will only have this effect if not used within a Tcl loop though.

exit [N]

This is not a modules specific command but another overloaded Tcl command and is similar to the break or continue commands. However, this command will cause the immediate cessation of this module and any additional ones on the command line. This module and the subsequent modules will not be listed as loaded. No environment commands will be performed in the current module.

getenv variable [value]

Returns value of environment variable. If variable is not defined, value is returned if set, _UNDEFINED_ is returned otherwise. The getenv command should be preferred over the Tcl global variable env to query environment variables.

When modulefile is evaluated in display mode, getenv returns variable name prefixed with dollar sign (e.g., $variable).

New in version 4.0.

getvariant variant [value]

Returns value of designated variant. If variant is not defined, value is returned if set, an empty string is returned otherwise. The getvariant command should be preferred over the ModuleVariant Tcl array to query a variant value.

When modulefile is evaluated in display mode, getvariant returns variant name enclosed in curly braces (e.g., {variant}).

New in version 4.8.

is-avail modulefile...

The is-avail command returns a true value if any of the listed modulefiles exists in enabled MODULEPATH. If a list contains more than one modulefile, then each member acts as a boolean OR operation. If an argument for is-avail is a directory and a modulefile exists in the directory is-avail would return a true value.

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

New in version 4.1.

is-loaded [modulefile...]

The is-loaded command returns a true value if any of the listed modulefiles has been loaded or if any modulefile is loaded in case no argument is provided. If a list contains more than one modulefile, then each member acts as a boolean OR operation. If an argument for is-loaded is a directory and any modulefile from the directory has been loaded is-loaded would return a true value.

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

is-saved [collection...]

The is-saved command returns a true value if any of the listed collections exists or if any collection exists in case no argument is provided. If a list contains more than one collection, then each member acts as a boolean OR operation.

If MODULES_COLLECTION_TARGET is set, a suffix equivalent to the value of this variable is appended to the passed collection name. In case no collection argument is provided, a true value will only be returned if a collection matching currently set target exists.

New in version 4.1.

is-used [directory...]

The is-used command returns a true value if any of the listed directories has been enabled in MODULEPATH or if any directory is enabled in case no argument is provided. If a list contains more than one directory, then each member acts as a boolean OR operation.

New in version 4.1.

module [sub-command] [sub-command-options] [sub-command-args]

Contains the same sub-commands as described in the module man page in the Module Sub-Commands section. This command permits a modulefile to load or unload other modulefiles. No checks are made to ensure that the modulefile does not try to load itself. Often it is useful to have a single modulefile that performs a number of module load commands. For example, if every user on the system requires a basic set of applications loaded, then a core modulefile would contain the necessary module load commands.

The --not-req option may be set for the load, unload and switch sub-commands to inhibit the definition of an implicit prereq or conflict requirement onto specified modules.

On try-load sub-command, if specified modulefile is not found thus loaded, no implicit prereq requirement is defined over this module.

Command line switches --auto, --no-auto and --force are ignored when passed to a module command set in a modulefile.

module-alias name modulefile

Assigns the modulefile to the alias name. This command should be placed in one of the modulecmd.tcl rc files in order to provide shorthand invocations of frequently used modulefile names.

The parameter modulefile may be either

  • a fully qualified modulefile with name and version
  • a symbolic modulefile name
  • another modulefile alias
module-forbid [options] modulefile...

Forbid use of modulefile. An error is obtained when trying to evaluate a forbidden module. This command should be placed in one of the modulecmd.tcl rc files.

module-forbid command accepts the following options:

  • --after datetime
  • --before datetime
  • --not-user {user...}
  • --not-group {group...}
  • --message {text message}
  • --nearly-message {text message}

If --after option is set, forbidding is only effective after specified date time. Following the same principle, if --before option is set, forbidding is only effective before specified date time. Accepted date time format is YYYY-MM-DD[THH:MM]. If no time (HH:MM) is specified, 00:00 is assumed. --after and --before options are not supported on Tcl versions prior to 8.5.

If --not-user option is set, forbidding is not applied if the username of the user currently running modulecmd.tcl is part of the list of username specified. Following the same approach, if --not-group option is set, forbidding is not applied if current user is member of one the group specified. When both options are set, forbidding is not applied if a match is found for --not-user or --not-group.

Error message returned when trying to evaluate a forbidden module can be supplemented with the text message set through --message option.

If --after option is set, modules are considered nearly forbidden during a number of days defined by the nearly_forbidden_days modulecmd.tcl configuration option (see MODULES_NEARLY_FORBIDDEN_DAYS), prior reaching the expiry date fixed by --after option. When a nearly forbidden module is evaluated a warning message is issued to inform module will soon be forbidden. This warning message can be supplemented with the text message set through --nearly-message option.

If a module-forbid command applies to a modulefile also targeted by a module-hide --hard command, this module is unveiled when precisely named to return an access error.

Forbidden modules included in the result of an avail sub-command are reported with a forbidden tag applied to them. Nearly forbidden modules included in the result of an avail or a list sub-command are reported with a nearly-forbidden tag applied to them. See Module tags section in module.

The parameter modulefile may leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

New in version 4.6.

module-hide [options] modulefile...

Hide modulefile to exclude it from available module search or module selection unless query refers to modulefile by its exact name. This command should be placed in one of the modulecmd.tcl rc files.

module-hide command accepts the following options:

  • --soft|--hard
  • --hidden-loaded
  • --after datetime
  • --before datetime
  • --not-user {user...}
  • --not-group {group...}

When --soft option is set, modulefile is also set hidden, but hiding is disabled when search or selection query's root name matches module's root name. This soft hiding mode enables to hide modulefiles from bare module availability listing yet keeping the ability to select such module for load with the regular resolution mechanism (i.e., no need to use module exact name to select it)

When --hard option is set, modulefile is also set hidden and stays hidden even if search or selection query refers to modulefile by its exact name.

When --hidden-loaded option is set, hidden state also applies to the modulefile when it is loaded. Hidden loaded modules do not appear on list sub-command output, unless --all option is set. Their loading or unloading informational messages are not reported unless the verbosity of Modules is set to a level higher than verbose. Hidden loaded modules are detected in any cases by state query commands like is-loaded.

If --after option is set, hiding is only effective after specified date time. Following the same principle, if --before option is set, hiding is only effective before specified date time. Accepted date time format is YYYY-MM-DD[THH:MM]. If no time (HH:MM) is specified, 00:00 is assumed. --after and --before options are not supported on Tcl versions prior to 8.5.

If --not-user option is set, hiding is not applied if the username of the user currently running modulecmd.tcl is part of the list of username specified. Following the same approach, if --not-group option is set, hiding is not applied if current user is member of one the group specified. When both options are set, hiding is not applied if a match is found for --not-user or --not-group.

If the --all option is set on avail, aliases, whatis or search sub-commands, hiding is disabled thus hidden modulefiles are included in module search. Hard-hidden modules (i.e., declared hidden with --hard option) are not affected by --all and stay hidden even if option is set. --all option does not apply to module selection sub-commands like load. Thus in such context a hidden module should always be referred by its exact full name (e.g., foo/1.2.3 not foo) unless if it has been hidden in --soft mode. A hard-hidden module cannot be unveiled or selected in any case.

If several module-hide commands target the same modulefile, the strongest hiding level is retained which means if both a regular, a --soft hiding command match a given module, regular hiding mode is considered. If both a regular and a --hard hiding command match a given module, hard hiding mode is retained. A set --hidden-loaded option is retained even if the module-hide statement on which it is declared is superseded by a stronger module-hide statement with no --hidden-loaded option set.

Hidden modules included in the result of an avail sub-command are reported with a hidden tag applied to them. Hidden loaded modules included in the result of a list sub-command are reported with a hidden-loaded tag applied to them. This tag is not reported on avail sub-command context. See Module tags section in module.

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

New in version 4.6.

Changed in version 4.7: Option --hidden-loaded added.

module-info option [info-args]

Provide information about the modulecmd.tcl program's state. Some of the information is specific to the internals of modulecmd.tcl. option is the type of information to be provided, and info-args are any arguments needed.

module-info alias name

Returns the full modulefile name to which the modulefile alias name is assigned

module-info command [commandname]

Returns the currently running modulecmd.tcl's command as a string if no commandname is given.

Returns 1 if modulecmd.tcl's command is commandname. commandname can be: load, unload, reload, source, switch, display, avail, aliases, list, whatis, search, purge, restore, help or test.

New in version 4.0.

module-info loaded modulefile

Returns the names of currently loaded modules matching passed modulefile. The parameter modulefile might either be a fully qualified modulefile with name and version or just a directory which in case all loaded modulefiles from the directory will be returned. The parameter modulefile may also be a symbolic modulefile name or a modulefile alias.

This command only returns the name and version of designated loaded module. The defined variants of the loaded module are not included in the returned string.

New in version 4.1.

module-info mode [modetype]

Returns the current modulecmd.tcl's mode as a string if no modetype is given.

Returns 1 if modulecmd.tcl's mode is modetype. modetype can be: load, unload, remove, switch, display, help, test or whatis.

module-info name

Return the name of the modulefile. This is not the full pathname for modulefile. See the Modules Variables section for information on the full pathname.

This command only returns the name and version of currently evaluating modulefile. The defined variants are not included in the returned string. See getvariant command or ModuleVariant array variable to get defined variant values for currently evaluating modulefile.

module-info shell [shellname]

Return the current shell under which modulecmd.tcl was invoked if no shellname is given. The current shell is the first parameter of modulecmd.tcl, which is normally hidden by the module alias.

If a shellname is given, returns 1 if modulecmd.tcl's current shell is shellname, returns 0 otherwise. shellname can be: sh, bash, ksh, zsh, csh, tcsh, fish, tcl, perl, python, ruby, lisp, cmake, r.

module-info shelltype [shelltypename]

Return the family of the shell under which modulefile was invoked if no shelltypename is given. As of module-info shell this depends on the first parameter of modulecmd.tcl. The output reflects a shell type determining the shell syntax of the commands produced by modulecmd.tcl.

If a shelltypename is given, returns 1 if modulecmd.tcl's current shell type is shelltypename, returns 0 otherwise. shelltypename can be: sh, csh, fish, tcl, perl, python, ruby, lisp, cmake, r.

module-info specified

Return the module designation (name, version and variants) specified that led to current modulefile evaluation.

module-info symbols modulefile

Returns a list of all symbolic versions assigned to the passed modulefile. The parameter modulefile might either be a full qualified modulefile with name and version, another symbolic modulefile name or a modulefile alias.

module-info tags [tag]

Returns all tags assigned to currently evaluated modulefile as a list of strings if no tag name is given (see Module tags section in module)

When tags are assigned to specific module variants, they are returned only if this variant is the one currently evaluated.

Returns 1 if one of the tags applying to currently evaluated modulefile is tag. Returns 0 otherwise.

New in version 4.7.

module-info type

Returns either C or Tcl to indicate which module command is being executed, either the C version or the Tcl-only version, to allow the modulefile writer to handle any differences between the two.

module-info usergroups [name]

Returns all the groups the user currently running modulecmd.tcl is member of as a list of strings if no name is given.

Returns 1 if one of the group current user running modulecmd.tcl is member of is name. Returns 0 otherwise.

If the Modules Tcl extension library is disabled, the id(1) command is invoked to fetch groups of current user.

New in version 4.6.

module-info username [name]

Returns the username of the user currently running modulecmd.tcl as a string if no name is given.

Returns 1 if username of current user running modulecmd.tcl is name. Returns 0 otherwise.

If the Modules Tcl extension library is disabled, the id(1) command is invoked to fetch username of current user.

New in version 4.6.

module-info version modulefile

Returns the physical module name and version of the passed symbolic version modulefile. The parameter modulefile might either be a full qualified modulefile with name and version, another symbolic modulefile name or a modulefile alias.
module-tag [options] tag modulefile...

Associate tag to designated modulefile. This tag information will be reported along modulefile on avail and list sub-commands (see Module tags section in module). Tag information can be queried during modulefile evaluation with the module-info tags modulefile command. module-tag commands should be placed in one of the modulecmd.tcl rc files.

module-tag command accepts the following options:

  • --not-user {user...}
  • --not-group {group...}

If --not-user option is set, the tag is not applied if the username of the user currently running modulecmd.tcl is part of the list of username specified. Following the same approach, if --not-group option is set, the tag is not applied if current user is member of one the group specified. When both options are set, the tag is not applied if a match is found for --not-user or --not-group.

The parameter modulefile may also be a symbolic modulefile name or a modulefile alias. It may also leverage a specific syntax to finely select module version (see Advanced module version specifiers section below).

Tags inherited from other modulefile commands or module states cannot be set with module-tag. Otherwise an error is returned. Those special tags are: auto-loaded, forbidden, hidden, hidden-loaded, loaded and nearly-forbidden.

When tag equals sticky or super-sticky, designated modulefile are defined Sticky modules.

New in version 4.7.

module-version modulefile version-name...

Assigns the symbolic version-name to the modulefile. This command should be placed in one of the modulecmd.tcl rc files in order to provide shorthand invocations of frequently used modulefile names.

The special version-name default specifies the default version to be used for module commands, if no specific version is given. This replaces the definitions made in the .version file in former modulecmd.tcl releases.

The parameter modulefile may be either

  • a fully or partially qualified modulefile with name / version. If name is . (dot) then the current directory name is assumed to be the module name. (Use this for deep modulefile directories.)
  • a symbolic modulefile name
  • another modulefile alias
module-virtual name modulefile

Assigns the modulefile to the virtual module name. This command should be placed in rc files in order to define virtual modules.

A virtual module stands for a module name associated to a modulefile. The modulefile is the script interpreted when loading or unloading the virtual module which appears or can be found with its virtual name.

The parameter modulefile corresponds to the relative or absolute file location of a modulefile.

New in version 4.1.

module-whatis string

Defines a string which is displayed in case of the invocation of the module whatis command. There may be more than one module-whatis line in a modulefile. This command takes no actions in case of load, display, etc. invocations of modulecmd.tcl.

The string parameter has to be enclosed in double-quotes if there's more than one word specified. Words are defined to be separated by whitespace characters (space, tab, cr).

prepend-path [-d C|--delim C|--delim=C] [--duplicates] variable value...

Append or prepend value to environment variable. The variable is a colon, or delimiter, separated list such as PATH=directory:directory:directory. The default delimiter is a colon :, but an arbitrary one can be given by the --delim option. For example a space can be used instead (which will need to be handled in the Tcl specially by enclosing it in " " or { }). A space, however, can not be specified by the --delim=C form.

A reference counter environment variable is also set to increase the number of times value has been added to environment variable. This reference counter environment variable is named by suffixing variable by _modshare.

When value is already defined in environment variable, it is not added again except if --duplicates option is set.

If the variable is not set, it is created. When a modulefile is unloaded, append-path and prepend-path become remove-path.

If value corresponds to the concatenation of multiple elements separated by colon, or delimiter, character, each element is treated separately.

Changed in version 4.1: Option --duplicates added

prereq modulefile...

See conflict.

remove-path [-d C|--delim C|--delim=C] [--index] variable value...

Remove value from the colon, or delimiter, separated list in variable. See prepend-path or append-path for further explanation of using an arbitrary delimiter. Every string between colons, or delimiters, in variable is compared to value. If the two match, value is removed from variable if its reference counter is equal to 1 or unknown.

When --index option is set, value refers to an index in variable list. The string element pointed by this index is set for removal.

Reference counter of value in variable denotes the number of times value has been added to variable. This information is stored in environment variable_modshare. When attempting to remove value from variable, relative reference counter is checked and value is removed only if counter is equal to 1 or not defined. Otherwise value is kept in variable and reference counter is decreased by 1.

If value corresponds to the concatenation of multiple elements separated by colon, or delimiter, character, each element is treated separately.

Changed in version 4.1: Option --index added

set-alias alias-name alias-string

Sets an alias or function with the name alias-name in the user's environment to the string alias-string. For some shells, aliases are not possible and the command has no effect. When a modulefile is unloaded, set-alias becomes unset-alias.

set-function function-name function-string

Creates a function with the name function-name in the user's environment with the function body function-string. For some shells, functions are not possible and the command has no effect. When a modulefile is unloaded, set-function becomes unset-function.

New in version 4.2.

setenv variable value

Set environment variable to value. The setenv command will also change the process' environment. A reference using Tcl's env associative array will reference changes made with the setenv command. Changes made using Tcl's env associative array will NOT change the user's environment variable like the setenv command. An environment change made this way will only affect the module parsing process. The setenv command is also useful for changing the environment prior to the exec or system command. When a modulefile is unloaded, setenv becomes unsetenv. If the environment variable had been defined it will be overwritten while loading the modulefile. A subsequent unload will unset the environment variable - the previous value cannot be restored! (Unless you handle it explicitly ... see below.)

source-sh shell script [arg...]

Evaluate with shell the designated script with defined arguments to find out the environment changes it does. Those changes obtained by comparing environment prior and after script evaluation are then translated into corresponding modulefile commands, which are then applied during modulefile evaluation as if they were directly written in it.

When modulefile is unloaded, environment changes done are reserved by evaluating in the unload context the resulting modulefile commands, which were recorded in the MODULES_LMSOURCESH environment variable at load time.

Changes on environment variables, shell aliases, shell functions and current working directory are tracked.

Shell could be specified as a command name or a fully qualified pathname. The following shells are supported: sh, dash, csh, tcsh, bash, ksh, ksh93, zsh and fish.

New in version 4.6.

system string

Run string command through shell. On Unix, command is passed to the /bin/sh shell whereas on Windows it is passed to cmd.exe. modulecmd.tcl redirects stdout to stderr since stdout would be parsed by the evaluating shell. The exit status of the executed command is returned.

uname field

Provide lookup of system information. Most field information are retrieved from the tcl_platform array (see the tclvars(n) man page). Uname will return the string unknown if information is unavailable for the field.

uname will invoke the uname(1) command in order to get the operating system version and domainname(1) to figure out the name of the domain.

field values are:

  • sysname: the operating system name
  • nodename: the hostname
  • domain: the name of the domain
  • release: the operating system release
  • version: the operating system version
  • machine: a standard name that identifies the system's hardware
unset-alias alias-name

Unsets an alias with the name alias-name in the user's environment.

unset-function function-name

Removes a function with the name function-name from the user's environment.

New in version 4.2.

unsetenv variable [value]

Unsets environment variable. However, if there is an optional value, then when unloading a module, it will set variable to value. The unsetenv command changes the process' environment like setenv.

variant [--boolean] [--default value] name value...

Declare module variant name with list of accepted value and instantiate it in the ModuleVariant array variable.

Variant's value is selected through the module designation that leads to the modulefile evaluation. See Advanced module version specifiers section to learn how variants could be specified.

Selected variant value is transmitted to the evaluating modulefile. A value must be specified for variant name and it must corresponds to a value in the accepted value list. Otherwise an error is raised.

When the --default option is set, variant name is set to the value associated with this option in case no value is specified for variant in module designation.

If the --boolean option is set, variant name is defined as a Boolean variant. No list of accepted value should be defined in this case. All values recognized as Boolean value in Tcl are accepted (i.e., 1, true, t, yes, y, on, 0, false, f, no, n or off). Boolean variants are instantiated in ModuleVariant using Tcl canonical form of Boolean value (i.e., 0 or 1).

A variant which is not defined as a Boolean variant cannot define Boolean values in its accepted value list, exception made for the 0 and 1 integers. An error is raised otherwise.

A variant cannot be named version. An error is raised otherwise.

New in version 4.8.

versioncmp version1 version2

Compare version string version1 against version string version2. Returns -1, 0 or 1 respectively if version1 is less than, equal to or greater than version2.

New in version 4.7.

x-resource [resource-string|filename]

Merge resources into the X11 resource database. The resources are used to control look and behavior of X11 applications. The command will attempt to read resources from filename. If the argument isn't a valid file name, then string will be interpreted as a resource. Either filename or resource-string is then passed down to be xrdb(1) command.

modulefiles that use this command, should in most cases contain one or more x-resource lines, each defining one X11 resource. The DISPLAY environment variable should be properly set and the X11 server should be accessible. If x-resource can't manipulate the X11 resource database, the modulefile will exit with an error message.

Examples:

x-resource /u2/staff/leif/.xres/Ileaf

The content of the Ileaf file is merged into the X11 resource database.

x-resource [glob ~/.xres/ileaf]

The Tcl glob function is used to have the modulefile read different resource files for different users.

x-resource {Ileaf.popup.saveUnder: True}

Merge the Ileaf resource into the X11 resource database.

Modules Variables

ModulesCurrentModulefile

The ModulesCurrentModulefile variable contains the full pathname of the modulefile being interpreted.

ModuleTool

The ModuleTool variable contains the name of the module implementation currently in use. The value of this variable is set to Modules for this implementation.

New in version 4.7.

ModuleToolVersion

The ModuleToolVersion variable contains the version of the module implementation currently in use. The value of this variable is set to 4.8.0 for this version of Modules.

New in version 4.7.

ModuleVariant

The ModuleVariant array variable contains an element entry for each defined variant associated to the value of this variant (e.g., the $ModuleVariant(foo) syntax corresponds to the value of variant foo if defined). A Tcl evaluation error is obtained when accessing an undefined variant in ModuleVariant array. Use preferably the getvariant command to retrieve a variant value when this variant state is not known.

The list of the currently defined variants can be retrieved with [array names ModuleVariant] Tcl code.

New in version 4.8.

Locating Modulefiles

Every directory in MODULEPATH is searched to find the modulefile. A directory in MODULEPATH can have an arbitrary number of sub-directories. If the user names a modulefile to be loaded which is actually a directory, the directory is opened and a search begins for an actual modulefile. First, modulecmd.tcl looks for a file with the name .modulerc in the directory. If this file exists, its contents will be evaluated as if it was a modulefile to be loaded. You may place module-version, module-alias and module-virtual commands inside this file.

Additionally, before seeking for .modulerc files in the module directory, the global modulerc file and the .modulerc file found at the root of the modulepath directory are sourced, too. If a named version default now exists for the modulefile to be loaded, the assigned modulefile now will be sourced. Otherwise the file .version is looked up in the module directory.

If the .version file exists, it is opened and interpreted as Tcl code and takes precedence over a .modulerc file in the same directory. If the Tcl variable ModulesVersion is set by the .version file, modulecmd.tcl will use the name as if it specifies a modulefile in this directory. This will become the default modulefile in this case. ModulesVersion cannot refer to a modulefile located in a different directory.

If ModulesVersion is a directory, the search begins anew down that directory. If the name does not match any files located in the current directory, the search continues through the remaining directories in MODULEPATH.

Every .version and .modulerc file found is interpreted as Tcl code. The difference is that .version only applies to the current directory, and the .modulerc applies to the current directory and all subdirectories. Changes made in these files will affect the subsequently interpreted modulefile.

If no default version may be figured out, an implicit default is selected when this behavior is enabled (see MODULES_IMPLICIT_DEFAULT in module). If disabled, module names should be fully qualified when no explicit default is defined for them, otherwise no default version is found and an error is returned. If enabled, then the highest numerically sorted modulefile, virtual module or module alias under the directory will be used. The dictionary comparison method of the lsort(n) Tcl command is used to achieve this sort. If highest numerically sorted element is an alias, search continues on its modulefile target.

For example, it is possible for a user to have a directory named X11 which simply contains a .version file specifying which version of X11 is to be loaded. Such a file would look like:

#%Module1.0
##
##  The desired version of X11
##
set ModulesVersion "R4"

The equivalent .modulerc would look like:

#%Module1.0
##
##  The desired version of X11
##
module-version "./R4" default

If the extended default mechanism is enabled (see MODULES_EXTENDED_DEFAULT in module) the module version specified is matched against starting portion of existing module versions, where portion is a substring separated from the rest of version string by a . character.

When the implicit default mechanism and the Advanced module version specifiers are both enabled, a default and latest symbolic versions are automatically defined for each module name (also at each directory level in case of deep modulefile). Unless a symbolic version, alias, or regular module version already exists for these version names.

If user names a modulefile that cannot be found in the first modulepath directory, modulefile will be searched in next modulepath directory and so on until a matching modulefile is found. If search goes through a module alias or a symbolic version, this alias or symbol is resolved by first looking at the modulefiles in the modulepath where this alias or symbol is defined. If not found, resolution looks at the other modulepaths in their definition order.

When locating modulefiles, if a .modulerc, a .version, a directory or a modulefile cannot be read during the search it is simply ignored with no error message produced. Visibility of modulefiles can thus be adapted to the rights the user has been granted. Exception is made when trying to directly access a directory or a modulefile. In this case, the access issue is returned as an error message.

Depending on their name, their file permissions or the use of specific modulefile commands, modulefile, virtual module, module alias or symbolic version may be set hidden which impacts available modules search or module selection processes (see Hiding modulefiles section below).

Hiding modulefiles

A modulefile, virtual module, module alias or symbolic version whose name or element in their name starts with a dot character (.) or who are targeted by a module-hide command are considered hidden. Hidden modules are not displayed or taken into account except if they are explicitly named (e.g., foo/1.2.3 or foo/.2.0 not foo). If module has been hidden with the --soft option of the module-hide command set, it is not considered hidden if the root name of the query to search it matches module root name (e.g., searching foo will return a foo/1.2.3 modulefile targeted by a module-hide --soft command). If module has been hidden with the --hard option of the module-hide command set, it is always considered hidden thus it is never displayed nor taken into account even if it is explicitly named.

A modulefile, virtual module, module alias or symbolic version who are targeted by a module-hide --hard command and a module-forbid command or whose file access permissions are restricted are considered hard-hidden and forbidden. Such modules are not displayed or taken into account. When explicitly named for evaluation selection, such modules are unveiled to return an access error.

A symbolic version-name assigned to a hidden module is displayed or taken into account only if explicitly named and if module is not hard-hidden. Non-hidden module alias targeting a hidden modulefile appears like any other non-hidden module alias. Finally, a hidden symbolic version targeting a non-hidden module is displayed or taken into account only if not hard-hidden and explicitly named to refer to its non-hidden target.

The automatic version symbols (e.g., default and latest) are unaffected by hiding. Moreover when a regular default or latest version is set hidden, the corresponding automatic version symbol takes the left spot. For instance, if foo/default which targets foo/1.2.3 is set hard-hidden, the default automatic version symbol will be set onto foo/2.1.3, the highest available version of foo.

When loading a modulefile or a virtual module targeted by a module-hide --hidden-loaded command, this module inherits the hidden-loaded tag. Hidden loaded modules are not reported among list sub-command results.

If the --all is set on avail, aliases, whatis or search sub-commands, hidden modules are taken into account in search. Hard-hidden modules are unaffected by this option.

If the --all is set on list sub-command, hidden loaded modules are included in result output.

Advanced module version specifiers

When the advanced module version specifiers mechanism is enabled (see MODULES_ADVANCED_VERSION_SPEC in module), the specification of modulefile passed on Modules specific Tcl commands changes. After the module name a version constraint and variants may be added.

Version specifiers

After the module name a version constraint prefixed by the @ character may be added. It could be directly appended to the module name or separated from it with a space character.

Constraints can be expressed to refine the selection of module version to:

  • a single version with the @version syntax, for instance foo@1.2.3 syntax will select module foo/1.2.3
  • a list of versions with the @version1,version2,... syntax, for instance foo@1.2.3,1.10 will match modules foo/1.2.3 and foo/1.10
  • a range of versions with the @version1:, @:version2 and @version1:version2 syntaxes, for instance foo@1.2: will select all versions of module foo greater than or equal to 1.2, foo@:1.3 will select all versions less than or equal to 1.3 and foo@1.2:1.3 matches all versions between 1.2 and 1.3 including 1.2 and 1.3 versions

Advanced specification of single version or list of versions may benefit from the activation of the extended default mechanism (see MODULES_EXTENDED_DEFAULT in module) to use an abbreviated notation like @1 to refer to more precise version numbers like 1.2.3. Range of versions on its side natively handles abbreviated versions.

In order to be specified in a range of versions or compared to a range of versions, the version major element should corresponds to a number. For instance 10a, 1.2.3, 1.foo are versions valid for range comparison whereas default or foo.2 versions are invalid for range comparison.

Range of versions can be specified in version list, for instance foo@:1.2,1.4:1.6,1.8:. Such specification helps to exclude specific versions, like versions 1.3 and 1.7 in previous example.

If the implicit default mechanism is also enabled (see MODULES_IMPLICIT_DEFAULT in module), a default and latest symbolic versions are automatically defined for each module name (also at each directory level for deep modulefiles). These automatic version symbols are defined unless a symbolic version, alias, or regular module version already exists for these default or latest version names. Using the mod@latest (or mod/latest) syntax ensures highest available version will be selected.

Variants

After the module name, variants can be specified. Module variants are alternative evaluation of the same modulefile. A variant is specified by associating a value to its name. This specification is then transmitted to the evaluating modulefile which instantiates the variant in the ModuleVariant array variable when reaching the variant modulefile command declaring this variant.

Variant can be specified with the name=value syntax where name is the declared variant name and value, the value this variant is set to when evaluating the modulefile.

Boolean variants can be specified with the +name syntax to set this variant on and with the -name or ~name syntaxes to set this variant off. The -name syntax is not supported on ml command as the minus sign already means to unload designated module. The ~name and +name syntaxes could also be defined appended to another specification word (e.g., the module name, version or another variant specification), whereas -name syntax must be the start of a new specification word.

Boolean variants may also be specified with the name=value syntax. value should be set to 1, true, t, yes, y or on to enable the variant or it should be set to 0, false, f, no, n or off to disable the variant.

Shortcuts may be used to abbreviate variant specification. The variant_shortcut configuration option associates shortcut character to variant name. With a shortcut defined, variant could be specified with the <shortcut>value syntax. For instance if character % is set as a shortcut for variant foo, the %value syntax is equivalent to the foo=value syntax.

Specific characters used in variant specification syntax cannot be used as part of the name of a module. These specific characters are +, ~, = and all characters set as variant shortcut. Exception is made for + character which could be set one or several consecutive times at the end of module name (e.g., name+ or name++).

New in version 4.4.

Changed in version 4.8: Use of version range is allowed in version list

Changed in version 4.8: Support for module variant added

Modulefile Specific Help

Users can request help about a specific modulefile through the module command. The modulefile can print helpful information or start help oriented programs by defining a ModulesHelp subroutine. The subroutine will be called when the module help modulefile command is used.

Modulefile Specific Test

Users can request test of a specific modulefile through the module command. The modulefile can perform some sanity checks on its definition or on its underlying programs by defining a ModulesTest subroutine. The subroutine will be called when the module test modulefile command is used. The subroutine should return 1 in case of success. If no or any other value is returned, test is considered failed.

Modulefile Display

The module display modulefile command will detail all changes that will be made to the environment. After displaying all of the environment changes modulecmd.tcl will call the ModulesDisplay subroutine. The ModulesDisplay subroutine is a good place to put additional descriptive information about the modulefile.

ENVIRONMENT

See the ENVIRONMENT section in the module man page.

SEE ALSO

module, ml, Tcl(n), TclX(n), id(1), xrdb(1), exec(n), uname(1), domainname(1), tclvars(n), lsort(n)

NOTES

Tcl was developed by John Ousterhout at the University of California at Berkeley.

TclX was developed by Karl Lehenbauer and Mark Diekhans.

Contributing

Thank you for considering contributing to Modules!

Support questions

Please use the modules-interest mailing list for questions. Do not use the issue tracker for this.

Asking for new features

Please submit your new feature wishes first to the modules-interest mailing list. Discussion will help to clarify your needs and sometimes the wanted feature may already be available.

Reporting issues

  • Describe what you expected to happen.
  • If possible, include a minimal, complete, and verifiable example to help us identify the issue.
  • Describe what actually happened. Run the module command in --debug mode and include all the debug output obtained in your report.
  • Provide the current configuration and state of your Modules installation by running the module config --dump-state command.
  • Provide the name and content of the modulefiles you try to manipulate.

Submitting patches

  • Whether your patch is supposed to solve a bug or add a new feature, please include tests. In case of a bug, explain clearly under which circumstances it happens and make sure the test fails without your patch.
  • If you are not yet familiar with the git command and GitHub, please read the don't be afraid to commit tutorial.
Start coding
Running the tests

Run the basic test suite with:

make test

This only runs the tests for the current environment. GitHub Actions and Cirrus CI will run the full suite when you submit your pull request.

Running test coverage

Generating a report of lines that do not have test coverage can indicate where to start contributing or what your tests should cover for the code changes you submit.

Run make test COVERAGE=y which will automatically setup the Nagelfar Tcl code coverage tool in your modules development directory. Then the full testsuite will be run in coverage mode and a modulecmd-test.tcl_m annotated script will be produced:

make test COVERAGE=y
# then open modulecmd-test.tcl_m and look for ';# Not covered' lines
Building the docs

Build the docs in the doc directory using Sphinx:

cd doc
make html

Open _build/html/index.html in your browser to view the docs.

Read more about Sphinx.

Coding conventions

  • Maximum line length is 78 characters

  • Use 3 spaces to indent code (do not use tab character)

  • Adopt Tcl minimal escaping style

  • Procedure names: lowerCameCase

  • Variable names: nocaseatall

  • Curly brace and square bracket placement:

    if {![isStateDefined already_report]} {
   setState already_report 1
}
Emacs settings for coding conventions

This is an example emacs configuration that adheres to the first two coding conventions. You may wish to add this to your .emacs or .emacs.d/ to modify your tcl-mode:

(add-hook 'tcl-mode-hook
   (lambda ()
     (setq indent-tabs-mode nil)
     (setq tcl-indent-level 3)
     (setq tcl-continued-indent-level 3)
     (font-lock-add-keywords nil '(("^[^\n]\\{79\\}\\(.*\\)$" 1
                                    font-lock-warning-face prepend)))))

Submitting installation recipes

  • If you want to share your installation tips and tricks, efficient ways you have to write or organize your modulefiles or some extension you made to the module command please add a recipe to the cookbook section of the documentation.
  • Create a directory under doc/example and put there the extension code or example modulefiles your recipe is about.
  • Describe this recipe through a reStructuredText document in doc/source/cookbook. It is suggested to have an Implementation, an Installation and an Usage example sections in that document to get as much as possible the same document layout across recipes.
  • Submit a patch with all the above content.

Design notes

Developer notes on feature specifications and code design.

Advanced module version specifiers

Configuration

  • Introduce advanced_version_spec option name

    • off by default in v4 as previously soft@1 could be a module name

    • on by default in v5

    • in case extended_default is disabled

      • means short-hand notation cannot be used

        • for soft/1.1 query soft@1 returns nothing

    • in case implicit_default is disabled

      • means a default should be found over version range or list in selection context
Specification

  • Following Spack spec

  • same grammar used whatever the context

    • command-line or as argument to modulefile command (like command)

  • versions are specified whether

    • as specific words (separated by " ")
    • or as suffix to module name

  • change command specifications which were previously accepting list of modules

    • like module1 module2/vers module3

    • now these modules could express versions appended to their name with @

      • like module1@1.8 module2@vers module3

    • or these versions could be defined as words next to module name

      • like module1@1.8 module2 @vers module3

    • as a consequence, it denies use of @ in module names

    • such change requires an option to be enabled to avoid breaking compat

  • single version could be specified with soft@vers

    • which matches soft/vers modulefile

  • version could be specified as range

    • soft@:vers or soft@vers: or soft@vers1:vers2

    • Tcl-dictionarily determine what is between specified range

    • extended_default is always considered on when matching range

      • as 2.10 is included in @1:3 whatever the configuration

    • to be specified in a range or compared to a range, version major element should match an hexadecimal number

      • which also means be only composed by [0-9af] characters
      • for instance 10a, 1.2.3, 1.foo, 10.2.good are versions valid for range comparison
      • but 10g, default, foo.2, .1.3.4 are versions invalid for range comparison

    • a version range using in its definition version invalid for range comparison raises error

      • for instance @bar:foo

    • existing module versions invalid for range comparison are ignored

      • which means versions 10g, default, .1.13.4 or new are excluded from result for a @1.10: range query

    • when range is defined as @major:major.minor, version matching major version but above major.minor are excluded

      • for instance @1:1.10 will matches 1.0 and 1.8 but not 1.12

  • version could be specified as list

    • soft@vers,vers,vers

    • version specified could be text, like if symbolic version names are used

    • should benefit from extended default specification

      • to just express version with their major release number for instance

    • an empty string among list is considered as a specification error

      • for instance soft@vers,vers, or soft@vers,,vers

    • any version in list can be a single version or a version range

      • like soft@1.2,1.4:1.6,1.8
      • helps to designate all versions except a few ones

  • when using extended default syntax

    • version selection is performed same way for @vers than for /vers
    • described in extended default design

  • when icase is enabled for selection context and multiple directories match module name

    • for instance query is ICase@1.1,1.2,1.4 and following modules exist: ICASE/1.1, icase/1.2, iCaSe/1.3 and iCaSe/1.4
    • as no ICase directory exists, and a version in highest directory icase matches query (1.2), icase/1.2 is returned
    • if query is iCaSe@1.1,1.2,1.4, iCaSe/1.4 will be selected as iCaSe directory matches query module name
    • if query is ICase@1.1,1.4 or icase@1.1,1.4, as no version match in highest directory iCaSe/1.4 will be selected

  • in case of deep modulefiles

    • specified version is matched at the level directly under specified module name

      • not below levels

      • for instance soft@vers, will match soft/vers, not soft/deep/vers

      • to specify version for deep modules:"soft/deep@vers

      • to ease version comparison deep version cannot be specified after the @ character like soft@deep/vers

        • such specification will raise an error

  • advanced version specifier cannot be used with full path modulefile

    • when a full path modulefile is specified any advanced version set afterward is treated literally
    • for instance /path/to/modulefiles/mymod@1.2 will lead to the access of file mymod@1.2 in directory /path/to/modulefiles

  • in case version is specified multiple times

    • lastly mentioned (read from left to right) value is retained (it overwrite previous values)

    • like module@1.8 @2.0 or module@1.8@2.0

    • beware of version specified over a fully qualified modulefile like in soft/1.8@1.10" or "soft/1.8 @1.10

      • it resolves to soft/1.8/1.10 as advanced version specified is treated as an additional directory level

  • in case modulefile is named module@vers in filesystem

    • it is not found when option advanced_version_spec is enabled
    • as it is translated to module/vers

  • when special characters like ? or * are used in version name or value

    • they are evaluated as Tcl glob pattern on return all matching modules context

    • they are treated literally on single module selection and compatibility check context, no wildcard meaning is applied

    • like currently done when specifying module version on command-line

      • which leads to errors as no corresponding module is found:

        $ module load loc_dv6/*
ERROR: Unable to locate a modulefile for 'loc_dv6/*'

  • if version range or list does not contain a defined default

    • in a selection context

      • highest version is returned if implicit_default is enabled

      • error returned if implicit_default is disabled

        • even if version range or list specifies non-existent modules and only one existent module

    • in a compatibility expression context

      • range or list is matched against loaded environment whether the implicit_default state
      • when no match found and evaluation are triggered, selection context applies

  • when version is specified over an alias

    • should proceed like for real modulefile

    • when alias equal to a bare module with no version

      • foo is alias on bar modulefile, bar is a file (not a dir with version modulefiles)
      • query alias@:2 should behave like query alias/2

    • when alias equal to a module/version modulefile

      • foo is alias on bar/3 modulefile
      • query alias@:2 should behave like query alias/2

  • Contexts where it could be used

Note

Advanced version specifier does not apply for the moment to the Module identification to select one module context. Adding support for this context will require a significant rework on module alias and symbolic version registering and resolving code.

  • impact of advanced version specifier implementation over code

    • question especially over auto_handling code like conflict and prereq handling

    • it should not impact triggers and actions

    • but consist in an overall change of procedures comparing queries against loaded environment

      • procedures like doesModuleConflict

    • also adapting getModules to restrict version possibilities to what has been specified

      • for instance with query soft@1,2 should only return versions matching

  • prereq/conflict persistency

    • LMPREREQ and LMCONFLICT content should reflect specified version constraint

    • it could be expressed in these variables somewhat like it is specified to the

      • prereq/conflict modulefile commands

      • for instance MODULES_LMPREREQ=soft/1.10&bar@1.8,1.10&foo@<2|foo@3<4

      • delimiters characters are :, & and |

        • so use of characters * , *@, , is not an issue
        • but for : which express version ranges it should be substituted to <

  • prereq/conflict specification

    • could consolidate different version set for same module on the same prereq/conflict list

      • to indicate a preferred order (if available)
      • like prereq foo@1.8 foo@1.10
      • or prereq foo @1.8 foo@1.10

    • also to trigger alternative requirement resolution in case first one failed

      • as each module version specification leads to one evaluation only

        • even if multiple modulefiles correspond to this specification

          • like prereq soft@1.8,1.9,1.10 will lead to soft/1.10 load

        • best candidate is chosen from matches

          • in case implicit_default is disabled an explicit default should be part of the list or range for the triggered evaluation to succeed

      • whereas prereq soft@1.8 soft@1.9 soft@1.10 will lead to a tentative load

        • of soft/1.8, then soft/1.9 if it failed then soft/1.8 if it also failed

    • one module version specification may match multiple loaded modules

      • like conflict soft@1.8,1.9,1.10 matches loaded modules soft/1.8 and soft/1.10
      • similar to situations where requirement or conflict is expressed over module generic name, like soft, and multiple versions of module are loaded
Corner cases

  • When icase is enabled on all contexts and multiple directories match same icase module name

    • for instance following modules exist: ICASE/1.1, icase/1.2, iCaSe/1.3 and iCaSe/1.4

    • a module avail -i icase will sort iCaSe/1.4 as the highest entry

    • however a module load -i icase@1.1,1.2,1.4 command will load icase/1.2

      • as icase directory matches query and version 1.2 is found in icase directory

    • but a module load -i icase@1.1,1.4 command will load iCaSe/1.4

      • as no version 1.1 nor 1.4 is found in icase directory

Default and latest version specifiers

Configuration

  • Rely on advanced_version_spec option

    • No need for an extra configuration option

    • When option is on it automatically enables use of these two new version specifiers

    • When advanced_version_spec is off, mod@latest is considered as a modulefile name to find

    • in case implicit_default is disabled

      • means a default symbol or a latest symbol should be found defined to respectively make use of the @default or @latest specifiers
      • a not found error is otherwise raised, as specified version does not exist
Specification

  • When a default or a latest symbol is defined

    • @default or respectively @latest will resolve to the symbol target

  • When no default or latest symbol is defined

    • @default and @latest point to the highest existing version (also called the implicit default)
    • note that if a default symbol is defined but not a latest symbol, @default points to the defined default and @latest points to the highest version that exists

  • default or latest version specifiers can be employed:

    • in single version specification: @default or @latest
    • in version list: @vers1,default or @latest,default,vers2

  • default or latest version specifiers cannot be employed in version range

    • otherwise an error is raised

  • default or latest version specifiers can also be specified with the traditionnal mod/version syntax

    • for instance mod/default or mod/latest

  • When a default or latest modulefile exists

    • @default or respectively @latest will resolve to the existing modulefile
    • no automatic symbol will be recorded in this case as default or latest are regular versions

  • To be included in module search result, the version specifiers should fully match search query or search query should not target a specific module version

    • the automatically defined symbolic versions are included in results for queries like mod@latest or mod
    • but not for queries like mod@la, mod@def, mod@lat*, mod@def??lt

  • Automatically defined default and latest version specifiers are not displayed to avoid overloading output:

    • on module list output
    • on module avail output
    • those two sub-commands only display symbolic versions manually defined

  • Alternative module names deduced from the automatically defined version specifiers need to be tracked

    • in loaded environment for each targeted module loaded

    • to keep track loaded module is default or latest version

    • thus keeping ability to answer queries like is-loaded mod@latest from further modulefile evaluation or module command-line

    • this information is kept in the MODULES_LMALTNAME environment variable, along other alternative names

    • Auto symbols in this variable are recorded with a as| prefix to distinguish them from other alternative names

      • for instance mod/1.2&mod/regular_symbol&as|mod/latest
      • it helps to filter auto symbols from regular symbols that need to be displayed

Extended default

Configuration

  • introduce extended_default option name

  • off by default in v4 as previously soft/1 was an error

  • on by default in v5, option could even disappear

  • set as a separate option than advanced_version_spec

    • as it affects also basic soft/vers version specification
    • seem easier to understand for user if concepts are distinguished
Specification

  • Take partial version identifier and returns matches

    • for soft/10.1.2.4 soft/10.1.2.3 soft/10.1.1
    • query soft/10 returns highest among 3
    • query soft/10.1 returns highest among 3
    • query soft/1 returns nothing
    • query soft/10.1.2 returns highest among soft/10.1.2.4 soft/10.1.2.3

  • In situation where soft/1.1(default) soft/1.2 soft/2.1 soft/2.2

    • query soft/1 returns soft/1.1
    • query soft/2 returns soft/2.2

  • Character considered as version number separator: .

    • list: . and -
    • - was also considered initially, but cannot determine in all case the highest version specified after this character (may find a hash name, strings like rc, alpha, beta, etc)
    • not possible with + as it is used by variant specification

  • Does not apply to the root part of module name

    • e.g. foo.2

  • If implicit_default is disabled

    • it makes extended_default inoperant if queried version does not include a defined default
    • even if only one modulefile matches query
    • with situation described above query soft/1 returns soft/1.1
    • but query soft/2 returns an error as no default is found among version 2 modulefiles

  • Contexts where it could be used

Hide or forbid modulefile

Configuration
  • No specific configuration
Use cases

  • Restrict usage of particular software to a limited set of user (RestrictUsage)

    • Included in module specification result for a not granted user: no
    • Visible for a not granted user on a full avail: no
    • Visible for a not granted user on a targeted avail: no
    • Visible for a not granted user once loaded on a list: yes
    • Load tentative for a not granted user: error

  • Allow usage of particular software once clearance has been obtained (AllowOnceCleared)

    • Included in module specification result for a not granted user: yes
    • Visible for a not granted user on a full avail: yes
    • Visible for a not granted user on a targeted avail: yes
    • Visible for a not granted user once loaded on a list: yes
    • Load tentative for a not granted user: error

  • Expire a software after a given date (Expire)

    • Included in module specification result for a not granted user: no after expiration date
    • Visible for a not granted user on a full avail: no after expiration date
    • Visible for a not granted user on a targeted avail: no after expiration date
    • Visible for a not granted user once loaded on a list: yes, even after expiration date
    • Load tentative for a not granted user: error after expiration date

  • Disclose a software after a given date (Disclose)

    • Included in module specification result for a not granted user: no prior disclosure date
    • Visible for a not granted user on a full avail: no prior disclosure date
    • Visible for a not granted user on a targeted avail: no prior disclosure date
    • Visible for a not granted user once loaded on a list: yes, even prior disclosure date
    • Load tentative for a not granted user: error prior disclosure date

  • Hide software not of interest for current user (HideNotOfInt)

    • Included in module specification result for a not granted user: yes
    • Visible for a not granted user on a full avail: no, unless specific option set
    • Visible for a not granted user on a targeted avail: yes
    • Visible for a not granted user once loaded on a list: yes
    • Load tentative for a not granted user: success

  • Hide software only useful for other software as dependency (HideDep)

    • Included in module specification result for a not granted user: yes
    • Visible for a not granted user on a full avail: no, unless specific option set
    • Visible for a not granted user on a targeted avail: yes
    • Visible for a not granted user once loaded on a list: yes
    • Load tentative for a not granted user: success

  • Hide dependency software once loaded (HideDepOnceLoaded)

    • Included in module specification result for a not granted user: see HideDep
    • Visible for a not granted user on a full avail: see HideDep
    • Visible for a not granted user on a targeted avail: see HideDep
    • Visible for a not granted user once loaded on a list: no, unless specific option set
    • Load tentative for a not granted user: see HideDep
Specification

  • 2 new modulefile commands are introduced for the needs described above: module-hide and module-forbid

  • module-hide removes visibility of specified modules

    • module-hide acts when modules are searched (avail, whatis and search sub-commands) or selected (load, unload, display, etc sub-commands)

    • Visibility is however enabled if hidden module is specifically searched

      • On all context hidden module mod/1.0 is included in result for instance if mod/1.0 or mod@1.0,2.0 are specficied
      • But hidden module mod/1.0 is excluded from result if mod@:2.0 or mod@1: are specified
      • And is also excluded from result if mod or mod/* are specified when mod/1.0 is set default
      • Unless if search is made to select one module since in this context a mod search query is equivalent to mod/default
      • Hard-hidden modules are not disclosed even if specifically searched

    • Excluded from module resolution result

      • Unless precisely specified on the following selection contexts:

      • For example, the hidden module mod/1.0

        • is included in module load mod/1.0 result
        • is excluded from module load mod/1 result, even if default symbol targets it
        • is excluded from module load mod result, unless if default symbol targets it (as query is equivalent to mod/default)
        • is excluded from module load mod@:2 result, even if default symbol targets it
        • is included in module load mod@1.0,2.0 result
        • is included/excluded the same way for prereq and conflict sub-commands than load sub-command
        • is matched by is-loaded and info-loaded sub-commands querying it once loaded
        • is excluded from module whatis result
        • is included/excluded the same way for whatis sub-command than avail sub-command
        • is excluded from module avail result
        • is excluded from module avail m* result
        • is included in module avail mod/1.0 result
        • is excluded from module avail mod/1 result, even if default symbol targets it
        • is excluded from module avail mod result, even if default symbol targets it (as query is NOT equivalent to mod/default in this context)
        • is excluded from module avail mod@:2 result, even if default symbol targets it
        • is included in module avail mod@1.0,2.0 result

    • Included in module resolution result if --all option of avail, whatis, search and aliases sub-commands is set

      • --all option does not apply to is-avail sub-command to make it coherent with load sub-command (eg. a is-avail mod returning true implies load mod effectively loading a module)

    • Visibility of a module targeted by a module-hide command, with regular hiding level defined, acts similarly than for a file whose name is prefixed by a dot character on Unix platform

    • If --soft option is set on module-hide command, module hiding is weakened

      • Module is always included in resolution result for the following contexts

      • For example, the hidden module mod/1.0

        • is included in module load mod/1.0 result
        • is included in module load mod/1 result
        • is included in module load mod result
        • is included in module load mod@:2 result
        • is included in module load mod@1.0,2.0 result
        • is included/excluded the same way for prereq and conflict sub-commands than load sub-command
        • is matched by is-loaded and info-loaded sub-commands querying it once loaded
        • is excluded from module whatis result
        • is included/excluded the same way for whatis sub-command than avail sub-command
        • is excluded from module avail result
        • is excluded from module avail m* result
        • is included in module avail mod/1.0 result
        • is included in module avail mod/1 result
        • is included in module avail mod result
        • is included in module avail mod@:2 result
        • is included in module avail mod@1.0,2.0 result

    • If --hard option is set on module-hide command, hiding is hardened and designated modules are never unveiled

      • Designated modules are strictly hidden, also referred as hard-hidden

        • --all option of avail sub-command cannot unveil them

      • Excluded from module resolution result, which means it is always excluded from resolution on following context:

      • For example, the hard-hidden module mod/1.0

        • is excluded from module load mod/1.0 result
        • is excluded from module load mod result, even if default symbol targets it
        • is excluded from module load mod/1 result, even if default symbol targets it
        • is excluded from module load mod@:2 result, even if default symbol targets it
        • is excluded from module load mod@1.0,2.0 result
        • is included/excluded the same way for prereq and conflict sub-commands than load sub-command
        • is matched by is-loaded and info-loaded sub-commands querying it once loaded
        • is excluded from any avail query result
        • is included/excluded the same way for whatis sub-command than avail sub-command

      • Visibility of a module targeted by a module-hide --hard command acts like if no modulefile exists on filesystem

    • If --hidden-loaded option is set on module-hide, hiding also applies to specified modules once they are loaded

      • Hidden once loaded modules do not appear on module list

        • Unless --all option is set on list sub-command

      • Hidden once loaded modules load or unload is not reported

        • If this evaluation has been triggered automatically

          • By an automated module handling mechanism for instance
          • Which means user has not explicitely asked the module load or unload

        • And was automatically loaded, in case of an automatic unload

          • Which means the automatic unload of an hidden loaded module will be reported if it was manually loaded

        • And if verbosity level is lower than verbose2 level

        • And if no issue occurs during hidden module evaluation

        • Switch of hidden modules is not reported

          • If both switched-off and switched-on modules are set hidden
          • If switched-off module were automatically loaded
          • And if the switch evaluation has been automatically triggered

      • When those modules are loaded, a hidden-loaded tag is applied to them and recorded in MODULES_LMTAG environment variable to keep track of their hidden status

      • Hidden once loaded status does not affect is-loaded: these modules will always be reported if they match is-loaded queries

  • module-forbid disallow evaluation of specified modules

    • It does not imply hiding, but can be of course coupled with module-hide calls

    • Evaluation of targeted modules is forbidden

      • Error is rendered prior evaluation when trying to load, display, help, test, path, whatis them
      • Note that for whatis evaluation mode, an error is reported only if a module is referred by its exact name which is not the case on search sub-command as no module is specified, just a keyword to search
      • No error occurs when unloading a module that were set forbidden after it was loaded by user

    • As it impacts module evaluation, module-forbid is only effective when it targets modulefiles or virtual modules

      • Module alias or symbolic version are not impacted by module-forbid directives
      • Even if they match some module-forbid statements, they are still resolved to their target and these targets do not inherit the forbidden tag set on their alias, symbol.
      • When a module-forbid command targets a directory, this directory is still resolved to its target, but the target inherits the forbidden tag as it matches the name specified on module-forbid command

    • When combined with a module-hide --hard command, designated modules is unveiled if referred by its exact name and set in error

      • Thus an error is obtained when trying to reach module instead of not finding it (which is the regular behavior for hard-hidden modules)

  • module-hide accepts options that change its behavior:

    • --hidden-loaded: hides module from loaded module list
    • --soft: lightweight module hide
    • --hard: highest hiding level
    • --not-user: specify a list of users unaffected by hide mechanism
    • --not-group: specify a list of groups whose member are unaffected by hide mechanism
    • --before: enables hide mechanism until a given date
    • --after: enables hide mechanism after a given date

  • module-forbid accepts options that change its behavior:

    • --not-user: specify a list of users unaffected by forbid mechanism
    • --not-group: specify a list of groups whose member are unaffected by forbid mechanism
    • --before: enables forbid mechanism until a given date
    • --after: enables forbid mechanism after a given date
    • --message: supplements error message obtained when trying to evaluate a forbidden module with given text message
    • --nearly-message: supplements warning message obtained when evaluating a nearly forbidden module with given text message

  • Each use case expressed above are covered by following command:

    • RestrictUsage: module-hide --hard
    • AllowOnceCleared: module-forbid
    • Expire: module-forbid --after + module-hide --hard --after
    • Disclose: module-hide --hard --before
    • HideNotOfInt: module-hide --soft
    • HideDep: module-hide --soft
    • HideDepOnceLoaded: module-hide --soft --hidden-loaded

  • module-hide and module-forbid accept the specification of several modules

    • For instance module-hide mod1 mod2...

    • Advanced module version specifiers are supported if relative module option is enabled

    • Full path specification are not supported, as modulerc are not evaluated when reaching a modulefile specified as full path

      • For instance, /path/to/modulefiles/.modulerc is not evaluated when loading /path/to/modulefiles/mod/1.0

      • Thus module-hide and module-forbid commands set in this modulerc files are not evaluated

      • If module is specified as full path, no error is returned, but it will have no effect as demonstrated above

        • Unless on very specific cases, where a global rc file defines these hidden/forbidden commands for the full path modules

  • --not-user and --not-group specification is only supported on Unix platform

    • These 2 options raise an error when used on Windows platform
    • In which case relative module-hide or module-forbid command is made ineffective as well as remaining content of the modulerc script hosting them
    • Error message is clearly seen when trying to load related modules and indicate where to find the erroneous command

  • --before and --after are also supported by module-hide to phase-out modules prior to forbid their evaluation

  • --before and --after accept a date time as value

    • Accepted date time format is YYYY-MM-DD[THH:MM]

    • If no time value is specified (just a date like 2020-08-01), 00:00 is assumed

      • So 2020-08-01 is translated into 2020-08-01T00:00

    • An error is raised if submitted date time value does not match accepted date time format

    • if both --before and --after options are set and before date is greater than after date

      • targeted module is always hidden/forbidden
      • no error is returned

  • --before and --after options are not supported on Tcl version below 8.5

    • Prior 8.5, clock scan command does not have a -format option
    • This option is required to support defined date time format
    • An error is raised when --before or --after options are used over a Tcl version below 8.5

  • --message option adds additional text to the access denied error message

    • Newline set in text message are preserved, which could help to control text output format

    • Message content is set along forbidden module specification

      • Message recorded for matching module specification will be printed
      • Message recorded on other matching specification will be ignored, only message from retained matching specificaton is printed
      • Firstly evaluated module-forbid command that matches module specification is retained with its message property

  • a module matching a module-forbid statement whose --after limit is close is considered nearly forbidden

    • nearly-forbidden tag applies to such module
    • matched module-forbid statement should of course not be disabled for current user of group due to --not-user or --not-group option values
    • the near range is defined by the nearly_forbidden_days configuration, which equals to 14 (14 days) by default
    • this configuration accepts an integer value which represents a number of days prior forbiding starts to be effective for module
    • nearly_forbidden_days configuration can be set at configure time with --with-nearly-forbidden-days option or afterward with the config sub-command (which sets the MODULES_NEARLY_FORBIDDEN_DAYS environment variable)
    • when evaluating a nearly-forbidden module, a warning message is reported to indicate that module access will soon be denied

  • --nearly-message option adds additional text to the access will be denied warning message

    • Newline set in text message are preserved, which could help to control text output format

    • Message content is set along nearly-forbidden module specification

      • Message recorded for matching module specification will be printed
      • Message recorded on other matching specification will be ignored, only message from retained matching specificaton is printed
      • Firstly evaluated module-forbid command that matches module specification is retained with its message property

  • module-hide and module-forbid are intended to be used in modulerc files

    • as they impact modulefile resolution
    • they also need to be enabled in modulefile context as global/user rc files are evaluated as modulefile, not modulerc

  • several module-hide calls for the same module will supersede each other

    • definition with the highest hiding level wins
    • which means the most restrictive call wins
    • a --hidden-loaded status set is kept even if corresponding module-hide call is not the highest one
    • the multiple definitions can come accross different modulerc files (global, modulepath or modulefile rc levels)

  • Module specification passed as argument to module-hide and module-forbid are matched exactly against available modules

    • Exception made when extended_default or icase mechanisms are enabled
    • Which means wildcard characters like * or ? are treated literally

  • Auto symbols (@default and @latest) are adapted when a latest version is hidden

    • Auto symbols are applied to this version if it is selected specifically (for instance loaded by its full name)
    • Auto symbols are applied to another version when hidden latest is not selected specifically, even if specified with @latest auto symbol

  • Auto-symbols cannot be set hidden

    • When a defined default or latest symbol is set hidden, it is replaced by a default or latest auto-symbol targetting highest available module version
    • Targeting an auto-symbol with a module-hide command, will have no effect

  • When module specification of module-hide targets:

    • A symbolic version

      • This symbol only is hidden
      • Modulefile targeted by hidden symbolic version stays visible

    • An alias

      • This alias only is hidden
      • Modulefile targeted by hidden alias stays visible

    • A modulefile targeted by either symbolic version or alias

      • This modulefile is hidden and all symbolic versions targeting it
      • Aliases targeting modulefile stays visible (thus resolving alias in load or whatis contexts make hidden modulefile target visible unless if set hard-hidden)

  • Hidden alias or symbolic version should not appear in the list of alternative names of loaded modules

    • Unless this alias or symbolic version is not hard-hidden and is used to designate the module to load

    • When default symbolic version is set hidden

      • also remove parent module name from the list of alternative names
      • if resolution query corresponds to parent module name, unhide default symbol unless if hard-hidden

  • On avail sub-command

    • Hidden symbolic versions are not reported along module they target

      • Unless for non-hard-hidden symbols specifically designated in search query

    • A --default filtered search considers search query matches default symbol

      • So default symbolic version will appear in result unless if hard-hidden

  • Different hiding level are considered

    • -1: module is not hidden
    • 0: soft hiding (applied with module-hide --soft)
    • 1: regular hiding (applied with dot name module or default module-hide command)
    • 2: hard hiding (applied with module-hide --hard)

  • Hiding threshold

    • is 0 by default, which means module is considered hidden if its hiding level is greater or equal to 0
    • is raised to 2 when --all option is applied, which means module is considered hidden if its hiding level is greater or equal to 2

Insensitive case

Configuration

  • Introduce the icase option name

    • Which is made persistent through the MODULES_ICASE environment variable

    • Many people asked for separate options as applying an icase approach to a module loading context is not seen desired by everybody whereas it is in an module search context

    • Defined levels of enablement are:

      • never
      • search
      • always

    • icase option will be set by default to

      • never in v4 not to change existing behaviors
      • search in v5 as it seems to be a general improvement for everybody

    • A command-line switch --icase (short form -i) is added

      • Was there in Modules 3 (for search sub-command only)
      • When set, equals to an always icase mode

  • No immediate need for a --no-icase command-line switch

    • Combining configuration option and --icase command-line switch seems sufficient
Specification

  • When enabled, match query string in a case insensitive manner

    • query soFT returns soft, SOFT, soFT, SOft, sOft and so on

  • In case multiple files correspond to the same icase word

    • like soft, soFT, SoFt, SOFT filenames
    • query SOFT returns SOFT (exact match is returned among possibilities)
    • query SoFt returns SoFt (exact match is returned among possibilities)
    • query SOft returns soft (highest dictionarily-sorted match is returned among possibilities)
    • query soFt returns soft (highest dictionarily-sorted match is returned among possibilities)

  • When icase is enabled for search context it applies to

  • When icase is enabled for all context it applies to

    • search context like described above

    • module specification passed as argument in following contexts:

    • module alias and symbolic version resolution triggered by

      • module-info alias
      • module-info version
      • getPathToModule
      • isModuleEvaluated

    • the gathering of all module alias and symbolic version targeting a given module

      • this is processed by getAllModuleResolvedName procedure

      • which is called by

        • doesLoadingModuleMatchesName
        • cmdModuleLoad

  • Note that whatis specification passed as argument to the search sub-command is always matched in a case insensitive manner

Corner cases

  • When looking for the best match among loaded modules to select one module to unload, lastly loaded module, or firstly loaded module depending on unload_match_order configuration, will be returned

    • When insensitive case is enabled, last or first icase match will be returned even if an exact match is present among the loaded module list

    • This behavior has an impact in case multiple available modules correspond to the same insensitive case string

      • For instance iCaSe and IcAsE modules

  • When icase is enabled on all contexts and multiple directories match same icase module name

    • for instance following modules exist: ICASE/1.1, icase/1.2, iCaSe/1.3 and iCaSe/1.4

    • a module avail -i icase will sort iCaSe/1.4 as the highest entry

    • however a module load -i icase command will load icase/1.2

      • as icase directory matches query

    • and also module load -i ICase command will load icase/1.2

      • as no directory ICase exists, result is returned from highest directory: icase

Module selection contexts

Description of the different contexts where a module is expressed to get resolved to modulefiles.

Module identification to select one module

  • specification expresses exactly one module

  • must qualify version to select in case a module has multiple versions

  • expression may

    • be absolute like mod/version
    • be relative to the modulefile currently being evaluated like ./version or /version
    • refer to symbolic version or module alias

  • specification used to resolve argument passed to the following commands:

    • module-version
    • module-alias
    • module-virtual (cannot use symbolic version or module alias, should define an actual module)
    • module-info symbols
    • module-info version
    • module-info loaded

  • relies on getModuleNameVersion procedure to get absolute name

Module version specification to return all matching modules

  • specification expresses one or multiple modules

  • expression may

    • be absolute like mod/version
    • refer to implicit or explicitly defined default version like mod
    • refer to extended default version like mod/1 (to refer to mod/1.1.1)
    • refer to symbolic version or module alias
    • specify multiple versions as list (mod@1,2) or range (mod@:1 or mod@1:2)

  • all modules matching specification are retrieved with getModules

  • specification used to resolve argument passed to the following commands:

    • avail
    • paths
    • whatis

  • used to get all alias and symbolic version of loaded modules treated by:

    • list

  • used to get all existing aliases and symbolic version (no module specification) by:

    • aliases

  • used to get all existing modulefiles (no module specification) by:

    • search
Module version specification to select one module

  • specification expresses one or multiple modules

  • relies on the module version specification to return all matching modules

  • then among matching modules, one is selected with getPathToModule:

    • the one set has the default version (also called the explicit default version)
    • or the highest version in case no explicit default is found among results (also called the implicit default version)
    • nothing is returned in case no explicit default is found among matching modules and implicit default version mechanism is disabled

  • module selection may trigger another match retrieval

    • for instance in case selection leads to an alias that resolves to a bare module name

  • specification used to resolve argument passed to the following commands:

    • load
    • unload (with attempt to match against loaded modules to resolve argument)
    • switch
    • help
    • test
    • display
    • path
    • is-avail

  • used to resolve aliases or symbolic versions treated by following commands:

    • paths
    • search
Module version specification to check compatibility

  • specification expresses one or multiple modules

  • expression may

    • be absolute like mod/version
    • refer to implicit or explicitly defined default version like mod
    • refer to extended default version like mod/1 (to refer to mod/1.1.1)
    • refer to symbolic version or module alias
    • specify multiple versions as list (mod@1,2) or range (mod@:1 or mod@1:2)

  • version specification is matched against loaded or loading modules with variety of procedures:

    • getLoadedMatchingName
    • getLoadedWithClosestName
    • isModuleEvaluated

  • specification used to resolve argument passed to the following commands:

    • is-loaded
    • info-loaded
    • prereq
    • conflict

  • for prereq command when version specification does not have a match among loaded or loading modules

    • version specification is used to select one module per module specification
    • for instance prereq mod@:1.8 mod@1.10 triggers load tentative of default version among mod@:1.8 then if this tentative fails load of mod@1.10 is attempted
    • another example with prereq mod mod@1.10, which triggers load tentative of mod/default then if it fails load of mod@1.10 is attempted

Module tags

Configuration
  • No specific configuration
Specification

  • Introduce one new modulefile command to set tags on modulefiles: module-tag

    • A tag is a piece of information associated to individual modulefiles

    • That is reported along module name on avail or list sub-command results

    • This piece of information could lead to specific behaviors when handling modulefile over the different module sub-commands or modulefile evaluation modes

      • For instance, a module with a sticky tag set on it cannot be unloaded

  • 3 different kind of tag exist:

    • Those inherited from module state, consequence of a specific modulefile command or module action

      • This kind of tag cannot be defined with the module-tag command

        • An error is returned otherwise
        • Specific modulefile command should be used instead as such tag may have specific properties that should also be defined along
        • Easier for everybody to only have one way to define such tags and not 2 different commands

      • Every tag holding specific properties should have its dedicated modulefile command to define it

    • Those set with module-tag that lead to a specific behavior

    • Those set with module-tag that are purely informational, no specific behavior

  • Tags inherited from module state:

    • hidden: module not visible, not reported by default in avail result (tag acquired through module-hide)
    • hidden-loaded: loaded module not reported by default in list result (tag acquired through module-hide --hidden-loaded)
    • forbidden: module that cannot be loaded (tag acquired through module-forbid)
    • nearly-forbidden: module that soon cannot be loaded (tag acquired through module-forbid)
    • loaded: loaded module
    • auto-loaded: module automatically loaded by another module

  • Tags set with module-tag associated to a specific behavior:

    • sticky: loaded module cannot be unloaded unless forced or reloaded (see Sticky modules)
    • super-sticky: loaded module cannot be unloaded even if forced, it stills can be unloaded if reloaded afterward (see Sticky modules)

  • Tags inherited from module state cannot be set with module-tag command

    • An error is otherwise thrown

  • Modules project may introduce in the future new tags inherited from new states or new specific behaviors

    • These new tags will supersede tags set by users using the same naming
Defining

  • module-tag [options] tag modspec...

    • Apply tag to all modules matching modspec module specification
    • Advanced module version specifiers are supported if relative module option is enabled
    • Full path module specification is not supported, as modulerc are not evaluated when reaching a modulefile specified as full path
    • One tag could be applied to multiple module specifications with a single module-tag command call

  • module-tag accepts the following options:

    • --not-user: specify a list of users unaffected by specified tagging
    • --not-group: specify a list of groups whose member are unaffected by specified tagging

  • --not-user and --not-group specification is only supported on Unix platform

    • These 2 options raise an error when used on Windows platform
    • In which case relative module-tag command is made ineffective as well as remaining content of the modulerc script hosting them
    • Error message is clearly seen when trying to load related modules and indicate where to find the erroneous command

  • module-tag is intended to be used in modulerc files

    • to be easily fetched during avail sub-command processing

    • they also need to be enabled in modulefile context as global/user rc files are evaluated as modulefile, not modulerc

    • it enables to dissociate environment changes described in the modulefile from the properties of this modulefile

      • as these properties are usually site-specific
      • and modulefile are automatically generated by a build tool
      • but properties are not always related and set by the build tool
Persistency

  • The MODULES_LMTAG environment variable holds all tag information applying to loaded modules

    • Following the same syntax scheme than MODULES_LMCONFLICT
    • For instance module/vers&tag&tag:module2&tag

  • The loaded tag is not recorded in MODULES_LMTAG environment variable

    • As it obviously applies to all loaded modules defined in LOADEDMODULES environment variable

  • The auto-loaded tag is not recorded in MODULES_LMTAG environment variable

    • Auto loaded modules are currently listed in the MODULES_LMNOTUASKED environment variable
Reporting

  • Defined tags are reported on avail and list sub-command results

    • Reported along modulefile name, within angle brackets (following the HTML tag fashion)
    • Each tag separated by a colon
    • For instance module/version <tag1:tag2>
    • Tags are right-aligned on each column
    • One space character at least separates module name and version or list of symbolic version from tag list

  • Tags applying to module alias are reported on avail reports only

    • Where the module alias stands for itself in the report

    • On list reports, alias is reported along its modulefile target

      • So the tags applying to the alias are not reported
      • Also these tags of the alias are not inherited by alias' target

  • Tags applying to symbolic version are never reported

    • As symbols are never reported alone on avail or list reports
    • Always reported along their modulefile target
    • Also these tags of the symbolic versions are not inherited by symbol's target

  • Some tags are not reported on avail output:

    • hidden-loaded: correspond to hiding module from loaded list, not from available list

  • Some tags are not reported on list output:

    • loaded: as every modules reported on list are loaded
    • forbidden: forbidden module cannot be loaded, so it cannot be found among loaded module list
    • hidden: correspond to hiding module from availabilities, not from loaded list

  • When reported in JSON output format

    • tags are listed under the tags key

  • Default --long report does not contain tag information

    • Not to exceed the 80-column output limit by default
Abbreviations

  • Tag abbreviations are used to translate tag names when reporting them on avail or list sub-command output

  • The tag_abbrev configuration defines the abbreviations to apply to each tag

    • Set by default at configure time to auto-loaded=aL:loaded=L:hidden=H:hidden-loaded=H:forbidden=F:nearly-forbidden=nF:sticky=S:super-sticky=sS

      • Note that by default, hidden and hidden-loaded tags share the same abbreviation, as they operate on separate contexts (respectively avail and list contexts)

    • Configuration value consists in a key=val pair value, each key pair are separated by a : character

      • Follow the same syntax than colors configuration

    • If an existing tag name is not part of the configuration, it means no abbreviation applies to it

    • If a tag name has an empty string abbreviation defined it is not reported

      • Unless if there is an SGR color configuration defined for this tag

    • The MODULES_TAG_ABBREV environment variable is used to set a specific value for tag_abbrev configuration

      • If MODULES_TAG_ABBREV is set to an empty string, no tag abbreviation applies

  • In case default value or environment value of tag_abbrev is badly set

    • a warning message is returned
    • value is ignored
    • if nor the environment nor the default value is correct then no abbreviation applies to tag

  • Tags are not translated to their defined abbreviation in JSON output format

SGR

  • If a tag name or tag abbreviation has an SGR code defined in the color list, this SGR code is applied to the module name this tag refer to

    • Tag name or abbreviation is not reported by itself in this case

    • As it is now represented by the SGR applied to module name

    • If an abbreviation exists for a tag, SGR code should be defined for this abbreviation in color list

      • An SGR code set for tag full name does not apply on the abbreviation of this tag

  • If multiple tags apply to the same modules and have an SGR code defined for them in the color list

    • All these SGR codes are rendered one after the other over the module name
    • For instance if 2 tags apply, the first one will be rendered over the first half of the module name, the second tag over the second half of

  • Tags use by default background color change to stand out

    • As module kind (alias, directory, etc) is mainly represented with foreground color change by default,

  • In case if there are more tags to graphically render than character in module name

    • The remaining tags are reported by there name or abbreviation and SGR applies over this name or abbreviation

  • The MODULES_TAG_COLOR_NAME environment variable is used to define the tags whose name (or abbreviation if set) should be reported

    • Their name does not vanish if a SGR code is defined in the color list for them

    • Their SGR code is not rendered over the module name

    • Instead the SGR is applied to the reported tag name (or tag abbreviation if set)

    • MODULES_TAG_COLOR_NAME is bound to the tag_color_name configuration

    • MODULES_TAG_COLOR_NAME contains the list of tag name (or abbreviation), each tag separated with colon character (:)

    • If an abbreviation is defined for a tag and one want it to be reported by itself not rendered over module name

      • This abbreviation should be set in MODULES_TAG_COLOR_NAME
      • Not the full tag name this abbreviation refers to
Querying

  • The tags sub-command of module-info modulefile command enables modulefile to know what tags apply to itself

    • module-info tags returns a list of all the tags applying to currently evaluated module
    • an empty list is returned when called from a modulerc evaluation context or if no tag applies to current modulefile

  • Tags cannot be queried to select modules

    • Symbolic versions or variants can be used to select modules

Output configuration

Configuration
Specification

  • Output configuration is available

    • For avail sub-command, on regular and --terse modes
    • For list sub-command, on regular and --terse modes

  • Output configuration is not available (but could be added later on)

  • Output configuration defines the content of the output not its format

  • These configurations are set by default with the elements reported by default

    • avail_output: modulepath:alias:dirwsym:sym:tag:key
    • avail_terse_output: modulepath:alias:dirwsym:sym:tag
    • list_output: header:idx:variant:sym:tag:key
    • list_terse_output: header

  • The above default value could be superseded:

  • Accepted elements in value lists are:

    • For avail options: modulepath, alias, dirwsym, sym, tag, key
    • For list options: header, idx, variant, sym, tag, key

  • If the -o/--output options are wronly specified

    • An error is raised and evaluation terminates

    • It may happen in the following situations

      • No value set after the option
      • Option set on unsupported module sub-command
      • Element set in option value unsupported by module sub-command
      • Elements set in option value not separated by colon character (:)
      • Option set on unsupported output format (--long or --json)
      • The above situations apply whether command is called from the terminal or within a modulefile

  • For all these new configuration options

    • accepted value is a list of strings separated by colon character (:)
    • order of elements in the list does not matter
    • an empty string is a valid value (means only the modulefile name has to be reported)

  • If the MODULES_AVAIL_OUTPUT/MODULES_AVAIL_TERSE_OUTPUT/MODULES_LIST_OUTPUT/MODULES_LIST_TERSE_OUTPUT env vars are wrongly specified

    • Their value is ignored

    • So the default value takes precedence, unless a -o/--output option is specified

    • Value in environment variable is wronly specified for instance in the following situations

      • Element set in option value unsupported by module sub-command
      • Elements set in option value not separated by colon character (:)

  • avail_output supersedes avail_report_dir_sym and avail_report_mfile_sym configurations

    • Enabled avail_report_dir_sym corresponds to adding dirsym value to avail_output

    • Enabled avail_report_mfile_sym corresponds to adding sym value to avail_output

    • Both avail_report_dir_sym and avail_report_mfile_sym could be removed safely as:

      • it was not possible to define them at configure time
      • or change default value with an environment variable

  • Some output content cannot be controlled at the moment with the output options:

    • Hidden modules is exclusively controlled by --all option to get these modules unveiled
    • Indepth/no-indepth output is exclusively controlled by --no-indepth/--indepth and related configuration option

  • When modulepath element is removed from an avail-related option

    • all the modulefiles returned from all searched modulepaths are merged and sorted as a single set
    • a module appearing in several modulepaths is only reported once
    • tags or symbols applying to a lower priority module with same name are still reported
Output key

  • An output key is added to print a legend explaining the output

    • Meaning of (), {} or <> is explained
    • Default version is colored specifically
    • Every tag shortened to a defined abbreviation
    • Every tag colored specifically
    • Every variant type set (variant=value, +boolvariant, -boolvariant, %shortcutvariantvalue, etc)
    • Legend entries only concern elements that can be found in reported output
    • Legend entries are not reported on JSON output mode

  • Output key is enabled by default on avail and list sub-command output

    • Key is reported at the end of the output
    • No key section is reported if no element need to be described (no color, no symbol, no tag, etc)

Source shell script in modulefile

Configuration
  • No specific configuration
Specification

  • Execute a shell script and include the environment changes it performs in modulefile

    • Environment changes done by shell script are gathered and evaluated in modulefile context through corresponding modulefile commands
    • Goal is to get the same environment after loading a modulefile using a source-sh than if shell script targeted by the source-sh where directly sourced in shell session

  • Environment changes tracked are (all environment elements module can handle):

    • Value set to variable: transformed into setenv modulefile command
    • Variable unset: transformed into unsetenv modulefile command
    • Path element added to variable: transformed into append-path or prepend-path modulefile command
    • Path element removed from variable: transformed into remove-path modulefile command
    • Current working directory change: transformed into cd modulefile command
    • Shell alias definition: transformed into set-alias
    • Shell alias unset: transformed into unset-alias
    • Shell function definition: transformed into set-function
    • Shell function unset: transformed into unset-function

  • Depending on modulefile evaluation mode, source-sh has different behaviors:

    • load: execute script to gather its environment changes, transform them in modulefile commands and evaluate them
    • unload: undo environment changes made on load mode
    • display: execute script to gather its environment changes and report resulting command name and arguments for display
    • help, test and whatis: no operation

  • Limitation: code in modulefile cannot rely on the environment changes done in script targeted by a source-sh command

    • For instance an environment variable set in shell script cannot be used to define another variable later in modulefile

    • This will work on load, unload and display modes, as script is run and/or analyzed and corresponding modulefile commands are evaluated in modulefile context

    • But it will not work on the other modes, as script is not run and analyzed there

      • To simplify processing as script need to be run and analyzed if not yet loaded, but if already loaded changes recorded in environment for tracking should be used instead
      • To avoid a negative impact on performances on the whatis global evaluation

  • Result of source-sh command evaluation is stored into the environment when modulefile is loaded

    • To keep track of environment changes made by source-sh script evaluation

    • In order to undo these changes when unloading modulefile and report corresponding modulefile commands when displaying modulefile

    • Environment variable MODULES_LMSOURCESH is used for this need

      • Using following syntax: mod&shell script arg|cmd|cmd|...&shell script|cmd:mod&shell script arg|cmd

      • Example value: foo/1&sh /tmp/source.sh|append-path PATH /tmp|cd /tmp

      • Characters used to structure information in MODULES_LMSOURCESH (:, & and |) are escaped

        • Respectively to <EnvModEscPS>, <EnvModEscS1> and <EnvModEscS2>
        • If found in environment changes to record

      • Actual bodies of shell alias and shell functions are not recorded in MODULES_LMSOURCESH, an empty body is recorded instead

        • Example value: foo/1&sh /tmp/source.sh|set-alias alfoo {}|set-function funcfoo {}

    • When unloading modulefile, content found for this modulefile in MODULES_LMSOURCESH variable is evaluated to reverse environment changes

      • When reaching a source-sh modulefile command, recorded content is evaluated through a modulefile Tcl interpreter in unload mode, to get the reversed effect

    • When displaying modulefile

      • If it is loaded

        • the content found for this modulefile in MODULES_LMSOURCESH variable is evaluated in display mode to report each resulting modulefile command
        • script is evaluated to fetch shell alias and function definition which are not recorded in MODULES_LMSOURCESH

      • If not loaded, script is evaluated to gather environment changes and report each resulting modulefile command

  • Script targeted by a source-sh command has to be run and environment prior this run and after this run have to be compared to determine the environment changes the script performs

    • The shell to use to run script has to be specified to source-sh

    • This shell will be run to execute the following sequence:

      1. output current environment state
      2. source the script with its args
      3. output environment state after script source

    • Script output is kept to return it in an error message in case its execution fails

    • This comparison determines the environment changes performed by script which are then translated into corresponding modulefile commands

    • Shell is run in a mode where neither its user nor its system-wide configuration is evaluated to avoid side effect

    • Shell needs current environment to correctly evaluate script

      • it seems desirable to run shell in an empty environment to get the full list of changes it does over the environment

      • but the script may need the environment to be defined to correctly operate

        • for instance it needs the PATH to be set to execute external commands

      • so shell run inherits current environment to ensure script will be properly evaluated

      • but as a consequence if an environment change performed by script is already done prior script run, this environment change will not be seen

  • Limitation: a variable already set by another module will not be seen set sourced script

    • if this module is unloaded and if it does not trigger the unload of the modulefile using source-sh, variable will be unset as source-sh script has not increased variable reference counter

  • Note: environment change done through source-sh will not preserve change order occurring in sourced script, as all changes are analyzed after sourcing script

  • Most shell will not get the full environment content when spawned as a sub-process

    • For instance shell aliases are not transmitted into sub-shells
    • As a result the environment prior script source will most of the time have no shell alias defined

  • Note: shells may have specific behaviors

    • dash cannot pass arguments to the sourced script
    • ksh93 reads the full script prior sourcing it which may cause issue if a shell alias is defined in script then used in script

  • Limitation: implementation does not currently support:

    • the zsh shell when used as sh or as ksh
    • the mksh shell
    • the BSD variant of the sh shell
    • the Windows cmd shell

  • Note: if sourced script produces shell alias or function, these alias or function may not be compatible with the current shell of the user

  • Note: the mechanism described here only applies for shell script as to understand the impact the script execution over the user environment, this environment need to be compared prior and after execution

  • source-sh modulefile command relies of the sh-to-mod procedure of modulecmd.tcl

    • sh-to-mod procedure handles the evaluation of the targeted script and comparison of the environment prior and after this comparison
    • sh-to-mod returns as a result the list of modulefile commands describing the environment changes made by the analyzed script

  • sh-to-mod is a module sub-command, calling the sh-to-mod procedure and outputting resulting modulefile commands

    • with a #%Module header
    • to enable users to convert the environment changes made by script in modulefiles

  • There is no need to also make sh-to-mod a modulefile command and source-sh a module sub-command

  • Shell to use to source targeted script may be specified as a command name or as a path name

    • When specified as a command name (eg. bash), command location will be determined based on currently set PATH
    • When specified as a path name (eg. /bin/bash), this path name determines command location

  • Prior executing shell and making it sourcing script:

    • Shell is checked to ensure an executable command matches it
    • Script is checked to ensure it exists
    • Those tests are done prior executing to avoid it if one check fails and provide a consistent error message whatever the shell used

  • Environment changes to output as result should be enclosed and escaped

    • Enclosed if they contains space character (like white-space, tab and newline characters)
    • Escaped if they contains curly braces, as output is formatted as Tcl code, to avoid parsing issue

  • If an error occurs during targeted script evaluation

    • Error is thrown which leads to either a modulefile evaluation error or a module sub-command error
    • Error and output messages reported by evaluated script will be reported along error by modulefile command or module sub-command

  • To get prior and resulting environment state

    • env command is not used to gather environment state through exec source execution

      • it would simplify environment state parsing, as same command would be used for every shell
      • but it is an external command, so requires extra processing and an additional requirement
      • moreover it does not return shell functions in general, only exported Bash functions

    • Shell builtin commands are used to query existing environment variables, aliases, functions and current working directory

      • which provides best processing efficiency
      • but leads to specific output parsing for each shell

    • A separator string %ModulesShToMod% is printed between each kind of environment item (variable, alias, etc) and also prior and after script evaluation

      • to separate each output kind and then be able to split them for separate analysis

  • De-duplication of path entries is applied for changes on path-like environment variables

    • If the same path entry appears several times in the newly prepended entries for a variable, the first occurrence of this entry is kept others are dropped

    • If the same path entry appears several times in the newly appended entries for a variable, the first occurrence of this entry is kept others are dropped

    • De-duplication is not applied for path entries:

      • appearing in both the new prepended entries and newly appended entries
      • appearing in newly prepended entries or newly appended entries and in entries defined prior script evaluation

  • An environment variable equaling to the path separator character (:) prior script evaluation is considered as undefined prior script evaluation to avoid misleading analysis

Sticky modules

Configuration
  • No specific configuration
Specification

  • Once loaded a sticky module cannot be unloaded unless forced or reloaded.

  • If module is super-sticky instead of sticky, it cannot be unloaded even if forced, only if it is reloaded afterward

  • Stickyness definition relies on Module tags

    • module-tag sticky foo/1.0 defines module foo/1.0 as sticky
    • module-tag super-sticky foo/1.0 defines module foo/1.0 as super-sticky

  • Stickyness specified over a symbolic version or a module alias has no effect

    • module-tag allows to specify a symbolic module version or a module alias
    • but associated tag will apply to the symbolic version or alias only
    • as modulefile targetted by symbol or alias does not inherit their tags
    • so a sticky or super-sticky tag set on a symbolic version or alias has no effect

  • Sticky module can be swapped with another version from same module when stickness is defined over module parent name

    • For instance if stickyness is defined over module foo, foo/1.0 can be swapped with foo/2.0

    • Such swap could occur from a restore or a switch sub-command

    • As soon as stickyness is defined over a precise module version name (like foo/1.0) such module cannot be swapped by another version of foo module

    • Stickyness defined over module parent name (like foo) means any version from module foo must be loaded

    • When stickyness is defined for several module versions using advanced version specifiers like foo@1: or foo@1.0,2.0

      • it means stickyness applies to the module versions
      • thus they cannot be swapped by another version

    • In case stickness is defined over module parent name and another module-tag defines stickyness over specific module version name

      • it means stickyness applies to the module version
      • thus these versions targetted specifically with module-tag cannot be swapped by another version from same module

  • When a super-sticky module depends on a non-super-sticky module

    • If a forced purge command occurs, the dependent module will be unloaded
    • Which let the super-sticky module with a missing dependency
Current limitations

  • When swapping a sticky module explicitely targetted by the module-tag command and which is the default version

    • For instance module-tag sticky foo/1.0 and module-version foo/1.0 default
    • If specified swapped-on module is the generic module name, for instance foo
    • switch sub-command raises an error even if the sticky module is the default version (either implicit or explicitely set) for this module

Variants

Configuration

  • Variant mechanism requires the advanced_version_spec option to be enabled

    • Variants can be seen as an extension of the advanced module version specifiers
Specification
Defining

  • variants are defined for modulefiles within concerned modulefiles with the variant modulefile command

    • variants are bound to the modulefile their target
    • especially modulefile evaluation may be guided by the value selected for the variants
    • so it makes more sense to have variants defined within modulefiles rather in modulercs

  • variant --default? defaultval? --alias? {aliasname...?} name value ...?

  • name restrained to the accepted variable name

    • which should correspond to a sequence of one or more characters that are a letter, digit or underscore
    • if we want to match Spack here also: [A-Za-z0-9_][A-Za-z0-9_.-]*
    • raise error otherwise

  • a variant name that has already been defined as variant or variant alias for this modulefile evaluation overwrites previous definition

    • it is up to modulefile author to ensure there is no duplicate
    • no need this way to check in modulecmd.tcl code if variant has already been defined

  • may have a default value (when --default argument is passed)

    • if not, variant is undefined

    • argument --default should be associated a defaultval argument

      • raise error otherwise

    • default value should be part of possible value list

      • error is raised otherwise when default value is used, as it does not correspond to an accepted value

        • ex: variant name --default 1.8 1.0 2.0

  • may be set as a Boolean variant (when --boolean argument is passed)

    • a Boolean variant may equal true or false

  • must have a list of accepted values unless if defined as a Boolean variant

    • passed value(s) must correspond to those accepted

      • raise error otherwise
      • raised error is a global error to signal specification issue, not a modulefile error

    • an error is raised if one or more accepted values are defined on a Boolean variant

    • non-Boolean variants cannot define Boolean value in their accepted value list

      • as Boolean values are translated to their canonical form (0 or 1) when specified
      • an error is raised otherwise
      • exception made for the 0 and 1 integers

  • may be aliased (when argument --alias is passed) in which case:

    • argument --alias should be associated one or a list of alias names

      • raise error otherwise

    • defined alias names should be valid variant name (i.e.: [A-Za-z0-9_][A-Za-z0-9_.-]*)

      • raise error otherwise

    • should not be already defined as variant or variant alias

      • raise error otherwise

    • variant alias could negate its target (when alias name is prefixed by -)

      • meaning if alias variant is set to false its target is set to true

      • only possible if targeted variant is Boolean

        • raise error otherwise

      • no support of ~ as an alternative negated prefix to only get one way to write negation

  • require a value to be set

    • raise error if no value set for variant (no value specified and no defined default)

  • to validate variant value is coherent against other variant values or any other test

    • it is up to the modulefile author to write the appropriate test after all variant declaration
    • and then raise error if variant combination is not appropriate
    • as there is no way to automatically determine when all variants are declared
Evaluating

  • when reaching variant command in modulefile

  • set a key name in array ModuleVariant if variant name has been specified or if it owns a default value

    • variable ModuleVariant($name) is not defined if variant not specified and no default for it

      • error is raised if used in this case

  • if variant name can be aliased, if alias is set it specifies the value of variant name

    • evaluated from left to right, in case variant is set and also its aliases
    • negating aliases sets its reversed value to Boolean variant

  • variant alias is not instantiated in ModuleVariant array

    • therefore accessing $ModuleVariant($aliasname) will raise an error

  • raise error if variant name:

    • has been specified but passed value is incorrect
    • is wrongly declared in modulefile

  • then variable $ModuleVariant(name) could be used to adapt modulefile evaluation

  • to know all variant currently set, use [array names ModuleVariant]

    • but beware that only the variant already evaluated will be set in array
    • must use after all the variant to be set in the modulefile to accurately get all variant defined

  • if variant accepted values are Booleans, variable could be directly used in conditional tests

    • like if {$ModuleVariant($name)} {...

  • quoting Tcl doc, Booleans could have many forms:

    • If string is any of 0, false, no, or off, then Tcl_GetBoolean stores a zero value at *boolPtr.
    • If string is any of 1, true, yes, or on, then 1 is stored at *boolPtr.
    • Any of these values may be abbreviated, and upper-case spellings are also acceptable

  • variants with a shortcut defined for them, are resolved to their full name and transmitted this way to the modulefile

    • therefore no entry in ModuleVariant array is made for the shortcut name

  • as special variant version will not be implemented on Modules 4.8, an error is raised if a variant is named version

    • to ensure no usage of this variant name will be made before its special mechanism availability

  • special variant version is instanciated as variable ModuleVariant(version) if declared

    • will be set to value that may be specified with the @ shortcut

    • will initially work as any other variant (accepted list of value, default value)

      • using variant version --default 1.8 1.8 1.10
      • is equivalent of having two modulefiles mod/1.8 and mod/1.10
      • with a default defined: module-version mod/1.8 default

    • FUTURE: could be refined later on to accept range then select latest version or defined default if in range

    • note that ModuleVariant(version) is set only if a version variant is declared in modulefile

    • querying [module-info name] returns modulefile designation appended by the version variant: @versionvalue

  • at the end of the modulefile evaluation, if module specification contains a variant which is not defined in modulefile

    • an error is raised

  • for the different evaluation modes:

    • load, display, help and test applies evaluation mechanism described above

      • for the display, help and test modes it helps to see how the modulefile reacts to the different variant value
      • on display mode, the variant command is reported in addition of its evaluation

    • unload evaluates the mechanism described above but the specified variants are fetched from loaded module persistent information

      • version variant value is fetched from the module specification to identify the module to unload

      • other variant specification on the unload command are dropped once matching loaded module is identified

        • however this specification remains available when querying [module-info specified]

      • variant values are defined within modulefile context by the evaluation of the variant modulefile commands

        • like it is done during a load modulefile evaluation
        • this way variables related to variant are instantiated the same way whether module is loading or unloading
        • so it is easier for modulefile author to understand how the modulefile code is evaluated

      • variant evaluated during modulefile unload may produce an error

        • if variant value not found defined or if value recorded in persistency does not match an allowed value
        • should encourage variants to be consistent between the load and unload evaluation phases

    • whatis ignores all variants from the module specification (only the module names are retained)

      • like for setenv or *-path modulefile commands on this mode, variant will set the ModuleVariant array with a empty string for each variant defined in the modulefile
      • this is done to avoid the undefined variable error if these variables are used later in the modulefile
      • FUTURE: if the different version variant values are considered as different modulefiles in the future, then whatis may evaluates the version variant from module specification
getvariant

  • A getvariant modulefile command is added following the same principle than getenv

    • A variant name is passed as argument
    • Corresponding variant value is returned if it is defined
    • If not defined the value if not defined argument is returned
    • By default value if not defined argument equals to the empty string

  • getvariant should be preferred to accessing variant value in ModuleVariant array

  • On display evaluation mode, getvariant returns the variant name enclosed in curly braces

    • Which helps to report where the variant is used in environment variable content
    • The variant name and enclosing curly braces are graphically enhanced with va SGR code
Persistency

  • once a module is loaded its defined variants are made persistent in user environment

    • to keep track of how loaded modules were evaluated
    • and enable query commands on these loaded modules without need to reevaluate modulefile

  • variants defined are made persistent in MODULES_LMVARIANT environment variable

    • following same approach than for MODULES_LMPREREQ

    • each loaded module with defined variants (default value or specifically set) will expose:

      • these variants value
      • and if the value is the default one and if this default was specifically asked
      • in a record with following syntax:
      • loadedmodule&boolvariantname1|isbooleanvariant|isdefaultvalue&variantname2|value2|value3...|isbooleanvariant|isdefaultvalue

    • for each variant it is recorded if the variant is a Boolean variant

      • which enables to compare value in a Boolean way
      • and to report variant value with the +variant or -variant syntax

    • for each variant it is recorded if the value set corresponds to the variant default value or not

      • such information is useful to save collection when pin version mechanism is disabled
      • on such setup the variant definition should not recorded in collection if this is the default value which is set

    • in addition to know if the variant value is default or not, it is recorded if the default value was:

      • specifically asked (isdefaultvalue=1)
      • or automatically set (isdefaultvalue=2)
      • this information will be useful in the FUTURE to determine if a variant may be automatically swapped by another

    • each loadedmodule record are joined in MODULES_LMVARIANT separated by : character

  • variant alias are also made persistent in MODULES_LMVARIANTALTNAME environment variable

    • each loaded module with defined variants (default value or specifically set) which could be aliased will expose their aliases in a record with following syntax

      • loadedmodule&variantname1|aliasname1|-aliasname2&variant2|aliasname3...

    • each loadedmodule record are joined in MODULES_LMVARIANTALTNAME separated by : character

  • when persistency information is corrupted

    • a missing or non Boolean isdefaultvalue information means variant value is not the default
    • a missing or non Boolean isbooleanvariant information means variant is not a Boolean variant
    • a non-Boolean value set on a Boolean variant means variant equals 0

  • Boolean variants are stored in the form +name or -name

    • which enables to determine this variant is of Boolean type

    • and check against query using different Boolean representations

      • like serial=0, serial=on, serial=false, etc.

  • when the special variant version is defined for a loaded module

    • the value of this variant is part of loaded module identification

    • @versionvalue is appended to the module name, for instance mod@1.2

    • such identification is then recorded in persistency variables to designate loaded module (like LOADEDMODULES, MODULES_LMPREREQ, MODULES_LMVARIANT, etc)

    • this way in case a modulefile allows the load of two of its versions in the user environment, it is possible to distinguish these two loaded versions (to unload one of them for instance)

    • with this identification, it is possible to distinguish a traditional module (identified by mod/version) from a module using version variant (identified by mod@version)

    • note that if a modulefile mod/1.2 defines a version variant, it will be identified as mod/1.2@versionvalue

      • so the version variant should not be defined if each version of the module has its own modulefile
      • version variant is useful if a single modulefile is used to instantiate every version of the module

    • FUTURE: when it will be possible to override the shortcut for version variant it will be important to identify version value in loaded module identification string with a designation that is not dependent of the selected shortcut

  • loaded module identification stops at the module name and version variant (if defined)

    • other variants are not considered as part of the identification
    • as it is not foreseen useful to have the same module loaded multiple times with different variant values, unless for version variant
Specifying

  • following Spack spec

  • same grammar used whatever the context

    • command-line or as argument to modulefile command (like command)

  • variants are specified whether

    • as specific words (separated by " ")
    • or as suffix to module name

  • change command specifications which were previously accepting list of modules

    • like module1 module2 module3

    • now these modules could express variants appended to their name

      • like module1@1.8+debug module2~shared module3

    • or these variants could be defined as words next to module name

      • like module1@1.8 +debug module2 shared=false module3

    • as a consequence it denies:

      • use of +, ~ and = in module names
      • and use of - as first character of module names

    • also a command-line argument starting with the - character is not anymore considered as an invalid option

      • it is considered as an element of the module specification (potential negated boolean variant)
      • unless if set prior the sub-command designation
      • or set on sub-commands that do not accept module specification as argument

    • such change requires an option to be enabled to avoid breaking compat

      • this is why to enable variant, the advanced_version_spec option has to be enabled

  • a valued-variant is specified by name=value

    • this kind of variant cannot be directly appended to module name
    • thus it must be specified as a separate word

  • a Boolean variant can be specified with its bare name prefixed by +, - or ~

    • when directly appended to module name string (no space) only + and ~ are recognized

      • - in this case is retained as part of previous name/value

    • the negation prefix - is not supported on the ml command

      • as this - prefix means to unload a module on this command

    • negation prefix plus Boolean variant name should not equal a command-line option short form

      • command-line option takes precedence
      • for instance the -t will always be treated as --terse and not the negation of a t variant

    • beware that the negation prefix ~ when used as a separate word may trigger the tilda resolution of the currently running shell

      • if a username matches a Boolean variant name, using the ~name form on the shell command-line will leads to the resolution of the HOME directory path of user name

    • module name could end with one or more + characters

      • it could be distinguished from a Boolean variant specification as no character should follow these trailing +

    • Boolean variant could also be specified using the name=value form

      • in which case, it should be written as a separate word

      • value could be any syntax recognized as a true or false string

        • false: 0, false, no, or off
        • true: 1, true, yes, or on
        • Any of these values may be abbreviated, and upper-case spellings are also acceptable.

    • when specified Boolean value is translated to its canonical form (0 or 1)

  • variant may be specified with a shortcut if any set (see Variant shortcut)

    • a shortcut is appended to the module designation word or specified as separate word, combined or not with other variant

      • for instance for the @ shortcut: module@versspec+boolvar, module+boolvar@versspec, module +boolvar@versspec

    • even if a shortcut is set, the variant could also be expressed as valued variant name

  • in case variant is specified multiple times

    • lastly mentioned (read from left to right) value is retained (it overwrites previous values)

      • a merge all passed values in list is not the methodology retained here

        • same handling way whatever the variant properties

    • like name=value1 name=value2

      • or name=value name=value
      • or name=value1,value2 name=value3
      • or name=value1 name=value2,value3
      • or @vers1 version=vers2
      • or for boolean +name~name
      • or ~name -name
      • or ~name name=value1 name=value2,value3
      • or in case of variant aliases +name~aliastoname

    • at the specification time variant aliases are not known

      • so the full module specification has to be transmitted toward the modulefile to determine what is the value at the most right position
      • for instance name=value1 aliasname=value2 with aliasname being an alias of name
      • specification can just be cleared from the obvious duplicates (same variant name defined multiple times on the line)

  • when special characters like ? or * are used in variant name or value

    • they are treated literally, no wildcard meaning is applied

    • like currently done when specifying module version on command-line

      • which leads to errors as no corresponding module is found:

        $ module load loc_dv6/*
ERROR: Unable to locate a modulefile for 'loc_dv6/*'

  • when a variant is specified but it does not correspond to a variant defined in the evaluated modulefile

    • an error is raised at the end of modulefile evaluation
    • need to wait for the end of modulefile evaluation to be sure the variant is defined nowhere in modulefile code

  • special variant version has to be specified with @ shortcut or by its full variant name (version=value)

    • traditional separator character / cannot be used to specify variant version
    • if used, a modulefile named mod/version will be searched and a module not found error will be raised

  • specification may be passed to commands to verify a given module and variant combination is loaded

    • which should be performed without evaluating modulefiles

    • like for is-loaded sub-command:

      • module is-loaded hdf5+parallel
      • or hdf5@1.8 parallel=true
      • or hdf5 -serial
      • or hdf5 serial=0

    • checks rely on the content of the MODULES_LMVARIANT and MODULES_LMVARIANTALTNAME variables

      • which store variants set for loaded modules and eventual variant aliases of variant set

    • with this information it is possible to compare query against what is loaded

      • a variant specified on query which is not part of the variables means a different module/variant combination

        • even if variant from query is not valid for module, which cannot be known

    • verification mechanism of a sub-command like is-loaded should be preserved

      • which means a query not mentioning a specific value for a variant should match a loaded module which specify a variant value that differs from this variant default

    • the module identification part in specification may be resolved from a symbolic version or a module alias to an actual modulefile

  • the @loaded specification is translated into the name, version and variant list of corresponding loaded module

    • for instance mod/3.0 foo=val1 is loaded so mod@loaded is translated into mod/3.0 foo=val1
    • in case the @loaded specification is followed by variant specified, those variant specifications are ignored
    • following the above example, mod@loaded foo=val2 is translated into mod/3.0 foo=val1

  • variant can also be specified to designate module on module-tag, module-hide or module-forbid commands

    • tags may only apply when a given variant of a module is loaded
    • it may be useful to decommission a given variant of a module prior others
    • or to forbid the use of a given variant to some users

  • variants specified on search context are ignored

    • search context taking a module specification as argument only look for module name and version

      • no variant evaluation occurs on such context
      • it concerns the avail, whatis, is-avail, path and paths sub-commands

    • if variants are defined within module specification, they are not taken into account by search commands

    • for instance avail mod foo=var returns all versions of foo module whether they support the foo variant or not

    • FUTURE: may be revised if variants are evaluated on search context

  • variant cannot be specified over the module-alias, module-version, module-virtual commands

  • variant passed as argument to module-info alias, version or symbols will not match anything

  • module-info loaded only accepts modulefile as argument, not variant specification

    • it also only return loaded module name and version, without the variant set
Variant in requirement specification

  • prereq/conflict specification

    • could consolidate different variation set for same module on the same prereq/conflict list

      • to indicate a preferred order (if available)
      • like prereq netcdf -debug netcdf +debug
      • or prereq netcdf var=val1 netcdf var=val2 netcdf

    • in last example, could not currently consolidate definition into prereq netcdf var=val1,val2,default

      • in case of requirement alternatives, all possibilities should be written as independent

        • like prereq module@vers variant=val1 module@vers variant=val2

      • to clearly indicate a priority order to apply when for instance attempting to load these requirements

      • FUTURE: a value selection mechanism, like when selecting a module version among others, would help here

  • prereq/conflict persistency

    • MODULES_LMPREREQ and MODULES_LMCONFLICT content should reflect specified variant constraint

    • it could be expressed in these variables as it is specified over the prereq/conflict modulefile commands

      • for instance MODULES_LMPREREQ=hdf5/1.10&mpi@1.8 +shared variant=name&netcdf

      • use of characters `` , ``+, ~, , is not an issue

        • as delimiters characters in these environment variables are :, & and |
Variant shortcut

  • shortcuts can be set to abbreviate variant names and simplify their specification

    • a shortcut abbreviates name= into a unique character
    • when using shortcut, variant value is specified as <shortcut>value
    • for instance, if the % is set as the shortcut for a toolchain variant, value foss21a is specified as %foss21a

  • shortcut can be set through the variant_shortcut configuration option

    • this option holds a colon separated list of shortcut definitions
    • each definition have the following form: variantname=shortcut_character
    • for instance: toolchain=%:foo=^

  • shortcut must be:

    • a single character
    • excluding characters already used for other concerns or in module names (-, +, ~, /, @, =, [a-zA-Z0-9])
    • when set through config sub-command or --with-variant-shortcut installation option: an error is raised when a shortcut is badly specified
    • if a badly specified shortcut ends up in modulecmd configuration, it is purely ignored

  • shortcut does not apply to Boolean variants

    • as shortcuts are intended to be a prefix, they do not add benefit compared to -, + or ~ Boolean prefixes
    • however a shortcut could be defined on a boolean variant (e.g., %true or %0)

  • by default, when advanced_version_spec is enabled, the @ character is set as a shortcut for the version variant

    • this shortcut is not referred in MODULES_VARIANT_SHORTCUT thus it cannot be unset

    • FUTURE: superseding of this @ shortcut in MODULES_VARIANT_SHORTCUT may be introduced in the future

      • but currently entries in MODULES_VARIANT_SHORTCUT for version variant are ignored

  • variant shortcuts could be used on the command-line or within modulefiles even if it is not recommended to use them in the latter case

    • as if user updates the variant_shortcut configuration option, it will broke underlying modulefiles using a de-configured shortcuts

  • module designation in collection does not use variant shortcuts

    • when shortcut configuration is changed it should not impact collection definition
Reporting

  • Defined variants are reported on list sub-command results

    • Reported joined to modulefile name, within curly braces
    • Each variant definition separated by a colon
    • For instance module/version{vr=val:+boolvr}

  • Variants are reported by default on list sub-command

    • as they qualify what exact flavor of the module is loaded
    • so it enables users to really catch what has been loaded exactly
    • They can be removed from output using the output configuration mechanism

  • Variants defined by modulefiles are not reported currently on avail sub-command as it requires to evaluate the modulefiles

    • FUTURE: this could be implemented later on, but such evaluation of all modulefiles would be really expensive
    • it will take a lot more time to get avail results (unless a valid cache is found)

  • A specific color key is defined to highlight variants: va

  • Variant report depends on variant type

    • valued variant: variant=value
    • boolean variant: +variant or -variant
    • valued variant with shortcut set: <shortcut>value (for instance if % is a defined shortcut: %value)
    • in case a shortcut is defined over a Boolean variant, Boolean report prevails over shortcut

  • Variant aliases are not reported

    • to keep output tight

  • Special variant version is reported right after the module name

    • with @ shortcut
    • using variant highlight color if any
    • for instance: module@version{othervariant=value}

  • Variants are reported on the Loading, Unloading and Switching informational messages

    • As they qualify what exact flavor of the module is loaded, unloaded or switched

    • They are put along the module name and version designation

    • They are reported using their short form, like for list sub-command to keep output tight

    • Separated between each other by space character

    • Each variant specification is enclosed between single quotes if it contains a space character

    • The whole variant specification is enclosed between curly braces ({}) and separated from module name version by space character

      • Easier this way to distinguish variant specification from module name version on informational messages where multiple module designation are reported

    • These designations have to be recorded

      • prior module evaluation and based on variant specification (passed as argument) in order to be ready for any report prior the end of modulefile evaluation (in case of error for instance)
      • then refined after module evaluation with the variant accurately set in loaded environment

  • Variants are also reported along module name and version in the auto_handling informational messages

Recording collection

  • The variant specification set for loaded modules has to be recorded when saving it in a collection

    • Restoring such environment should apply the variant specified on the module to load
    • Lines like the following one could now be found in collection: module load hdf5@1.10 +parallel
    • Important to distinguish multiple modules specified on a single line from a module specified with variant defined

  • In case the collection_pin_version configuration option is disabled variant set to their default value should not be recorded in collection

    • Following the same scheme than for module version

    • When saving collection, the is-default-value information stored in persistency variable (MODULES_LMVARIANT) helps to know whether to value set to a variant is or not the default one

    • The save mechanism will rely on this information to exclude or not the variant specification in the generated collection output

    • Within this is-default-value hint, the was-this-default-specified-by-user sub-information is not preserved when saving collection

      • if collection is not pinned, default value is excluded whether it was specifically set by user or not
Comparing module specification including variants

  • When a module specification has to be compared on a non-loaded or non-loading modules context

    • If this specification contains variants

      • There is no variant set on non-loaded or non-loading modules we are comparing to

      • Specified variants are ignored, match is only performed over module name and version

      • Applies especially to the search commands taking a module specification as argument

        • no variant evaluation occurs on such context
        • Namely the avail, whatis, is-avail, path and paths sub-commands

    • If this specification does not contain variant

      • There is no variant set on non-loaded or non-loading modules we are comparing to
      • Match is performed over module name and version

  • When a module specification has to be compared against loaded or loading modules

    • If this specification contains variants

      • It should be matched against the variants set on loaded or loading modules
      • No variant set for loaded or loading module means no match

    • If this specification does not contain variant

      • Loaded or loading modules match is only made on their name
      • No comparison occurs over the variants set on loaded or loading modules

  • To compare variant set on loaded or loading modules

    • A ismodlo flag is added to the modEq procedure

    • With this flag it is known if modEq operates on a:

      • non-loaded or non-loading context (0),
      • loading context (1)
      • loaded context (2)

    • Variants set on loading or loaded modules will be fetched in case ismodlo is equal to 1 or 2

    • Loaded or loading modules are passed to modEq by their name/version designation

      • No change here
      • And no change required in all procedures that perform such comparison

    • Alternative names should also be tested like main module name with variants set

      • As the alternative names currently apply to module name and version only
      • Name and version could be compared on their own
      • Then variants could be compared
      • Which means all applying names are compared then if a match is found variants are compared

  • Specific comparison occurs when restoring collection

    • When a collection is restored an unspecified variant for a given module to load means this variant has to be set at its default value

    • So when comparing against loaded environment, an unspecified variant in collection only matches if variant set on loaded environment is the default one

    • Collection procedures now rely on the modEq procedure

      • ismodlo flag is set to 3 to indicate a collection context
      • This collection context leads to also compare simplified module names (in addition to alternative names)
      • And to treat an unspecified variant on tested pattern as a default variant value

  • There is no need to compare variants on following procedures

    • getLoadedWithClosestName

      • Find a loaded module whose name and version is the closest to passed specification
      • Variant specified on loaded modules or on specification is ignored here

    • modStartNb

      • Only used to compare module name and versions
      • Used by getLoadedWithClosestName and isModuleHidden

    • modEqStatic

      • Always used over non-loaded or non-loading context
      • Used by findModules and getModules

    • getEqArrayKey

    • cmdModuleSearch

    • cmdModuleSwitch

    • getModules

      • Used by cmdModuleAvail, getPathToModule, isStickynessReloading, cmdModulePaths, cmdModuleSearch and cmdModuleAliases

    • getPathToModule

      • Which calls to getModules
      • Used by cmdModulePath, cmdModuleSearch, cmdModuleSwitch, cmdModuleLoad, cmdModuleUnload, cmdModuleTest, cmdModuleHelp, getAllModuleResolvedName, is-avail, getSimplifiedLoadedModuleList and cmdModuleDisplay

    • getAllModuleResolvedName

  • Variant comparison is needed on following procedures

    • setModuleDependency

    • getUnmetDependentLoadedModuleList

    • getDirectDependentList

    • cmdModuleLoad

    • conflict

    • getLoadedMatchingName

    • doesModuleConflict

    • getModuleTag

      • Useful when a tag is defined only when a specific variant is set

    • collectModuleTag

      • Useful when a tag is defined only when a specific variant is set

    • getModuleHidingLevel

      • Useful when a module with a specific variant value set is defined hidden
      • FUTURE: if variants are reported on avail, hiding a variant specific value would have an effect on this sub-command

    • isModuleHidden

      • Useful when a module with a specific variant value set is defined hidden
      • FUTURE: if variants are reported on avail, hiding a variant specific value would have an effect on this sub-command
Specific impact

  • When loading a module with variant, if this module is already loaded but with a different variant set an error is raised

  • Tags applying specifically on variants do not appear over avail result

    • As variant are not treated on avail mode
    • However if a module is loaded and tags apply to the variant selected, these tags will appear on the module designation within avail result

  • Variant specification is ignored on avail and whatis sub-commands

    • If a forbidden tag applies to a specific module variant
    • If this variant is the one specified as argument to avail or whatis sub-command
    • The module will still be reported on avail or evaluated on whatis

  • Hiding a specific variant of a module will not hide the module from search results

    • As variant are not treated on search context like on avail sub-command

  • Sticky modules can be swapped by another sticky modules if the stickiness applies to the generic module name

    • It stays true even if module has variants
    • Which means if stickiness designate module version or several versions, sticky module cannot be changed once loaded
    • Variant change cannot either occur
    • FUTURE: this may be revised to allow variant change if needs arise

  • Stickiness can be defined over specific variant value, like any other tag

    • In case stickiness applies to the default value of a variant

    • When swapping sticky modules by another still matching the sticky rule

    • The variant with default value has to be explicitly specified for the swap to be allowed

      • As it cannot be guessed prior loading module that the default value of the variant will match the sticky rule
      • It applies to both sticky module swap context: restore and switch

  • On module-info tags, currently defined tags need to get fetched when called

    • As variant specified prior module-info tags call may change the list of tags that apply
    • Especially when a variant inherits its default value as it is not specified when loading module

  • module-info specified returns module name version and variants specified to designate the currently evaluating module

  • module-info name only returns module name and version, variants are not part of the result

    • Variants can be fetched in the ModuleVariant array within evaluation context
Corner cases

  • When loading a variant which is an optional requirement for an already loaded module

    • If this optional requirement is loaded without specifying its variant value to get the default variant value
    • Currently it is not seen as an optional requirement at the load time
    • Thus dependent module is not reloaded
    • FUTURE: the deps evaluation mode that will be triggered prior module load may fix this current limitation

License

Modules is distributed under the GNU General Public License, either version 2 or (at your option) any later version (GPL v2+).