Tips for Code Reuse in Modulefiles

Although Modules allow one to manage many packages and versions of packages on a system, managing the modulefiles themselves can become error prone, especially since the differences between modulefiles for different versions of the same package are often rather small. This can lead to issues with the consistency of the module definitions.

E.g., let us say you have a package foo, version 1.0 with a corresponding modulefile. A bit later, foo version 2.0 comes out, and you add a modulefile for that, likely copying the 1.0 version and editing a few lines. Then a bug is discovered in the version 1.0 modulefile, e.g. an environment variable that would be useful in some cases was omitted, or a typo in the help function. The version 1.0 modulefile is fixed, but did you remember to fix the 2.0 modulefile which, since it was created by copying the version 1.0 modulefile, likely has the same error?

This page includes some tips to reduce the amount of redundant code in your modulefiles (i.e. increase the amount of code reuse) so that they are easier to manage and more consistent.

A Simple Example

We return to our "foo" example. A fairly traditional set up would have a "foo" directory somewhere under your MODULEPATH with modulefiles 1.0 and 2.0 underneath it. Each of 1.0 and 2.0 will likely be largely the same, having basically the same help text, whatis, and variables set, with only some changes to some text and paths to reflect the version difference. Example code for this and other cases discussed in this document can be found in the "doc/example/tips-for-code-reuse" subdirectory of this package. The modulefiles for this simple example start in the "foo" subdirectory. For example, the version 1.0 modulefile might look something like

#%Module1.0
##
## foo

proc ModulesHelp { } {
   puts stderr "
FooA: A simple example of modulefile code reuse
Version 1.0

blah, blah, blah
"
}
module-whatis "foo version 1.0"

conflict foo
prepend-path PATH /software/foo/1.0/bin
prepend-path MANPATH /software/foo/1.0/share/man
prepend-path LD_LIBRARY_PATH /software/foo/1.0/lib

Although the above is simple enough, there is still much redundancy between the version 1.0 and 2.0 files, and we do not bother showing the file for 2.0 because it is so similar (only 5 chars change between the 1.0 and 2.0 versions, basically all the "1.0" strings become "2.0"). The command "module avail foo" shows versions 1.0 and 2.0, and loading either will set the environmental variables PATH, MANPATH, and LD_LIBRARY_PATH appropriately, conflict with itself (so only one version of foo can be loaded at a time), and define the help and whatis texts.

To refactor this to increase code reuse, we copy one of the original modulefiles to "common", and change all the references to the version number to a "$version" variable. To avoid confusion with the previous version, we change the package name from "foo" to "fooA", and the new versions can be found in the "fooA" subdirectory of "doc/example/tips-for-code-reuse". After adding some comments, etc, the common module becomes

#Common modulefile for fooA
#Expects the following variables to have been set
#   version: the version of fooA


proc ModulesHelp { } {
   global version
   puts stderr "
FooA: A simple example of modulefile code reuse
Version $version

This is a simple example of code reuse in modulefiles.
We have a couple versions of fooA, and the only differences
in what the modulefiles for the different fooA versions do
is that some paths include the version number.

"
}
module-whatis "fooA version $version"
    
conflict fooA
set rootdir /software/fooA/$version
prepend-path PATH $rootdir/bin
prepend-path MANPATH $rootdir/share/man
prepend-path LD_LIBRARY_PATH $rootdir/lib

We then create a pair of small stub files to replace "1.0" and "2.0". These simply just set the version variable appropriately, and then "source" the "common" file. For "1.0", the file would look like

#%Module1.0
set version 1.0

set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/common

A similar file would be created for "2.0", just changing the version number in the "set version" line. The "set moduledir" line sets the Tcl variable moduledir to contain the name of the directory in which the final modulefile (i.e. "1.0") is located, so we can find the "common" file.

The common file then handles all the work of defining the PATH, MANPATH, and LD_LIBRARY_PATH environmental variables, using the value of the Tcl variable "version" passed to it by the stub file. It also handles defining the help procedure and the whatis text, and the conflict with itself.

Note that the "common" file does NOT start with the Modules magic "#%Module1.0"; this will keep "common" from showing up in "module avail fooA". The "1.0" and "2.0" files do need to start with the magic "#%Module1.0" tag so the module command will "see" them. Because of this, the "module avail fooA" command will just show the 1.0 and 2.0 versions as expected.

The files for this example are in the "doc/example/tips-for-code-reuse/fooA" directory.

A Simple Example, revisited

The "fooA" case above is fairly common, and we can actually improve upon what we did above. To avoid confusion, we will repeat using the package name "fooB", and the files for this example will be in the "doc/example/tips-for-code-reuse/fooB" directory.

Using introspection in the modulefiles, we can get the version number of fooB from the name of the modulefile. So we add code to the top of "common" to default the version variable from the modulename.

We then copy the "1.0" stub modulefile to ".generic", and remove the line which sets the version. Because is starts with a leading ".", the generic modulefile will not be displayed in "module avail" and such. Since this generic modulefile is now version independent, we can replace "1.0" and "2.0" with symlinks to the .generic file.

The new common file looks like

#Common modulefile for fooB
#Expects the following variables to have been set
#   version: the version of fooB, defaults to last component of module tag

if [info exists version]==0 {
   set version [file tail [module-info version [module-info name] ] ]
}

proc ModulesHelp { } {
   global version
   puts stderr "
FooB: A simple example of modulefile code reuse, revisited
Version $version

This our second visit to simple example of code reuse in modulefiles.
Because the all of the differences between the different fooB versions
modulefiles is contained in the version number, we replace the stub
modulefiles for the two versions with symlinks to a generic version
file, and infer the version from the tag given to the module command.

The generic modulefile, .generic, does not appear in 'module avail' 
commands because of the leading period (.).  Indeed, this generic modulefile
does is not even 'bar' specific, and typically we actually put the file
in an utilities directory outside the MODULEPATH and symlink to it from
multiple packages.

"
}
module-whatis "fooB version $version"
    
conflict fooB
set rootdir /software/fooB/$version
prepend-path PATH $rootdir/bin
prepend-path MANPATH $rootdir/share/man
prepend-path LD_LIBRARY_PATH $rootdir/lib

The .generic stub file looks like

#%Module######################
##
## generic modulefile, 
##
## This just sources the file common in the same directory
##
## Usage:
## Just symlink this file to .generic in the directory with common, and
## then make symlink to .generic for all versions of the app

set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/common

Again, the common file handles setting the environmental variables PATH, MANPATH, and LD_LIBRARY_PATH, using the Tcl variable version which was defaulted from modulename. This works even in the case where the user does not specify the full module path, and the modulecmd defaults the version (e.g. if the user types "module load fooB" without specifying the version of fooB.) The common file also handles defining the help procedure and whatis text. When the command "module avail fooB" is issued, the common file does not get listed (because it does not start with the magic "#%Module1.0" tag), nor does the .generic file get listed (because it starts with a period (".")), but the two 1.0 and 2.0 symlinks do get listed, just as one wants.

This replaces a file for each version of fooB with one common file doing all the work, one generic file, and a bunch of symlinks. And the .generic file is not even specific to fooB. Indeed, at our site, we put the generic modulefile in an "utilities" directory outside of the MODULEPATH, and just symlink it to ".generic" in each application directory, and then symlink the versions to this symlinked ".generic". So we end up with one "common" file for each application, one shared generic modulefile, and a bunch of symlinks.

A More Complicated Example

Although the simple cases like the "foo" example above are not uncommon, many packages are more complicated. We now consider the fictitious "bar" package (example modulefiles in "example/tips-for-code-reuse/bar" directory). This is admittedly a contrived example, but it displays some of the ways in which we can modify the above simple cases to handle the more complicated needs of some packages. Our "bar" package has the follow characteristics:

  • We have two versions of "bar" built, "1.0" and "2.0"
  • For each bar version, we have builds for three different threading models: nothreads, pthreads, and openmp.
  • We expect users to use commands specifying both the bar version and the threading model, e.g. "module load fooB/1.0/pthreads" or "module load fooB/2.0/openmp"
  • The path to the installation directories include both the version number of "bar" and the threading model
  • The package requires the environmental variable BAR_LICENSE_FILE to be set appropriately
  • All builds of bar version 1.0 use one license file which is different than that used by bar version2.0
  • Bar 2.0 nothreads and openmp builds use the same license file, with a completely different name than that used by bar version 1.0.
  • The bar 2.0 pthreads build uses its very own license file with a very different filename than those used by bar version 1.0 or the other threading models of version 2.0.
  • The "nothreads" builds have a prerequisite on "fooB", with bar 1.0 wanting fooB version 1.1, and bar 2.0 wanting fooB version 3.2

Even with the above exceptions, there are still more similarities than differences between the various modulefile definitions. Like in the "fooA" case, we will have a "common" script that does almost all the work, and stub files for each of the variants, but in this case the stub files are expected to define some more variables. Our "common" file, which can be found in the "example/tips-for-code-reuse/bar" directory, looks like name "fooB", and the files for this example will be in the

#Common modulefile for bar
#Expects the following variables to have been set
#   version: the version of bar, 
#      defaults to last component of module tag
#   threadingmodel: one of 'nothreads', 'openmp' or 'pthreads'
#   licensefile: or will default based on version

if [info exists version]==0 {
   set version [file tail [module-info version [module-info name] ] ]
}

#Default to nothreads
if [info exists threadingmodel]==0 {
   set threadingmodel nothreads
}

#Default licensefile based on version
if [info exists licensefile]==0  {
   if [string equal $version 1.0] {
      set licensefile /somepath/to/version1/licenseFile.lic
   } else {
      set licensefile /a/completely/different/licenseFile.lic
   }
}

proc ModulesHelp { } {
   global version threadingmodel licensefile

   set threadstr $threadingmodel
   if [ string equal $threadingmodel nothreads ] {
      set threadstr {no threading support}
   }
   puts stderr "
Bar: Not so simple modulefile example
Version $version
Threading Model: $threadstr

This is a more complicated example of code reuse in modulefiles.
We have multiple versions of bar on the system, and each version
comes with multiple variants for different threading support modules.
Plus we assume that the env var BAR_LICENSE_FILE needs to be set
differently depending on the bar version.

Using license file: $licensefile

"
}
module-whatis "bar version $version with threading $threadingmodel"
    
conflict bar
set rootdir /software/bar/$version/$threadingmodel
prepend-path PATH $rootdir/bin
prepend-path MANPATH $rootdir/share/man
prepend-path LD_LIBRARY_PATH $rootdir/lib
setenv BAR_LICENSE_FILE $licensefile

Here we have extended the parameters which are allowed to be passed in from the build specific "stub" files; in addition to the version of "bar", the stub files are expected to have set the variables "threadingmodel" and optionally "licensefile". If "licensefile" was not set, the common script will default it based on the version of "bar". The rest of the "common" file is similar to the "fooA" case, except that we use both "version" and "threadingmodel" in setting the paths, and now define BAR_LICENSE_FILE based on "licensefile". Again, almost all the work is done in the common file, including setting the relevant environmental variables, defining the help procedure and whatis text, and preventing multiple versions of bar being loaded at the same time.

We now create stubfiles for each of the three threading models underneath directories for each bar version. These can all be found under the "example/tips-for-code-reuse/bar" directory. The one for bar 1.0 and openmp looks like

#%Module1.0
set version 1.0
set threadingmodel openmp

set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/../common

The pthreads version is almost identical, just setting threadingmodel to "pthreads" instead of "openmp". The "openmp" build for bar version 2.0 is also very similar, differing only in value "version" is set to. In all three cases, these stubfiles are basically the same as those used in the "fooA" example; we set a couple of Tcl variables (version and threadingmodel), and then source the common file which does all the work. The licensefile variable is left unset so the common file will default it correctly based on the bar version.

The stub file for the pthreads build for bar version 2.0 is as below:

#%Module1.0
set version 2.0
set threadingmodel pthreads

set licensefile /special/license/file/for/bar/2.0/with/pthreads/license.lic

set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/../common

This version is similar to the previous three, with the addition of explicitly setting the "licensefile" variable. This value will be used in the common file to set BAR_LICENSE_FILE, rather than the default which would have been set in the common file had no value been provided.

Finally, the two "nothreads" build also have a slightly different stub file than the other builds. The one for bar version 2.0 is shown below:

#%Module1.0
set version 2.0
set threadingmodel nothreads

prereq fooB/3.2

set moduledir [file dirname $ModulesCurrentModulefile]
source $moduledir/../common

Here we add a "prereq fooB/3.2" to require that the correct version of fooB was loaded. The bar version 1.0 stubfile is almost the same, just changing the setting of the version variable from 2.0 to 1.0, and changing the version of fooB needed as a prerequisite.

So instead of having six somewhat lengthy modulefiles (~40 lines each), one for each combination of bar version and threading model, we put almost all the work into a single, slightly longer (55 lines) common file, with six small (8 lines or less) stub files which mostly just set variables to define the behavior of the common file.

When customization is needed for a specific build, we showed two ways of accommodating such. First, for simple, more-or-less "one-off" cases, you can add some code to the appropriate "stubfiles" to handle the added complexity, like was done with the "prereq fooB" for the "nothreads" builds in the examples.

Alternatively, one can add some additional logic to the common script to accommodate the complexity, like was done with the license file differences between builds. This is the recommended way to handle complexity which is likely to recur in multiple places. Generally, this is done by:

  • adding a parameter to be passed from the stub file to the common file and setting appropriately in the stub files (e.g. licensefile in bar/2.0/pthreads)
  • adding logic to the common file to default it appropriately (e.g. licensefile for the other builds)

The above was only a simple example of what could be done, but it shows how modulefiles can leverage the fact that the modulefiles are evaluated in a full programming language to increase the amount of code re-use, which should reduce errors, improve consistency, and in general make things more manageable over the long run.

The tips given above should work with both 3.x and 4.x versions of the Tcl based environmental packages. They have been explicitly tested on versions 3.2.10 and 4.3.0.