Version control your Modulefiles using Git

The Modules concept is a great way to manage versions of tools on a common server, but managing the Modulefiles themselves is error-prone when not controlled using a SCM like Git.

Goals

  • To be able to create, edit, and test Modulefiles without the risk of breaking other users.

  • To be able to track changes to the system-wide Modulefiles using Git.

  • To enable testing of new tool versions (with their associated Modulefiles) before making the new version generally available (since the new Modulefile will not be public until pushed).

Assumptions

Details are tweakable via TCL variables

  • You have Environment Modules version 4.1 or later installed on your system, and Git version 2.4 or later.

  • You have a Unix user named modules that exists only for managing the Modulefiles, and controlling updates to them.

  • The path /home/modules/modulefiles is in your modulerc file:

    module use --append /home/modules/modulefiles

  • Other Unix users (not named modules) use the modules from /home/modules/modulefiles.

Principles of Operation

First we create a git repo for the Modulefiles.

Then we install a Modulefile named localmodules that, when loaded, switches MODULEPATH to a locally-created git clone of the Modulefiles. When unloaded, it switches MODULEPATH back to the default.

After this, any time a user wants to edit the Modulefiles, he works in his local git repo. After editing, testing, and committing to the local git repo, git push updates the main repository, which (assuming the user knows the password for user modules) automatically updates /home/modules/modulefiles.

Implementation

localmodules
#%Module4.1
##
## When loaded, use a local git repo for our Modulefiles,
## instead of the system-wide location that is typically used.
##
## This is useful for editing and testing Modulefiles.
##
## Tune to your system:
## $specialuser, $localmoddir, $globalmodfetch, $globalmodpush
##
## Author: Scott Johnson <scottjohnsoninsf@gmail.com>
##
## Originally developed at:
## github.com/scottj97/environment-modules-in-git


eval set [array get env HOME]

# Special Unix username that owns the main Modulefiles repository
set specialuser modules

# Where each user's local copy should go:
set localmoddir $HOME/modulefiles

# Where `git fetch` can find the main repository.
# This must be a filesystem path, since it will also be used by Modules:
set globalmodfetch /home/modules/modulefiles

# Where `git push` should write to the main repository:
set globalmodpush ssh://modules@localhost/home/modules/modulefiles



proc ModulesHelp {} {
   global localmoddir
   puts stderr "localmodules: switch to local Git repo for Modulefiles\n"
   puts stderr {This is useful for editing and testing Modulefiles.}
   puts stderr {Usage:}
   puts stderr {   $ module load localmodules}
   puts stderr "Then edit and test modulefiles in $localmoddir"
   puts stderr {When complete, git commit and push, then}
   puts stderr {   $ module unload localmodules}
}

module-whatis   {switch MODULEPATH to local git repo}

# Runs `system` and dies if error code is nonzero
proc safeSystem cmd {
   set retcode [system $cmd]
   if {[expr {$retcode != 0}]} {
      error "`$cmd` returned non-zero exit code: $retcode"
   }
}

# Make sure $localmoddir is what we expect, so we don't clobber
# anybody's work if they happen to have something unexpected here
proc ensureProperLocalmoddir {} {
   global localmoddir
   global globalmodfetch
   global globalmodpush

   # Make sure it's a directory
   if {![file isdirectory $localmoddir]} {
      error "a file named $localmoddir already exists, and\
         I don't want to clobber it"
   }

   # Make sure it has the expected .git remote setup
   if {![file isdirectory [file join $localmoddir .git]]} {
      error "expected git repo inside $localmoddir, found none"
   }
   safeSystem "if \[ `git -C $localmoddir remote get-url origin` !=\
      \"$globalmodfetch\" ]; then exit 1; fi"
   safeSystem "if \[ `git -C $localmoddir remote get-url --push origin` !=\
      \"$globalmodpush\" ]; then exit 1; fi"
}

# No $localmoddir exists, so run `git clone` to create
proc createLocalmoddir {} {
   global localmoddir
   global globalmodfetch
   global globalmodpush

   safeSystem "git clone $globalmodfetch $localmoddir"
   safeSystem "git -C $localmoddir remote set-url --push origin $globalmodpush"
}

proc checkRepoStatus {} {
   global localmoddir
   safeSystem "git -C $localmoddir remote update"
   safeSystem "git -C $localmoddir status"
}

# Special modules user not allowed to load or unload this.
# Why? Because it would defeat the purpose of tracking changes, if the
# changes were committed by this anonymous special user.
eval set [array get env USER]
if {$USER == $specialuser} {
   error "special user $specialuser should not load this module"
}

# create directory if necessary
if [module-info mode load] {
   if {![file exists $localmoddir]} {
      createLocalmoddir
   }
   ensureProperLocalmoddir
}

## These two work for load, but not for unload
## (the $globalmodfetch doesn't get put back in):
##   remove-path MODULEPATH $globalmodfetch
##   append-path MODULEPATH $localmoddir

## These two work for load, but not for unload
## (the $globalmodfetch doesn't get put back in):
##   module unuse $globalmodfetch
##   module use --append $localmoddir

## This method assumes that $MODULES_REPO is part
## of MODULEPATH, e.g. in your modulerc:
##   setenv MODULES_REPO /home/modules/modulefiles
##   module use /\$MODULES_REPO


if [module-info mode load] {
   setenv MODULES_REPO $localmoddir
}
if [module-info mode unload] {
   # unsetenv gets converted to setenv since we're unloading
   unsetenv MODULES_REPO $globalmodfetch
}

if [module-info mode load] {
   puts stderr "\nSwitched to local modulefiles in $localmoddir"
   puts stderr {When editing and testing complete, git commit and push, then:}
   puts stderr "   $ module unload localmodules\n"
   checkRepoStatus
}

Installation

First convert the existing Modulefiles into a git repo at /home/modules/modulefiles, as user modules:

cd /home/modules/modulefiles
git init
git add .
git commit -m 'Initial checkin of existing Modulefiles'
# Enable updates when receiving pushes:
git config --local receive.denyCurrentBranch updateInstead

Edit your global $MODULESHOME/init/modulerc file to use the new env var MODULES_REPO instead of hard-coding the path (requires Modules 4.1). Your modulerc should have:

setenv MODULES_REPO /home/modules/modulefiles
module use /\$MODULES_REPO

(The extra slash required before $MODULES_REPO is a bug to be fixed in 4.2.3.)

Copy the localmodules file from the Modules source tree to your repo:

cd /home/modules/modulefiles
curl --output localmodules https://raw.githubusercontent.com/cea-hpc/modules/main/doc/example/modulefiles-in-git/modulefiles/localmodules

Edit paths in the top of localmodules, if your installation differs from the assumptions, then:

git add localmodules
git commit -m 'Add localmodules from github.com/cea-hpc/modules'

Usage example

As a regular user (i.e. anyone but user modules):

module load localmodules
cd $HOME/modulefiles

Edit, test, then:

git commit -am 'Make some edits'
git push
module unload localmodules

After a successful push and unload, it is safe to delete your local $HOME/modulefiles directory if you wish.