Sync remote application directories on first load¶

Some workstations and laptops mount a network share holding all kind of software installations that may be used on this machine, alongside their own local, faster disk. Reading application files directly from this network share on every use works, but it is slower than reading them from local disk, and it stops working entirely when the machine goes offline.

This recipe describes how to keep modulefiles pointing at application directories on a network share mounted under /remote_apps, while transparently copying, with the rsync tool, the directory of a given application to /local_apps the first time the corresponding module is loaded. Every later load of the same module finds its application directory already synced locally and skips the copy.

Implementation¶

A .module_appdir_map file, located at the root of the /remote_apps network share, maps each module name and version to the basename of its application directory under /remote_apps:

/remote_apps/.module_appdir_map¶

foo/2.1 foo-2.1-build3

The .modulerc file at the root of the modulepath reads this map file and applies the remote tag, with the module-tag modulefile command, to every module listed in it whose application directory has not been synced locally yet -- tracked by the presence of a .<basename>.synced marker file under /local_apps:

modulefiles/.modulerc¶

#%Module5.7

# tag every module listed in the remote application directory map as
# 'remote', unless its application directory has already been synced to
# local disk (see siteconfig.tcl for how the sync itself is triggered)
set mapfile [file join /remote_apps .module_appdir_map]
if {[file readable $mapfile]} {
   set fid [open $mapfile r]
   set fdata [split [read $fid] "\n"]
   close $fid
   foreach fline $fdata {
      if {[llength $fline] != 2} {
         continue
      }
      lassign $fline modnamevr appdir
      set syncedfile [file join /local_apps ".$appdir.synced"]
      if {![file exists $syncedfile]} {
         module-tag remote $modnamevr
      }
   }
}

The actual sync is performed by a before-modulefile-eval hook, registered with the add-hook siteconfig command (see the Hook API design notes and the Hooks section of module(1) man page). The hook procedure returns immediately if the evaluation mode is not load, or if the module being evaluated is not tagged remote -- checked with module-info tags, run in the modulefile Tcl interpreter reached through getCurrentModfileInterpName, exactly as already documented for a hook procedure that needs to run modulefile commands. Otherwise, it looks up the application directory basename mapped to the module in .module_appdir_map, and skips the sync if this directory was already marked as synced by a previous load. Otherwise it copies the application directory with rsync and, on success, touches the .<basename>.synced marker file so later loads skip the copy.

Because the modulepath root .modulerc tags a module remote ahead of the sync actually happening, this tag would otherwise still be recorded as applying to the module once loaded, which is misleading once its application directory has just been synced locally. The remote tag is therefore added to the non_exportable_tags configuration option, introduced in Modules v5.7 together with the hook API, so it is dropped from the tag list persisted once a module is loaded, without affecting how it is reported beforehand, for instance on an avail listing. The remote tag is also given its own abbreviation and color, so modules whose application directory has not been synced yet stand out on an avail or spider listing. These module config calls are set in /etc/environment-modules/initrc, evaluated once when the module shell function initializes with autoinit, the only context module config is usable from within a file evaluated by Modules itself:

initrc¶

#%Module5.7

# give the 'remote' module tag its own abbreviation and color, and make
# sure it is not persisted onto a module once it gets loaded (see the "Sync
# remote application directories on first load" cookbook recipe)
module config tag_abbrev "+remote=R"
module config colors "+R=38;5;202"
module config tag_color_name +remote
module config non_exportable_tags +remote

The sync itself is triggered by the hook procedure, defined and registered in siteconfig.tcl:

siteconfig.tcl¶

#
# siteconfig.tcl - Site specific configuration script that copies, the first
#   time a module tagged 'remote' is loaded, its application directory from
#   a remote network share to local disk with rsync, so this and every later
#   load of the same module read from local disk instead of the network
#   share.
#
# Author: Xavier Delaruelle <xavier.delaruelle@cea.fr>
# Compatibility: Modules v5.7+
#
# Installation: put this file in the 'etc' directory of your Modules
#   installation. Refer to the "Modulecmd startup" section in the
#   module(1) man page to get this location.

# root of the mounted network share and of its local counterpart
set g_remoteAppDir /remote_apps
set g_localAppDir /local_apps

# return the application directory basename mapped to given bare module name
# and version, or an empty string if this module has no mapped directory
proc getAppDirBasename {modname} {
   set mapfile [file join $::g_remoteAppDir .module_appdir_map]
   if {![file readable $mapfile]} {
      return {}
   }
   set fid [open $mapfile r]
   set fdata [split [read $fid] "\n"]
   close $fid
   foreach fline $fdata {
      if {[llength $fline] == 2 && [lindex $fline 0] eq $modname} {
         return [lindex $fline 1]
      }
   }
   return {}
}

# copy application directory from the remote network share to local disk, on
# the first load of a module tagged 'remote' (see the modulepath root
# .modulerc file for how this tag gets applied)
proc syncRemoteAppDir {modfile modname modnamevr modspec mode requested} {
   if {$mode ne {load}} {
      return
   }
   set itrp [getCurrentModfileInterpName]
   if {![interp eval $itrp {module-info tags remote}]} {
      return
   }

   set appdir [getAppDirBasename $modname]
   if {$appdir eq {}} {
      return
   }

   set syncedfile [file join $::g_localAppDir ".$appdir.synced"]
   if {[file exists $syncedfile]} {
      return
   }

   report "Syncing '$appdir' application directory from remote share..."
   file mkdir $::g_localAppDir
   set srcdir [file join $::g_remoteAppDir $appdir]
   set destdir [file join $::g_localAppDir $appdir]
   if {[catch {exec rsync -a --delete $srcdir/ $destdir/} errMsg]} {
      reportError "Failed to sync '$appdir' from remote share\n$errMsg"
      return
   }

   # mark this application directory as synced so it does not get copied
   # again on a later load
   close [open $syncedfile w]
}
add-hook before-modulefile-eval syncRemoteAppDir

# vim:set tabstop=3 shiftwidth=3 expandtab autoindent:

Since the hook fires before the modulefile itself is evaluated, and the sync is performed with a blocking exec call, module load waits for the copy to complete before the modulefile that relies on the now-local application directory gets evaluated. A hook procedure cannot abort the evaluation it wraps, so a sync failure is reported as an error but does not prevent the module from loading afterward, even though its application directory may still be missing locally in that case.

Compatible with Modules v5.7+

Installation¶

Create site-specific configuration directory if it does not exist yet:

$ mkdir /etc/environment-modules

Copy the site-specific configuration script and initialization file of this recipe:

$ cp example/sync-remote-appdir/siteconfig.tcl /etc/environment-modules/
$ cp example/sync-remote-appdir/initrc /etc/environment-modules/

Note

Defined location for the site-specific configuration script may vary from one installation to another. To determine the expected location for this file on your setup, check the value of the siteconfig configuration option:

$ module config siteconfig

Adapt modulefiles/.modulerc to your modulepath, and copy it at its root, next to the modulefiles it applies to. Finally, create /remote_apps/.module_appdir_map on the network share, with one <module_name_and_version> <software_install_directory_basename> entry per line for each application directory that should be synced this way.

Usage example¶

The application directory of foo/2.1 has not been synced locally yet, so it shows up tagged remote on an avail listing, using the abbreviation and color configured for this tag:

$ module avail foo
--------------- /path/to/modulefiles ---------------
foo/2.1 <R>

Key:
<module-tag>  <R>=remote

Loading it triggers the sync, then proceeds with the load once the copy completes:

$ module load -v foo/2.1
Syncing 'foo-2.1-build3' application directory from remote share...
Loading foo/2.1

Once loaded, the module no longer carries the remote tag, since its application directory now lives on local disk:

$ module list
Currently Loaded Modulefiles:
 1) foo/2.1

A later load, after the module has been unloaded, finds the .foo-2.1-build3.synced marker file and skips the sync entirely, and the module no longer shows up tagged remote on avail either:

$ module unload foo/2.1
$ module avail foo
--------------- /path/to/modulefiles ---------------
foo/2.1
$ module load -v foo/2.1
Loading foo/2.1