Hide and forbid modules

module is often used to give access to hundreds of different software to a large variety of users, each of them provided in several versions. In such situation users ends up facing thousands of modulefiles which could be confusing.

On the other side, site's staff needs to decommission older software version to ensure for instance that only the best matches for their hardware resources remain. Such work needs specific care to avoid breaking the computing workflow of users.

This recipe provides examples on how to hide or/and forbid modulefiles to better cope with software life cycles and help both user and staff to get their bearings in large environments.

Implementation

The module-hide and module-forbid modulefile commands block respectively the visibility or the load evaluation of the modulefiles they target. These two commands have a variety of options to inhibit their action or display specific messages. Combining both commands enables to hide and forbid specified modulefiles.

Compatible with Modules v4.6+

Note

Modules v4.7+ is required to use the --hidden-loaded option of module-hide command.

Usage examples

The use cases below describe situations where hiding or/and forbidding use of modulefiles can be helpful.

The following set of modules will be used in the examples of this recipe. Each use case will progressively add module-hide and module-forbid statements in modulepath's .modulerc file to improve adapt module visibility or forbid access to them.

$ module avail
-- ../example/hide-and-forbid-modules/modulefiles --
appA/1.0  bioappA/1.0  chemappA/1.0  deplibA/1.0
appA/2.0  bioappA/2.0  chemappA/2.0  deplibA/2.0
appB/1.0  bioappB/1.0  chemappB/1.0  deplibB/1.0
appB/2.0  bioappB/2.0  chemappB/2.0  deplibB/2.0

Limiting view to the useful software only

With a large number of available modules it is interesting to reduce user's visibility to only report the modules of interest.

Some modules may require other to be loaded as dependency. This is the case for appA and appB which depend on deplibA or deplibB. These two dependency libraries are not interesting by themselves so they could be hidden by default but they should stay findable for appA or appB modules to load their dependency.

module-hide --soft could be used to address this need. Designated module will be hidden unless searched. Following lines are added to modulepath's .modulerc file

# hide modules only loaded as dependency
module-hide --soft deplibA
module-hide --soft --hidden-loaded deplibB

As a result deplibA and deplibB are not returned anymore on a global avail sub-command:

$ module avail
-- ../example/hide-and-forbid-modules/modulefiles --
appA/1.0  appB/2.0     bioappB/1.0   chemappA/2.0
appA/2.0  bioappA/1.0  bioappB/2.0   chemappB/1.0
appB/1.0  bioappA/2.0  chemappA/1.0  chemappB/2.0

Yet these modules are still found when loading the modules of the application requiring them:

$ module load appA
Loading appA/2.0
  Loading requirement: deplibA/2.0

In some cases it is desirable to hide such dependency modules also when they are loaded. This could be achieved by using the --hidden-loaded option of the module-hide command. Hidden loaded modules do not appear by default on list sub-command output unless --all is set. Queries like is-loaded still detect such modules as loaded even if hidden. In addition loading or unloading informational messages related to these modules are not reported unless a verbosity mode higher than verbose is configured.

$ module load appB
$ module list
Currently Loaded Modulefiles:
 1) appB/2.0
$ module list --all
Currently Loaded Modulefiles:
 1) deplibB/2.0   2) appB/2.0
$ module is-loaded deplibB
$ echo $?
0

Going further, among scientific applications some are only useful for a given scientific field. A site may provide many software covering many scientific fields but a user may only be concerned by one of these fields. In our example software are provided for biology users (bioappA and bioappB) and other software for chemistry users (chemappA and chemappB).

Say every biology users are part of a bio Unix group and every chemistry users are member of a chem Unix group. Available software visibility could be improved by only reporting the bio software to the bio users and the chem software to the chem users.

# hide modules not from user's scientific field
module-hide --soft --not-group bio bioappA bioappB
module-hide --soft --not-group chem chemappA chemappB

With the above statements put in modulepath's .modulerc file, the bio software are not seen anymore by chem users:

$ module avail
-- ../example/hide-and-forbid-modules/modulefiles --
appA/1.0  appB/1.0  chemappA/1.0  chemappB/1.0
appA/2.0  appB/2.0  chemappA/2.0  chemappB/2.0

However as they are softly hidden, these modules can still be seen if queried or if --all option is used:

$ module avail bioappA
-- ../example/hide-and-forbid-modules/modulefiles --
bioappA/1.0  bioappA/2.0
$ module avail --all
-- ../example/hide-and-forbid-modules/modulefiles --
appA/1.0  bioappA/1.0  chemappA/1.0  deplibA/1.0
appA/2.0  bioappA/2.0  chemappA/2.0  deplibA/2.0
appB/1.0  bioappB/1.0  chemappB/1.0  deplibB/1.0
appB/2.0  bioappB/2.0  chemappB/2.0  deplibB/2.0

Software requiring administrative actions prior usage

Some software may require an administrative step to get allowed to use them, like the signature of a user agreement. Such software should be reported among the available modules but should not be used prior the administrative step achieved.

In our example, the chemappA application requires a user charter to be signed and then sent to the site staff. The access to the chemappA is controlled by the chemappA Unix group: a user needs to be part of this group to access and use the application.

# forbid use of modules unless software user agreement is signed
set msg {User agreement for ChemAppA application must be validated to use it
1. Please connect to https://ChemAppA.example.org
2. Read the Term of Use and sign it
3. Send back this signed agreement to our service desk}
module-forbid --not-group chemappA --message $msg chemappA

With the above module-forbid statement put in modulepath's .modulerc file, the software is still visible but its load is denied unless if the user is part of the chemappA group.

$ module avail chemappA
-- ../example/hide-and-forbid-modules/modulefiles --
chemappA/1.0  chemappA/2.0
$ module load chemappA
ERROR: Access to module 'chemappA/2.0' is denied
  User agreement for ChemAppA application must be validated to use it
  1. Please connect to https://ChemAppA.example.org
  2. Read the Term of Use and sign it
  3. Send back this signed agreement to our service desk

A specific message is provided through the --message option to guide the user to complete the required administrative step. Once group membership is acquired, the module can be seamlessly loaded.

$ id --groups --name
chemappA chem
$ module load -v chemappA
Loading chemappA/2.0

Note

Do not forget to protect the access to the directory where the software is installed to really ensure that only allowed users can use it.

Software limited to particular users

Some applications may be restricted to a limited set of users. For instance because such application should not be disclosed or because it requires a license that is paid only by a few users.

In our example, the appC application works with token-based licenses. The only users that can use this software are those that have paid for a license token. Other users should not access nor even see the availability of this application. Users that have bought a license token are added to the appC Unix group.

# fully hide and forbid modules unless user owns a license token
module-hide --hard --not-group appC appC
set msg {Access is restricted to owners of license token}
module-forbid --not-group appC --message $msg appC

The above statements have been added in modulepath's .modulerc file. The module-hide --hard command is used to completely remove visibility for non-authorized users.

$ module avail
-- ../example/hide-and-forbid-modules/modulefiles --
appA/1.0  appB/1.0  chemappA/1.0  chemappB/1.0
appA/2.0  appB/2.0  chemappA/2.0  chemappB/2.0
$ module avail appC
$ module load appC
ERROR: Unable to locate a modulefile for 'appC'

The module-forbid statement added for appC helps to get a clear error message for the non-authorized users that are aware of the existence of the module name and version (instead of getting a modulefile location error).

$ module load appC/2.0
ERROR: Access to module 'appC/2.0' is denied
  Access is restricted to owners of license token

Alternatively such restrictions on modulefiles can be achieved by adapting file permission mode instead of adding statements in modulepath's .modulerc file.

$ chmod 640 ../example/hide-and-forbid-modules/modulefiles/appC/*
$ chgrp appC ../example/hide-and-forbid-modules/modulefiles/appC/*

However restricting file permission mode does not enable to authorize several Unix groups or users to access those modulefiles or to have specific error messages unlike when module-hide and module-forbid commands are used.

Note

Do not forget to protect the access to the directory where the software is installed to really ensure that only the authorized users can use it.

Software life cycle

When providing a new software version, it may be interesting to have a test phase, for instance during one week, to make this new version checked by some some pilot users prior the general availability.

In our example, we are currently November 13th and chempappB/2.0 has been installed two days ago. A test phase is ongoing and users that are member of the pilot Unix group can already access this software prior its general availability set for November 18th.

# test new version of chemappB prior general availability
module-hide --before 2020-11-18 --not-group pilot chemappB/2.0

With the above statement added to modulepath's .modulerc file, common users will not see the new module until November 18th.

$ date
Fri 13 Nov 2020 02:04:21 PM CET
$ module avail chemappB
-- ../example/hide-and-forbid-modules/modulefiles --
chemappB/1.0

At some point software need to be decommissioned to ensure for instance that buggy or under-optimized versions are not used anymore by users.

In our example, appA/1.0 needs to be removed as a newer version is available for appA and this new version fixes a lot of issues that were encountered with version 1.0. Decommission date for appA/1.0 is planned on November 18th.

# decommission of old version of appA
set nearmsg {appA/1.0 will be decommissioned, please use appA/2.0}
set msg {appA/1.0 is decommissioned, please use appA/2.0}
module-hide --hard --after 2020-11-18 --not-group eol appA/1.0
module-forbid --after 2020-11-18 --not-group eol \
   --nearly-message $nearmsg --message $msg appA/1.0

The above statements added to modulepath's .modulerc file will remove visibility and access to the decommissioned module on November 18th. Only users part of the eol Unix group will still keep an access to the software (which is useful in case some users are unfortunately stuck on this specific version of appA). Specific messages are set to guide users toward selecting another version for appA.

$ module avail appA
-- ../example/hide-and-forbid-modules/modulefiles --
appA/1.0  appA/2.0
$ module load appA/1.0
Loading appA/1.0
  WARNING: Access to module will be denied starting '2020-11-18'
    appA/1.0 will be decommissioned, please use appA/2.0
  Loading requirement: deplibA/1.0

Prior decommission date, module will still be visible and loadable. However when loading the module a warning message will appear when the expiry date will be close. The nearly_forbidden_days module configuration defines starting when such warning message should be reported prior decommission date (14 days by default).

$ module config nearly_forbidden_days
Modules Release 4.6.0 (2020-09-16)

- Config. name ---------.- Value (set by if default overridden) ---------------
nearly_forbidden_days     14

Once the decommissioned date is over, appA/1.0 has disappeared unless for users member of the eol Unix group.

$ date
Thu 19 Nov 2020 02:46:27 PM CET
$ module avail appA
-- ../example/hide-and-forbid-modules/modulefiles --
appA/2.0
$ module load appA/1.0
ERROR: Access to module 'appA/1.0' is denied
  appA/1.0 is decommissioned, please use appA/2.0

Note

Do not forget to protect the access to the directory where the software is installed prior its general availability and after its decommissioned date.