Source shell script in modulefile

Configuration

• No specific configuration

Specification

• Execute a shell script and include the environment changes it performs in modulefile

  • Environment changes done by shell script are gathered and evaluated in modulefile context through corresponding modulefile commands

  • Goal is to get the same environment after loading a modulefile using a source-sh than if shell script targeted by the source-sh where directly sourced in shell session

• Environment changes tracked are (all environment elements module can handle):

  • Value set to variable: transformed into setenv modulefile command

  • Variable unset: transformed into unsetenv modulefile command

  • Path element added to variable: transformed into append-path or prepend-path modulefile command

  • Path element removed from variable: transformed into remove-path modulefile command

  • Current working directory change: transformed into cd modulefile command

  • Shell alias definition: transformed into set-alias

  • Shell alias unset: transformed into unset-alias

  • Shell function definition: transformed into set-function

  • Shell function unset: transformed into unset-function

  • Shell completion definition: transformed into complete

  • Shell completion unset: transformed into uncomplete

• Depending on modulefile evaluation mode, source-sh has different behaviors:

  • load, refresh and scan: execute script to gather its environment changes, transform them in modulefile commands and evaluate them

  • unload: undo environment changes made on load mode

  • display: execute script to gather its environment changes and report resulting command name and arguments for display

  • help, test and whatis: no operation

• Limitation: code in modulefile cannot rely on the environment changes done in script targeted by a source-sh command

  • For instance an environment variable set in shell script cannot be used to define another variable later in modulefile

  • This will work on load, unload and display modes, as script is run and/or analyzed and corresponding modulefile commands are evaluated in modulefile context

  • But it will not work on the other modes, as script is not run and analyzed there

    • To simplify processing as script need to be run and analyzed if not yet loaded, but if already loaded changes recorded in environment for tracking should be used instead

    • To avoid a negative impact on performances on the whatis global evaluation

• Result of source-sh command evaluation is stored into the environment when modulefile is loaded

  • To keep track of environment changes made by source-sh script evaluation

  • In order to undo these changes when unloading modulefile and report corresponding modulefile commands when displaying modulefile

  • Environment variable __MODULES_LMSOURCESH is used for this need

    • Using following syntax: mod&shell script arg|cmd|cmd|...&shell script|cmd:mod&shell script arg|cmd

    • Example value: foo/1&sh /tmp/source.sh|append-path PATH /tmp|cd /tmp

    • Characters used to structure information in __MODULES_LMSOURCESH (:, & and |) are escaped

      • Respectively to <EnvModEscPS>, <EnvModEscS1> and <EnvModEscS2>

      • If found in environment changes to record

    • Actual bodies of shell alias, shell functions and shell completions are not recorded in __MODULES_LMSOURCESH, an empty body is recorded instead

      • Example value: foo/1&sh /tmp/source.sh|set-alias alfoo {}|set-function funcfoo {}|complete bash foocmd {}

  • When unloading modulefile, content found for this modulefile in __MODULES_LMSOURCESH variable is evaluated to reverse environment changes

    • When reaching a source-sh modulefile command, recorded content is evaluated through a modulefile Tcl interpreter in unload mode, to get the reversed effect

  • When displaying modulefile

    • If it is loaded

      • the content found for this modulefile in __MODULES_LMSOURCESH variable is evaluated in display mode to report each resulting modulefile command

      • script is evaluated to fetch shell alias, function and completion definitions which are not recorded in __MODULES_LMSOURCESH

    • If not loaded, script is evaluated to gather environment changes and report each resulting modulefile command

• Script targeted by a source-sh command has to be run and environment prior this run and after this run have to be compared to determine the environment changes the script performs

  • The shell to use to run script has to be specified to source-sh

  • This shell will be run to execute the following sequence:

    1. output current environment state

    2. source the script with its args

    3. output environment state after script source

  • Script output is kept to return it in an error message in case its execution fails

  • This comparison determines the environment changes performed by script which are then translated into corresponding modulefile commands

  • Shell is run in a mode where neither its user nor its system-wide configuration is evaluated to avoid side effect

  • Shell needs current environment to correctly evaluate script

    • it seems desirable to run shell in an empty environment to get the full list of changes it does over the environment

    • but the script may need the environment to be defined to correctly operate

      • for instance it needs the PATH to be set to execute external commands

    • so shell run inherits current environment to ensure script will be properly evaluated

    • but as a consequence if an environment change performed by script is already done prior script run, this environment change will not be seen

• Limitation: a variable already set by another module will not be seen set sourced script

  • if this module is unloaded and if it does not trigger the unload of the modulefile using source-sh, variable will be unset as source-sh script has not increased variable reference counter

• Limitation: Script will not get access to shell aliases, functions or completions defined in interactive mode or shell profile configurations

• Note: environment change done through source-sh will not preserve change order occurring in sourced script, as all changes are analyzed after sourcing script

• Most shell will not get the full environment content when spawned as a sub-process

  • For instance shell aliases are not transmitted into sub-shells

  • As a result the environment prior script source will most of the time have no shell alias defined

• Note: shells may have specific behaviors

  • dash cannot pass arguments to the sourced script

  • ksh93 reads the full script prior sourcing it which may cause issue if a shell alias is defined in script then used in script

• Limitation: implementation does not currently support:

  • the zsh shell when used as sh or as ksh

  • the mksh shell

  • the BSD variant of the sh shell

  • the Windows cmd shell

• Note: if sourced script produces shell alias or function, these alias or function may not be compatible with the current shell of the user

• Shell completion is defined for the shell used to evaluate script

  • complete commands are generated even if current user shell is not the same as the shell used to evaluate script

  • but the completion definition will produce a no-operation on current user shell when complete command is evaluated

• Note: the mechanism described here only applies for shell script as to understand the impact the script execution over the user environment, this environment need to be compared prior and after execution

• source-sh modulefile command relies of the sh-to-mod procedure of modulecmd.tcl

  • sh-to-mod procedure handles the evaluation of the targeted script and comparison of the environment prior and after this comparison

  • sh-to-mod returns as a result the list of modulefile commands describing the environment changes made by the analyzed script

• sh-to-mod is a module sub-command, calling the sh-to-mod procedure and outputting resulting modulefile commands

  • with a #%Module header

  • to enable users to convert the environment changes made by script in modulefiles

• There is no need to also make sh-to-mod a modulefile command and source-sh a module sub-command

• Shell to use to source targeted script may be specified as a command name or as a path name

  • When specified as a command name (e.g., bash), command location will be determined based on currently set PATH

  • When specified as a path name (e.g., /bin/bash), this path name determines command location

• Prior executing shell and making it sourcing script:

  • Shell is checked to ensure an executable command matches it

  • Script is checked to ensure it exists

  • Those tests are done prior executing to avoid it if one check fails and provide a consistent error message whatever the shell used

• Environment changes to output as result should be enclosed and escaped

  • Enclosed if they contains space character (like white-space, tab and newline characters)

  • Escaped if they contains curly braces, as output is formatted as Tcl code, to avoid parsing issue

• If an error occurs during targeted script evaluation

  • Error is thrown which leads to either a modulefile evaluation error or a module sub-command error

  • Error and output messages reported by evaluated script will be reported along error by modulefile command or module sub-command

• To get prior and resulting environment state

  • env command is not used to gather environment state through exec source execution

    • it would simplify environment state parsing, as same command would be used for every shell

    • but it is an external command, so requires extra processing and an additional requirement

    • moreover it does not return shell functions in general, only exported Bash functions

  • Shell builtin commands are used to query existing environment variables, aliases, functions, completions and current working directory

    • which provides best processing efficiency

    • but leads to specific output parsing for each shell

  • A separator string %ModulesShToMod% is printed between each kind of environment item (variable, alias, etc) and also prior and after script evaluation

    • to separate each output kind and then be able to split them for separate analysis

• Path entries found in the newly prepended entries or in the newly appended entries for a variable are never de-duplicated: every occurrence is kept and passed on the corresponding prepend-path or append-path command

• The --duplicates option is added to a prepend-path or append-path command when one of the entries it adds:

  • is already found in the variable value prior script evaluation

  • is found several times in the newly prepended entries or in the newly appended entries

  • is found in both the newly prepended entries and the newly appended entries

  • Such an entry is genuinely meant to end up at this specific position in the resulting value

  • Without this option, prepend-path and append-path's default de-duplication behavior would silently absorb the entry, and the path_entry_reorder configuration option could relocate it, which would not correctly reproduce the change made by the script

• An environment variable equaling to the path separator character (:) prior script evaluation is considered as undefined prior script evaluation to avoid misleading analysis

• Path-like variable change is recorded as a setenv command instead of prepend-path and append-path commands when the newly prepended and newly appended parts would each need a different delimiter character to be expressed

  • Prior and after variable values are compared to find a common part between them: what is found before this common part is expressed as a prepend-path command and what is found after it as an append-path command

  • Each of these two commands may need its own delimiter character, determined from the character found in the after value right before (for the prepended part) or right after (for the appended part) the common part, when this character is not the variable path separator

  • When the delimiter character determined for the prepended part differs from the one determined for the appended part, describing the change as a prepend-path and an append-path command would not correctly reproduce the resulting value, as each command would independently split and de-duplicate the full variable value using its own delimiter

  • A setenv command directly setting the after value is generated instead in this situation, to always accurately reflect the change made by the script

• Environment variables made for Modules private use are filtered-out from the environment changes produced

  • LOADEDMODULES, _LMFILES_ and any variable prefixed by __MODULES_ are concerned

  • Changes relative to Modules state are ignored this way

  • If script loads in turn a modulefile, environment changes reported will not report the loaded module but only the environment changes it does

  • Modules configuration variable (prefixed by MODULES_) are still taken into account

• There may be several shell completion changes found for the same command on particular shell like fish

  • It produces multiple complete modulefile commands

  • But in __MODULES_LMSOURCESH variable, a single entry is recorded as a single uncomplete command should be generated when unloading modulefile

  • If completion definition is partial for a command when loading modulefile, on fish shell previous completion definition is cleared right before setting those defined in modulefile

  • When a completion definition difference is spotted after script evaluation, new completion definition is set which means previous definition is cleared (either intrinsically by shell or for fish due to the definition clear specifically added)

• Fish shell allows nested definition of shell functions

  • A separator string %ModulesSubShToMod% is printed between each function definition not to get main and nested functions mixed up

  • Such sub-separator is only applied on fish shell

• bash-eval shell is added in version 5.4

  • Instead of sourcing a bash shell script, execution output of this script is evaluated

• --ignore option is added in version 5.4

  • This option accepts a list of shell element separated by : character among:

    • envvar

    • alias

    • function

    • chdir

    • complete

  • Shell elements changed by sourced script and listed on --ignore option are ignored (no modulefile command relative to these changes are produced)

• FUTURE: this feature may be extended to translate environment changes made by tools like Spack, Lmod or pkg-config. It may provide this way bridges between different realms.