Ignore files in modulepath

When walking down the content of a modulepath directory to search for modulefiles, every file found is checked to determine if it is a modulefile and every sub-directory is walked down. Some modulepath directories contain files that are not modulefiles: helper scripts sourced by modulefiles or software installation files when modulefiles are mixed with the software trees they describe.

Currently the only way to make the module command skip such files is to give them a name starting with a dot character. This is not always possible or desirable and it does not avoid these files to be tested or read. When the amount of non-modulefiles is very large (millions of files), the cachebuild sub-command takes a huge amount of time and memory as an entry is recorded in the cache file for every invalid modulefile found (see issue #562 and the discussion on the modules-interest mailing-list linked there).

Objective is to let modulepath maintainers describe the content of their modulepath directory that should be ignored when searching for modulefiles. Ignored files and directories are treated as if they do not exist: they are not tested, not walked down, not reported and not recorded in cache files.

modulepath-ignore modulerc command

A new modulefile command is created: modulepath-ignore. It takes one or more file path patterns as arguments:

modulepath-ignore pattern1 [pattern2 ...]

An error is raised if no argument is provided.

This command is registered as a command alias in the modulerc Tcl interpreter (g_modrcAliases in execute-modulerc). It is also registered in the modulefile interpreter per-mode alias table (like module-hide), as global and user rc files are evaluated as modulefiles through the source sub-command mechanism. The command itself raises an error when evaluated from a file which is neither a rc file nor evaluated in the global/user rc context (tracked by the rc_running state), so a regular modulefile using it gets a not allowed in modulefile error. It is set no-operation on refresh and scan evaluation modes.

The command may be called several times: patterns accumulate in their definition order into a single ordered list.

The initial specification (issues #561 and #562) also proposed a modulepath-only command to switch from an accept all, ignore some logic to an ignore all, accept some logic. This second command is dropped: as pattern syntax follows the gitignore syntax (see below), its negated patterns already cover this need. For instance to only take .mod files into account:

modulepath-ignore * !*/ !*.mod

* ignores everything, !*/ re-includes directories so the walk down continues, !*.mod re-includes files with the .mod extension.

The use case reported on the mailing-list (helper Tcl scripts stored next to modulefiles) is solved with:

modulepath-ignore *.tcl

Pattern syntax

Patterns follow the exact same syntax and behavior as git with .gitignore files:

  • A pattern without slash (except an optional trailing one) matches files or directories at any depth below the directory of the rc file defining it. For instance *.tcl matches lib.tcl and mod/1.0/lib.tcl.

  • A pattern with a slash at its start or in its middle is anchored to the directory of the defining rc file. For instance mod/1* matches mod/1.0 but not other/mod/1.0.

  • A pattern ending with a slash only matches directories.

  • * matches anything except a slash. ? matches any single character except a slash. [...] matches one character from a range or class.

  • A leading **/ means match in any directory (**/foo equals foo). A trailing /** matches everything inside the designated directory. a/**/b matches with zero or more intermediate directories.

  • A pattern starting with ! negates a previous match: the element is re-included. To match a file name starting with a literal !, escape it with a backslash (\!).

  • When several patterns match an element, the last matching pattern decides.

  • It is not possible to re-include a file if one of its parent directories is ignored, as ignored directories are not walked down (same behavior as git).

As patterns are passed as Tcl command arguments and not read from a dedicated file, the .gitignore file-format rules (comment lines, blank lines, trailing space stripping) do not apply here.

Pattern anchoring and definition scope

modulepath-ignore is honored in .modulerc files located within a modulepath directory. Preferred location is the .modulerc file at the root of the modulepath, as this file determines how to walk through the modulepath content.

No specific restriction is set when the command is used in a .version file: it proceeds like it does in a .modulerc file, consistently with the other modulerc commands which are all accepted in .version files. The purpose of .version files is however to set the default version of a directory, so module documentation will designate .modulerc as the place where modulepath-ignore should be defined.

Each pattern is recorded along with its anchor directory, which is the directory of the rc file defining it. For instance modulepath-ignore 1* in /path/to/modpath/foo/.modulerc produces a pattern equivalent to /path/to/modpath/foo/1*. An element is matched against a pattern by first checking the pattern anchor directory is one of its parent directories, then matching its path relative to this anchor against the pattern.

Recording the anchor directory along the pattern means the modulepath root directory does not need to be known when the command is evaluated, and patterns naturally apply only to the sub-tree of the rc file defining them.

As rc files of parent directories are evaluated before those of deeper directories (see next section), patterns from deeper rc files are appended after those of parent directories. Combined with the last matching pattern decides rule, deeper rc files take precedence over parent ones, like git does with nested .gitignore files.

When modulepath-ignore is evaluated from a rc file not located within a modulepath directory (global rc, user rc), it registers global patterns. Such patterns have no anchor directory: they apply to every modulepath, matched against element paths relative to the root of the walked modulepath, as if they were defined in the .modulerc file at this root. As global and user rc files are evaluated at startup, before any modulepath walk, global patterns come first in the pattern list: patterns defined within a modulepath take precedence over them through the last matching pattern decides rule.

Evaluation of rc files during modulepath walk

Currently findModules collects the .modulerc and .version files found during the modulepath walk as modulerc entries in its result list, and getModules evaluates them afterward, in sorted order, during its consolidation phase.

This is too late for modulepath-ignore: patterns must be known when the walk occurs. Modulerc evaluation is thus moved to an earlier stage, in the findModulesFromDirsAndFiles procedure:

  • No ordering requirement is set on the element list returned by getFilesInDirectory: the rc files of a fetched directory are directly looked up by name in the element array built from this list, and the remaining elements are sorted in lexical order when inserted in the walked list.

  • findModulesFromDirsAndFiles evaluates the rc files of a directory as soon as its content is fetched, prior trimming the element list for the stop_mod optimization: their ignore patterns and hiding rules are in place before any sibling element of this directory is analyzed or processed. Once evaluated and recorded in the result list, these rc files are withdrawn from the element list: only the rc files that are not part of a fetched directory content flow through the walked list and get evaluated and recorded by the walk loop (modulepath root rc file from the initial listing, limited access rc files walked directly from a cache file listing).

  • The walk becomes depth-first: the content of a directory is inserted right after it in the walked list instead of being appended at its end. Module resolution and symbol definition results depend on the rc file evaluation order: a breadth-first walk would evaluate the rc files of all first-level directories before any deeper rc file, changing the resolution result when a rc file of one sub-tree sets a symbol like default on modules of another sub-tree. Root elements are walked in lexical order (sorted by findModules) to keep a deterministic evaluation order for the rc files of first-level directories. Deeper sibling directories are walked in lexical order too, as element paths are sorted when inserted in the walked list.

  • At the start of the walk, the .modulerc file at the root of the walked directory is evaluated if it exists and is readable, then every rc file already recorded in the result list transmitted to the walk procedure is evaluated in lexical order. This transmitted list is empty on a regular walk, but on cache use it holds the entries of the evaluated cache file: these rc files are not walked through, as they were readable when cache was built, but the patterns they may define apply to the walk down of the limited access elements. Containing this evaluation in findModulesFromDirsAndFiles keeps findModulesInCacheFile free of any rc evaluation logic.

  • On a no-indepth search, .version files at a different depth level than the search target are currently neither recorded nor evaluated (unlike .modulerc files, which have no such restriction). This behavior is kept when evaluation moves to walk time, so no extra .version evaluation occurs. As a consequence, patterns defined in a .version file lying at a different depth level than a no-indepth search target are not applied on such a search: one more reason to prefer defining patterns in .modulerc files.

The g_rcfilesSourced tracking array still guards against evaluating the same rc file twice. modulerc entries are still recorded in the result list of findModules: they are needed by the cache build process and to transmit cache file entries to the walk procedure. As every rc file is now evaluated by findModulesFromDirsAndFiles whether it comes from a directory walk or from a cache file, the consolidation phase of getModules does not evaluate rc files anymore: it just withdraws the modulerc entries from the result list.

Side benefits of this change:

  • rc files are now evaluated in walk order, which guarantees parent directory rc files are evaluated before deeper ones.

  • trimElemListForStopmod does not need to source rc files anymore for the stop_mod search optimization: the hiding and ignore rules it needs are in place as rc files are evaluated prior its call.

Impact on module cache

The cache build process (formatModuleCacheContent) relies on findModules to collect modulepath content: ignored elements are excluded from its results, so no modulefile-content, modulefile-invalid or directory-related entry is recorded for them in the generated .modulecache file. This solves the initial problem of issue #562: non-modulefiles matched by ignore patterns are neither read nor recorded, whatever their number, and cache size stays proportional to the number of actual modulefiles.

getLimitedAccessesInDirectory performs its own walk of the modulepath directory: it must also apply the ignore patterns so that ignored files or directories with limited access rights are not recorded as limited-access-file or limited-access-directory entries. Patterns are already known at this stage as this procedure is called after findModules in the cache build process.

The rc file exemption from pattern matching also applies to this walk: a limited access rc file matching an ignore pattern (which happens with the ignore all, accept some idiom, as * matches .modulerc itself) is still recorded as a limited access entry. Without this exemption, such a file would be skipped by this walk but still collected by the module search walk, where rc files are exempt: its content would then get plainly recorded in the cache file despite its restricted access.

No new cache file command is introduced and no existing command signature changes: the minimal Modules version advertised in the cache file header is unchanged.

When a cache file is used:

  • ignored elements are simply absent from it, so search and direct access behaviors are consistent with the no-cache case

  • limited access entries recorded in cache are re-walked through findModulesFromDirsAndFiles: ignore patterns are applied on this walk too. The rc files recorded in cache are evaluated at the start of this walk (their content is obtained from the evaluated cache file), so the patterns they may define apply to the walked down limited access elements.

As ignored elements are excluded from the cache file at build time, the ignore pattern definitions have to be static. If patterns are defined dynamically (e.g., depending on user, group or environment), elements ignored for the user building the cache are definitively absent from the cache file, whatever the dynamic conditions evaluate to for the user of the cache. This limitation must be clearly stated in the module documentation: modulepath-ignore definitions should not be conditional.

The same caution applies to global patterns defined in the user rc file: they are honored when this user builds a cache file, so elements they match are excluded from a cache file potentially shared with other users.

Disabling ignore patterns

A new boolean configuration option controls whether modulepath-ignore definitions are applied: modulepath_ignore. It is enabled by default and can be disabled through the MODULES_MODULEPATH_IGNORE environment variable, the config sub-command or the --no-modulepath-ignore command-line switch for one execution.

A show_ignored option name associated with a --ignored switch (git wording, as in git status --ignored) was first considered then rejected: disabling the mechanism does not show the ignored elements, it just stops ignoring them, and they may still not appear anywhere if they are not valid modulefiles in the end. The option is thus named after the mechanism it enables or disables.

When this option is disabled, modulepath-ignore commands still evaluate without error but the patterns they register are not applied: every element of the modulepath directories is walked through and tested as if no pattern were defined.

Disabling this option helps troubleshooting, for instance to determine whether a modulefile is not found because it is ignored. Note that a cache file built with patterns applied does not contain the ignored elements: combining with the --ignore-cache option is required to also search elements excluded from cache file. Similarly, running cachebuild with modulepath_ignore disabled records every element in the cache file, as if the feature was not used.

Interaction with existing mechanisms

  • ignored_dirs configuration option: this option globally lists directory names to ignore (version control directories by default). modulepath-ignore complements it on a per-modulepath basis with a much finer pattern syntax. The ignored_dirs check is integrated in the matcher procedure, as an early test made prior applying patterns: the behavior of this option is preserved, as negated patterns cannot re-include a matching directory and directories keep being ignored when the modulepath_ignore option is disabled.

  • Backup or version control files (*~, *,v, #*#): the walk procedures historically skipped these file names. This check is also integrated in the matcher procedure, with the same properties as the ignored_dirs one: negated patterns cannot re-include such files and they keep being ignored when the modulepath_ignore option is disabled.

  • Dot files: files whose name starts with a dot are hidden but still tested and recorded in cache. Ignore patterns match dot files like any other file and provide a stronger exclusion (no test, no record).

  • module-hide --hard: hard-hidden modules also behave as nonexistent for users, but their modulefiles are still tested for validity and recorded in cache files, and hiding is applied at query filtering time. modulepath-ignore acts earlier, at walk time, with no I/O on ignored elements. modulepath-ignore is meant for content that is not modulefiles, module-hide for actual modulefiles to withhold.

  • Virtual modules: a virtual module whose target file matches an ignore pattern is still resolved, as its target is directly read, not found through the modulepath walk.

Implementation notes

  • Patterns are compiled once, at definition time, into an ordered list of entries stored in a global structure (e.g., ::g_modulepathIgnorePatterns), each entry holding: anchor directory (empty for global patterns, meaning the root of the walked modulepath), compiled regular expression, negation flag, directory-only flag.

  • Translating a gitignore pattern into a Tcl regular expression handles the *, ?, [...], ** and anchoring rules. Tcl string match is not sufficient (* would match slashes, no ** support).

  • A matcher procedure (e.g., isPathIgnored modpath fpelt isdir) applies the last matching pattern decides rule and is called from the element loop of findModulesFromDirsAndFiles and from the walk loop of getLimitedAccessesInDirectory. The walked modulepath is passed to resolve the anchor of global patterns. The rc file exemption is implemented within this procedure and the procedure returns false straight away when the modulepath_ignore option is disabled.

  • Matching is pure string processing: for modulepaths with a large amount of non-modulefiles it replaces one file read per element (magic cookie check) by one in-memory pattern match, which is where the performance gain of this feature comes from.

  • Code must stay compatible with Tcl 8.5.

Impacted files

  • tcl/mfcmd.tcl: modulepath-ignore procedure and pattern compilation helper

  • tcl/interp.tcl.in: command alias registration in modulerc interpreter

  • tcl/init.tcl.in and every file listed in doc/source/devel/add-new-config-option.rst (shell completion scripts, man pages, config testsuite): modulepath_ignore configuration option and its --no-modulepath-ignore command-line switch

  • tcl/modfind.tcl.in: isPathIgnored matcher procedure, rc file evaluation moved into findModulesFromDirsAndFiles, ignore matching in walk loop, trimElemListForStopmod simplification

  • tcl/cache.tcl.in: ignore matching in getLimitedAccessesInDirectory walk

  • share/nagelfar/syntaxdb_modulerc.tcl: new command for lint sub-command

  • share/vim/syntax/modulefile.vim and share/emacs/lisp/modulefile-mode.el: syntax highlighting

  • doc/source/modulefile.rst: command reference (including the static definition recommendation), doc/source/changes.rst and NEWS.rst

  • testsuite: dedicated testfiles for search and direct access behaviors (modules.20-locate), listing sub-commands (modules.90-avail, modules.92-spider), cache build content and cache use (modules.30-cache), plus lint coverage of the new command