Continuous integration

This document describes the continuous integration (CI) set up for Modules: what each workflow tests, when it runs, and how to work with the coverage and dependency-update tooling that goes along with it.

All CI is implemented as GitHub Actions workflows, defined under .github/workflows/. Every workflow that builds and tests Modules follows the same overall shape: install system packages for the target platform, ./configure with a job-specific option combination, make, run the testsuite (see Testsuite structure), make install, re-run the testsuite against the installed tree, then make uninstall.

Overview

linux_tests.yaml

Build/testsuite across Tcl versions, configure option combinations and Linux distributions. Runs on push, pull request.

macos_tests.yaml

Build/testsuite on macOS. Runs on push, pull request.

windows_tests.yaml

Build/testsuite on native Windows, Cygwin and MSYS2. Runs on push, pull request.

lint_tests.yaml

Static analysis (Nagelfar, ShellCheck). Runs on push, pull request.

differential_shellcheck.yml

ShellCheck diff annotations on the changed lines only. Runs on pull request (to main).

easybuild_tests.yaml

Downstream integration: EasyBuild framework test suite against Modules. Runs on push, pull request.

scorecard.yml

OSSF Scorecard supply-chain security analysis. Runs on push (to main), weekly, manual.

All of the above live under .github/workflows/.

Every build/test workflow (all except differential_shellcheck.yml and scorecard.yml) triggers on push to any branch except c-main and c-3.2 (legacy imported-history branches that are not active development targets) and on every pull_request.

Build/test workflows

Linux (linux_tests.yaml)

The largest and most detailed workflow, made of one job per Tcl version/environment combination:

tcl86-nolibtclenvmodules, tcl85-nolibtclenvmodules

Tcl 8.6 and Tcl 8.5 (the latter built from source) with the optional C helper library (see lib/ in the repository layout) disabled via --disable-libtclenvmodules.

tcl85-2, tcl86, tcl85, tcl91, tcl90

Further Tcl 8.5/8.6/9.0/9.1 combinations (Tcl 9.0/9.1 built from source), each passing a different, deliberately varied set of ./configure options (pager, color, versioning, quarantine, icase, domain name detection, multilib, siteconfig, locked configs, ...). Spreading option coverage across jobs this way, rather than repeating the same configure line everywhere, is what exercises the option surface described in Add new configuration option; when adding a new option, consider whether an existing job's option set is the natural place to add it. tcl91 additionally checks out the repository into a working directory whose name contains + and other shell-special characters, to catch path-quoting regressions.

el8, el9

Full build, test, install and RPM/SRPM packaging inside rockylinux:8/almalinux:9 containers, exercising the RPM packaging path (make srpm rpm) that the other jobs don't cover.

Most jobs also set COVERAGE: y (see Coverage below) and run under xvfb-run where the testsuite needs an X11 display (X resource handling tests, see Testsuite structure).

macOS (macos_tests.yaml)

A single macos job on macos-15 (Apple Silicon), using Tcl/Tk installed via Homebrew. It additionally compiles the testutil-*.c helper libraries used by the testsuite as dylibs (clang -dynamiclib, universal arm64/arm64e) to exercise DYLD-based library insertion, the macOS equivalent of the LD_PRELOAD quarantine tests run elsewhere.

Windows (windows_tests.yaml)

Four jobs on windows-2022, covering the different ways Modules can run on Windows:

native-cmd

Native Windows Tcl (installed from the BAWT Tcl distribution), built via make dist-win and installed/tested/uninstalled through the generated INSTALL.bat/TESTINSTALL.bat/UNINSTALL.bat scripts, driven from a cmd shell.

native-pwsh

Same native Windows install, but through the PowerShell variant (INSTALL_PWSH.bat/TESTINSTALL_PWSH.ps1).

cygwin

Build and test inside a Cygwin environment, using the regular ./configure/make/ script/mt flow like the Unix workflows.

msys

Same, inside an MSYS2 MSYS environment.

Lint (lint_tests.yaml)

Runs the lint DejaGnu tool (Nagelfar for Tcl files, ShellCheck for sh/bash/ksh scripts) via script/mt lint, i.e. the same static analysis described as the lint tool in Testsuite structure. On failure it uploads lint.log as a build artifact.

Differential ShellCheck (differential_shellcheck.yml)

Runs only on pull requests targeting main (not on push). Unlike the full-repository ShellCheck run in the lint tool, this uses the redhat-plumbers-in-action/differential-shellcheck action to lint only the lines actually changed by the pull request and post results as inline PR review annotations and GitHub code-scanning alerts (requires the extra security-events: write and pull-requests: write permissions granted to this job only).

EasyBuild integration (easybuild_tests.yaml)

Builds and installs Modules, then clones the latest tagged release of the EasyBuild framework (a major downstream consumer of Modules) and runs its own test suite (python -m test.framework.suite) configured to use the just-built Modules as its modules tool. This is the one workflow that verifies Modules against a real downstream project rather than against its own testsuite, catching regressions the in-repo testsuite wouldn't see.

Scorecard analysis (scorecard.yml)

Runs OSSF Scorecard, a supply-chain security posture checker, on every push to main, weekly (Saturdays at 02:30 UTC), and on manual dispatch. Results are published to the public OSSF API and uploaded as a SARIF file to GitHub's Code scanning. This is unrelated to build/test coverage; it checks repository practices (branch protection, pinned dependencies, ...). See .github/security-insights.yml for how this feeds into the project's OpenSSF Best Practices self-assessment.

Coverage

Most build/test jobs set COVERAGE: y, which (as described in Testsuite structure) runs the testsuite against a Nagelfar-instrumented modulecmd.tcl and produces marked-up tcl/*.tcl_m files. After the testsuite runs, each job converts that coverage data (script/nglfar2ccov per Tcl file, plus gcov for the optional C helper library in lib/) and uploads it via the codecov/codecov-action. codecov.yml at the repository root configures Codecov to wait for 7 builds before notifying (so the PR check reflects combined coverage from all the Linux/macOS jobs that report it, not just the first one to finish) and to fail the project coverage status if the CI run itself failed.

Dependency updates

.github/dependabot.yml configures Dependabot to check the github-actions ecosystem (i.e. the uses: action references in .github/workflows/) weekly and open pull requests to bump pinned versions, keeping the actions used by all of the above workflows current.