Apptainer/Singularity HPC Runtime Fixes

Pain point

Containers fail to start on HPC due to security policy and filesystem constraints (namespace disabled,

permission denied, failed to restore cwd, etc.).

Repro/diagnostic steps

1. Capture exact error text and command (including --fakeroot, --nv, -B binds).
2. Identify HPC policy constraints: setuid allowed? user namespaces enabled? filesystem mount options?

Root causes (common)

• User namespace disabled by admins (common in HPC security posture).
• fakeroot requires features not available on your system.
• Bind mount targets not writable / cwd on restricted FS.

Fix workflow

1. Prefer "non-fakeroot" workflows unless building images; bind only necessary paths.
2. Use admin-supported container mode (setuid vs userns) per site policy.
3. Relocate execution to a writable directory and bind scratch explicitly.

Expected result

• Container runs with predictable binds, access to data, and correct GPU enablement where needed.

References

• https://stackoverflow.com/questions/69375369/how-to-run-singularity-container-on-hpc-cluster-error-failed-to-create-user
• https://stackoverflow.com/questions/73618551/error-failed-to-create-user-namespace-user-namespace-disabled-even-after-dis
• https://github.com/apptainer/apptainer/issues/2769