Back to Cookbook

Apptainer/Singularity HPC Runtime Fixes

Resolve user-namespace and permission blockers without guesswork.

Troubleshoot the most common Apptainer/Singularity failures on HPC: user namespace disabled, fakeroot permission errors, and bind/cwd restore failures.

CommunitySubmitted by CommunityWork12 min

INGREDIENTS

🐙GitHub🔍Web

PROMPT

You are OpenClaw. Ask for the apptainer/singularity version, exact command line, and the HPC filesystem constraints (home/scratch, NFS, noexec). Then propose alternative run modes (without fakeroot), a minimal bind strategy, and validation steps. Provide a short message template the user can send to HPC admins if a policy change is required.

Pain point

Containers fail to start on HPC due to security policy and filesystem constraints (namespace disabled,

permission denied, failed to restore cwd, etc.).

Repro/diagnostic steps

  1. Capture exact error text and command (including --fakeroot, --nv, -B binds).
  2. Identify HPC policy constraints: setuid allowed? user namespaces enabled? filesystem mount options?

Root causes (common)

  • User namespace disabled by admins (common in HPC security posture).
  • fakeroot requires features not available on your system.
  • Bind mount targets not writable / cwd on restricted FS.

Fix workflow

  1. Prefer "non-fakeroot" workflows unless building images; bind only necessary paths.
  2. Use admin-supported container mode (setuid vs userns) per site policy.
  3. Relocate execution to a writable directory and bind scratch explicitly.

Expected result

  • Container runs with predictable binds, access to data, and correct GPU enablement where needed.

References

  • https://stackoverflow.com/questions/69375369/how-to-run-singularity-container-on-hpc-cluster-error-failed-to-create-user
  • https://stackoverflow.com/questions/73618551/error-failed-to-create-user-namespace-user-namespace-disabled-even-after-dis
  • https://github.com/apptainer/apptainer/issues/2769
Tags:#containers#hpc#apptainer#singularity