I’m naive about HPC, but it’s always surprised me a bit that HPC software seems to be a fairly distinct ecosystem. I’m not doubting it’s for good reasons, but why is a special package manager needed for this domain?
The big differentiator is really the degree to which people want to tune and customize their builds in HPC vs. other communities, and the diversity of hardware that needs to be supported. Things that stand out to me:
- different applications' needs to customize the builds of their dependencies
- e.g., one app might need HDF5 with MPI support, another might want it
built without. Those are two different, incompatible HDF5 builds.
- need for specific microarchitecture builds to take advantage of vectorization
- need for GPU support (for NVIDIA, AMD, *and* Intel GPUs)
- need to use sometimes vendor-specific tuned system libraries like MPI, cray-libsci, mol
- need for specific *versions* of dependencies (solvers, mesh libs, etc.) for
numerical reproducibility
- need to integrate across languages, e.g. C, C++, Fortran, Python, Lua, perl, R,
and dare I even say Yorick.
So I'd say Spack is not so much a special package manager as a much more general one. You can use it for combinatorial deployments at HPC centers, dev workflows for people dealing with multi-physics and other complex codes, and as sort of a distributed poly repo with lock files.
The intent was never to be specific to HPC, and I would love to see broader adoption outside this community.