Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Spack – scientific software package manager for supercomputers, Linux, and macOS

spack.io

Post reply on HN
Print view View original

71–80 of 107 posts

Re: Spack – scientific software package manager for supercomputers, Linux, and macOS

#71

Post by eviks »

tgamblin wrote: post #15

Original author here -- since I know that people click the comments to decide whether to read further, here's a (biased?) summary: Spack is aimed at end users, developers, and admins. It's used primarily in HPC at the moment but it would be great to see use outside of the HPC community. Spack supports: 1. Basic installation (spack install foo, foo@3.1.2, foo@3.1.2 +option, etc.) 2. Reproducible environments, via `spa…

Compared to homebrew, do prebuilt binaries have to be installed in a predefined path they were built in or is it also flexible?

Also, is there an indication/warning whether the package is prebuilt before you install?

Re: Spack – scientific software package manager for supercomputers, Linux, and macOS

#72

Post by tgamblin »

eviks wrote: post #71
tgamblin wrote: post #15

Original author here -- since I know that people click the comments to decide whether to read further, here's a (biased?) summary: Spack is aimed at end users, developers, and admins. It's used primarily in HPC at the moment but it would be great to see use outside of the HPC community. Spack supports: 1. Basic installation (spack install foo, foo@3.1.2, foo@3.1.2 +option, etc.) 2. Reproducible environments, via `spa…

Compared to homebrew, do prebuilt binaries have to be installed in a predefined path they were built in or is it also flexible? Also, is there an indication/warning whether the package is prebuilt before you install?

> do prebuilt binaries rerouted to be installed in a predefined path they were built in or is it also flexible?

Prebuilt binaries don't have to go in a fixed prefix -- we'll relocate RPATHs, shebangs, strings in text files, and (as long as you build in a longer path than you install to -- see `padded_length` at https://spack.readthedocs.io/en/latest/config_yaml.html) strings in binaries.

There is not currently a way to see before you install if an install will be from binary or from source, but that is something we are working on.

Re: Spack – scientific software package manager for supercomputers, Linux, and macOS

#73

Post by pxc »

tgamblin wrote: post #36
yjftsjthsd-h wrote: post #34

Earlier quoted context omitted.

Very neat:) Given the similarity in features, could you comment on how it compares to nix?

Main differences with Nix: * Spack has a dependency solver (concretizer). nixpkgs is managed by humans who do the dependency solving for you over time. What this means practically is that you can install 5 versions of some package with different dependencies and flag combinations with a few commands in Spack, while with Nix you would probably need to check out different commits of the nixpkgs repo and maybe do some h…

> with Nix you would probably need to check out different commits of the nixpkgs repo and maybe do some hacking on nix derivations to get the same thing. Spack is fundamentally designed to help with combinatorics.

Typically you'd use the overlay mechanism combined with package overrides for this, so you would have to write some Nix code, but you can do it out-of-tree (the most common location of choice would be inline in your configuration repo).

One of the usability issues with this is that the APIs for overriding dependencies and flags in Nix packages are sometimes a bit language ecosystem-specific. The uniformity of Spack's approach strikes me as an even bigger win over Nixpkgs here than the fact that you can append those tweaks directly to CLI invocations without saving any code to a file. :)

> Spack installation hashes are what Nix would call full configuration hashes -- they're configuration metadata hashes, not content hashes of the installation. So you could say we're not quite as committed to exact binary reproducibility, but you could also say that this allows us to support relocatable binaries (which Nix does not).

Nix supports content-addressed store paths but it's still not very widely used yet, and it's opt-in. A normal system with the feature enabled will have a mix of configuration-addressed and content-addressed paths.

Spack looks really, really cool. Thank you for advancing the state of the art. :D

I hope Spack's package collection is seeing and will continue to see exponential growth like Nixpkgs and Guix are! It seems like the stronger design fundamentals in these newer package management ecosystems can really enable that.

Re: Spack – scientific software package manager for supercomputers, Linux, and macOS

#74

Post by bee_rider »

mellosouls wrote: post #57
Moissanite wrote: post #31

Earlier quoted context omitted.

For Americans who don't want to click the link: "Spack" is basically a milder British-English version of "retard". Imagine how well the "Spaz programming language" would do in your market.

It's certainly not milder.

FWIW “retard” is not very mild at all in US-English nowadays, it has been essentially removed from even mildly polite conversation (which is to say, if I’m hanging out with friends, and we’re all happily swearing away as one might with friends, we’d still avoid it, it is firmly in the “slur” category).

This is a moderately recent occurrence in the US; around here I want to say very strong pushback started to occur around… I dunno, 15 or so years ago?

Anyway, I’m not sure how that compares. But “less serious than retard” sets the ceiling pretty high, for the US audience.

Re: Spack – scientific software package manager for supercomputers, Linux, and macOS

#75

Post by eviks »

tgamblin wrote: post #72
eviks wrote: post #71

Earlier quoted context omitted.

Compared to homebrew, do prebuilt binaries have to be installed in a predefined path they were built in or is it also flexible? Also, is there an indication/warning whether the package is prebuilt before you install?

> do prebuilt binaries rerouted to be installed in a predefined path they were built in or is it also flexible? Prebuilt binaries don't have to go in a fixed prefix -- we'll relocate RPATHs, shebangs, strings in text files, and (as long as you build in a longer path than you install to -- see `padded_length` at https://spack.readthedocs.io/en/latest/config_yaml.html ) strings in binaries. There is not currently a way…

So how would I know whether the prebuilt binaries were built in a longer path than I install to?

Is there at least a way to block source builds if the app is not in the binary cache to avoid any surprises (basically have manual override/confirmation instead of automatic fallbacks to source builds)?

Re: Spack – scientific software package manager for supercomputers, Linux, and macOS

#76

Post by pxc »

331c8c71 wrote: post #35
yjftsjthsd-h wrote: post #34

Earlier quoted context omitted.

Very neat:) Given the similarity in features, could you comment on how it compares to nix?

+100 From what I saw the package definitions are in python. Given python is really an imperative language I don't see how spack could be comparably powerful for definitions and reuse.

After using it for ~8 years, I rather like Nixlang. At the same time, I don't know Python well or particularly like it, so Spack's DSL just doesn't look as comfy to me. It reminds me of RPM spec macros a little bit, which I guess are fine but I don't love them. I'd rather see Nix learn from Spack than myself jump ship.

But I think it would be very silly to dismiss Spack over something like the fact that their DSL is Python-based.

It's fine to be partial to what you know, and there are lots of good reasons to prefer Nix today, at well as to think that improving Nix might be better for you than adopting Spack.

But I think we should go a bit deeper than this in evaluating competing projects. Nix is about more than just being in some FP club, you know?

Re: Spack – scientific software package manager for supercomputers, Linux, and macOS

#77

Post by tgamblin »

eviks wrote: post #75
tgamblin wrote: post #72

Earlier quoted context omitted.

> do prebuilt binaries rerouted to be installed in a predefined path they were built in or is it also flexible? Prebuilt binaries don't have to go in a fixed prefix -- we'll relocate RPATHs, shebangs, strings in text files, and (as long as you build in a longer path than you install to -- see `padded_length` at https://spack.readthedocs.io/en/latest/config_yaml.html ) strings in binaries. There is not currently a way…

So how would I know whether the prebuilt binaries were built in a longer path than I install to? Is there at least a way to block source builds if the app is not in the binary cache to avoid any surprises (basically have manual override/confirmation instead of automatic fallbacks to source builds)?

> So how would I know whether the prebuilt binaries were built in a longer path than I install to?

All the public ones are built with very long padding; if you make your own you're currently on your own.

> Is there at least a way to block source builds if the app is not in the binary cache to avoid any surprises (basically have manual override/confirmation instead of automatic fallbacks to source builds)?

Yep:

    spack install --cache-only ...

Re: Spack – scientific software package manager for supercomputers, Linux, and macOS

#78

Post by eslaught »

I'm surprised that no one here has talked about the build times yet. As a new user, the first thing you notice out of the box is that even on a relatively beefy system, you spend ~2 hours getting to a working state, even when you're building a relatively modest set of packages. This is particularly surprising since one of the points of Spack is to integrate with the existing system software, and so much of what you're building already exists and works fine on the system.

In the past when I've talked to people about this, the answer I got was "well, set it up once for your team and don't reinstall it all the time". Which is kind of awful from a "cattle, not pets" perspective. Several other design decisions in Spack (like the lack of certain kinds of configuration isolation by default, and the general difficulty of configuring certain kinds of things) make it feel like a "pets" system. I've gotten instructions from people on using Spack to install things that have been impossible to follow because they fiddled with something in their environment and forgot to tell me.

The best thing I can say about Spack is that all the other systems are worse. I just wish they could get around to fixing some of this stuff already.

Re: Spack – scientific software package manager for supercomputers, Linux, and macOS

#79

Post by EntrePrescott »

eslaught wrote: post #78

I'm surprised that no one here has talked about the build times yet. As a new user, the first thing you notice out of the box is that even on a relatively beefy system, you spend ~2 hours getting to a working state, even when you're building a relatively modest set of packages. This is particularly surprising since one of the points of Spack is to integrate with the existing system software, and so much of what you'r…

> like the lack of certain kinds of configuration isolation by default

could you please explain a little what kind of "configuration isolation" you mean?

> and the general difficulty of configuring certain kinds of things

like what for example?

Re: Spack – scientific software package manager for supercomputers, Linux, and macOS

#80

Post by rekado »

eslaught wrote: post #78

I'm surprised that no one here has talked about the build times yet. As a new user, the first thing you notice out of the box is that even on a relatively beefy system, you spend ~2 hours getting to a working state, even when you're building a relatively modest set of packages. This is particularly surprising since one of the points of Spack is to integrate with the existing system software, and so much of what you'r…

Guix has a different take on this problem: it provides package transformations to modify the build graph on the command line (or in a Scheme file), which don't preclude the use of binary substitutes and reproducible builds.

All stuff is cached in /gnu/store so people won't have to rebuild packages, and with `guix publish` people can share built binaries with others (discovery via Avahi optional).

Post reply on HN