Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NixCon Live

2020.nixcon.org

Post reply on HN
Print view View original

21–30 of 50 posts

Re: NixCon Live

#21

Post by microtonal »

nikivi wrote: post #9
zhenyavinogrdov wrote: post #8

Earlier quoted context omitted.

The situation for installing haskell programs with nix is great, practically everything from hackage is available in the binary cache.

Is this harder to achieve for the Rust ecosystem (due to some properties of Haskell perhaps)? Or just no one put in the effort in doing it? Is it expensive to maintain all the binaries for all of Hackage?

Is this harder to achieve for the Rust ecosystem (due to some properties of Haskell perhaps)?

You can actually do this today. The buildRustCrate function in nixpkgs creates a derivation for a single Rust crate. You can combine this with crate2nix [1] to make derivations for a crate and all its dependencies in a Cargo.lock file. Every compiled crate will be a separate output in the Nix store and you can upload them to a binary cache such as Cachix.

This is what I have been using for CI in many of my Rust projects in combination with Cachix to get very fast builds.

[1] https://github.com/kolloch/crate2nix

Re: NixCon Live

#22

Post by catern »

the_duke wrote: post #20
arianvanp wrote: post #16

Earlier quoted context omitted.

I'm confused with your first point? > No concept of source packages. There is only a binary cache (hydra) and the derivations in nixpkgs. NixOS _Only_ has a concept of source packages. The cache is a (sound) optimisation om top of it. just disable the binary cache and you can build all packages from source.

Nix packages don't contain any source code. The package definition describes how to fetch the source code from a source (like a Git repo or a hosted archive) and build it. The built result only contains what is necessary at runtime. A sizeable amount of packages don't even fetch source code but a prebuilt binary which is then fixed up to work with Nix. There is a source cache, but it is optional. As an example, check…

Note that the source code is referenced by a hash, so it can't change without changing the package. Also, the source code of all packages built by Hydra is on cache.nixos.org alongside the resulting binaries.

Re: NixCon Live

#24

Post by the_duke »

catern wrote: post #22
the_duke wrote: post #20

Earlier quoted context omitted.

Nix packages don't contain any source code. The package definition describes how to fetch the source code from a source (like a Git repo or a hosted archive) and build it. The built result only contains what is necessary at runtime. A sizeable amount of packages don't even fetch source code but a prebuilt binary which is then fixed up to work with Nix. There is a source cache, but it is optional. As an example, check…

Note that the source code is referenced by a hash, so it can't change without changing the package. Also, the source code of all packages built by Hydra is on cache.nixos.org alongside the resulting binaries.

True, which is absolutely sufficient for most use cases.

I'm currently doing some work for ML and data science companies where full reproducibility and introspection is very much desired.

So you need to run your own source cache to provide that guarantee, because you can't count on cache.nixos.org still providing the source code from a package built 4 years ago.

But that's why I love the IPFS cache efforts. [1] Running your own node to pin all required sources should then be relatively easy.

[1] https://blog.ipfs.io/2020-09-08-nix-ipfs-milestone-1/

Re: NixCon Live

#25

Post by ryukafalz »

the_duke wrote: post #24
catern wrote: post #22

Earlier quoted context omitted.

Note that the source code is referenced by a hash, so it can't change without changing the package. Also, the source code of all packages built by Hydra is on cache.nixos.org alongside the resulting binaries.

True, which is absolutely sufficient for most use cases. I'm currently doing some work for ML and data science companies where full reproducibility and introspection is very much desired. So you need to run your own source cache to provide that guarantee, because you can't count on cache.nixos.org still providing the source code from a package built 4 years ago. But that's why I love the IPFS cache efforts. [1] Runni…

Software Heritage is also helpful here, and Guix is integrating with it - see e.g. https://guix.gnu.org/blog/2019/connecting-reproducible-deplo...

Re: NixCon Live

#26

Post by est31 »

catern wrote: post #22
the_duke wrote: post #20

Earlier quoted context omitted.

Nix packages don't contain any source code. The package definition describes how to fetch the source code from a source (like a Git repo or a hosted archive) and build it. The built result only contains what is necessary at runtime. A sizeable amount of packages don't even fetch source code but a prebuilt binary which is then fixed up to work with Nix. There is a source cache, but it is optional. As an example, check…

Note that the source code is referenced by a hash, so it can't change without changing the package. Also, the source code of all packages built by Hydra is on cache.nixos.org alongside the resulting binaries.

Are those sources target independent or specific for each new build of the package? That is, is there a new source code package on hydra when one of the dependencies changes? Or does it only change if the package itself changes?

Also, they are only available when hydra builds the package anyways, right? So if some package is not built by hydra (like how it used to be for the texlive packages), it'll still download the sources from the various places they are hosted.

As for the hash, it's good that the source code is hashed, but my main concern was that it was downloading from external sources in the first place. This is bad for privacy, as those hosts know I'm downloading from them, as well as for reliability, because the hosts might not have as good uptime as a debian package mirror.

Re: NixCon Live

#27

Post by est31 »

the_duke wrote: post #24
catern wrote: post #22

Earlier quoted context omitted.

Note that the source code is referenced by a hash, so it can't change without changing the package. Also, the source code of all packages built by Hydra is on cache.nixos.org alongside the resulting binaries.

True, which is absolutely sufficient for most use cases. I'm currently doing some work for ML and data science companies where full reproducibility and introspection is very much desired. So you need to run your own source cache to provide that guarantee, because you can't count on cache.nixos.org still providing the source code from a package built 4 years ago. But that's why I love the IPFS cache efforts. [1] Runni…

Debian's archive reaches back 15 years now: https://snapshot.debian.org/

It also contains source code.

Re: NixCon Live

#28

Post by microtonal »

est31 wrote: post #26
catern wrote: post #22

Earlier quoted context omitted.

Note that the source code is referenced by a hash, so it can't change without changing the package. Also, the source code of all packages built by Hydra is on cache.nixos.org alongside the resulting binaries.

Are those sources target independent or specific for each new build of the package? That is, is there a new source code package on hydra when one of the dependencies changes? Or does it only change if the package itself changes? Also, they are only available when hydra builds the package anyways, right? So if some package is not built by hydra (like how it used to be for the texlive packages), it'll still download th…

Are those sources target independent or specific for each new build of the package? That is, is there a new source code package on hydra when one of the dependencies changes? Or does it only change if the package itself changes?

Fixed-output derivations are used for sources (they are content-addressed in the store), so the latter.

Re: NixCon Live

#29

Post by wtetzner »

nikivi wrote: post #3

I hope Nix takes off for one reason. I don't want people to do same builds again and again especially for Rust it's painful. I want to install binaries with cargo instead of doing the same build again and again. Similar to https://cachix.org I don't honestly know why that isn't the case yet, perhaps storing binaries takes too much storage that Mozilla doesn't have money for. Or perhaps it can't do guarantees like the…

I think another good reason is that it lets you specify language-agnostic dependencies, including build tools etc. If Nix really takes off, we can stop dealing with language-specific package managers.
Post reply on HN