Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nix × IPFS – Milestone 1

blog.ipfs.io

Post reply on HN
Print view View original

71–80 of 92 posts

Re: Nix × IPFS – Milestone 1

#71

Post by sterlind »

wesnel wrote: post #69
adsjhdashkj wrote: post #40

Tangent, what's the best way to use Nix-like features in a mainstream distro? Eg, i've been evaluating PopOS recently because i want a Mac equivalent OS. Ie, i don't want driver/config issues. However one problem i never see solved from any "normal" distros is reproducible systems. Hypothetically if i used Nix as a Desktop my config would be bulletproof. .. but then i'm going through a fair amount of work configuring…

Personally, I use NixOS in conjunction with Home Manager[0] to more conveniently use Nix to manage my user-level config. The configuration that I manage with Home Manager includes my Emacs config, my Git config, and my Bash config. Additionally, almost all of the other programs I use on a daily basis are at least installed through it. Since it’s primarily only used for managing some packages, the configuration proces…

I've found home-manager to be more trouble than it's worth for most things:

* Config files aren't always stored in text (e.g. KDE Plasma, Gnome)

* It's all-or-nothing usually, unless you can find a way to make the home-manager config your "base"

* Changing settings isn't integrated, and requires editing/rebuilding/reopening (e.g. I can't just hit Ctrl+ to increase font size in VS Code, I have to edit a config file, run home-manager switch, then usually reopen)

* Plugins, especially things like extension stores, are a hack and a half. You have to hunt for hashes, then change it every time your old version falls off the CDN.

It's a nice idea, in theory, but it only works in a vacuum.

Re: Nix × IPFS – Milestone 1

#72

Post by sterlind »

Ericson2314 wrote: post #65
teacpde wrote: post #50

Earlier quoted context omitted.

I understand why changing comments doesn’t matter, but how do you decide which dependencies don’t need to be rebuilt when certain part of the source code is changed?

Nix does this simply and stupidly --- if you can observ it, it's part of the cache key. This is wonderful because it's totally sound. All efficiency methods (basically, hiding things, as the other comment says) is left to the build plan itself. Switching to Nix for batch jobs is switching from DOS to an OS with actual process isolation. A complete paradigm shift.

It does still require trust, though. You can lie about the contents since the hash only covers the build inputs, so someone could break the sandboxing.

Of course, since builds are reproducible you can catch this if you build it yourself and compare the actual content-hashes.

Re: Nix × IPFS – Milestone 1

#73

Post by Ericson2314 »

Foxboron wrote: post #25
nmfisher wrote: post #2

Interesting use case for IPFS, which has often felt like an (admittedly cool) solution in search of a problem. Can someone enlighten me as to real-world examples where actually reproducible builds are critical?

Confusingly enough, when NixOS talks about reproducible builds they are not talking about https://reproducible-builds.org/ . They are talking about reproducible systems. NixOS doesn't solve reproducible builds as-in "bit-for-bit identical binary distributed files", but it does solve "same input" -> "functionally same system". This is the core idea of NixOS and what makes the package manager and the spawned ecosystem…

The floating content-addressed derivations we've worked on as part of this Nix × IPFS work get's us a lot closer to reproducible builds by making non-determinstic builds far more noticiable.

See these:

https://github.com/NixOS/rfcs/pull/62

https://github.com/NixOS/nix/issues/4087

Re: Nix × IPFS – Milestone 1

#74

Post by leppr »

shp0ngle wrote: post #67
imhoguy wrote: post #9

Open source plus distributed p2p file sharing is the killer combo. I don't get why public stuff like NPM, DEB or Docker registries haven't switched to use it as primary way of distribution. P2P, such as IPFS and recently BitTorrent 2.0 with its hash tree per file, is the only free (as beer and speech) reliable way to host things online forever - at least as long there is the last of the veterans who keeps a copy and…

BitTorrent 2.0 is not recent, no. The current bittorrent company is interested in chasing ICOs, I don’t think they work on new versions of the protocol anymore? edit: yeah it’s from 2008

Going on a tangent but this article that came out today is a good read for anyone that didn't follow what happened to BitTorrent the company: https://www.theverge.com/21459906/bittorrent-tron-acquisitio...

Re: Nix × IPFS – Milestone 1

#75

Post by whyrusleeping »

Ericson2314 wrote: post #63
georgyo wrote: post #48

Earlier quoted context omitted.

Say what you will about IPFS and ProtocolLabs, but I really admire that they made IPFS completely blockchain agnostic. There is no requirement to buy or receive coins to use IPFS. And multiple different implementations of persistent stores can use the same IPFS network to ensure durability. This means that FileCoin, other coins, static hosting providers, self-hosting can all coexist and strengthen the same network. E…

Yeah the layering in Protocol Lab's work is really nice. The tendency of funded / for-profit software is to eschew layering, so it's easy to try out but ultimately ill-fitting and inflexible. This is a core problem of capitalism with IT. IPFS, libp2p, IPLD, Filecoin etc. all resist that temptation, and I think it will help them greatly in the end.

I really appreciate hearing things like this :) We put a ton of work into making the layering of our projects just right so that they can be reused over and over even if other things we're working on fail. It often does make it slower for us to make progress (could take soooo many shortcuts if we tried to collapse the stack) but I really do believe it's worth the effort.

Re: Nix × IPFS – Milestone 1

#76

Post by Ericson2314 »

sterlind wrote: post #72
Ericson2314 wrote: post #65

Earlier quoted context omitted.

Nix does this simply and stupidly --- if you can observ it, it's part of the cache key. This is wonderful because it's totally sound. All efficiency methods (basically, hiding things, as the other comment says) is left to the build plan itself. Switching to Nix for batch jobs is switching from DOS to an OS with actual process isolation. A complete paradigm shift.

It does still require trust, though. You can lie about the contents since the hash only covers the build inputs, so someone could break the sandboxing. Of course, since builds are reproducible you can catch this if you build it yourself and compare the actual content-hashes.

I would say the trust issues with build remotes and substitutors are fairly independent of the caching policy?

The caching model is about when the drv changes ((drv, output) is the cache key, basically), the trust model is one one can download something to avoiding build to fill in the value for the new cache key.

Note also that with the new floating content-addressed derivations, the trust stuff is much easier to think about.

Re: Nix × IPFS – Milestone 1

#77

Post by Foxboron »

Ericson2314 wrote: post #73
Foxboron wrote: post #25

Earlier quoted context omitted.

Confusingly enough, when NixOS talks about reproducible builds they are not talking about https://reproducible-builds.org/ . They are talking about reproducible systems. NixOS doesn't solve reproducible builds as-in "bit-for-bit identical binary distributed files", but it does solve "same input" -> "functionally same system". This is the core idea of NixOS and what makes the package manager and the spawned ecosystem…

The floating content-addressed derivations we've worked on as part of this Nix × IPFS work get's us a lot closer to reproducible builds by making non-determinstic builds far more noticiable. See these: https://github.com/NixOS/rfcs/pull/62 https://github.com/NixOS/nix/issues/4087

Making it noticeable isn't going to get you closer to reproducible builds though. There is still a quite a lot of effort that needs to go into patching individual packages and fix the non-determinism introduced by the compilation process which NixOS can't solve by adding tricks.

And as an outsider I'm unable to read the RFC and understand what is going on and what tangible issues it solves for Nix.

Re: Nix × IPFS – Milestone 1

#78

Post by jononor »

arianvanp wrote: post #3

Does IPFS actually solve the problem they set out here though? IPFS is a distributed CDN; but not very good for storing things persistently or reliably from my experience. At the moment; the nixos cache is stored in very durable and reliable S3 storage; with very high durability guarantees. Why is that not good enough? sure it's centralised. But IPFS doesn't offer distributed durability; it offers a CDN. It doesn't s…

As a org supoporting a curated content-set, one could implement IPFS backed by S3 persistence (or some other cloud bucket). Possibly as an "open frontend" part of ones CDN infrastructure, where others can also freely "peer" and contribute CDN power. The benefits for package management I believe can be pretty good, given widespread deployment. At our office, or hackerspace there are many computers which are likely to have the packages. Though until these things are enabled by default (can we ever get there?) I suspect only large IT departments or very interested people will set it up, unfortunately.

Re: Nix × IPFS – Milestone 1

#79

Post by linsomniac »

takeda wrote: post #56
na85 wrote: post #31

Earlier quoted context omitted.

In a lot of the world, upload speeds are very limited and ISPs charge an arm and a leg if you exceed your upstream data caps. On top of that, Bit torrent has a tendency to overwhelm routers and make them slow to a crawl. Those are a lot of downsides to ask your users to accept just so you can cheap out on download servers.

Perhaps there are routers that don't handle well large amounts of data, but from my experience what typically is happening are two things: 1. the fairness in TCP is done per TCP connection, bittorrent opens many connections so they take over the whole bandwidth. You can limit number of connections or throttle the bandwidth used by BT (of course you will get slower speeds), maybe there are some other ways. 2. when you…

3. Bufferbloat. It's still a thing. dslreports has a bufferbloat score, one of my coworkers has been really impacted by it. http://www.dslreports.com/speedtest

Re: Nix × IPFS – Milestone 1

#80

Post by takeda »

linsomniac wrote: post #79
takeda wrote: post #56

Earlier quoted context omitted.

Perhaps there are routers that don't handle well large amounts of data, but from my experience what typically is happening are two things: 1. the fairness in TCP is done per TCP connection, bittorrent opens many connections so they take over the whole bandwidth. You can limit number of connections or throttle the bandwidth used by BT (of course you will get slower speeds), maybe there are some other ways. 2. when you…

3. Bufferbloat. It's still a thing. dslreports has a bufferbloat score, one of my coworkers has been really impacted by it. http://www.dslreports.com/speedtest

I heard about bufferbloat, but I didn't think it was related to bittorrent maximizing throughput
Post reply on HN