Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nix as OS X Package Manager

ariya.ofilabs.com

Post reply on HN
Print view View original

181–190 of 209 posts

Re: Nix as OS X Package Manager

#181

Post by Keats »

geocar wrote: post #153
Keats wrote: post #69

Earlier quoted context omitted.

We wrote an article about the same thing https://blog.wearewizards.io/why-docker-is-not-the-answer-to... (Tom has a background in physics as well)

You are assuming "that reproducibility is good". I believe this assumption needs to be challenged.

So let's say you are deploying 10 servers. Are you fine with all of them being different?

Or to get out of computers, you have a drug where they say it cures cancer 100% of the time and no one can reproduce the results, it's fine for you?

Re: Nix as OS X Package Manager

#182

Post by pmahoney »

geocar wrote: post #173
sciurus wrote: post #171

Earlier quoted context omitted.

It sure sounds like you are advocating Docker as a way to produce and distribute a http://martinfowler.com/bliki/SnowflakeServer.html We've been down this road before. Hacking something until it works, then imaging it and passing it on to your coworkers is a maintenance nightmare.

I appreciate how difficult it is to understand someone when you begin the conversation by assuming you understand them. For example, did you read that article? You know Docker has none of those problems > The first problem with a snowflake server is that it's difficult to reproduce. Should your hardware start having problems, this means that it's difficult to fire up another server to support the same functions. dock…

I appreciate your presentation of an alternate viewpoint.

The "build from source" camp provides solutions to a couple problems that at first glance seem to be difficult in a "principled mutation of binary image" model:

- Moving to a different architecture, e.g. x86 to ARM or ppcle. In build from source, I essentially change some parameters and components in the lowest level and rebuild.

- Other big changes to base components, e.g. compiling for Linux/libc vs. building atop a unikernel (obviously heavily dependent on what the upper layers are doing, but I can imagine wanting to run similar code atop multiple bases).

- Sharing common components. This is not really a "from source" thing, but is an important part of Nix. If two images include components in different orders, in Nix the common components will be shared (at the arguably coarse granularity of a Nix package), while the linear structure of Docker layers means once two images diverge, everything further down the chain will be its own copy, even if the same bits.

Re: Nix as OS X Package Manager

#183

Post by zenlikethat »

colemickens wrote: post #31
rdtsc wrote: post #15

When I first heard about Docker, I thought it did what Nix does. Then someone handed me a docker image saying, "I got this working on my laptop, deploy this. Isn't this great!". I had no idea how they arrived at that configuration. If something happened to them or that image, I wouldn't have known how to reproduce that "working state". So I am happy to see Nix / Guix become popular. It is what I imagined package mana…

I mean, that's exactly what a Dockerfile is for. I don't know of anyone seriously using Docker that isn't using Dockerfiles exclusively. I only know of a single time when it's valuable to "commit" a new image (recovering logs from a stopped container). None of which is to detract from Nix. Nix+Docker is a powerful combination. It takes Docker's declarative nature all the way down to the compiler used to build the bit…

Tangential, but if you want to recover logs from a stopped container, why not use 'docker cp'?

Re: Nix as OS X Package Manager

#184

Post by zenlikethat »

lfam wrote: post #50
arianvanp wrote: post #43

Earlier quoted context omitted.

I did this too until there was a glibc vulnerability. and it took ages for a 'patch' to come out that had to be installed in an obscure way. For a toy project this is not a problem of course. But before I'd use nixos in production, it needs a better story for security patches. If you try to look up the word 'security' you get viagra spam on their wiki. and there is no security mailinglist whatsoever iirc. I'm sure th…

In Guix, we address this problem with grafting. Rather than rebuild everything that depends on the fixed glibc, we build the fixed glibc, then rewrite all references to the old glibc to refer to the new one. That rewriting happens in new copies of the referring packages, of course, since the "store" is immutable. https://www.gnu.org/software/guix/manual/html_node/Security-... https://savannah.gnu.org/forum/forum.php?…

Doesn't this affect the promised reproducibility of builds?

Re: Nix as OS X Package Manager

#185

Post by geocar »

coldtea wrote: post #180
geocar wrote: post #153

Earlier quoted context omitted.

You are assuming "that reproducibility is good". I believe this assumption needs to be challenged.

Any significant contribution you might have for software engineering in regards to that? Aside from merely believing that. I can't even tell what you mean. If I want to rebuild my machine same as before, or deploy build 100s of servers at different points in time with the same setup, reproducibility is necessary. There's not even one argument against that. If you have any, please oblige us. Should deployments be a su…

> If I want to rebuild my machine same as before, or deploy build 100s of servers at different points in time with the same setup, reproducibility is necessary.

No it isn't.

Reproducible is talking about the process or steps needed to make a machine. You do not need to repeat the steps to produce the first machine to produce the second machine. You can simply clone the first machine, and Docker makes this very easy to do.

Keats begins his article with the statement offered, without justification:

In this post I'm going to assume that reproducibility is good and necessary.

Since I don't believe that, and in fact believe the opposite, I take issue with the entire article. Indeed, he goes on to construct a straw man comparing the exact opposite of what I propose -- a non-interactive "dockerfile" and not even a particularly good one, with a very well thought-out "purely functional" build system.

> Any significant contribution you might have for software engineering in regards to that?

I don't know of any big technology company that doesn't design their live infrastructure around ephemeral machines cloned from a template.

They don't all use Docker, and they don't all use interactive development, however, but these two tools can make it easier for people who (for example) don't have Google's economy of scale.

> Should deployments be a surprise?

No they should not.

Deployments take less time with my interactive method because you do not have to wait to "rebuild everything" which can take hours.

Remember heartbleed? People using nixos had to rebuild every component that used OpenSSL, then test them, then find something else broke and repeat the process. Only once they had finished building their new process could they roll out this process to all machines, doing one more final rebuild. Even if done perfectly this has a minimum of two iterations.

I could simply create a new clone, get it working, then tag it as the next release. New instances can then be cloned from the release, monitored, and the old instances destroyed. The entire process took under an hour for the entire fleet, including reading about heartbleed on HN. This is clearly almost as fast as running `apt-get dist-upgrade` on each of the live systems, but without any of the risk.

The organisational complexity, however is admittedly much higher than "wash, wait, repeat", which is why most system administrators could not even begin to use live clones until virtualization and tooling had become much more commonplace.

> I can't even tell what you mean.

And yet you feel qualified to tell me I'm wrong.

Re: Nix as OS X Package Manager

#186

Post by geocar »

pmahoney wrote: post #182
geocar wrote: post #173

Earlier quoted context omitted.

I appreciate how difficult it is to understand someone when you begin the conversation by assuming you understand them. For example, did you read that article? You know Docker has none of those problems > The first problem with a snowflake server is that it's difficult to reproduce. Should your hardware start having problems, this means that it's difficult to fire up another server to support the same functions. dock…

I appreciate your presentation of an alternate viewpoint. The "build from source" camp provides solutions to a couple problems that at first glance seem to be difficult in a "principled mutation of binary image" model: - Moving to a different architecture, e.g. x86 to ARM or ppcle. In build from source, I essentially change some parameters and components in the lowest level and rebuild. - Other big changes to base co…

"Sharing common components" is useful, and the fact that any combination of Nix packages is a valid combination is not something that is guaranteed by other package management systems[1].

However there's another, perhaps simpler solution: I can have a separate "wu-ftpd" container, and a separate "hylafax" container. This admittedly might not have been as easy when nix started, but it is very very easy now.

[1]: https://cr.yp.to/slashpackage/studies.html

Re: Nix as OS X Package Manager

#187

Post by davexunit »

zenlikethat wrote: post #184
lfam wrote: post #50

Earlier quoted context omitted.

In Guix, we address this problem with grafting. Rather than rebuild everything that depends on the fixed glibc, we build the fixed glibc, then rewrite all references to the old glibc to refer to the new one. That rewriting happens in new copies of the referring packages, of course, since the "store" is immutable. https://www.gnu.org/software/guix/manual/html_node/Security-... https://savannah.gnu.org/forum/forum.php?…

Doesn't this affect the promised reproducibility of builds?

No, there's no compromise there.

Re: Nix as OS X Package Manager

#188

Post by geocar »

Keats wrote: post #181
geocar wrote: post #153

Earlier quoted context omitted.

You are assuming "that reproducibility is good". I believe this assumption needs to be challenged.

So let's say you are deploying 10 servers. Are you fine with all of them being different? Or to get out of computers, you have a drug where they say it cures cancer 100% of the time and no one can reproduce the results, it's fine for you?

Straw man.

Reproducible in this case is talking about the process or steps needed to make a machine. You do not need to repeat the steps to produce the first machine to produce the second machine. You can simply clone the first machine, and Docker makes this very easy to do.

That I can clone a machine does not mean that I can clone the interaction between a patient and their drugs, but if I could it would most certainly be preferable to trying to repeat the experiments with larger and larger populations.

Re: Nix as OS X Package Manager

#189

Post by lfam »

zenlikethat wrote: post #184
lfam wrote: post #50

Earlier quoted context omitted.

In Guix, we address this problem with grafting. Rather than rebuild everything that depends on the fixed glibc, we build the fixed glibc, then rewrite all references to the old glibc to refer to the new one. That rewriting happens in new copies of the referring packages, of course, since the "store" is immutable. https://www.gnu.org/software/guix/manual/html_node/Security-... https://savannah.gnu.org/forum/forum.php?…

Doesn't this affect the promised reproducibility of builds?

You end up with a new, grafted dependency graph, which can be reproduced just like the un-grafted graph. Is there some other reproducibility issue you see?

Re: Nix as OS X Package Manager

#190

Post by coldtea »

geocar wrote: post #185
coldtea wrote: post #180

Earlier quoted context omitted.

Any significant contribution you might have for software engineering in regards to that? Aside from merely believing that. I can't even tell what you mean. If I want to rebuild my machine same as before, or deploy build 100s of servers at different points in time with the same setup, reproducibility is necessary. There's not even one argument against that. If you have any, please oblige us. Should deployments be a su…

> If I want to rebuild my machine same as before, or deploy build 100s of servers at different points in time with the same setup, reproducibility is necessary. No it isn't. Reproducible is talking about the process or steps needed to make a machine. You do not need to repeat the steps to produce the first machine to produce the second machine. You can simply clone the first machine, and Docker makes this very easy t…

>Reproducible is talking about the process or steps needed to make a machine. You do not need to repeat the steps to produce the first machine to produce the second machine. You can simply clone the first machine, and Docker makes this very easy to do.

Cloning is not the answer since it's all or nothing -- where reproducibility can be achieved piecemeal.

Only offering cloning means that starting with the same initial Dockerfile and making some needed updates to it, one can't be sure that the NON updated parts will be producible. So you don't get flexible reuse.

Only offering cloning also means that when you need to update only a part (e.g. because of security reasons) you can't just use the same dockerfile with the changed component and be sure that you'll get an otherwise identical system. So you don't get flexible updating.

Cloning also required an already set-in-stone configured system to be the prototype.

But until one reaches that, they have to experiment with various configurations (to end with the specific Docker image they want). Without reproducibility, they can't be guaranteed that their various test configurations under test are the same and only their latest changes differ. Perhaps another thing they left as is broke too.

>I don't know of any big technology company that doesn't design their live infrastructure around ephemeral machines cloned from a template.

That's just an argument from popularity. And not necessarily a popularity they opted for, while also being provided with the alternative.

That they use "ephemeral machines cloned from a template" is just what they have to do -- not what would be ideal.

Besides, reproducibility doesn't preclude cloning as part of the process -- it's a superset of it, offering way more flexibility.

>I could simply create a new clone, get it working, then tag it as the next release.

But if you mess with the clone it's not a clone anymore. And if a whole team has to mess with the clone over some period of time to update or fix it, there's hell to keep track what went on.

Nobody argues that you can't simple install cloned images as they are.

Post reply on HN