Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Packagers don't know best

vagabond.github.io

Post reply on HN
Print view View original

121–127 of 127 posts

Re: Packagers don't know best

#121

Post by dvanduzer »

tmoertel wrote: post #104
dvanduzer wrote: post #67

Earlier quoted context omitted.

If I understand you correctly, you're saying the dependency graph and the "change-flow network" are completely orthogonal. Separation of concerns is a value of good software projects. But there are practical realities that the author of the article enumerates specifically. If there is a tight coupling between his application and a handful of upstream libraries, packagers are far more likely to break his application b…

They're not orthogonal; they're two directed graphs with the same vertices and different edges. If the dependency graph G = ( V , E ) has a vertex for every software project and an edge x -> y iff downstream project y depends on upstream project x , then the change-flow network is the graph C = ( V , F ), where there is an edge x -> y in F iff there is a downstream path between x and y in G and also y requires an upd…

You seem to misunderstand how static linking works.

If P statically links to its own version of L2, then L2 is just a part of P. The fact that there may be a dynamically linked version of L2 elsewhere on the system is irrelevant.

Consider:

  L -> L2 -> P

  L -> L2 -> Q

  L -> L2 -> R
If the authors of L2 release a new version that P and Q are happy with, but creates an extremely subtle segfault condition in R, then what?

The packager could just wait to release the upgrade to L2 until all downstream packages have compatible releases.

The packager could backport a subset of the L2 patches that is still compatible with R (Redhat does this a lot).

The packager could silently curse the author of R for not statically linking the necessary frozen-in-time version of L2 and thus bypassing this problem entirely.

Re: Packagers don't know best

#122

Post by regularfry »

mwcampbell wrote: post #113
regularfry wrote: post #111

Earlier quoted context omitted.

Splitting a package into docs, libs and executables makes a lot of sense. Splitting those further, so you've got umpteen "independent" packages which 95% of users are just going to have to manually recombine to get the functionality the upstream package provides out of the box, can get pathological. Debian has historically been particularly bad at this, and Ubuntu inherited that tendency.

Please give an example of a pathological case in Debian.

Ruby in etch was pretty absurd, from memory.

Re: Packagers don't know best

#123

Post by tmoertel »

dvanduzer wrote: post #121
tmoertel wrote: post #104

Earlier quoted context omitted.

They're not orthogonal; they're two directed graphs with the same vertices and different edges. If the dependency graph G = ( V , E ) has a vertex for every software project and an edge x -> y iff downstream project y depends on upstream project x , then the change-flow network is the graph C = ( V , F ), where there is an edge x -> y in F iff there is a downstream path between x and y in G and also y requires an upd…

You seem to misunderstand how static linking works. If P statically links to its own version of L2, then L2 is just a part of P. The fact that there may be a dynamically linked version of L2 elsewhere on the system is irrelevant. Consider: L -> L2 -> P L -> L2 -> Q L -> L2 -> R If the authors of L2 release a new version that P and Q are happy with, but creates an extremely subtle segfault condition in R, then what? T…

> If P statically links to its own version of L2, then L2 is just a part of P. The fact that there may be a dynamically linked version of L2 elsewhere on the system is irrelevant.

No, it's highly relevant because when a security fix lands for L2, it takes longer to propagate to users if projects like P bundle their own versions of L2 as part of their releases. In that case, users must wait for the project developers to work the already-released L2 fixes into their own bundled versions of L2 and then release new versions of the projects before any downstream users get the fix. But if P and other projects use the same version of L2 that everybody else does, everybody gets the fix right away.

> If the authors of L2 release a new version that P and Q are happy with, but creates an extremely subtle segfault condition in R, then what? ...

> The packager could silently curse the author of R for not statically linking the necessary frozen-in-time version of L2 and thus bypassing this problem entirely.

More likely, the packager would patch L2 to fix the problem with R and then talk to the upsteam L2 developers to get the patch included in L2 proper. This way, R's users get the fix right away and the problem gets eliminated at its source, in L2, rather than papered-over in R's private copy of L2.

As I wrote in my original post, one of the big benefits of the "no bundling" policy is to make sure that patches flow upsteam to where they belong instead of piling up in downstream repos where they do good for only one dependent project instead of all dependent projects.

Re: Packagers don't know best

#124

Post by marssaxman »

homeomorphic wrote: post #108
shtylman wrote: post #37

Earlier quoted context omitted.

This is a fallacy. Just because a bug was exposed in a library you used does not mean your software will be exposed in the same way. Your use of the lib may not even overlap with the bug exposure. And I seriously doubt you carefully comb all the libs you select on a project for "security" before you release something.

Say there's a vulnerability or something in zlib's decompression, as GP suggested. That's gonna affect pretty much all software using zlib (i.e. a LOT). On a system where the package maintainers took care of having all packages use the system zlib, the whole problem is fixed by ONE ENTITY (the zlib maintainer or team) waking up and patching one package. Every user updates the zlib library package through their packag…

Funny that you call the packaging scheme the "classical model" - to me that's the "[relatively] new and weird" scheme. I'd say that "everybody builds their own binaries" is the classical model.

This is the argument that comes up every time. Perhaps it matters to people who are running complex servers hosting an array of services. In my life, it never comes up. I am either using a personal machine or managing a server which is responsible for one single service.

The idea that upgrading one library could affect the behavior of hundreds of programs is terrifying. How do I know they all still work? I don't. I have to go test them all. What this means is that I never update any libraries at all, on a linux machine, because I can't know in advance what the upgrade might break.

My concern is not about keeping everything up to date; it is about keeping everything working. If there is a bug in one program then I want to update that program and only that program and no other programs at all. Then I can evaluate the behavior of the new program. If it is worse than the old behavior, I can hopefully go back to the old version. If it is better, then I can keep it. Nothing else should change.

This is exactly what I get on Mac OS X, and it's what I get when I build apps with statically linked libraries on Linux: stuff works until I break it, and then I know what I broke, so I can fix it.

When I let package managers update things for me, my system becomes an unknowable chaos of changing behavor. Instead, I simply never update anything until I am ready to pave the machine and start from scratch. I install everything I might want to use, then I disable updates and leave it alone until I am ready to start over.

Re: Packagers don't know best

#125

Post by homeomorphic »

marssaxman wrote: post #124
homeomorphic wrote: post #108

Earlier quoted context omitted.

Say there's a vulnerability or something in zlib's decompression, as GP suggested. That's gonna affect pretty much all software using zlib (i.e. a LOT). On a system where the package maintainers took care of having all packages use the system zlib, the whole problem is fixed by ONE ENTITY (the zlib maintainer or team) waking up and patching one package. Every user updates the zlib library package through their packag…

Funny that you call the packaging scheme the "classical model" - to me that's the "[relatively] new and weird" scheme. I'd say that "everybody builds their own binaries" is the classical model. This is the argument that comes up every time. Perhaps it matters to people who are running complex servers hosting an array of services. In my life, it never comes up. I am either using a personal machine or managing a server…

> Funny that you call the packaging scheme the "classical model" - to me that's the "[relatively] new and weird" scheme. I'd say that "everybody builds their own binaries" is the classical model.

I stand corrected.

> In my life, it never comes up. I am either using a personal machine or managing a server which is responsible for one single service.

I just went back through my own update log a couple of months: You don't use anything that uses libxml [1], ffmpeg (audio/video Swiss army knife) [2], poppler (popular PDF library) [3] or openSSL [4]? Do you pay attention to the security notices of every program you have that bundled one of those? Are you sure that upstream is paying attention to the security ntoices of those libraries?

> The idea that upgrading one library could affect the behavior of hundreds of programs is terrifying. How do I know they all still work? I don't. I have to go test them all. What this means is that I never update any libraries at all, on a linux machine, because I can't know in advance what the upgrade might break.

For non-rolling distributions (for example Debian stable, Ubuntu, Fedora, RedHat) packages don't have their behavior changed throughout the lifetime of a release . Security and bug fixes are backported to whatever version is in place for the lifespan of the distribution's release. This is part of the point of stable releases, and something that's routinely forgotten by those who hound package maintainers for newer versions of things!

As an example, let's consider [1]. Ubuntu 13.04 uses libxml2 version 2.9.0 plus some Debian/Ubuntu patches. They released version 2.9.0+dfsg1-4ubuntu4.1 with the following changelog [5]:

  * SECURITY UPDATE: multiple use after free issues
    - debian/patches/CVE-2013-1969.patch: properly reset pointers in
      HTMLparser.c, parser.c.
    - CVE-2013-1969
Only the security fix is applied, and after upgrade you know that libxml2 will continue as it has done for the lifetime of your distro release, except it's no longer vulnerable to CVE-2013-1969. Moreover, this applies to every program using libxml (i.e. a lot) with one update. Again: There's no other change in behavior for libxml!

> This is exactly what I get on Mac OS X, and it's what I get when I build apps with statically linked libraries on Linux: stuff works until I break it, and then I know what I broke, so I can fix it.

Stuff seems to work until you break it. Then a security hole is found, and it turns out things aren't really working very well at all. To me it seems like a lot of work to keep track of such things by yourself!

[1] http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-1969

[2] http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2496

[3] http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-1790

[4] http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-0169

[5] https://launchpad.net/ubuntu/+source/libxml2/2.9.0+dfsg1-4ub...

Firefox and a few other packages, mostly web browsers who are making it hard to provide security backports, are notable exceptions.

Re: Packagers don't know best

#126

Post by dagw »

tmoertel wrote: post #20

The packagers actually do know what’s best. What they do makes patches flow faster not only downstream but also upstream. Improvements and fixes get to more people and get to them faster. Unbundling upstream libraries from downstream projects flattens the change-flow network, reducing the time it takes for things to get fixed and for the fixes to propagate. For example, say that project P uses library L and bundles a…

In principle I agree, however in my experience the time from me submitting a patch to L that I need for my new feature in P to work until that patch makes it into a stable release of L that packagers actually ship can be month.

In that time frame I'm stuck between not shipping a new version of P (often unacceptable as I've got users to answer to) or shipping my own slightly modified version of L.

Post reply on HN