Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Packagers don't know best

vagabond.github.io

Post reply on HN
Print view View original

101–110 of 127 posts

Re: Packagers don't know best

#101

Post by grosskur »

mwcampbell wrote: post #73
dvanduzer wrote: post #70

Earlier quoted context omitted.

Reading your comment, I am starting to understand why everyone is so confused about this. Linux and OS X both have the same underlying options for static or shared libraries. There is a large amount of "enterprise" software that is distributed just like a .dmg file. There is plenty of middle ground between having everything dynamically linked and everything statically linked. The author of the article believes that p…

As a sysadmin, which do you prefer: self-contained software distributions with all the dependencies included, or packages that use the host distro's package manager, use system versions of libraries wherever possible, and otherwise integrate well with the host system? The latter seems better to me, but it's an honest question, not rhetorical.

False dichotomy---bundling dependencies doesn't preclude integrating well with the host system. See Basho's latest Riak package for Ubuntu 12.04:

http://s3.amazonaws.com/downloads.basho.com/riak/1.3/1.3.2/u...

It integrates well with the host system (init script, "riak" user/group, data in /var/lib/riak, logs in /var/log/riak, etc.) and bundles dependencies in /usr/lib/riak.

In my experience, bundling dependencies is often the only practical way to install a complex app. Take Sentry as another example:

https://github.com/getsentry/sentry

The current version (5.4.5) depends on 37 Python packages:

  BeautifulSoup==3.2.1
  Django==1.4.5
  Pygments==1.6
  South==0.7.6
  amqp==1.0.11
  anyjson==0.3.3
  billiard==2.7.3.28
  celery==3.0.19
  cssutils==0.9.10
  distribute==0.6.31
  django-celery==3.0.17
  django-crispy-forms==1.2.8
  django-indexer==0.3.0
  django-paging==0.2.5
  django-picklefield==0.3.0
  django-social-auth==0.7.23
  django-social-auth-trello==1.0.3
  django-static-compiler==0.3.3
  django-templatetag-sugar==0.1
  gunicorn==0.17.4
  httpagentparser==1.2.2
  httplib2==0.8
  kombu==2.5.10
  logan==0.5.6
  nydus==0.10.6
  oauth2==1.5.211
  pynliner==0.4.0
  python-dateutil==1.5
  python-openid==2.2.5
  pytz==2013b
  raven==3.3.11
  redis==2.7.6
  sentry==5.4.5
  setproctitle==1.1.7
  simplejson==3.1.3
  six==1.3.0
  wsgiref==0.1.2
Ruby or Node apps have dependency trees of similar or greater size. It takes an enormous amount of effort to roll all of these as individual packages (yes, I've done it) and it's a colossal waste of time once you realize you can have a working package in under a minute with virtualenv, pip, and fpm:

  $ virtualenv --distribute /opt/sentry
  $ /opt/sentry/bin/pip install sentry
  $ fpm -n sentry -v 5.4.5 -s dir -t deb /opt/sentry

Re: Packagers don't know best

#102

Post by demallien »

dvanduzer wrote: post #69
freshhawk wrote: post #43

Earlier quoted context omitted.

Well put, and this particular problem can easily be handled by having a prominent doc section called "Information for packagers" that outlines all this stuff. This isn't a new problem and seems to be best handled by engaging with the packagers and putting a small amount of effort into helping them, it's easy and pays enormous dividends.

The "information for packagers" doc section is often referred to as a Makefile. It enumerates minimum versions of shared libraries, as well as explicit versions of static libraries.

The problem isn't with minimum versions, the problem is with maximum versions, which are potentially unknown at the time a project is released, i.e. the code works with the latest real ease of a library, but a future release of the library may break the project.

The only real way to avoid that is for a project to include a bunch of compliance tests, that validate that an underlying library correctly performs the operations the project needs it to do. But this is actually a lot of work, so in reality it will almost never be done. Which leads us back to the discussion about the wisdom of packagers changing libs without understanding the on downstream projects.

Re: Packagers don't know best

#103

Post by spullara »

The real issue with bundling software is that you can't pull in a security patch. You actually have the same issue with internal packages at large companies. If you can stay on the current release you can drastically reduce the effect of security bugs.

Re: Packagers don't know best

#104

Post by tmoertel »

dvanduzer wrote: post #67
tmoertel wrote: post #61

Earlier quoted context omitted.

You're conflating the dependency graph with the change-flow network. The first represents how projects rely on other projects; the second represents how changes must propagate to reach all users. Once you understand the difference, you'll understand why debundling is the sensible response to the sea of large-scale interdependent software-development projects that characterizes most FOSS ecosystems.

If I understand you correctly, you're saying the dependency graph and the "change-flow network" are completely orthogonal. Separation of concerns is a value of good software projects. But there are practical realities that the author of the article enumerates specifically. If there is a tight coupling between his application and a handful of upstream libraries, packagers are far more likely to break his application b…

They're not orthogonal; they're two directed graphs with the same vertices and different edges.

If the dependency graph G = (V, E) has a vertex for every software project and an edge x -> y iff downstream project y depends on upstream project x, then the change-flow network is the graph C = (V, F), where there is an edge x -> y in F iff there is a downstream path between x and y in G and also y requires an update and re-release when x changes (e.g., because it bundles a copy of x in its releases).

So if there is a change to project x, for it to flow to all affected dependents, you must update all downstream neighbors of x in the change-flow network C.

For example, consider the following dependency graph, in which library L is used by downstream library L2, and L2 by project P:

    L -> L2 -> P
If none of the projects bundle their upstream dependencies in their own releases, then the corresponding change-flow network has no edges, and updating any project requires only re-releasing its own package to satisfy all dependencies:

    L
    L2
    P
But if L2 bundles a copy of L, and P bundles a copy of L2, then the corresponding network looks like this:

    L -> L2
    L -> P

    L2 -> P

    P
A change to L requires re-releasing not only L but also L2, and P. A change to L2 requires re-releasing L2 and also P.

Does that make more sense now?

Re: Packagers don't know best

#105

Post by dfc »

omaranto wrote: post #81
dfc wrote: post #33

Look at this ubuntu erlang package, it depends on 40 other packages, as well. That isn’t even the worst of it, if you type ‘erl’ it tells you to install ‘erlang-base’, which only has a handful of dependencies, none of which are any of these erlang libraries! That package is a dummy package that depends on erlang-base and the rest of the base erlang platform. You would have to force dpkg to ignore dependencies in orde…

You misunderstood the point the author made about erlang-base: it's not that he or she somehow installed Erlang without installing erlang-base, but rather that if in an Ubuntu system you try to run 'erl' before installing any Erlang packages at all, you receive a message telling you something like "to get the erl command install the package 'erlang-base'" and if you go and do that, you don't get the Erlang standard l…

Nice catch. It did not occur to me that people would run a program without installing it, use command-not-found and/ignore the suggests list when installing a package. As another commenter pointed out this has nothing to do with splitting packages up. Its a toss up between user error or bug in c-n-f.

Re: Packagers don't know best

#106

Post by dvanduzer »

tmoertel wrote: post #104
dvanduzer wrote: post #67

Earlier quoted context omitted.

If I understand you correctly, you're saying the dependency graph and the "change-flow network" are completely orthogonal. Separation of concerns is a value of good software projects. But there are practical realities that the author of the article enumerates specifically. If there is a tight coupling between his application and a handful of upstream libraries, packagers are far more likely to break his application b…

They're not orthogonal; they're two directed graphs with the same vertices and different edges. If the dependency graph G = ( V , E ) has a vertex for every software project and an edge x -> y iff downstream project y depends on upstream project x , then the change-flow network is the graph C = ( V , F ), where there is an edge x -> y in F iff there is a downstream path between x and y in G and also y requires an upd…

I mean, sure, just typeset it in LaTeX and it'll breeze right past your Fortune 500 IT department's change management board.

Re: Packagers don't know best

#107

Post by homeomorphic »

randallsquared wrote: post #75
vacri wrote: post #64

Earlier quoted context omitted.

Then you fall back to this 'superior' individual packaging method. I don't see how this is a counter.

If you packaging partisans manage to convince developers of the merits of your position, the individual packaging method won't be easily available: it'll be "wait for the OS maintainers to decide to include the newer version" or "make; make test; make install" (i.e., be your own packager).

We "packaging partisans" are just saying we find classical packaging superior to the every-piece-of-software-on-its-own model, not that the latter isn't an OK fallback if there is no package.

Re: Packagers don't know best

#108

Post by homeomorphic »

shtylman wrote: post #37
asuffield wrote: post #30

Earlier quoted context omitted.

Then one day somebody finds a critical bug in zlib and you need to go and fetch an update for every single application you have ever installed on your system. Then it turns out half of them don't have updates.

This is a fallacy. Just because a bug was exposed in a library you used does not mean your software will be exposed in the same way. Your use of the lib may not even overlap with the bug exposure. And I seriously doubt you carefully comb all the libs you select on a project for "security" before you release something.

Say there's a vulnerability or something in zlib's decompression, as GP suggested. That's gonna affect pretty much all software using zlib (i.e. a LOT). On a system where the package maintainers took care of having all packages use the system zlib, the whole problem is fixed by ONE ENTITY (the zlib maintainer or team) waking up and patching one package. Every user updates the zlib library package through their package manager (which informs them that they need to), and the vulnerability goes away for thousands of programs.

If, on the other hand, those thousands of programs all bundled zlib, the user won't be safe until hundreds of maintainers wake up and do (repeatedly the same) patching. Or even worse, if there isn't even a package management system, as some apparently want, the user has to also go fetch the fixed programs from thousands of upstreams. Oh, and the user also has to know about the vulnerability. Not gonna happen!

As we can see, the classical model reduces work duplication, reduces patching times and manpower need, and certainly takes a big responsibility off the user's shoulders.

Re: Packagers don't know best

#109

Post by homeomorphic »

viraptor wrote: post #72
Afforess wrote: post #16

Earlier quoted context omitted.

Your argument is disingenuous. While it might prevent a bug from spreading due to older/differing embedded libraries, it is equally likely to cause new bugs, when library signatures change, and some library function is suddenly gone. When you bundle the libraries yourself, you only have to target the libraries you included. When you let the package manager do the magic for you, you have to target every version of the…

> when library signatures change, and some library function is suddenly gone. No, no, no. This is not how you fix critical / security issues in a well maintained system. You either backport a single patch that fixes the problem without changing any signatures, or if you support a very old, incompatible software you reimplement the fix yourself. Then the release is not a new library. It's the old one + fix. This is wh…

I think this is a key point that many people overlook the importance of when pressuring, say, Debian or Ubuntu maintainers for new package versions inside of a single release.

Re: Packagers don't know best

#110

Post by homeomorphic »

bcl wrote: post #28
m-r-a-m wrote: post #12

My favorite interesting packaging choice is TeX Live in Fedora 18 [1]. There are about 4500 texlive-* packages (out of around 35000 binary packages in Fedora total). The packagers split up the packages based on upstream metadata. [1] https://bugzilla.redhat.com/show_bug.cgi?id=949626

s/interesting/insane/

Why is this insane? Seriously, what problems does it cause, hypothetical or otherwise?
Post reply on HN