Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The fundamental problem of programming language package management

blog.ezyang.com

Post reply on HN
Print view View original

21–30 of 73 posts

Re: The fundamental problem of programming language package management

#21

Post by mercurial »

innguest wrote: post #14
chrisfarms wrote: post #2

Nix, NixOS, Nix ... a thousand times Nix. I can't believe the article doesn't mention it. I've been using NixOS as my OS for development, desktop and we're in the middle of transitioning to using it for production deployments too. Nix (the package manager not the distribution) solves so many of the discussed problems. And NixOS (the linux distribution) ties it all together so cleanly. I keep my own fork of the Nixpkg…

Yes, thank you! But people still want to "solve" the problem the hard way, I guess. Also, "Java" is mentioned twice in the article but I can't find mention of Ocaml Functors. I thought they solved most package problems even before Nix was around?

Functors are part of the Ocaml module system, but they don't really have any relation to package management with version numbers and dependencies.

Re: The fundamental problem of programming language package management

#22

Post by ownedthx »

Apache Ivy is built on Java and could be a starting point for the uber package manager. It is extremely extensible and can resolve depencies against maven style repos, file systems, and really any other storage mechanism. It can resolve against multiple repos simultaneously.

The real problem is that its so powerful and hard to ramp up on... The docs aren't sufficient for its overall complexity. That all aside, if the will were there, it could be the git of package managers.

Re: The fundamental problem of programming language package management

#23

Post by pmahoney »

bad_user wrote: post #12
davexunit wrote: post #6

Earlier quoted context omitted.

Yes! The Haskell community has been using Nix to great effect and I would like to see other programmers catch on as well. Here is a great talk about using Nix targeted at Python programmers: http://pyvideo.org/video/3036/rethinking-packaging-developme... In addition to Nix, there is also a newer project: GNU Guix. Guix is built on top of Nix but replaces the custom package configuration language with Scheme, among ot…

You could of course use a platform that doesn't depend on OS dependent binaries (like the JVM) and a package manager that likes ad-hoc and easily created repositories and that has lots of plugins available (Maven, or derivates like Gradle, SBT or Leiningen). I worked with a lot of platforms, such as PHP, Perl, Ruby, Python, Node.js and .NET. I felt the pain of pip, easy_install, setup-tools, virtualenv, bundler, gems…

There are Java jars with native binaries, just sayin. Though I do like Maven and Java quite a bit.

https://bitbucket.org/xerial/sqlite-jdbc

https://github.com/twall/jna

Re: The fundamental problem of programming language package management

#24

Post by avsm »

> The Git of package management doesn't exist yet

We've taken a pretty good shot at this in the OCaml ecosystem the via OPAM package manager (https://opam.ocaml.org).

* OPAM composes its package universe from a collection of remotes, which can be fetched either via HTTP(S), Git, Hg or Darcs. The resulting package sets are combined locally into one view, but can be separated easily. For instance, getting a view into the latest XenAPI development trees just requires "opam remote add xapi-dev git://github.com/xapi-project/opam-repo-dev".

* The same feature applies to pinning packages ("opam pin add cohttp git://github.com/avsm/ocaml-cohttp#v0.6"). This supports local trees and remote Git/Hg/Darcs remotes (including branches).

* OCaml, like Haskell, is statically typed, and so recompiles all the upstream dependencies of a package once its updated. This lets me work on core OCaml libraries that are widely used, and just do an "opam update -u" to recompile all dependencies to check for any upstream breakage. We did not go for the very pure NixOS model due to the amount of time it takes to compile distinct packages everywhere. This is a design choice to balance composability vs responsiveness, and Nix or 0install are fine choices if you want truely isolated namespaces.

* By far the most important feature in OPAM is the package solver core, which resolves version constraints into a sensible user-facing solution. Rather than reinvent the (rather NP-hard) solver from scratch, OPAM provides a built-in simple version and also a CUDF-compatible interface to plug into external tools like aspcud, which are used by other huge repositories such as Debian to handle their constraints.

This use of CUDF leads to some cool knobs and utilities, such as the OPAM weather service to test for coinstallability conflicts: http://ows.irill.org/ and the solver preferences that provide apt-like preferences: https://opam.ocaml.org/doc/Specifying_Solver_Preferences.htm...

* Testing in a decentralized system is really, really easy by using Git as a workflow engine. We use Travis to test all incoming pull requests to OPAM, much like Homebrew does, and can also grab a snapshot of a bunch of remotes and do bulk builds, whose logs are then pushed into a GitHub repo for further analysis: https://github.com/ocaml/opam-bulk-logs (we install external dependencies for bulk builds by using Docker for Linux, and Xen for *BSD: https://github.com/avsm/docker-opam).

All in all, I'm very pleased with how OPAM is coming along. We use it extensively for the Mirage OS unikernel that's written in OCaml (after all, it makes sense for a library operating system to demand top-notch package management).

If anyone's curious and wants to give OPAM a spin, we'd love feedback on the 1.2beta that's due out in a couple of weeks: http://opam.ocaml.org/blog/opam-1-2-0-beta4/

Re: The fundamental problem of programming language package management

#25

Post by akavel »

This topic reminded me of some very interesting thoughts from Joe Armstrong, that I remember seeing posted somewhere (HN?) some time ago -- "Why do we need modules at all?":

    [...]
    The basic idea is
    
        - do away with modules
        - all functions have unique distinct names
        - all functions have (lots of) meta data
        - all functions go into a global (searchable) Key-value database
        - we need letrec
        - contribution to open source can be as simple as
          contributing a single function
        - there are no "open source projects" - only "the open source
          Key-Value database of all functions"
        - Content is peer reviewed
    
    These are discussed in no particular order below:
    [...]
Full thread: http://thread.gmane.org/gmane.comp.lang.erlang.general/53472

Re: The fundamental problem of programming language package management

#26

Post by mercurial »

tel wrote: post #20
mercurial wrote: post #7

Earlier quoted context omitted.

> A way of specifying an ABI for a packages instead of a version number? Technically impossible for many languages (have fun figuring out what it would look like in Perl...). And even when it's possible, it's not a guarantee: you can have a semantic change without an ABI change. Cargo, Rust's newfangled package manager, supposes semantic versioning, and I think it's a sane attitude.

Stronger and stronger typing makes this ABI guarantee stronger and stronger. Dependent typing could package theorems about the properties of your interface and then ensure that all matches satisfy those properties. You'd like depend upon their theorems to prove things about your own program knowing that nothing can break.

Yes, but something like Idris isn't mainstream. 99% of the languages out there don't allow you to express this kind of invariant.

Re: The fundamental problem of programming language package management

#27

Post by mrottenkolber »

calpaterson wrote: post #3

The downside he mentions to "pinned versions" actually applies to everything on this page. If you don't pay attention to security updates, you will be vulnerable whether or you forgot about your pinned versions or you forgot about your stable distribution. "Stable" distributions have an additional downside he doesn't mention: when you upgrade every package all at once it's a LOT more effort than if you had upgraded t…

> Dealing with multiple library changes at once is an order of magnitude more difficult than dealing with them one-at-a-time.

From my experience exactly the opposite is true. Compare uprading Slackware to keeping an Arch Linux running. With Slackware, I have to sit down for an hour, do the upgrade, read the notices that come along with it, maybe see if it will break any of my custom packages. This happes once or twice a year (security upgrades are completely painless as they don't break things). With Arch Linux I need to do that every day. If I don't have time to do it for a month, the system is basically broken beyond recognition...

Re: The fundamental problem of programming language package management

#28

Post by mrottenkolber »

calpaterson wrote: post #16
mcherm wrote: post #11

Earlier quoted context omitted.

> Dealing with multiple library changes at once is an order of magnitude more difficult than dealing with them one-at-a-time. Curious... I have exactly the opposite experience. I find that a certain amount of time is required to carefully regression-test my application code after upgrading a library. Doing this 23 times for my 23 different dependencies that need to be upgraded can be quite costly. If I, instead, upgr…

Do you have an automated testing suite? That might explain the difference in experience. I use a lot of automated tests

I don't think its the testing thats work intensive, its fixing things that broke. The more often you upgrade the more often things will break.

Re: The fundamental problem of programming language package management

#29

Post by xsace »

Dependency management (ie class/script/resource loading) is too coupled with the programming language execution environment.

It's not something you can make generic like a file/folder based version control tool. It's like asking for the Git of unit testing/continuous integration or whatever, not going to happen.

Re: The fundamental problem of programming language package management

#30

Post by xsace »

akavel wrote: post #25

This topic reminded me of some very interesting thoughts from Joe Armstrong, that I remember seeing posted somewhere (HN?) some time ago -- "Why do we need modules at all?": [...] The basic idea is - do away with modules - all functions have unique distinct names - all functions have (lots of) meta data - all functions go into a global (searchable) Key-value database - we need letrec - contribution to open source can…

I can't see how that can be better than a module cohesive API.
Post reply on HN