Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The fundamental problem of programming language package management

blog.ezyang.com

Post reply on HN
Print view View original

11–20 of 73 posts

Re: The fundamental problem of programming language package management

#11

Post by mcherm »

calpaterson wrote: post #3

The downside he mentions to "pinned versions" actually applies to everything on this page. If you don't pay attention to security updates, you will be vulnerable whether or you forgot about your pinned versions or you forgot about your stable distribution. "Stable" distributions have an additional downside he doesn't mention: when you upgrade every package all at once it's a LOT more effort than if you had upgraded t…

> Dealing with multiple library changes at once is an order of magnitude more difficult than dealing with them one-at-a-time.

Curious... I have exactly the opposite experience. I find that a certain amount of time is required to carefully regression-test my application code after upgrading a library. Doing this 23 times for my 23 different dependencies that need to be upgraded can be quite costly. If I, instead, upgrade all of the libraries at once and perform my extensive regression testing just once, I save a great deal of effort.

That's if everything goes smoothly. If something does NOT go smoothly and I encounter an error, then I need to determine which upgrade caused the problem. Most of the time (85% perhaps?), that turns out to be easy and obvious just by looking at the error that presents itself. In the remaining cases, I simply roll back half of the package upgrades and start binary-searching to identify the culprit (or culprits in the case of a conflict between libraries).

Re: The fundamental problem of programming language package management

#12

Post by bad_user »

davexunit wrote: post #6
chrisfarms wrote: post #2

Nix, NixOS, Nix ... a thousand times Nix. I can't believe the article doesn't mention it. I've been using NixOS as my OS for development, desktop and we're in the middle of transitioning to using it for production deployments too. Nix (the package manager not the distribution) solves so many of the discussed problems. And NixOS (the linux distribution) ties it all together so cleanly. I keep my own fork of the Nixpkg…

Yes! The Haskell community has been using Nix to great effect and I would like to see other programmers catch on as well. Here is a great talk about using Nix targeted at Python programmers: http://pyvideo.org/video/3036/rethinking-packaging-developme... In addition to Nix, there is also a newer project: GNU Guix. Guix is built on top of Nix but replaces the custom package configuration language with Scheme, among ot…

You could of course use a platform that doesn't depend on OS dependent binaries (like the JVM) and a package manager that likes ad-hoc and easily created repositories and that has lots of plugins available (Maven, or derivates like Gradle, SBT or Leiningen).

I worked with a lot of platforms, such as PHP, Perl, Ruby, Python, Node.js and .NET. I felt the pain of pip, easy_install, setup-tools, virtualenv, bundler, gems, cpan, pear, rvm, rbenv, npm, bower, apt-get or whatever else I used at some point or another. And I swear, in spite of all the criticism that Java or Maven get and in spite of all warts, in terms of packaging and deployment for me it's been by far the sanest. I mean, it's not without warts, heaven forbid to end up with classpath issues due to transitive dependencies, but at the very least it is tolerable.

Re: The fundamental problem of programming language package management

#13

Post by danielweber »

When I sit around and think "what's the biggest improvement I could make personally to the computing world?" there's always this voice in my head saying "kidnap whoever is building the next package management system and lock them in a deep dark box."

There seems this fundamental disconnect between people making languages about how people use their languages. I don't have time to follow your Twitter feed, because I'm working on a lot of different things. I know it's important to you, the Language Developer, and so you think it should be important to me, the Language User. But I have dozens of things to keep track of, and all of them imagine that they're the most important thing in my world.

It's like the old office culture mocked in "Office Space" where the guy has 7 different bosses, each imagining their own kingdom is the most important.

Re: The fundamental problem of programming language package management

#14

Post by innguest »

chrisfarms wrote: post #2

Nix, NixOS, Nix ... a thousand times Nix. I can't believe the article doesn't mention it. I've been using NixOS as my OS for development, desktop and we're in the middle of transitioning to using it for production deployments too. Nix (the package manager not the distribution) solves so many of the discussed problems. And NixOS (the linux distribution) ties it all together so cleanly. I keep my own fork of the Nixpkg…

Yes, thank you! But people still want to "solve" the problem the hard way, I guess.

Also, "Java" is mentioned twice in the article but I can't find mention of Ocaml Functors. I thought they solved most package problems even before Nix was around?

Re: The fundamental problem of programming language package management

#15

Post by talex5 »

shadowmint wrote: post #4

yeah yeah, I read the previous post ( http://www.standalone-sysadmin.com/blog/2014/03/just-what-we... ) too. Maybe this time we can talk about how to meaningfully solve these problems instead of just fighting pointlessly about if old tools are so great should be used for everything. Decentralized package management huh? How would that work? A way of specifying an ABI for a packages instead of a version number? A way…

"Decentralized package management huh? How would that work?"

http://0install.net/ does this (sad to see it wasn't mentioned in the article). Basically:

1. Use URIs rather than short names to identify packages.

2. Scope dependencies so different applications can see different versions of the same library where necessary.

Here's an OSNews article from 2007 about such things:

http://www.osnews.com/story/16956/Decentralised-Installation...

Re: The fundamental problem of programming language package management

#16

Post by calpaterson »

mcherm wrote: post #11
calpaterson wrote: post #3

The downside he mentions to "pinned versions" actually applies to everything on this page. If you don't pay attention to security updates, you will be vulnerable whether or you forgot about your pinned versions or you forgot about your stable distribution. "Stable" distributions have an additional downside he doesn't mention: when you upgrade every package all at once it's a LOT more effort than if you had upgraded t…

> Dealing with multiple library changes at once is an order of magnitude more difficult than dealing with them one-at-a-time. Curious... I have exactly the opposite experience. I find that a certain amount of time is required to carefully regression-test my application code after upgrading a library. Doing this 23 times for my 23 different dependencies that need to be upgraded can be quite costly. If I, instead, upgr…

Do you have an automated testing suite? That might explain the difference in experience. I use a lot of automated tests

Re: The fundamental problem of programming language package management

#17

Post by bad_user »

mcherm wrote: post #11
calpaterson wrote: post #3

The downside he mentions to "pinned versions" actually applies to everything on this page. If you don't pay attention to security updates, you will be vulnerable whether or you forgot about your pinned versions or you forgot about your stable distribution. "Stable" distributions have an additional downside he doesn't mention: when you upgrade every package all at once it's a LOT more effort than if you had upgraded t…

> Dealing with multiple library changes at once is an order of magnitude more difficult than dealing with them one-at-a-time. Curious... I have exactly the opposite experience. I find that a certain amount of time is required to carefully regression-test my application code after upgrading a library. Doing this 23 times for my 23 different dependencies that need to be upgraded can be quite costly. If I, instead, upgr…

Personally I agree with you, except that sometimes you find out that a library you depend on hasn't been upgraded and cannot be used in combination with your other libraries, due to some conflict somewhere. At which point one needs to drop the library from the project and that can prove to be costly, so it is better to identify libraries that aren't well maintained earlier rather than later. This isn't a problem for well established popular libraries, but is a problem if you're using newer, more cutting age stuff.

Re: The fundamental problem of programming language package management

#18

Post by davexunit »

vertex-four wrote: post #9
davexunit wrote: post #6

Earlier quoted context omitted.

Yes! The Haskell community has been using Nix to great effect and I would like to see other programmers catch on as well. Here is a great talk about using Nix targeted at Python programmers: http://pyvideo.org/video/3036/rethinking-packaging-developme... In addition to Nix, there is also a newer project: GNU Guix. Guix is built on top of Nix but replaces the custom package configuration language with Scheme, among ot…

> In addition to Nix, there is also a newer project: GNU Guix. Guix is built on top of Nix but replaces the custom package configuration language with Scheme, among other differences. https://gnu.org/software/guix/ Personally, I'm rather more keen on Nix; the language is pretty much designed for writing JSON-style configuration except as a formal programming language, which is what the vast majority of Nix code is (b…

>the language is pretty much designed for writing JSON-style configuration except as a formal programming language, which is what the vast majority of Nix code is (both package definitions and system configurations).

Guix uses an embedded domain specific language that is also designed for easily writing package recipes, but it uses s-expressions instead of something that is "JSON-style". Also, Nix build scripts are written in Bash, whereas Guix build scripts are written in Scheme. I think that makes Guix more consistent in its programming style.

>Additionally, with Nix, you can be close to certain that if you build something twice, you'll get the same result, because it can't access impure resources.

Guix has this same certainty because it uses the Nix daemon, and the defaults are a bit stricter than Nix.

>Finally, because Guix is a GNU project, the official repositories are going to go nowhere near non-free software.

That doesn't mean that you can't host your own non-free packages or use someone else's non-free packages. But yes, Guix does not ship with packages that ask the user to give up their freedom. To me, that's an advantage.

Re: The fundamental problem of programming language package management

#19

Post by tel »

innguest wrote: post #14
chrisfarms wrote: post #2

Nix, NixOS, Nix ... a thousand times Nix. I can't believe the article doesn't mention it. I've been using NixOS as my OS for development, desktop and we're in the middle of transitioning to using it for production deployments too. Nix (the package manager not the distribution) solves so many of the discussed problems. And NixOS (the linux distribution) ties it all together so cleanly. I keep my own fork of the Nixpkg…

Yes, thank you! But people still want to "solve" the problem the hard way, I guess. Also, "Java" is mentioned twice in the article but I can't find mention of Ocaml Functors. I thought they solved most package problems even before Nix was around?

There's a bit of missing context: the author is writing this most likely as part of his research on implementing Backpack—a systems which would being OCaml Functor-like things to Haskell.

Re: The fundamental problem of programming language package management

#20

Post by tel »

mercurial wrote: post #7
shadowmint wrote: post #4

yeah yeah, I read the previous post ( http://www.standalone-sysadmin.com/blog/2014/03/just-what-we... ) too. Maybe this time we can talk about how to meaningfully solve these problems instead of just fighting pointlessly about if old tools are so great should be used for everything. Decentralized package management huh? How would that work? A way of specifying an ABI for a packages instead of a version number? A way…

> A way of specifying an ABI for a packages instead of a version number? Technically impossible for many languages (have fun figuring out what it would look like in Perl...). And even when it's possible, it's not a guarantee: you can have a semantic change without an ABI change. Cargo, Rust's newfangled package manager, supposes semantic versioning, and I think it's a sane attitude.

Stronger and stronger typing makes this ABI guarantee stronger and stronger. Dependent typing could package theorems about the properties of your interface and then ensure that all matches satisfy those properties. You'd like depend upon their theorems to prove things about your own program knowing that nothing can break.
Post reply on HN