Comparison of Programming Language Package Managers • Hacker News Board

Re: Comparison of Programming Language Package Managers

#71

Post by gnoway » Sat, Jul 30, 2016, 4:55 AM UTC

nilliams wrote: post #34

mordocai wrote: post #18

Earlier quoted context omitted.

> * All current package managers are either language or OS specific. What if you have an application with code written in multiple languages? guix and nix both work cross-language and cross-distro. Still OS specific though since only linux AFAIK. Also, containers partially solve this problem. > * NPM didn't have any kind of integrity checks for its packages, and I assume most package managers don't either. If you dow…

I often see drive-by bashing of npm, but rarely any justification. What exactly is wrong with it? (Aside from the lack of package integrity check, which I'll grant, sucks).

I think it's not bashing of npm specifically so much as it is the node ecosystem it serves and depends on; at least in my mind it's difficult to separate node from npm. That said, for what it's trying to do (read a list of deps, resolve vs. registry, download and unpack) it seems to do a fine job of it.

My major complaint about npm is the choice to allow version range operators on dependency declarations. We know the node.js ecosystem places a high value on composability, so using lots of tiny modules which themselves depend on lots of tiny modules is the norm. This is a problem though because range operators get used liberally everywhere, so getting reproducible builds is like winning the lottery.

There are other things I don't like about using npm: node_modules/ is big and has a lot of duplication (even with npmv3), it's pretty slow, historically it has been unstable, its still crap on Windows, etc. - but for someone who has 'ensures reproducible builds' as part of their job description, the way its modules get versioned is its worst feature.

Re: Comparison of Programming Language Package Managers

#72

Post by CMCDragonkai » Sat, Jul 30, 2016, 6:50 AM UTC

mordocai wrote: post #18

anonymousguy wrote: post #14

Since I had starting looking for an alternative to NPM I have discovered a couple of things: * All current package managers are either language or OS specific. What if you have an application with code written in multiple languages? * NPM didn't have any kind of integrity checks for its packages, and I assume most package managers don't either. If you download a corrupt package, for example, you won't have any idea a…

> * All current package managers are either language or OS specific. What if you have an application with code written in multiple languages? guix and nix both work cross-language and cross-distro. Still OS specific though since only linux AFAIK. Also, containers partially solve this problem. > * NPM didn't have any kind of integrity checks for its packages, and I assume most package managers don't either. If you dow…

There was work to make nix work in Cygwin, it's been paused, but there was partial functionality. I remember one of the problems was Windows symlinks.

Re: Comparison of Programming Language Package Managers

#73

Post by CMCDragonkai » Sat, Jul 30, 2016, 6:51 AM UTC

chriswarbo wrote: post #11

knucklesandwich wrote: post #5

One of the things that I've found to be important for (and generally under-documented by) package managers is resolution strategy. pip, for instance, eagerly resolves the highest version of a dependency first found in the tree and doesn't backtrack AFAIK [1]. Cargo sounds as though it behaves similarly, but will actually backtrack [2]. I've also seen package managers like 0install that will use SAT solvers to solve d…

Resolution strategies are really interesting in Nix; specifically, because it doesn't have any ! All dependencies, etc. are passed in explicitly (either as function arguments, or by importing files).

Yep, currently the maintainers are the ones doing resolution in order to achieve package set harmoniousness.

Re: Comparison of Programming Language Package Managers

#74

Post by w4tson » Sat, Jul 30, 2016, 6:53 AM UTC

Surely if a package manger provides the ability to store in VCS then it's admitting defeat for reproducible builds. I'm thinking npm here

Re: Comparison of Programming Language Package Managers

#75

Post by nilliams » Sat, Jul 30, 2016, 8:42 AM UTC

gnoway wrote: post #71

nilliams wrote: post #34

Earlier quoted context omitted.

I often see drive-by bashing of npm, but rarely any justification. What exactly is wrong with it? (Aside from the lack of package integrity check, which I'll grant, sucks).

I think it's not bashing of npm specifically so much as it is the node ecosystem it serves and depends on; at least in my mind it's difficult to separate node from npm. That said, for what it's trying to do (read a list of deps, resolve vs. registry, download and unpack) it seems to do a fine job of it. My major complaint about npm is the choice to allow version range operators on dependency declarations. We know the…

For reproducible builds (or at least 'to get the same versions again') you should be using 'npm shrinkwrap'. (Of course there's probably more you should do to get true reproducible builds, but that goes for any package manager).

The range operators are important, else you'd never be able to resolve 2 packages that want a similar versioned sup-dependency e.g. jquery 1.12 because without range operators those 2 packages would have declared minor version differences (1.12.1 and 1.12.3) depending on when they were published. This would mean you'd always end up with duplicated dependencies.

I'd argue 'node_modules is big' is not a fault of npm. If the package or app you're trying to install generates a large node_modules dir, that is something you should take up with the package maintainer. See buble vs babel - buble has a way smaller dep tree.

npm is only slow in the ways that all other package managers are, when installing large dependency trees or native dependencies (like libSass) and it is way faster than say pip and rubygems in this regard. When I 'pip install -r requirements.txt' at work, I literally go and make a coffee.

Also never experienced any instability, though I may have been lucky. Certainly it has been very stable for the last year or so when I've been working with a lot. Could you elaborate on why it is crap on Windows? I did think all major issues (e.g. deep nesting problem) were now fixed ...

Re: Comparison of Programming Language Package Managers

#76

Post by berntb » Sat, Jul 30, 2016, 9:44 AM UTC

sivoais wrote: post #62

mfer wrote: post #59

Earlier quoted context omitted.

IIRC, and I could be wrong, CPAN was the first to go down the route that many modern toolchains now provide. We've looked at it. In fact, the original creator of Glide (Go package manager) wrote about Perl and CPAN when talking about Go at http://technosophos.com/2015/09/02/dont-let-go-be-condemned-... .

Yeah, I think it was. The only thing older was probably CTAN, but it didn't have the same structure. Other things from the Perl ecosystem that should be copied are: - CPAN Testers which automatically tests every package on multiple systems from Windows to Solaris. This helps identify portability, backcompat, and regression issues. - CPAN mirrors which ensure that there isn't a single point of failure. This might not…

CPAN Testers also test on different operating system versions. And hence also test the Perl binaries, for a lot of different versions, by literally running the combined test suites for all of the (public) Perl code.

I really, really don't get why this isn't copied to the rest of the open source language environments. (Masochism? :-) )

Re: Comparison of Programming Language Package Managers

#77

Post by rurban » Sat, Jul 30, 2016, 10:56 AM UTC

mfer wrote: post #59

sivoais wrote: post #55

Perl's CPAN has tooling for diffing versions through the use of MetaCPAN (a top-notch site which every language should try to emulate). For example, here is a diff of the URI distribution: https://metacpan.org/diff/file?target=ETHER%2FURI-1.71%2F&so... This information is also available through an API for integration into command line tools.

IIRC, and I could be wrong, CPAN was the first to go down the route that many modern toolchains now provide. We've looked at it. In fact, the original creator of Glide (Go package manager) wrote about Perl and CPAN when talking about Go at http://technosophos.com/2015/09/02/dont-let-go-be-condemned-... .

Well, yes. But CPAN does not provide src urls via git (github, sourceforge, ...). You need to upload your tar.gz to a central repo which is then distributed via mirrors.

All modern packagers provide now such src urls, esp. go went ahead with this idea.

Re: Comparison of Programming Language Package Managers

#78

Post by halostatue » Sat, Jul 30, 2016, 3:27 PM UTC

jacques_chester wrote: post #66

halostatue wrote: post #52

Earlier quoted context omitted.

Absolutely. That’s sort of a fundamental reason why I wrote Cartage. I like some of what Capistrano does, but the general insistence on being able to `bundle install` on the target server is questionable, and the earlier practice for Rails apps of keeping the gems in Git resulted in other problems. Heroku mostly does this right with its 'slug' system, and I made Cartage specifically to be able to make deployable pack…

Cloud Foundry has staging and run steps, much as Heroku does, to the point that a lot of Heroku buildpacks will run without modification ... if you're in a fully connected environment at staging time . It's not an accident: Cloud Foundry was in part consciously inspired by Heroku in the early days, so adopting the buildpacks model was natural. However, as I noted above, this model breaks for disconnected environments…

Yeah. That makes sense. My run steps are in Ansible, so it made sense for me to make something that‘s just a packager. The nice thing about Cartage is that it can make use of already-vendored packages (old way) or you can create the deployment package from a developer machine. I really should write it up one of these days, but I’m so busy getting my team’s pipeline fully fed that I haven’t had time to properly shout out about Cartage (because, frankly, it’s kind of awesome if I do say so myself).

Re: Comparison of Programming Language Package Managers

#79

Post by gnoway » Sat, Jul 30, 2016, 4:07 PM UTC

nilliams wrote: post #75

gnoway wrote: post #71

Earlier quoted context omitted.

I think it's not bashing of npm specifically so much as it is the node ecosystem it serves and depends on; at least in my mind it's difficult to separate node from npm. That said, for what it's trying to do (read a list of deps, resolve vs. registry, download and unpack) it seems to do a fine job of it. My major complaint about npm is the choice to allow version range operators on dependency declarations. We know the…

For reproducible builds (or at least 'to get the same versions again') you should be using 'npm shrinkwrap'. (Of course there's probably more you should do to get true reproducible builds, but that goes for any package manager). The range operators are important, else you'd never be able to resolve 2 packages that want a similar versioned sup-dependency e.g. jquery 1.12 because without range operators those 2 package…

Shrinkwrap is ridiculous. I'm expected to go look at every resolved dependency and individually add them if I want to update or not? No thanks; one app at my workplace defines ~50 top level dependencies, but this balloons to almost 1300 - and this is with npm v3 - after npm install. Ain't nobody got time for that.

Deep nesting is not 'solved' it just doesn't happen 100% of the time anymore. If you have conflicts, you still have deep trees. I suppose range operators help with this a little, but looking at what gets installed it doesn't seem to help that much; I still have duplicated dependencies.

I was mentally comparing npm to tools like maven, ivy and nuget, all of which are faster but also not interpreted. Not a fair comparison I guess.

Re: Comparison of Programming Language Package Managers

#80

Post by steveklabnik » Sat, Jul 30, 2016, 4:08 PM UTC

jacques_chester wrote: post #64

steveklabnik wrote: post #41

Earlier quoted context omitted.

Hm, even without the above stuff, `cargo fetch` should enable you to do offline builds just fine.

Can you point to a man page? We weren't able to find a reference doc on the cargo site.

"cargo --list" will show all commands, which shows fetch. "cargo fetch --help" then shows the help for fetch.

We certainly have some work to do on Cargo's docs. Please don't hesitate to open a thread on users.rust-lang.org if you run into some kind of issue. In addition, I actually live in Brooklyn; if Pivotal ever needs a hand with Rust stuff, just let me know, I'd be happy to swing by your office and talk about whatever.