Flat Tree Dependency Resolution in Npm v3 • Hacker News Board

Re: Flat Tree Dependency Resolution in Npm v3

#31

Post by randallsquared » Thu, Dec 31, 2015, 5:24 PM UTC

Killswitch wrote: post #24

vec wrote: post #10

So, assuming I have two libraries, A and B, that both require the same version of library C, do those libraries still get their own separate in-memory copies of C, or do they share a singleton? It's terrible practice, but it's not unheard of for an NPM module to monkey patch its dependencies, since before this the library could assume it had sole ownership of its whole subtree.

If depend on A and B and both A and B depend on the same version range as C, C is now a top-level dependency. Your node_modules will look like this: - Package_A - Package_B - Package_C It's only when A and B depend on different versions of C that cannot be resolved via semver as safe. - Package_A -- node_modules --- Package_C - Package_B -- node_modules --- Package_C I am pretty certain that monkey patching your depe…

Sadly, this is the result of the second situation:

    - Package_A
    - Package_C_vX
    - Package_B
    -- node_modules
    --- Package_C_vY

Re: Flat Tree Dependency Resolution in Npm v3

#32

Post by [deleted] » Thu, Dec 31, 2015, 5:27 PM UTC

Killswitch wrote: post #26

[deleted] wrote: post #14

Earlier quoted context omitted.

[deleted]

While I can understand the want for that, there's too much issues. The only reason npm does it this way is because the module that depends on B v1.3 and a module that depends on B v2.1 could be introducing some really bad bugs or breakage if you force all modules to use B v2.1. That's one of the reason bower is losing out.

[deleted]

Re: Flat Tree Dependency Resolution in Npm v3

#33

Post by tsuraan » Thu, Dec 31, 2015, 5:32 PM UTC

tvanantwerp wrote: post #17

city41 wrote: post #12

Earlier quoted context omitted.

Why do dev communities have to continually pay the Windows tax? If Microsoft keeps insisting on being an outlier in the world of OSes, shouldn't they be the ones burdened? I gotta say, as someone who never ever uses Windows but maintains a couple of open source packages, I'm really sick of the Windows only problems that crop up.

As someone who has no choice but to use Windows at work, I'm really sick of all the problems that crop up because so many tools use were written by people who won't use Windows and haven't tested anything on it.

That's a funny point of view. Windows is made by a multi-billion dollar company, rented by countless wealthy software development shops, but it's volunteer open-source programmers who are supposed to take on the burden of supporting it? Seriously, I can't even comprehend that thought process.

I write software for fun, I release it on github, and some people find it useful. If there are other people who don't, they can improve it for their own purposes, or they can find or write something better. I don't think anybody has a right to expect me to support Microsoft's product. That's insane.

Re: Flat Tree Dependency Resolution in Npm v3

#34

Post by theGimp » Thu, Dec 31, 2015, 5:35 PM UTC

vec wrote: post #25

theGimp wrote: post #13

That's a great improvement over the old behavior, but I'm wondering why the team didn't go a different route. Why not always store packages at the top level, and create a directory for each version that's required? For example: directory "A" contains two subdirectories: "v0.1.0" and "v0.1.1"

Because then they'd have to change the semantics of how `require()` works. As it is, node packages are not version aware at runtime. Instead NPM is designed to place packages such that any library will have the correct version first in their load path. Changing the directory structure to encode versions is probably the Right Way in a grand sense, but it's a much more substantial change than just promoting packages by…

I thought about that, but clearly they have enough information to figure out whether to load the package from the top level or a nested directory. (I'm guessing simply by bubbling up and looking for the "node_packages" directory.)

It doesn't seem too big of a leap to do it the way I was suggesting.

Either bubble up and look for "node_packages", but this time store a symlink there, or add a dot file that tells you "this is where I found package.json".

I'm sure there was a reason they didn't go for either of those solutions because I've followed the team's discussions when they were iojs, and they have very smart people. I was just curious about the reasoning :)

Re: Flat Tree Dependency Resolution in Npm v3

#35

Post by ploxiln » Thu, Dec 31, 2015, 5:36 PM UTC

kylecordes wrote: post #8

I understand that this is a hairy, messy problem they are solving, and that they traded off one aspect of the problem for another (Install order dependency! as a new "feature" in 2015!). But I wish they had aimed higher. The better goal would be that the entire state of the node modules directory is a pure function of the contents of the package.json file plus the platform details (compiler used for native modules).…

This "penalty" you propose would be a cosmetic thing, it wouldn't really make a difference. Many devs would choose to use the module as a dependency of their modules.

The real solution to the sub-dependency-insanity of npm is ... use fewer dependencies. Don't use dependencies that have sub-dependencies. Write more code to directly solve your problem.

This is a viewpoint you won't often hear, because it comes from someone who doesn't write js or use node ... I've just had to fix the deployment of some node applications written by frontend devs a few times. I've seen npm trees with over 1000 modules to do some very trivial stuff. Node actually seems nice, and it has all the basics you need built in.

I typically use C or Python, which pretty much never use nested dependencies, and have never felt the desire to install/update/manage/debug exponentially more dependencies.

Re: Flat Tree Dependency Resolution in Npm v3

#36

Post by [deleted] » Thu, Dec 31, 2015, 5:40 PM UTC

dc2 wrote: post #23

theGimp wrote: post #13

That's a great improvement over the old behavior, but I'm wondering why the team didn't go a different route. Why not always store packages at the top level, and create a directory for each version that's required? For example: directory "A" contains two subdirectories: "v0.1.0" and "v0.1.1"

When your actual Node .js file runs `require('a')`, it has no context to determine which version it is requiring. To expect Node to look for a package.json file at runtime is absurd. This presents a problem.

[deleted]

Re: Flat Tree Dependency Resolution in Npm v3

#37

Post by theGimp » Thu, Dec 31, 2015, 5:40 PM UTC

dc2 wrote: post #23

theGimp wrote: post #13

That's a great improvement over the old behavior, but I'm wondering why the team didn't go a different route. Why not always store packages at the top level, and create a directory for each version that's required? For example: directory "A" contains two subdirectories: "v0.1.0" and "v0.1.1"

When your actual Node .js file runs `require('a')`, it has no context to determine which version it is requiring. To expect Node to look for a package.json file at runtime is absurd. This presents a problem.

I'm guessing the current behavior for require is to bubble up the directory structure and look for "node_packages".

So resolving things at runtime is already happening.

I agree using package.json at runtime is not the best solution though.

A way to keep things clean and avoid using package.json is to use symlinks instead of downloading a fresh copy, which would make putting everything at the top level a possibility.

I'm sure that option was considered, but I'm curious about why it was not taken.

I suppose I should look at the discussion notes. Hope I'll remember to do that when I'm home!

Re: Flat Tree Dependency Resolution in Npm v3

#38

Post by curun1r » Thu, Dec 31, 2015, 5:42 PM UTC

dc2 wrote: post #23

theGimp wrote: post #13

That's a great improvement over the old behavior, but I'm wondering why the team didn't go a different route. Why not always store packages at the top level, and create a directory for each version that's required? For example: directory "A" contains two subdirectories: "v0.1.0" and "v0.1.1"

When your actual Node .js file runs `require('a')`, it has no context to determine which version it is requiring. To expect Node to look for a package.json file at runtime is absurd. This presents a problem.

That version information could be encoded into symlinks if it weren't for the need to support Windows.

Re: Flat Tree Dependency Resolution in Npm v3

#39

Post by feross » Thu, Dec 31, 2015, 5:45 PM UTC

ploxiln wrote: post #35

kylecordes wrote: post #8

I understand that this is a hairy, messy problem they are solving, and that they traded off one aspect of the problem for another (Install order dependency! as a new "feature" in 2015!). But I wish they had aimed higher. The better goal would be that the entire state of the node modules directory is a pure function of the contents of the package.json file plus the platform details (compiler used for native modules).…

This "penalty" you propose would be a cosmetic thing, it wouldn't really make a difference. Many devs would choose to use the module as a dependency of their modules. The real solution to the sub-dependency-insanity of npm is ... use fewer dependencies. Don't use dependencies that have sub-dependencies. Write more code to directly solve your problem. This is a viewpoint you won't often hear, because it comes from som…

I wouldn't recommend using fewer dependencies, but smaller ones. One of the best features of npm is that it solves the "dependency hell" problem. You can depend on modules freely and not worry about making it harder for users of your module to install it.

The issue comes when people create these huge "do-everything" modules. This is worth a read: https://github.com/sindresorhus/ama/issues/10#issuecomment-1...

Re: Flat Tree Dependency Resolution in Npm v3

#40

Post by tehnorm » Thu, Dec 31, 2015, 5:46 PM UTC

kylecordes wrote: post #8

I understand that this is a hairy, messy problem they are solving, and that they traded off one aspect of the problem for another (Install order dependency! as a new "feature" in 2015!). But I wish they had aimed higher. The better goal would be that the entire state of the node modules directory is a pure function of the contents of the package.json file plus the platform details (compiler used for native modules).…

Aimed higher is correct. I've always wondered why no one looked at the way git does this under the covers and applied it to npm. A directed graph that points to (with a sym link or similar mechanism) the correct package version on the file system.

That way you have the best of both worlds - dependencies don't collide and a single copy of a package version on the file system in node_modules.