Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Flat Tree Dependency Resolution in Npm v3

docs.npmjs.com

Post reply on HN
Print view View original

101–110 of 123 posts

Re: Flat Tree Dependency Resolution in Npm v3

#101

Post by zebracanevra »

nmjohn wrote: post #86
vfc1 wrote: post #74

Jspm solves this by installing the module in a directory appending the package version. There is no maximally flat tree, the tree is 100% flat. At most there are several versions of the same package side by side, but no nesting. It even supports circular dependencies.

Are there any trade-offs to this approach? This seems so obvious to me I'm confused why it is more widely used.

In node, require()'ing a dependency is stateless - it searches ./node_modules/ for the module, then ../n_m/ then ../../n_m/, etc, until it finds an appropriately named module.

In JSPM/SystemJS, require()'ing/importing a module is still by name (as it supports NPM modules), but the package.json file has to be parsed in order to map module names to an installed module version. Note, this mapping is only done in the developer environment - once you build a bundle all the mapping is statically compiled into one file.

Re: Flat Tree Dependency Resolution in Npm v3

#102

Post by JaRail »

bhouston wrote: post #100

I do not understand why they just didn't have a two level directory structure: node_modules/[module_name]/[version] Then it would be flat and support multiple versions of the same module in a way that is completely deterministic and also fully deduplicated. This new system is unnecessarily complex.

The file system layout is constrained by how dependency resolution works. For example, a require() call has a very (computationally) simple algorithm.

I certainly agree that your suggestion simplifies file-system layouts. The tradeoff is that the complexity shifts to other parts of the system.

That said, I'm not a fan of the v3 approach. I'd have preferred a central package cache with a structure similar to your suggestion. I'd add that each package in the cache should have all of its dependencies resolved in its own /node_modules/ dir with symlinks. Unfortunately, I still can't see a nice way to handle peer dependencies. Peer dependencies require the ability to walk up the file system to resolve, which you can't do with symlinks.

Re: Flat Tree Dependency Resolution in Npm v3

#103

Post by JaRail »

theGimp wrote: post #34
vec wrote: post #25

Earlier quoted context omitted.

Because then they'd have to change the semantics of how `require()` works. As it is, node packages are not version aware at runtime. Instead NPM is designed to place packages such that any library will have the correct version first in their load path. Changing the directory structure to encode versions is probably the Right Way in a grand sense, but it's a much more substantial change than just promoting packages by…

I thought about that, but clearly they have enough information to figure out whether to load the package from the top level or a nested directory. (I'm guessing simply by bubbling up and looking for the "node_packages" directory.) It doesn't seem too big of a leap to do it the way I was suggesting. Either bubble up and look for "node_packages", but this time store a symlink there, or add a dot file that tells you "th…

That doesn't handle peer dependencies. Those are resolved by walking up the filesystem. You can't walk up the file system with a symlinked structure. A single npm can be resolving different peer dependencies depending on where it is in the tree.

Re: Flat Tree Dependency Resolution in Npm v3

#104

Post by JaRail »

theGimp wrote: post #37
dc2 wrote: post #23

Earlier quoted context omitted.

When your actual Node .js file runs `require('a')`, it has no context to determine which version it is requiring. To expect Node to look for a package.json file at runtime is absurd. This presents a problem.

I'm guessing the current behavior for require is to bubble up the directory structure and look for "node_packages". So resolving things at runtime is already happening. I agree using package.json at runtime is not the best solution though. A way to keep things clean and avoid using package.json is to use symlinks instead of downloading a fresh copy, which would make putting everything at the top level a possibility.…

Symlinks break peer dependencies.

Re: Flat Tree Dependency Resolution in Npm v3

#105

Post by brianbarker »

V2's deep nesting recently caused trouble for us at work. We had errors where our node_modules folder was too long for Windows, due to the deep folder structure. Updating to v3 resolved the problem. Hopefully other flaws being discussed here can be worked out as npm evolves.

Re: Flat Tree Dependency Resolution in Npm v3

#106

Post by Silhouette »

feross wrote: post #39
ploxiln wrote: post #35

Earlier quoted context omitted.

This "penalty" you propose would be a cosmetic thing, it wouldn't really make a difference. Many devs would choose to use the module as a dependency of their modules. The real solution to the sub-dependency-insanity of npm is ... use fewer dependencies. Don't use dependencies that have sub-dependencies. Write more code to directly solve your problem. This is a viewpoint you won't often hear, because it comes from som…

I wouldn't recommend using fewer dependencies, but smaller ones. One of the best features of npm is that it solves the "dependency hell" problem. You can depend on modules freely and not worry about making it harder for users of your module to install it. The issue comes when people create these huge "do-everything" modules. This is worth a read: https://github.com/sindresorhus/ama/issues/10#issuecomment-1...

One of the best features of npm is that it solves the "dependency hell" problem. You can depend on modules freely and not worry about making it harder for users of your module to install it.

That statement is true in theory, as long as everyone follows semantic versioning and dependencies never have unexpected interactions because of bugs.

Unfortunately, in the real world, neither of these conditions holds perfectly. The moment you depend on any module in your package.json that directly or indirectly depends on any other module where any version is not precisely fixed, you are immediately running uncontrolled and probably untested (by you) code. Moreover, if anything does go wrong, you may also have considerable difficulty identifying and reproducing the exact combination of versions of everything that caused the failure.

I have spent far too many days over the past few months tracking down exactly these kinds of issues, sometimes in some of the best known and most popular modules available via npm. For example, as the article itself mentions, what you wind up with in your node_modules directory can depend on install order under the new scheme. I don't think this is a good thing.

I therefore agree with those who advocate minimising dependencies, within reason of course. I also advocate strict version numbering and 100% reproducible builds, which is unfortunately quite hard to achieve once indirect dependencies start entering the picture with Node and npm, under either the old or the new scheme.

Re: Flat Tree Dependency Resolution in Npm v3

#107

Post by Silhouette »

city41 wrote: post #98
tracker1 wrote: post #92

Earlier quoted context omitted.

Given windows is the vast majority of desktops, and that there is more variety between different Linux distros and OSX versions than there are in the decades of Windows APIs, I think you have things a little backwards. I've done far more dev targeting linux deployments the past few years, and much of it is far nicer on *nix than windows... but your viewpoint seems to be kind of arrogant. I haven't worked for a compan…

I don't mean to be arrogant, and if I am honestly it comes from ignorance. I've worked in MS shops before (I did .NET, COM, etc for nearly a decade). This whole thing blew up out of frustration at Windows specific issues cropping up more often than any other OS in environments like Node. I do think that's hard to deny. If that's arrogant, then I don't know what to say :-/

There is a real problem here, but I think you're laying the blame a little unfairly at Microsoft's doorstep. It's probably fair to say that at least until relatively recently Microsoft has gone to far greater lengths to ensure compatibility and longevity of code build on its platforms than any other organisation in the history of software development.

We used to value writing portable software as a skill, we used to value languages and libraries with robust specifications that could be used to write portable code, and portable code is also relatively future-proof code. I've worked on large projects that shipped on literally a dozen or more different platforms at any given time and were maintained for well over a decade with the significant variations in platforms that happen over that kind of time frame. Those projects built probably 95+% of the same source code for each platform, with platform-specific APIs and conventions carefully isolated.

That attitude and the related skills seem to have been much less valued in recent years, not least by the Linux community. (What, you want to build this C or C++ code with a tool chain other than GCC and friends?) If people carelessly scatter platform-specific code all over their projects, then of course they won't be easily portable, but more often than not such limitations are entirely artificial and could easily be avoided at negligible cost. In my experience, this is also true of a lot of libraries with the likes of Node and Python where native code is used in managed packages, and I think it's fair to consider that the resulting portability limitations are a potential disadvantage when choosing the language for a project.

Re: Flat Tree Dependency Resolution in Npm v3

#108

Post by Silhouette »

tsuraan wrote: post #33
tvanantwerp wrote: post #17

Earlier quoted context omitted.

As someone who has no choice but to use Windows at work, I'm really sick of all the problems that crop up because so many tools use were written by people who won't use Windows and haven't tested anything on it.

That's a funny point of view. Windows is made by a multi-billion dollar company, rented by countless wealthy software development shops, but it's volunteer open-source programmers who are supposed to take on the burden of supporting it? Seriously, I can't even comprehend that thought process. I write software for fun, I release it on github, and some people find it useful. If there are other people who don't, they ca…

I don't think anybody has a right to expect me to support Microsoft's product. That's insane.

True enough, though it's also fair to consider a language/platform/ecosystem inferior for some work if it relies on that kind of individual and often less portable contribution for its effectiveness. A lot of FOSS advocates will promote community support and a broad contributor base as advantages of that style of development, but there is another side to that coin, which is that sometimes you get just what you (didn't) pay for and you're on your own in terms of support or working around any problems.

Re: Flat Tree Dependency Resolution in Npm v3

#109

Post by tgamblin »

nawitus wrote: post #64
davexunit wrote: post #43

Earlier quoted context omitted.

>This is what we did with Dart's package manager[1]. It works like Bundler where it finds a single set of package versions that satisfy all of the constraints and there are no duplicate packages. Constraint solving is the status quo, and I am strongly opposed to it. Rather than saying "I want foo >= 1.0" in a package manager, you should have the build system test the environment for the features you need. This is wha…

Language specific package managers seem to actually work in real life and are cross platform, e.g. npm. But I'm not aware of anything for e.g. C++ which would offer the features of npm (mainly installing dependencies with a single command and supporting GNU/Linux, OS X and Windows). Is there a single non-language specific package manager which is cross platform, and supports a large percentage of packages for a given…

Spack [1,2] is our attempt to do this for HPC, where we have two things going on: 1) Our applications aren't single-language. They include Fortran, Python, C, C++, Yorick (!), Lua, etc. etc. 2) We have a major combinatorial versioning problem, because HPC clusters (and scientific software in general) rely on ABI-incompatible libraries like MPI, and MANY different compilers.

Spack is build-from-source and takes a lot of inspiration from Nix and its descendants, but it adds support for things like swapping out compilers/MPI versions/etc., as well as a syntax for composing new builds of things on the fly That is, you can build something with a slightly different version of a dependency without ever editing a package file. Our users to be able to build a package with 6 different compilers and 3 different MPI library versions, and have their 18 versions coexist on the same system. This has been super helpful for testing production code at LLNL, because we have to run our stuff on other peoples' often very different clusters. For instance, at LANL, we prefer PGI compilers. At LLNL we tend to like Intel or Clang.

One reason languages tend to grow their own package managers is that there are all kinds of peculiarities about how different languages manage modules/extensions. Have you seen how many different ways there are to deploy Python modules? It's very hard to have multiple versions of things coexisting in the same Python env, and things like pkg_resources require editing client code [3]. Ew. Spack tries to address this by allowing different language packages to implement their own module activation/deactivation logic, so you can kind of have a language-specfic package manager for each install of the language. I'd rather have something more like Nix's profiles, though.

I wouldn't say that the project is huge yet, but we've had some interest from mostly other HPC sites. It would be cool if the project caught on in the "real" world, too.

[1] https://github.com/llnl/spack

[2] http://www.computer.org/csdl/proceedings/sc/2015/3723/00/280...

[3] http://stackoverflow.com/questions/5265731

Re: Flat Tree Dependency Resolution in Npm v3

#110

Post by frank-weindel »

tlrobinson wrote: post #68

"dependency resolution depends on install order" Does this sound insane to anyone else? EDIT: I understand not wanting to modify node's `require` semantics, but this is an unacceptable sacrifice of consistency for efficiency. Surely it would have been possible for an `npm install --save x` to put the `node_modules` directory in a state identical to `npm install --save x && rm -rf node_modules && npm install`. It migh…

What's more is that standard `npm install` install order is ALWAYS alphabetic. Which causes some packages, purely arbitrarily, to influence directory structure.
Post reply on HN