Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Node.js needs a virtual file system

blog.platformatic.dev

Post reply on HN
Print view View original

31–40 of 275 posts

Re: Node.js needs a virtual file system

#31

Post by epolanski »

indutny wrote: post #29

Taking the question of whether this would be a useful addition to Node.js core or aside, it must be noted that this 19k LoC PR was mostly generated by Claude Code and manually reviewed by the submitter which in my opinion is against the spirit of the project and directly violates the terms of Developer's Certificate of Origin set in the project's CONTRIBUTING.md

Do as I say, not as I do.

On a more serious note, I think that this will be thoroughly reviewed before it gets merged and Node has an entire security team that overviews these.

Re: Node.js needs a virtual file system

#32

Post by Normal_gaussian »

yarn pnp is currently broken on Node v25.7+;

- https://github.com/yarnpkg/berry/issues/7065

- https://github.com/nodejs/node/issues/62012

This is because yarn patches fs in order to introduce virtual file path resolution of modules in the yarn cache (which are zips), which is quite brittle and was broken by a seemingly unrelated change in 25.7.

The discussion in issue 62012 is notable - it was suggested yarn just wait for vfs to land. This is interesting to me in two ways: firstly, the node team seems quite happy for non-trivial amounts of the ecosystem to just be broken, and suggests relying on what I'm assuming will be an experimental API when it does land; secondly, it implies a lot of confidence that this feature will land before LTS.

Re: Node.js needs a virtual file system

#34

Post by PaulHoule »

koolba wrote: post #22
MarleTangible wrote: post #11

Earlier quoted context omitted.

The number of files in the node modules folder is crazy, any amount of organization that can tame that chaos is welcomed.

And if you thought malware hiding in a mess of files was bad, just wait till you see it in two layers of container files.

Or worse yet, the performance load of anti-malware software that has to look inside ZIP files.

Look, most of us realized around 2004 or so that if you had a choice between Norton and the virus you would pick the virus. In the Windows world we standardized around Defender because there is some bound on how much Defender degrades the performance of your machine which was not the case with competitive antivirus software.

I've done a few projects which involved getting container file formats like ZIP and PDF (e.g. you know it's a graph of resources in which some of those resources are containers that contain more resources, right?) and now that I think of it you ought to be able to virus scan ZIP files quickly and intelligently but the whole problem with the antivirus industry is that nobody ever considers the cost.

Re: Node.js needs a virtual file system

#35

Post by PaulHoule »

Dangeranger wrote: post #18
PaulHoule wrote: post #5

Would be nice if node packages could be packed up in ZIP files so to avoid the security/metadata tax for small file access on Windows.

There are alternative package managers like Yarn that use zip files as a way to store each Node package.[0] [0] https://yarnpkg.com/advanced/pnp-spec#zip-access

... and of course JAR files in Java are just ZIP files with a little extra metadata and the JVM can unpack them in realtime just fine.

Re: Node.js needs a virtual file system

#36

Post by indutny »

epolanski wrote: post #31
indutny wrote: post #29

Taking the question of whether this would be a useful addition to Node.js core or aside, it must be noted that this 19k LoC PR was mostly generated by Claude Code and manually reviewed by the submitter which in my opinion is against the spirit of the project and directly violates the terms of Developer's Certificate of Origin set in the project's CONTRIBUTING.md

Do as I say, not as I do. On a more serious note, I think that this will be thoroughly reviewed before it gets merged and Node has an entire security team that overviews these.

As someone who was a part of the aforementioned security team I'm not sure I'd be interested in reviewing such volume of machine generated code, expecting trap at every corner. The implicit assumption that I observed at many OSS projects I've been involved with is that first time contributions are rarely accepted if they are too large in volume, and "core contributor" designation exists to signal "I put effort into this code, stand by it, and respect everyone's time in reviewing it". The PR in the post violates this social contract.

Re: Node.js needs a virtual file system

#37

Post by athorax »

indutny wrote: post #29

Taking the question of whether this would be a useful addition to Node.js core or aside, it must be noted that this 19k LoC PR was mostly generated by Claude Code and manually reviewed by the submitter which in my opinion is against the spirit of the project and directly violates the terms of Developer's Certificate of Origin set in the project's CONTRIBUTING.md

How exactly does it violate the Developer's Certificate of Origin clause?

Re: Node.js needs a virtual file system

#38

Post by mohsen1 »

Yarn, pnpm, webpack all have solutions for this. Great to see this becoming a standard. I have a project that is severely handicapped due to FS. Running 13k tests takes 40 minutes where a virtual file system that Node would just work with it would cut the run time to 3 minutes. I experimented with some hacks and decided to stay with slow but native FS solution.

What I really want is a way of swapping FS with VFS in a Node.js program harness. Something like

     node --use-vfs --vfs-cache=BIG_JSON_FILE
So basically Node never touches the disk and load everything from the memory

Re: Node.js needs a virtual file system

#40

Post by afavour »

pier25 wrote: post #39

The Node team has lost the plot IMO. By far the most critical issue is the over reliance on third party NPM packages for even fundamental needs like connecting to a database.

What would a Node-native database connection layer look like? What other platforms have that?

Databases are third party tech, I don’t think it’s unreasonable to use a third party NPM module to connect to them.

Post reply on HN