Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Node.js needs a virtual file system

blog.platformatic.dev

Post reply on HN
Print view View original

161–170 of 275 posts

Re: Node.js needs a virtual file system

#161

Post by oystersareyum »

syrusakbary wrote: post #119
indutny wrote: post #29

Taking the question of whether this would be a useful addition to Node.js core or aside, it must be noted that this 19k LoC PR was mostly generated by Claude Code and manually reviewed by the submitter which in my opinion is against the spirit of the project and directly violates the terms of Developer's Certificate of Origin set in the project's CONTRIBUTING.md

Fully disagree with this take. Not allowing AI assistance on PRs will likely decimate the project in the future, as it will not allow fast iteration speeds compared to other alternatives. Note aside, OpenJS executive director mentioned it's ok to use AI assistance on Node.js contributions: I checked with legal and the foundation is fine with the DCO on AI-assisted contributions. We’ll work on getting this documented.…

Allowing AI contributions results in lower quality contributions and allows wild things to come in and disrupt it, making it an unreliable dependency. We have seen big tech experience constant outages due to AI contributions as is...

Re: Node.js needs a virtual file system

#163

Post by dijksterhuis »

syrusakbary wrote: post #156
jaredklewis wrote: post #154

Earlier quoted context omitted.

AI coding is great, but iteration speed is absolutely not a desirable trait for a runtime. Stability is everything. Speed code all your SaaS apps, but slow iteration speeds are better for a runtime because once you add something, you can basically never remove it. You can't iterate. You get literally one shot, and if you add a awkward or trappy API, everyone is now stuck with it forever. And what if this "must have"…

Since when we accepted that we can’t go fast and offer stability at the same time? Time is highly correlated with expertise. When you don’t have expertise, you may go fast at expense of stability because you lack the experience to make good decisions to really save speed. This doesn’t hold true for any projects where you rely on experts, good processes and tight timelines (aka: Apollo mission)

it’s a well known true-ism you can have it cheap, correct or fast.

but you can only have two of them at the same time.

and we’re talking about FOSS here, so cheap kinda has to be one of them.

Re: Node.js needs a virtual file system

#164

Post by nine_k »

sauercrowd wrote: post #145
socalgal2 wrote: post #144

What's special about node.js here? Does golang, C#, python, ruby, java, etc... have a virtual file system? I get it, I've implemented things for tests, I'm just wondering if this shouldn't be solved at an OS level. --- update Let's put this another way, my code does effectively, child_process.spawn('something-that-reads-and-write-a-file') now I'm back to the same issue. To test I need a virtual file system. Node prov…

Go actually does https://pkg.go.dev/embed I do think it's more painful to distribute files when you're a distributed as a single binary vs scripts, since the latter has to figure out bundling of files anyway. But still - it does exist

Zip files are created in such a way that they can be a part of an executable file. (This is how self-extracting archives used to work.) Support for reading zip files is lightweight, and is present almost everywhere.

A ZIP fork embedded into the executable should be an obvious read-only VFS implementation. Bring your assets with you, even maybe build them with the standard zip utility.

It should take relatively few LOCs, provided that libzip is already linked into the executable anyway.

Re: Node.js needs a virtual file system

#165

Post by jaredklewis »

syrusakbary wrote: post #156
jaredklewis wrote: post #154

Earlier quoted context omitted.

AI coding is great, but iteration speed is absolutely not a desirable trait for a runtime. Stability is everything. Speed code all your SaaS apps, but slow iteration speeds are better for a runtime because once you add something, you can basically never remove it. You can't iterate. You get literally one shot, and if you add a awkward or trappy API, everyone is now stuck with it forever. And what if this "must have"…

Since when we accepted that we can’t go fast and offer stability at the same time? Time is highly correlated with expertise. When you don’t have expertise, you may go fast at expense of stability because you lack the experience to make good decisions to really save speed. This doesn’t hold true for any projects where you rely on experts, good processes and tight timelines (aka: Apollo mission)

IME there's a reason it's "move fast and break things" and not "move fast and don't break anything," because if the second was generally possible, we wouldn't even need this little aphorism.

And again, I'm not making a claim that the slow and steady tradeoff is best for all situations. Just that it is a great tradeoff for foundational platforms like a runtime. On a platform like postgresql or the JVM, the time from initial proposal to being released as a stable feature is generally years, and this pace I think has served those platforms well.

But I'm open to updating my priors. Do you think there are foundational platforms out there that iterate quickly and do a good job of it?

Re: Node.js needs a virtual file system

#166

Post by joshkel »

grey-area wrote: post #148
giancarlostoro wrote: post #137

Earlier quoted context omitted.

> I pointed the AI at the tedious parts, the stuff that makes a 14k-line PR possible but no human wants to hand-write: implementing every fs method variant (sync, callback, promises), wiring up test coverage, and generating docs. Is it slop if it is carefully calculated? I tire of hearing people use slop to mean anything AI, even when it is carefully reviewed.

Was 14k lines carefully reviewed? Seems unlikely.

Considering the many hundreds of technical comments over at the PR (https://github.com/nodejs/node/pull/61478), the 8 reviewers thanked by name in the article, and the stellar reputations of those involved, seems likely.

Re: Node.js needs a virtual file system

#167

Post by IgorPartola »

dakiol wrote: post #95
digikata wrote: post #53

Earlier quoted context omitted.

Large PRs could follow the practices that the Linux kernel dev lists follow. Sometimes large subsystem changes could be carried separately for a while by the submitter for testing and maintenance before being accepted in theory, reviewed, and if ready, then merged. While the large code changes were maintained, they were often split up into a set of semantically meaningful commits for purposes of review and maintenanc…

Nobody wants to review AI-generated code (unless we are paid for doing so). Open source is fun, that's why people do it for free... adding AI to the mix is just insulting to some, and boring to others. Like, why on earth would I spent hours reviewing your PR that you/Claude took 5 minutes to write? I couldn't care less if it improves (best case scenario) my open source codebase, I simply don't enjoy the imbalance.

In theory because the code being added is introducing a feature so compelling that it is worth it. In practice, that’s rarely the case.

My personal approach to open source is more or less that when I need a piece of software to exist that does not and there is no good reason to keep it private, it becomes open source. I don’t do it for fun, I do it because I need it and might as well share it. If someone sends me a patch that enhances my use case, I will work with them to incorporate it. If they send me a patch that only benefits them it becomes a calculus of how much effort would it take for me to review it. If the effort is high, my advice is to fork the project or make it easier for me to review. Granted I don’t maintain huge or vital projects, but that’s precisely why: I don’t need yet another programming language or runtime to exist and I wouldn’t want to work on one for fun.

Re: Node.js needs a virtual file system

#168

Post by IgorPartola »

gonzalohm wrote: post #147
tyre wrote: post #143

Earlier quoted context omitted.

Why do you care how much effort it took the engineer to make it? If there was a huge amount of tedium that they used Claude Code for, then reviewed and cleaned up so that it’s indistinguishable from whatever you’d expect from a human; what’s it to you? Not everyone has the same motivations. I’ve done open source for fun, I’ve done it to unblock something at work, I’ve done it to fix something that annoys me. If your…

Because sometimes programming is an art and we want people to do it as if it was something they cared about. I play chess and this is a bit like that. Why do I play against humans? Because I want to face another person like me and see what strategies they can come up with. Of course any chess bot is going to play better, but that's not the point

What about the other times?

Re: Node.js needs a virtual file system

#169

Post by nine_k »

conartist6 wrote: post #158
indutny wrote: post #29

Taking the question of whether this would be a useful addition to Node.js core or aside, it must be noted that this 19k LoC PR was mostly generated by Claude Code and manually reviewed by the submitter which in my opinion is against the spirit of the project and directly violates the terms of Developer's Certificate of Origin set in the project's CONTRIBUTING.md

Pain is a signal. Even if the trick is not minding, it's still inadvisable to burn your hand on an open flame. The pain is there to help you not get hurt. I do not think it is wise to brag that your solution to a problem is extremely painful but that you were impervious to all the pain. Others will still feel it. This code takes bandwidth to host and space on devices and for maintainers it permanently doubles the wor…

I don't see it to be such a pain.

> Bundle a full application into a Single Executable.

Embed a zip file into the executable, or something. Node sort of supports this since v25, see --build-sea. Bun and Deno support this for a longer time.

> Run tests without touching the disk.

This must be left to the host system to decide. Maybe I want them to touch the disk and leave traces useful for debugging. I'd go with tmpfile / tmpdir; whoever cares, knows to mount them as tmpfs, which sits in RAM. (Or a ramdisk under Windows.)

> Sandbox a tenant’s file access. In a multi-tenant platform, you need to confine each tenant to a directory without them escaping

This looks like a wrong tool, again. Run your Node app in a container (like you are already doing), mount every tenant's directory as a separate mount point into your container. (Similar with BSD jails.) This seems like the only problem that is not trivial to solve without a "VFS", but I'm not very certain that such a VFS would be as well-audited as Docker, or nsenter and unshare. The amount of work necessary for implementing that is too much for the niche benefit it would provide.

> Load code generated at runtime. See tmpfs for a trivial answer. For a less trivial answer, I don't see how Node's code loader is bound to a filesystem. If it can import via https, Just use ESM loader hooks and register() your loader, assuming you're running Node ≥ 20.6.

Re: Node.js needs a virtual file system

#170

Post by keeganpoppen »

joshkel wrote: post #166
grey-area wrote: post #148

Earlier quoted context omitted.

Was 14k lines carefully reviewed? Seems unlikely.

Considering the many hundreds of technical comments over at the PR ( https://github.com/nodejs/node/pull/61478 ), the 8 reviewers thanked by name in the article, and the stellar reputations of those involved, seems likely.

[flagged]
Post reply on HN