Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf] • Hacker News Board

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#171

Post by gkop » Sun, Mar 03, 2019, 7:41 PM UTC

IsaacSchlueter wrote: post #162

gkop wrote: post #153

Earlier quoted context omitted.

You have to run npm ci instead of npm install to get npm to respect the lock file. I don’t consider that remotely obvious. And this feature was just added to npm last year, 8 years after npm was invented!

That is incorrect. Both `npm install` and `npm ci` respect the lock file, and if a lock file is present, will make the `node_modules` tree match the lock file exactly. `npm ci` is optimized for a cold start, like on a CI server, where it's expected that `node_modules` will not be present. So, it doesn't bother looking in `node_modules` to see what's already installed. So, _in that cold start case_, it's faster, but i…

Thanks for the reply Isaac! This doesn’t match my first-hand experience unfortunately. Are there any circumstances under which npm install with a lockfile present deviates from the lockfile where npm ci does not?

For example, why did this person experience the changing lockfile? https://github.com/npm/npm/issues/17101

Or why do these docs say?

> Whenever you run npm install, npm generates or updates your package lock https://docs.npmjs.com/files/package-locks

Oh, this seems like what I experienced: https://stackoverflow.com/a/45566871/283398

It does appear that npm works somewhat differently than the “obvious” way we would expect package managers to work vis a vis lockfiles :(

At least npm ci gets the job done for my use case :)

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#172

Post by CamouflagedKiwi » Sun, Mar 03, 2019, 7:43 PM UTC

tyingq wrote: post #89

CamouflagedKiwi wrote: post #83

Earlier quoted context omitted.

> Imagine the K8S team porting from Go to some other language for similar reasons. k8s was originally written in Java, so they have already done this once.

I didn't know that, but it makes it a more interesting comparison. I imagine at least part of rewriting it in Go was to "eat your own dogfood".

My guess would be more on the side of native binaries (i.e. no runtime JVM dep) and lower memory use, but could be that too.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#173

Post by IsaacSchlueter » Sun, Mar 03, 2019, 7:43 PM UTC

mattashii wrote: post #119

amluto wrote: post #94

Earlier quoted context omitted.

Which is roughly the equivalent of every single human being downloading two npm packages per week. To me, this suggests that the real problem is that too many packages are being downloaded.

> Which is roughly the equivalent of every single human being downloading two npm packages per week The current human population of earth is about 7.7 billion, so that number should probably be closer to 1.17 npm packages per week per human being. That is still quite a lot, though

This highlights the problem of averages. Most (99.87% or so) humans download zero npm packages. But those that do, often download them in the thousands at a time. And yes, clean-room CI servers are a big part of that.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#174

Post by spdionis » Sun, Mar 03, 2019, 7:46 PM UTC

IsaacSchlueter wrote: post #162

gkop wrote: post #153

Earlier quoted context omitted.

You have to run npm ci instead of npm install to get npm to respect the lock file. I don’t consider that remotely obvious. And this feature was just added to npm last year, 8 years after npm was invented!

That is incorrect. Both `npm install` and `npm ci` respect the lock file, and if a lock file is present, will make the `node_modules` tree match the lock file exactly. `npm ci` is optimized for a cold start, like on a CI server, where it's expected that `node_modules` will not be present. So, it doesn't bother looking in `node_modules` to see what's already installed. So, _in that cold start case_, it's faster, but i…

> Both `npm install` and `npm ci` respect the lock file

This is not correct. `npm install` will update your dependencies, not install them, disregarding the package versions defined in the lock file.

It feels like you are not getting the point of having a lock file in the first place. It should be obvious that you can't do an install (which npm calls ci) if you don't have a lock file.

The lock file represents your actual dependencies. Package.json should only be used to explicitly update said dependencies.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#175

Post by z3t4 » Sun, Mar 03, 2019, 7:49 PM UTC

You shouldn't be afraid to make independent micro services. Using a different language makes it more likely that it can be deleted, and more easily rewritten. If you are forced into a monolith usually means the architecture has a complexity problem.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#176

Post by pjmlp » Sun, Mar 03, 2019, 7:49 PM UTC

tyingq wrote: post #167

pjmlp wrote: post #163

Earlier quoted context omitted.

You would be surprised at the quality of Android tooling stable releases, to the point that now there is Project Marble in place to try to improve its image. https://adtmag.com/articles/2019/01/17/project-marble.aspx If you want to have some reading fun, check /r/androiddev/ every time there is a "stable" release.

I bet k8s would look different if it were written by developers used to a more limited set of cpu, io, and memory resources.

Well, given the hops one has to go through for the privilege to work at Google, another level of code quality is to be expected.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#177

Post by nwlieb » Sun, Mar 03, 2019, 7:50 PM UTC

Thaxll wrote: post #150

tyingq wrote: post #99

Earlier quoted context omitted.

I'm not clear on why this is particularly CPU heavy, though: "the authorization service that determines whether a user is allowed to, say, publish a particular package"

It's not clear either, it makes sens to use Rust for CPU heavy task, but a CRUD service that do authentication would be fine in Nodejs since every low level crypto are using C. So I'm not sure exatly what they mean, tbh the paper is very light on details.

Some CPU heavy operations like crypto are not put in a threadpool. While it may be running C code, it will block the main thread while executing. See https://github.com/nodejs/node/issues/678

Perhaps the new worker threads may alleviate this, but I'm not sure (it's still an experimental API).

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#178

Post by simion314 » Sun, Mar 03, 2019, 7:51 PM UTC

weberc2 wrote: post #88

simion314 wrote: post #24

Earlier quoted context omitted.

But having options is always good, isn't the same with compiled languages? you keep the defaults options but if you want extra performance you try different compiler flags or different compiler if the language has more then one.

Honestly I’d rather have good defaults than lots of options. If one tool meets requirements out of the box, then it’s preferable to a tool that requires a lot of tooling.

But if you do not have options then what is a "good default" , you have a default option if you have more then an option, so you prefer software with no options?

Can you give examples of languages(compilers or VMs with bad defaults in your opinion?

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#179

Post by tyingq » Sun, Mar 03, 2019, 7:52 PM UTC

ambicapter wrote: post #161

derefr wrote: post #72

Earlier quoted context omitted.

Not so ironic—just because a practice is common among the median engineer in an ecosystem, doesn’t mean it will be common among the most experienced engineers in that ecosystem. Node gets a bad rap mostly for the fact that it has tons and tons of inexperienced engineers using it (probably as one of their first programming languages.) Same reason PHP got a bad rap back when. You can build solid software in both, by fi…

Do you have any advice for PHP best practices? I'm starting a job that will probably have me using PHP...

A fair amount of the pitfalls are already gone. Register_globals is off by default now, placeholders for sql queries are the default examples, etc.

You will probably hear that you should use Laravel or similar. I'd argue it's a pretty big hammer, so don't reach for it if you don't need it.

The biggest issue is probably still the breadth and inconsistency of the standard library. Too many ways to do the same thing. Also, the general issues of a dynamically typed language, sprinkled in with things like == vs ===.

I like the books this guy writes: https://github.com/codeguy

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#180

Post by lioeters » Sun, Mar 03, 2019, 7:58 PM UTC

adamnemecek wrote: post #12

I’ve been playing around with rust since it came out but only recently did I decide to use it for a part of a project. It’s a very pleasant language. I didn’t fight the borrow checker much (maybe due to prior experience). The language is nuts. It’s true what they say cargo is even better than the language, it’s just so easy to add packages to your project or to split your project into packages. Cargo is an amazing in…

> Modern development needs a workflow where you pull in a dependency, and work on it in tandem with your code. Achieving a good workflow for this is surprisingly hard.

Well put! I've struggled with this exact situation, and although I figured out a setup that works for me, it's still not ideal. In my experience, npm the package manager doesn't enable such workflows reliably (yet).