Live data from Hacker News

Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

rust-lang.org

211–220 of 307 posts

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#211

Earlier quoted context omitted.

Except that npm will gladly update your lock file when you run npm install which is insane.

…so would Cargo? If you install a new package, why wouldn’t you expect it to show up in your lock file?

No, you guys don't understand.. npm updates the package lock even when not adding a new package, i.e. the initial `npm install`. It's insane I'm think to go back to yarn again..

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#212

Earlier quoted context omitted.

Except that npm will gladly update your lock file when you run npm install which is insane.

Why is that insane? What else is supposed to happen when you install a package? EDIT: I misunderstood and thought you were talking about installing a package. If you're running `npm install` to just reinstall dependencies then yes the lockfile should not be modified. However it seems like that is indeed the case and you may be talking about a prior bug with NPM.

`npm install` is what you the developer would run when you first clone a project; it should install exactly what's in the package-lock.json file. Unfortunately, it sometimes doesn't do that.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#213

Hey folks! This is part of our whitepaper series. This means that the audience is CTOs of larger organizations, and so the tone and content are geared for that, more than HN. Please keep that in mind!

Are CTO's of largish organizations still not part of the hacker news audience ...? That seems a bit of a damning statement to be about the population of CTO's ...

It's expected that high-level executives of large organizations deal with planning and strategy rather than low-level technical details. They're not following language features or implementations.

However in that case, this whitepaper (and many others) are damning in how little they actually state and why so many technical decisions go wrong.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#214
post #158

Earlier quoted context omitted.

Yep, not that surprising though, given the anemic state of the JavaScript standard library.

The idea of a scripting language is that it does not have a std. It will be different in each environment. You for example don't want the same std in nodejs and the browser. Each runtime can choose what API's it want to expose.

Python is the batteries included scripting language. The two concepts are not mutually exclusive.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#215
post #204

Earlier quoted context omitted.

Anecdotal, but I recently used Gulp in a project to run some css clean-up tasks as part of a build process. The JS dependencies: "gulp" "gulp-clean-css" "gulp-postcss" "gulp-uglify" "autoprefixer" "postcss-uncss" "uncss" The number of node modules: just over 400. So I'm not at all surprised that this might create surprises when deploying JS services in production.

That’s an ongoing annoyance with using NPM in a security-conscious environment. It’s really easy to end up with thousands of submodules and the amount of time you’ll have an audit showing a vulnerable package can be many months while layers of dependencies slowly update. You can usually show that it’s not exploitable but the number of modules on the average project means you’ll be doing that all the time.

npm now automatically reports known vulnerable packages, right?

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#216

Earlier quoted context omitted.

> write non duplicated code ... Like how many string implementations are there across c code bases. In one of projects I needed foo::string that can share data with foo::variant without copying the data each time. So foo::string was implemented as a COW string - smart pointer for foo::string_data. std::string simply does not work in such requirements. So I am not sure I understand how cargo will help in this case. Ei…

Your problem isn't hard to solve in Rust. Reasoning about allocations is a foundational idea of Rust and is solved using lifetimes. Basically, lifetimes are allocators as a first-class language construct. You can reason about what happens if values are stack or heap allocated and specialize your code based on that. Lifetimes are definitely an advanced feature and I think you can do some crazy optimizations with them.…

npm is used a lot for front end code, incidentally. Node is an environment for a lot of build tooling.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#217
I have my doubts and confusion about this problem statement

> Most of the operations npm performs are network-bound and JavaScript is able to underpin an implementation that meets the performance goals. However, looking at the authorization service that determines whether a user is allowed to, say, publish a particular package, they saw a CPUbound task that was projected to become a performance bottleneck.

Oh, Really ???

So essentially Authorization service and I doubt the security algorithms computation are the main cause.

What i dont understand here is why is it not possible to write lower level JS or asm code to craft a well optimized code which V8 can totally nail to minimum CPU instructions required?

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#218
post #72

Earlier quoted context omitted.

Not so ironic—just because a practice is common among the median engineer in an ecosystem, doesn’t mean it will be common among the most experienced engineers in that ecosystem. Node gets a bad rap mostly for the fact that it has tons and tons of inexperienced engineers using it (probably as one of their first programming languages.) Same reason PHP got a bad rap back when. You can build solid software in both, by fi…

Do you have any advice for PHP best practices? I'm starting a job that will probably have me using PHP...

https://phptherightway.com/

http://www.phpthewrongway.com/

https://paragonie.com/blog/2017/12/2018-guide-building-secur...

Edit: formatting

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#219
post #73

Earlier quoted context omitted.

You've identified the problem, but I think you're wrong about the cause. It's not internet centralization that's the problem, it's the mundane fact that JavaScript does not have a large standard library. And JavaScript absolutely should have a large standard library, but it's not clear what organization would have the motivation to implement, support, and promote one. It's not impossible that a community-driven effor…

I think the biggest to a hurdle is the fact that there is no stdlib so nothing unused has to be shaken out. I think that’s a low barrier now though with more than adequate tooling. Another comment mentions a different stdlib for server vs browser but that’s also not a terribly hard problem. I think a good first pass would come from studying analytics from npm. What are the most used packages? The most stable? I know…

> I think a good first pass would come from studying analytics from npm. What are the most used packages? The most stable?

I think it would also make a lot of sense to look at what's in the Python and Ruby standard libraries.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#220
post #23

This entire article is a pretty damning report on JavaScript in general, but this sentence takes the cake (emphasis mine): > The process of deploying the new Rust service was straight-forward, and soon they were able to forget about the Rust service because it caused so few operational issues. At npm, the usual experience of deploying a JavaScript service to production was that the service would need extensive monito…

They also state that writing the service in Node took them an hour, two days for Go, and a week for Rust. Even taking into account their unfamiliarity with the language, it's probably fair to say that when switching to Rust, you'll usually spend more time writing and less time debugging. Whether that trade-off is worth it depends on the project.

> writing the service in Node took them an hour

I'm really skeptical of this unless it's just a wrapper for a thing that happens to already exist. It would be interesting to have comparative LOC numbers.

> At npm, the usual experience of deploying a JavaScript service to production was that the service would need extensive monitoring for errors and excessive resource usage necessitating debugging and restarts

So, they deployed it after an hour, but it wasn't finished until they stopped having to debug it in production?

Post reply on HN