Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

rust-lang.org

Post reply on HN
Print view View original

201–210 of 307 posts

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#201

Post by w0rd-driven »

curtis wrote: post #73
sheeshkebab wrote: post #16

Earlier quoted context omitted.

Deploying a service written in any language into production environment at scale of npmjs is far from straightforward. I think the satire here is that internet got so centralized lately that even a simple piece of code in JavaScript requires such a huge behemoth of an org running and maintaining all this monstrous infrastructure.

You've identified the problem, but I think you're wrong about the cause. It's not internet centralization that's the problem, it's the mundane fact that JavaScript does not have a large standard library. And JavaScript absolutely should have a large standard library, but it's not clear what organization would have the motivation to implement, support, and promote one. It's not impossible that a community-driven effor…

I think the biggest to a hurdle is the fact that there is no stdlib so nothing unused has to be shaken out. I think that’s a low barrier now though with more than adequate tooling. Another comment mentions a different stdlib for server vs browser but that’s also not a terribly hard problem.

I think a good first pass would come from studying analytics from npm. What are the most used packages? The most stable? I know lodash makes a lot of sense but there’s also underscore. I think the biggest hurdles are really political over technology as everyone has been so entrenched now that a one-size-fits-all stdlib would be hard. Not impossible, just hard. I do wish someone were working on it and I hate to say it but Google probably has the most skin in the game with v8 and Chrome yet I don’t really trust them not to abandon it. So who else is there? It wouldn’t be a very ‘sexy project’ either but still seems worth it to at least try.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#202

Post by burntsushi »

pcwalton wrote: post #146
paulddraper wrote: post #117

Earlier quoted context omitted.

> New java versions do break existing libraries or apps Have you used the Rust compiler? My experience tells me that any Github Rust project not updated in the last two years doesn't work with my Rust compiler. Whereas Java apps written a decade ago still compile and run on OpenJDK/Oracle often with zero or near zero changes. > Often an architect or software team will insist on using the Oracle JVM rather than the in…

> My experience tells me that any Github Rust project not updated in the last two years doesn't work with my Rust compiler. Please file bugs against the Rust compiler then, because that would be a serious violation of the compatibility rules!

It's definitely a thing. It's easier to see with applications that use a lock file rather than libraries that always try to fetch the latest dependencies. That's because there have been incompatible changes made, but within the "allowed breakage" policy. For example, try checking out 0.2.0 or 0.3.0 of ripgrep and building it with Rust 1.33. Both fail for different reasons. The former fails because rustc-serialize 0.3.19 was using an irrefutable pattern in a method without a body, and this was prohibited: https://github.com/rust-lang/rust/pull/37378 However, if you run `cargo update` first, then the build succeeds.

The 0.3.0 checkout fails because of a change in Cargo. Specifically, I think it used to define OUT_DIR while compiling the build script, but this was not intentional. It was only supposed to be set when running the build script. ripgrep 0.3.0 got caught here in the time after when the bug was introduced but before it was fixed. See: https://github.com/rust-lang/cargo/issues/3368 In this case, cargo update doesn't solve this. You cannot build ripgrep 0.3.0 with any recent version of the compiler. You need to go back to a version of the Rust distribution which ships a Cargo version that sets OUT_DIR when compiling build scripts. (Or, more likely, patch build.rs since it's a trivial fix.)

Of course, both of these things are fine on their own. But strictly speaking, from a user experience point of view, code does break from time to time. With that said, ripgrep 0.4.0 and newer all compile on Rust 1.33, and ripgrep 0.4.0 was released over two years ago. So, strictly speaking, it does show the GP is wrong in at least some circumstances. :-)

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#204

Post by acdha »

ddebernardy wrote: post #181
nothrabannosir wrote: post #10

This entire article is a pretty damning report on JavaScript in general, but this sentence takes the cake (emphasis mine): > The process of deploying the new Rust service was straight-forward, and soon they were able to forget about the Rust service because it caused so few operational issues. At npm, the usual experience of deploying a JavaScript service to production was that the service would need extensive monito…

Anecdotal, but I recently used Gulp in a project to run some css clean-up tasks as part of a build process. The JS dependencies: "gulp" "gulp-clean-css" "gulp-postcss" "gulp-uglify" "autoprefixer" "postcss-uncss" "uncss" The number of node modules: just over 400. So I'm not at all surprised that this might create surprises when deploying JS services in production.

That’s an ongoing annoyance with using NPM in a security-conscious environment. It’s really easy to end up with thousands of submodules and the amount of time you’ll have an audit showing a vulnerable package can be many months while layers of dependencies slowly update. You can usually show that it’s not exploitable but the number of modules on the average project means you’ll be doing that all the time.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#205

Post by dbaupp »

kristopolous wrote: post #120
atoav wrote: post #50

Earlier quoted context omitted.

Rust's package managment (cargo) is the best thing I have ever seen of it's kind. The very basic thing you can do is: cargo new funkyproject Which creates a new barebones rust project called "funkyproject". Every dependency specified in it's Cargo.toml will be automatically downloaded at build (if there is a new version). When a build is sucessful the versions of said dependency will be saved into a Cargo.lock file.…

> Every dependency specified in it's Cargo.toml will be automatically downloaded at build (if there is a new version). Why do people want this? The builds are no longer reproducible, security and edge case issues can come out of nowhere, api changes from an irresponsible maintainer can break things, network and resource failure can break the build, it's just a terrible idea. The proper use of a semvar system is entir…

You're misreading your parent. The download only happens for the first build using a new dependency. As they mention, once the version is written into the Cargo.lock file, that is the exact version that is used until there is an explicit update step run.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#206

Post by yawaramin »

Filligree wrote: post #132
tyingq wrote: post #99

Earlier quoted context omitted.

I'm not clear on why this is particularly CPU heavy, though: "the authorization service that determines whether a user is allowed to, say, publish a particular package"

Lots of users. Anything will be CPU-heavy if you give it enough work. Except the things that end up being memory-bound instead, but the NPM database isn't large enough for that.

Couldn't you implement the authorization logic in, say, Redis? Then the npm service is I/O-bound again, and everyone is doing the job they're optimized for.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#207

Post by echelon »

nothrabannosir wrote: post #10

This entire article is a pretty damning report on JavaScript in general, but this sentence takes the cake (emphasis mine): > The process of deploying the new Rust service was straight-forward, and soon they were able to forget about the Rust service because it caused so few operational issues. At npm, the usual experience of deploying a JavaScript service to production was that the service would need extensive monito…

This is my experience, too.

I wrote trumped.com and deployed it prior to the last presidential election. The frontend and assets have been redeployed, but the core rust service for speech generation hasn't been touched. I've never had a service this reliable, and it took so little effort!

Rust is the best language I've ever used, bar none, period. And I've used a countless many of them.

The only places where I won't write Rust are for small one-off scripts and frontend web code. (Even with wasm, Typescript would be tough to beat.)

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#208

Post by manigandham »

spdionis wrote: post #169
zukzuk wrote: post #59

Earlier quoted context omitted.

npm (and yarn) literally does exactly all of this, via `npm init funkyproject` and `package-lock.json`.

Except that npm will gladly update your lock file when you run npm install which is insane.

Why is that insane? What else is supposed to happen when you install a package?

EDIT: I misunderstood and thought you were talking about installing a package. If you're running `npm install` to just reinstall dependencies then yes the lockfile should not be modified. However it seems like that is indeed the case and you may be talking about a prior bug with NPM.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#209

Post by simion314 »

ambicapter wrote: post #161
derefr wrote: post #72

Earlier quoted context omitted.

Not so ironic—just because a practice is common among the median engineer in an ecosystem, doesn’t mean it will be common among the most experienced engineers in that ecosystem. Node gets a bad rap mostly for the fact that it has tons and tons of inexperienced engineers using it (probably as one of their first programming languages.) Same reason PHP got a bad rap back when. You can build solid software in both, by fi…

Do you have any advice for PHP best practices? I'm starting a job that will probably have me using PHP...

I think majority of the good practices are the same in PHp as in other languages like Java, many PHP issues were caused by creating SQL statements using user input(you can have same issues with beginner devs in most languages), so the best practice is to find a framework that does all the ugly part for you(the logins, session management, routing), there are large and small frameworks , I suggest use a popular one even is not perfect(to avoid the risk of hitting bugs if is not used that much or the risk of it getting abandoned or deprecated),

Don't get me wrong, if you need to do one small thing, like for example I had a desktop app and the user could submit feedback directly from the app, then a single PHP file was enough for this case(no dependencies, no frameworks), you get the submitted data, clean it and put it in the database or submit it to a third-party API that can handle it.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#210

Post by fwip »

spdionis wrote: post #169
zukzuk wrote: post #59

Earlier quoted context omitted.

npm (and yarn) literally does exactly all of this, via `npm init funkyproject` and `package-lock.json`.

Except that npm will gladly update your lock file when you run npm install which is insane.

Npm hasn't done this in over a year.
Post reply on HN