Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

rust-lang.org

Post reply on HN
Print view View original

251–260 of 307 posts

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#251

Post by catach »

mlthoughts2018 wrote: post #65
Ygg2 wrote: post #41

Earlier quoted context omitted.

> It’s the same with C or C++. These languages support a type of low-level memory and data model that is permissive in ways that can lead to errors if not used properly. To be honest, I don't think I know of single piece of C/C++ code that doesn't suffer from security exploits from "improper" memory use. Chrome does, Firefox does, Linux does. Hell car and pacemaker software suffers from exploits, most likely caused b…

This type of thinking is exactly why I stay away from the developer communities of Rust and Julia. Both communities seem to have this weird perception that their respective language categorically supersedes other tools in all use cases. Don’t get me wrong. Rust and Julia are very cool languages with a lot of use cases where they are great choices. But the communities come off as disproportionately full of zealots, wh…

As someone who does not stay away from the Rust community, my experience is that r/Rust is persistently inclusive when it comes to other languages:

https://www.reddit.com/r/rust/comments/a1ffb4/should_i_learn...

https://www.reddit.com/r/rust/comments/acpidm/should_i_start...

https://www.reddit.com/r/rust/comments/afk6mi/starting_a_new...

https://www.reddit.com/r/rust/comments/arndm5/does_rust_work...

https://www.reddit.com/r/rust/comments/an11l9/webdevelopemen...

https://www.reddit.com/r/rust/comments/6pg4y8/learning_rust_...

https://www.reddit.com/r/rust/comments/777spr/would_you_reco...

https://www.reddit.com/r/rust/comments/70ajv1/im_getting_int...

https://www.reddit.com/r/rust/comments/arhuv8/i_feel_like_ru...

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#252

Post by manigandham »

rolleiflex wrote: post #244
fwip wrote: post #210

Earlier quoted context omitted.

Npm hasn't done this in over a year.

The current version of npm does this and this is "correct behaviour". I got bitten by this a few weeks ago. For the passers-by, the only way to make npm behave expectedly in this specific case is to use "npm ci" instead of "npm install" . If you do not do this, npm will assume you want to update the packages to the latest version at all times, at all costs, even if you have a lock file in place, and even if you have…

Or use Yarn.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#253

Post by makkesk8 »

steveklabnik wrote: post #40
makkesk8 wrote: post #17

Interested to know why they didn't consider .net core.

This stuff happened before, or at least around the time, that .NET Core was released. So it either didn’t exist or was an extremely new option, at least.

According to the article they have been using rust in production for 1½ years, .net core 2 was released in 2017 so it definitely existed. If .net core was mature enough for their liking is however another matter.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#254

Post by acdha »

simion314 wrote: post #248
acdha wrote: post #231

Earlier quoted context omitted.

Java by default has a hard memory limit which when hit causes services to hang in a semi-responsive state rather than exiting cleanly, requiring every user to hand-tune limits and carefully setup health checks and monitoring. Two decades in, they added a non-standard option to instead exit. If they had instead made the default to act like almost everything else it would have worked with standard process monitors and…

Isn't this default better for average Joe that runs a Java desktop app, and developers that deploy apps should know how to change this defaults?

How is it better for something to stop working but not exit? It’s exceedingly uncommon for anyone to have correct handling for OOM exceptions so the most likely effect is that stuff partially works - servers accept requests but never respond, apps have menus/buttons which don’t work, etc.

Similarly, if developers who deployed apps knew enough to avoid this, we’d know by now because it wouldn’t happen so frequently. It does highlight who failed to do real health-checks (e.g. years ago most Hadoop services needed huge amounts of RAM to start without crashing but they’d be listed as okay) but it’s the kind of thing sysadmins have been cleaning up for decades.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#255

Post by steveklabnik »

paulddraper wrote: post #235
pcwalton wrote: post #146

Earlier quoted context omitted.

> My experience tells me that any Github Rust project not updated in the last two years doesn't work with my Rust compiler. Please file bugs against the Rust compiler then, because that would be a serious violation of the compatibility rules!

Unfortunately, I don't know enough to rewind history and do enough archaeology to find out whose "fault" it is. But as a possible example, typemap [1] which is a library featured on Awesome Rust [2] ("curated list of Rust code and resources"). Project uses cargo. Has a lock file with two deps. Last code commit was May 2017. And I can't get it to compile. [1] https://github.com/reem/rust-typemap [2] https://github.com…

What error do you get?

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#256

Post by acdha »

acdha wrote: post #222
steveklabnik wrote: post #215

Earlier quoted context omitted.

npm now automatically reports known vulnerable packages, right?

Yes - which is great for surfacing this, along with GitHub’s alerts, but unless it’s a direct dependency I find I’m usually just stuck researching the vector and waiting months for numerous layers of dependencies to update in sequence.

Oh, and to be clear: I think this is a problem with OSS sustainability – shipping updates takes real work – more than NPM, mildly exacerbated by the JS stdlib leading to more modules being used instead.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#257

Post by steveklabnik »

makkesk8 wrote: post #253
steveklabnik wrote: post #40

Earlier quoted context omitted.

This stuff happened before, or at least around the time, that .NET Core was released. So it either didn’t exist or was an extremely new option, at least.

According to the article they have been using rust in production for 1½ years, .net core 2 was released in 2017 so it definitely existed. If .net core was mature enough for their liking is however another matter.

.net core was released in August 2017. That’s basically one and a half years ago. Depending on how fuzzy the one and a half year time is, it may not have been released. Like it may not have literally been 18 months exactly. That’s all I’m saying.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#259

Post by pcwalton »

paulddraper wrote: post #235
pcwalton wrote: post #146

Earlier quoted context omitted.

> My experience tells me that any Github Rust project not updated in the last two years doesn't work with my Rust compiler. Please file bugs against the Rust compiler then, because that would be a serious violation of the compatibility rules!

Unfortunately, I don't know enough to rewind history and do enough archaeology to find out whose "fault" it is. But as a possible example, typemap [1] which is a library featured on Awesome Rust [2] ("curated list of Rust code and resources"). Project uses cargo. Has a lock file with two deps. Last code commit was May 2017. And I can't get it to compile. [1] https://github.com/reem/rust-typemap [2] https://github.com…

There's no lock file in that project.

But moreover, you stumbled upon one of the worst cases, a soundness problem that has yielded a lot of discussion because of that crate specifically: https://github.com/rust-lang/rust/issues/50781

The issue here is that the typemap crate was found to be relying on a compiler bug. The compiler bug could be exploited to write transmute in safe code: in other words, all the safety guarantees of Rust go out the window unless we break that crate. It's a bad situation, but I don't know what we could have done differently. I think everyone agrees it's not worth sacrificing all the safety properties of Rust to keep typemap building.

Re: Case Study: Npm uses Rust for its CPU-bound bottlenecks [pdf]

#260

Post by mlthoughts2018 »

Ygg2 wrote: post #81
mlthoughts2018 wrote: post #65

Earlier quoted context omitted.

This type of thinking is exactly why I stay away from the developer communities of Rust and Julia. Both communities seem to have this weird perception that their respective language categorically supersedes other tools in all use cases. Don’t get me wrong. Rust and Julia are very cool languages with a lot of use cases where they are great choices. But the communities come off as disproportionately full of zealots, wh…

For what is worth, I'm not a Rust dev. I did wrote some code in Rust, but by that definition I'm also a Fantom dev. I've just seen too many memory exploits in C/C++ to consider statements like "you can achieve memory safety in C/C++ if you're a True Scotsman" as anything other as logical fallacies.

> "I've just seen too many memory exploits in C/C++ to consider statements like "you can achieve memory safety in C/C++ if you're a True Scotsman" as anything other as logical fallacies."

The problem is that for each person who offers your anecdata, there's another person who offers the antithetical anecdata. For example, in my experience with two very large defense lab projects written in C and C++ spanning the 80s to the early 00s, both projects had exceptionally nice tooling to prevent memory issues, and it was pleasant to work with the code. Debugging time was usually spent on application logic bugs and very rarely, if ever, spent on memory leaks, segfaults, corrupt data, alignment issues, etc.

While I know this isn't directly applicable to Rust, I have also worked in a separate codebase that was a huge Haskell project and it was a nightmare of spaghetti code, performance bugs that were impossible to reason about, some segfaults, and it was extremely painful to descend into unsafe code for the occasional situations when it was required. Having the compiler to help us and leveraging the type system as part of the application design didn't really offer any significant benefits.

This doesn't prove anything except that, like usual, it's situation-specific and depends on relevant trade-offs at hand. It almost never has anything to do with the generic types of safety guarantees that a framework or language allows you to formally verify.

Post reply on HN