Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust compiler performance

kobzol.github.io

Post reply on HN
Print view View original

181–190 of 264 posts

Re: Rust compiler performance

#181

Post by ninkendo »

littlestymaar wrote: post #157
aloha2436 wrote: post #129

Earlier quoted context omitted.

> remains the same whether they pull source or binaries. I don't think that's exactly true, it's definitely _easier_ to sneak something into a binary without people noticing than it is to sneak it into rust source, but there hasn't been an underhanded rust competition for a while so I guess it's hard to be objective about that.

Pretty much nobody does those two things at the same time: - pulling dependencies with cargo - auditing the source code of the dependencies they're building You are either censoring and vetting everything or you're using dependencies from crates.io (ideally after you've done your due diligence on the crate), but should crates.io be compromised and inject malware in the crates' payload, I'm ready to bet nobody would n…

> Pretty much nobody does those two things at the same time: - pulling dependencies with cargo - auditing the source code of the dependencies they're building

Your “pretty much” is probably weaseling you out of any criticism here, but I fully disagree:

My IDE (rustrover) has “follow symbol” support, like every other IDE out there, and I regularly drill into code I’m calling in external crates. Like, just as often as my own code. I can’t imagine any other way of working: it’s important to read code you’re calling to understand it, regardless of whether it’s code made by someone else in the company, or someone else in the world.

My IDE’s search function shows all code from all crates in my dependencies. With everything equal regardless of whether it’s in my repo or not. It just subtly shades the external dependencies a slightly different color. I regularly look at a trait I need from another crate, and find implementations across my workspace and dependencies, including other crates and impls within the defining crate. Yes, this info is available on docs.rs but it’s 1000x easier to stay within my IDE, and the code itself is available right there inline, which is way more valuable than docs alone.

I think it’s insane to not read code you depend on.

Does this mean I’m “vetting” all the code I depend on? Of course not. But I’m regularly reading large chunks of it. And I suspect a large chunk of people work the way I do; There are a lot of eyeballs on public crates due to them being distributed as source, and this absolutely has a tangible impact on supply chain attacks.

Re: Rust compiler performance

#182

Post by baalimago »

diggan wrote: post #179
baalimago wrote: post #174

Earlier quoted context omitted.

>But we need more people to work on them. That's my point: I don't see how there could be people dedicated to work on an issue as grand as this in Rust's current organizational form. Especially considering all the gotchas, and continuous development of 'more fun' things (why work on open source if it's no fun?). That's why it's 'the bedrock'. To do something like that, Rust would need to be forked and later on rewrit…

> I don't see how there could be people dedicated to work on an issue as grand as this in Rust's current organizational form You're getting half ways there of giving actionable feedback, what exactly is the problem with the current organization structure that would prevent any "grand" issues like these? Is there a specific point in time when you felt like Rust stopped being able to work on these grand issues, or it h…

>what exactly is the problem with the current organization structure that would prevent any "grand" issues like these?

Well, it's summarized quite well here:

>"Performing large cross-cutting changes is also tricky because it will necessarily conflict with a lot of other changes being done to the compiler in the meantime. You can try to perform the modifications outside the main compiler tree, but that is almost doomed to fail, given how fast the compiler changes8. Alternatively, you try to land the changes in a single massive PR, which might lead to endless rebases (and might make it harder to find a reviewer). Or, you will need to do the migration incrementally, which might require maintaining two separate implementations of the same thing for a long time, which can be exhausting." - OP

A rigid organizational form (such as a company) can say: "Okay, we'll make an investment here and feature freeze until the refactors are done". I have a hard time seeing how the open source rust community who are doing this out of passion would get on board on such a journey. But maybe, who knows! The 'compilation people' would need to not only refactor to speed up the compilation times, they'd also need to encourage and/or perform refactors on all features being developed 'on main'. That, to me, sounds tedious and boring. Sort of like a job. Maybe something for the rust foundation to figure out.

Re: Rust compiler performance

#183

Post by littlestymaar »

moooo99 wrote: post #148
dgb23 wrote: post #143

Earlier quoted context omitted.

I had almost the exact opposite experience.

I share this impression. I’ve never worked much with low level languages with the exception of a few university assignments. I did last years advent of code in Zig and was quite productive and never really struggled with the language that much. Meanwhile, rust makes things a lot more complicated on my current project. The main benefit of rust over zig seems to be the maturity. Rust has had a decade+ to stabilize and…

Zig is C with a better syntax, easy things are easy to do, but for hard things you're on your own with no borrow checker friend to cover your back.

Also Rust makes you think more upfront (like string handling), which makes it more complicated for advent of code kind of stuff but vastly reduces bugs later on.

Re: Rust compiler performance

#184

Post by diggan »

baalimago wrote: post #182
diggan wrote: post #179

Earlier quoted context omitted.

> I don't see how there could be people dedicated to work on an issue as grand as this in Rust's current organizational form You're getting half ways there of giving actionable feedback, what exactly is the problem with the current organization structure that would prevent any "grand" issues like these? Is there a specific point in time when you felt like Rust stopped being able to work on these grand issues, or it h…

>what exactly is the problem with the current organization structure that would prevent any "grand" issues like these? Well, it's summarized quite well here: >"Performing large cross-cutting changes is also tricky because it will necessarily conflict with a lot of other changes being done to the compiler in the meantime. You can try to perform the modifications outside the main compiler tree, but that is almost doome…

I understand that quoted part as "it's tricky" rather than "It's impossible because no one wants to do it", just like many collaboration-efforts in FOSS. But you're right that it's probably for the foundation to figure out, a lone compiler-optimization geek isn't gonna be able to come up with a project-wide solution and force it through.

Haven't the Rust team already implemented "grand features" that took many years to get across the finish line? For example, GATs didn't look particularly fun, exciting or sexy, but somehow after being thought about and developed for like 5-6 years eventually landed in stable.

Edit: Also just remembered a lot of the work that "The Rust Async Working Group" has done, a lot which required a large collaborations between multiple groups within Rust. Seems to have worked out in the end too.

Re: Rust compiler performance

#185

Post by sapiogram »

maccard wrote: post #105
sapiogram wrote: post #82

Earlier quoted context omitted.

Thanks for actually including the slow repo in your comment. My results on a Ryzen 5900X: * Clean debug build: 1m 22s * Incremental debug build: 13s * Clean release build: 1m 51s * Incremental release build: 24s Incremental builds were done by changing one line in creates/symbolicator/src/cli.rs. It's not great, but it sounds like your experience was much worse for some reason.

Sorry - my clean build was actually including the dependency fetching, which is a large part of it. My experience was in 2023 which if we go by article roughly scales with compiler performance to 5 minutes or so

My build times also included fetching dependencies, which only took a few seconds on my modestly fast 8 MB/s network.

Re: Rust compiler performance

#186

Post by littlestymaar »

ninkendo wrote: post #181
littlestymaar wrote: post #157

Earlier quoted context omitted.

Pretty much nobody does those two things at the same time: - pulling dependencies with cargo - auditing the source code of the dependencies they're building You are either censoring and vetting everything or you're using dependencies from crates.io (ideally after you've done your due diligence on the crate), but should crates.io be compromised and inject malware in the crates' payload, I'm ready to bet nobody would n…

> Pretty much nobody does those two things at the same time: - pulling dependencies with cargo - auditing the source code of the dependencies they're building Your “pretty much” is probably weaseling you out of any criticism here, but I fully disagree: My IDE (rustrover) has “follow symbol” support, like every other IDE out there, and I regularly drill into code I’m calling in external crates. Like, just as often as…

You answer your own argument here:

> Does this mean I’m “vetting” all the code I depend on? Of course not.

Inspecting public facing parts of the code is one thing, finding nasty stuff obfuscated in a macro definition or in a Default or Debug implementation of a private type that nobody is ever going to check outside of auditors is a totally different thing.

> My IDE (rustrover) has “follow symbol” support

I don't know exactly how it works for RustRover, since I know Jetbrain has reimplemented some stuff on their own, but if it evaluates proc macros (like rust-analyzer) does, then by the time you step into the code it's too late, proc macros aren't sandboxed in any ways and your computer could be compromised already.

Re: Rust compiler performance

#187

Post by ninkendo »

littlestymaar wrote: post #186
ninkendo wrote: post #181

Earlier quoted context omitted.

> Pretty much nobody does those two things at the same time: - pulling dependencies with cargo - auditing the source code of the dependencies they're building Your “pretty much” is probably weaseling you out of any criticism here, but I fully disagree: My IDE (rustrover) has “follow symbol” support, like every other IDE out there, and I regularly drill into code I’m calling in external crates. Like, just as often as…

You answer your own argument here: > Does this mean I’m “vetting” all the code I depend on? Of course not. Inspecting public facing parts of the code is one thing, finding nasty stuff obfuscated in a macro definition or in a Default or Debug implementation of a private type that nobody is ever going to check outside of auditors is a totally different thing. > My IDE (rustrover) has “follow symbol” support I don't kno…

The point of my argument is not to say I’m vetting anything, but to say that there are tons of eyeballs on crates today, because of the fact that they are distributed as source and not a binary. It’s not a silver bullet but every little bit helps, every additional eyeball makes hiding things harder.

The original claim is that “pretty much no one” reads any of their dependencies, in order to support a claim that they should be distributed as binaries, meaning “if there was no source available at all in your IDE, it wouldn’t make a difference”, which is just a flatly wrong claim IMO.

A disagreement may be arising here about the definition of “audit” vs “reading” source code, but I’d argue it doesn’t matter for my point, which is that additional eyeballs matter for finding issues in dependencies, and seeing the source of your crates instead of a binary blob is essential for this.

Re: Rust compiler performance

#188

Post by Fiahil »

norir wrote: post #27

Compiler performance must be considered up front in language design. It is nearly impossible to fix once the language reaches a certain size without it being a priority. I recently saw here the observation that one can often get a 2x performance improvement through optimization, but 10x requires redesigning the architecture. Rust can likely never be rearchitected without causing a disastrous schism in the community,…

At some point, the community is also responsible for the demanding expectation of a "not slow" compiler.

What's "slow"? What's "fast"? It depends. It depends on the program, the programmer, his or her hardware, the day of the week, the hour of the day, the season, what he or she had for lunch, ...

It's a never ending quest.

I, for exemple, am perfectly happy with the current benchmark of the rust compiler. I find a x2 improvement absolutly excellent.

Re: Rust compiler performance

#189

Post by littlestymaar »

sureglymop wrote: post #156
littlestymaar wrote: post #52

In my experience working on medium-sized Rust projects (hundreds of thousands of LoCs, not millions), incremental compilation and mold pretty much solved the problem in practice. I still occasionally code on my 13 years old laptop when traveling and compilation time is fine even there (for cargo check and debug build, that is, I barely ever compile in release mode locally). What's painful is compiling from scratch, a…

Yes but are these the default? I want this to work pleasantly out of the box so we don't scare new users as quickly.

Incremental compilation has been the default for a few years now. Mold isn't and won't be anytime soon but AFAIK there's a PR to use lld by default and that should happen sooner than later. It's not exactly as fast as mold but it will still be a linking speed boost for everyone by default.

Re: Rust compiler performance

#190

Post by maccard »

pm215 wrote: post #164
maccard wrote: post #105

Earlier quoted context omitted.

Sorry - my clean build was actually including the dependency fetching, which is a large part of it. My experience was in 2023 which if we go by article roughly scales with compiler performance to 5 minutes or so

I do wonder how much the fetch-source-and-build-all-dependencies approach contributes to the poor perception of compile times -- your very first experience of "how long did cargo take to build this project" will be terrible because it will include downloading and building a ton of dependencies...

Agreed - I didn't do reliable benchmarks on how long was fetch + compile dependencies vs compiling the project itself.
Post reply on HN