> despite having a 64-core machine with 128 threads, Rust barely puts any of them to work. Rust is fast in theory, but if in practice they can't even get their compiler to squeeze any juice from the CPU, then what's the value of that language from a software engineering viewpoint?
Compilation is inherently pretty hard to parallelise, and various design decisions around how Rust modules/creates work make it even harder (as demonstrated by achieving greater parallelism here with many smaller crates). There's no particular reflection on the performance of rust code here, it's really a design/algorithms problem.
Cutting down Rust compile times from 30 to 2 minutes with one thousand crates
51–60 of 89 posts
Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates
#52The main issue here is: - in rust one semantic compilation unit is one crate - in C one semantic compilation unit is one file There are quite a bunch of benefits in the rust approach, but also drawbacks, like huge projects have to be split into multiple workspaces to maximize parallel building. Oversimplified the codegen-units setting tells the compiler into how many parts the compiler is allowed to split the a singl…
> Now it still seems strange (as in it looks like a performance bug) that most times rust was stuck in just one threat (instead of e.g. 8). Agreed, seems like there are some rustc performance bugs at play here.
Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates
#53Earlier quoted context omitted.
> Compilation is inherently pretty hard to parallelise I don't agree. In a large project there is going to be a lot of stuff that can be compiled in parallel without problems.
And Rust's module design makes this significantly more complicated than in superficially-similar languages. That's why splitting it into modules brings drastic improvements - you are effectively giving the compiler clear boundaries across which it doesn't need to propagate as much information.
Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates
#54Earlier quoted context omitted.
Except the Rust ecosystem lacks the solutions we have in C++ land to compile fast and have easy paralelisation of builds. Because C and C++ communities for historical reasons embrace binary libraries, and binary component frameworks like COM, so while in theory a full build from scratch takes similar time as Rust, in practice that isn't the case. Also note that D, a language as complex as C++, with three compilers, o…
> Also note that D, a language as complex as C++ While D is complex, it's a different beast than Rust. I suspect think various checks, from lifetime to trait resolution, might make it more complex to parallelize than C++.
Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates
#55Earlier quoted context omitted.
> Now it still seems strange (as in it looks like a performance bug) that most times rust was stuck in just one threat (instead of e.g. 8). Agreed, seems like there are some rustc performance bugs at play here.
I haven't dug into the details, but it may not even be a performance bug, depending on how you define 'bug': the Rust compiler is not fully parallel itself yet. That's a bug in the sense of something that needs to be improved and fixed, but isn't one in the sense of "unexpected bad behavior".
Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates
#56Earlier quoted context omitted.
I haven't dug into the details, but it may not even be a performance bug, depending on how you define 'bug': the Rust compiler is not fully parallel itself yet. That's a bug in the sense of something that needs to be improved and fixed, but isn't one in the sense of "unexpected bad behavior".
Makes sense. We'd appreciate some more eyeballs here for sure. Between HN and a Reddit thread, there are a few hypotheses floating around. I've shared a repro here for anyone interested: https://github.com/feldera/feldera/issues/3882
Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates
#57Earlier quoted context omitted.
C++ is fast in theory, but if they can't get C++ compiler (LLVM) to squeeze any juice from CPU, then what's the value of that language from a software engineering viewpoint. Hopefully, you can see why this reasoning is a problem. The main stumbling point being compilation speed != runtime speed.
I have been using make's -j flag to compile C++ projects with great success. The main point being, Rust focuses on the wrong type of optimizations.
Rust focuses on a different set of optimizations. I'm still working out when I prefer the Rust set or the C++ set or the Python set. I want to love Rust, but when doing exploratory work with unfamiliar APIs, the slow recompile loop and need to get everything correct for each incremental experimental build I do to try to figure out how something works is pretty painful. I don't know how much better it gets with familiarity. Rust is very nice when I fully understand the problem I'm solving and the environment I'm working in, and I vastly prefer it to C++ for that. But I frequently dive into unfamiliar codebases to make modifications.
Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates
#58Mostly a throwaway code with a heavy input from Claude, so the docs are in the code itself :-)
But in case anyone can find it useful:
Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates
#59That said, I could see how it would make writing the transpiler easier, so that's a win.
Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates
#60the rust compiler is so impressively slow
It becomes more evident when you consider the amount of work it is doing as well.