Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust compiler performance

kobzol.github.io

Post reply on HN
Print view View original

241–250 of 264 posts

Re: Rust compiler performance

#241

Post by littlestymaar »

ninkendo wrote: post #240
littlestymaar wrote: post #231

Earlier quoted context omitted.

I'm not moving the goal post, a supply chain attack is an adversarial situation it is not about spotting an issue occurring at random, it is about spotting an issue specially crafted to avoid detection. So in practice you are either able to spot every kind of issues , or none of the relevant ones because if there's one kind that reliably slips through, then you can be certain that the attacker will focus on this kind…

In such an environment I’m doomed anyway, even if I’m vetting code. I don’t understand why the goal has to be “the ability to spot attacks specifically designed to prevent you from detecting.” For what you’re describing, there seems to be no hope at all. It’s like if someone says “don’t pipe curl into bash to install software”, ok that may or may not be good advice. But then someone else says “yeah, I download the sc…

> It’s like if someone says “don’t pipe curl into bash to install software”, ok that may or may not be good advice. But then someone else says “yeah, I download the script first and give it a cursory glance to see what it’s doing”, wouldn’t you agree they’re marginally better off than the people who just do it blindly?

I don't agree with your comparison, in this case it's more like downloading, then running it without having read it and then every once in a while look at a snippet containing a feature that interest you.

The comparison to “download the script and read it before you run it” would be to download the crate's repo, read it and then vendor the code you've read to use as a dependency, which is what I'd consider proper vetting (in this case the attacker would need to be much more sophisticated to avoid detection, it's still possible but in this case at least you've actually gained something), but it's a lot more work.

Re: Rust compiler performance

#242

Post by pjmlp »

hinkley wrote: post #234
pjmlp wrote: post #161

Earlier quoted context omitted.

Not only language. Many of complaints towards Rust, or C++, are in reality tooling complaints. As shown on other ecosystems, the availability of interpreters or image based tooling are great ways to overcome slow optimizating compilers. C++ already had a go at this back in the early 90's with Energize C++ and Visual Age for C++ v4, both based on Common Lisp and Smalltalk from their respective owners. They failed on t…

I had a coworker who was using Rational back then, and found out one of its killer features was caching of pre compiled headers. Whoever changed them had to pay the piper of compilation, but everyone else got a copy shipped to them over the local network.

Yes, you are most likely talking about ClearMake, the build tool used by ClearCase.

It may have required dedicated infra team, but it had features that many folks only got to discover with git.

Better save those view description configurations safely.

Re: Rust compiler performance

#243

Post by feelamee »

norir wrote: post #27

Compiler performance must be considered up front in language design. It is nearly impossible to fix once the language reaches a certain size without it being a priority. I recently saw here the observation that one can often get a 2x performance improvement through optimization, but 10x requires redesigning the architecture. Rust can likely never be rearchitected without causing a disastrous schism in the community,…

maybe rustc will never be re-architectured (although it has already been rewritten once), but with developing rust standard there will come new Rust implementations. And there is a chance that they will prioritize performance when architecting.

Re: Rust compiler performance

#244

Post by feelamee »

juliangmp wrote: post #119

>[...] this will depend on who you ask, e.g. some C++ developers don’t mind Rust’s compilation times at all, as they are used to the same (or worse) build times Yeah pretty much. C++ is a lot worse when you consider the practical time spent vs compilation benchmarks. In most C++ projects I've seen/worked on, there were one or sometimes more code generators in the toolchain which slowed things down a lot. And it looks…

why do you run clang-tidy with compiler? Just use it interactively - with cland. These is much more useful to me

Re: Rust compiler performance

#245

Post by mcherm »

kibwen wrote: post #97
WhyNotHugo wrote: post #95

Earlier quoted context omitted.

One of the issue why compile times are so awful is that all dependencies must be compiled for each project. 20 different projects use the same dependency? They each need to recompile it. This is an effect of the language not having a proper ABI for compiling libraries as dynamically loadable modules, which in itself presents many other issues, including making distribution of software a complete nightmare.

> This is an effect of the language not having a proper ABI for compiling libraries as dynamically loadable modules No, this is a design decision of Cargo to default to using project-local cached artifacts rather than caching them at the user or system level. You can configure Cargo to do so if you'd like. The reason it doesn't do this by default is because Cargo gives crates great latitude to configure themselves vi…

> You can configure Cargo to do so if you'd like

How does one do this?

Re: Rust compiler performance

#246

Post by rstuart4133 »

feelamee wrote: post #243
norir wrote: post #27

Compiler performance must be considered up front in language design. It is nearly impossible to fix once the language reaches a certain size without it being a priority. I recently saw here the observation that one can often get a 2x performance improvement through optimization, but 10x requires redesigning the architecture. Rust can likely never be rearchitected without causing a disastrous schism in the community,…

maybe rustc will never be re-architectured (although it has already been rewritten once), but with developing rust standard there will come new Rust implementations. And there is a chance that they will prioritize performance when architecting.

The root cause of the problem is not the compiler. It's the language. If you compile a C program that links to a lot of libraries, the compiler compiles just the source code you wrote, and then the linker combines that with the pre-compiled libraries. Linking is a comparatively fast operation, so the total time is roughly what it took to compile just your source. Rust has chosen a design that requires it to compile not just your source, but also a fair chunk of the the libraries source code as well. Those libraries are usually many times the size of your source code.

Unsaid here is modern C++ programs suffer from the same problem, because modern C++ libraries are mostly .h files that are stuffed full of templates. The outcome is the same as Rust: slow compilation, even with a GNU C++ compiler.

The outcome is the same because Rust's generics have a lot in common with C++ templates. Both a generic Rust function and a C++ template are a kind of type safe macro. Like all macro's they generate source code, customised by the parameters the caller supplied, and because its customised that source code can't be pre-compiled and put into a library.

The downside of this is not just slow compile times. It's also means very fat binaries. And it means security issues in a library can't be fixed by shipping a new version of a dll, you have to recompile the original program.

Re: Rust compiler performance

#247

Post by majewsky »

nicoburns wrote: post #230
cesarb wrote: post #108

Earlier quoted context omitted.

> Why can't Cargo have a system like PyPI where library author uploads compiled binary Unless you have perfect reproducible builds, this is a security nightmare. Source code can be reviewed (and there are even projects to share databases of already reviewed Rust crates; IIRC, both Mozilla and Google have public repositories with their lists), but it's much harder to review a binary, unless you can reproducibly recrea…

> Unless you have perfect reproducible builds Or a trusted build server doing the builds. There is a build-bot building almost every Rust crate already for docs.rs.

docs.rs is just barely viable because it only has to build crates once (for one set of features, one target platform etc.).

What you propose would 1) have to build each create for at least the 8 Tier 1 targets, if not also the 91 Tier 2 targets. That would be either 8 or 99 binaries already.

Then consider that it's difficult to anticipate which feature combinations a user will need. For example, the tokio crate has 14 features [1]. Any combination of 14 different features gives 2^14 = 16384 possible configurations that would all need to be built. Now to be fair, these feature choices are not completely independent, e.g. the "full" feature selects a bunch of other features. Taking these options out, I'm guessing that we will end up with (ballpark) 5000 reasonable configurations. Multiply that by the number of build targets, and we will need to build either 40000 (Tier 1 only) or 495000 binaries for just this one crate.

Now consider on top that the interface of dependency crates can change between versions, so the tokio crate would either have to pin exact dependency versions (which would be DLL hell and therefore version locking is not commonly used for Rust libraries) or otherwise we need to build the tokio crate separately for each dependency version change that is ABI-incompatible somewhere. But even without that, storing tens of thousands of compiled variants is very clearly untenable.

Rust has very clearly chosen the path of "pay only for what you use", which is why all these library features exist in the first place. But because they do, offering prebuilt artifacts is not viable at scale.

[1] https://github.com/tokio-rs/tokio/blob/master/tokio/Cargo.to...

Re: Rust compiler performance

#248

Post by majewsky »

felipeccastro wrote: post #239
norir wrote: post #27

Compiler performance must be considered up front in language design. It is nearly impossible to fix once the language reaches a certain size without it being a priority. I recently saw here the observation that one can often get a 2x performance improvement through optimization, but 10x requires redesigning the architecture. Rust can likely never be rearchitected without causing a disastrous schism in the community,…

It is ironic how “rewrite it in Rust” is the solution to make any program fast, except the Rust compiler.

It's not ironic at all. Rust programs being fast is in large part due to shifting work from runtime to compile time.

Re: Rust compiler performance

#249

Post by lenkite »

johnfn wrote: post #87

> First, let me assure you - yes, we (as in, the Rust Project) absolutely do care about the performance of our beloved compiler, and we put in a lot of effort to improve it. I'm probably being ungrateful here, but here goes anyway. Yes, Rust cares about performance of the compiler, but it would likely be more accurate to say that compiler performance is, like, 15th on the list of things they care about, and they'll h…

I think Rust compiler team should just build a Rust interpreter for iterative, development mode. The compiler can only be sped up so far.

Re: Rust compiler performance

#250

Post by juliangmp »

feelamee wrote: post #244
juliangmp wrote: post #119

>[...] this will depend on who you ask, e.g. some C++ developers don’t mind Rust’s compilation times at all, as they are used to the same (or worse) build times Yeah pretty much. C++ is a lot worse when you consider the practical time spent vs compilation benchmarks. In most C++ projects I've seen/worked on, there were one or sometimes more code generators in the toolchain which slowed things down a lot. And it looks…

why do you run clang-tidy with compiler? Just use it interactively - with cland. These is much more useful to me

We do use clangd and it provides many checks, but I wanted to include clang-tidy in the CI. Though since me and my colleagues combine ff-only and ‘must build successfully before merging’, adding like 5 minutes to the CI is really annoying.

Iirc the reason why clangd is so much faster than clang-tidy is that clangs omits certain checks and limits which code it analyzes.

Post reply on HN