Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cutting down Rust compile times from 30 to 2 minutes with one thousand crates

feldera.com

Post reply on HN
Print view View original

71–80 of 89 posts

Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates

#71

Post by pdimitar »

ay wrote: post #58

I have just went through this with a project of mine, though unfortunately the code wasn’t autogenerated, so I needed to do a lot of mind-numbingly boring search-and-replace commands. I have cobbled together a little utility that allowed to automate the process somewhat. Mostly a throwaway code with a heavy input from Claude, so the docs are in the code itself :-) But in case anyone can find it useful: https://github…

Zero documentation? Do you expect potential users t9 figure it out by themselves?

Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates

#72

Post by upmind »

ameixaseca wrote: post #60
sitzkrieg wrote: post #46

the rust compiler is so impressively slow

It actually isn't. It becomes more evident when you consider the amount of work it is doing as well.

I agree that it is doing a LOT of work, but I believe OP and many others will compare it to other languages and notice that the Rust compiler is a LOT slower.

Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates

#74

Post by ay »

pdimitar wrote: post #71
ay wrote: post #58

I have just went through this with a project of mine, though unfortunately the code wasn’t autogenerated, so I needed to do a lot of mind-numbingly boring search-and-replace commands. I have cobbled together a little utility that allowed to automate the process somewhat. Mostly a throwaway code with a heavy input from Claude, so the docs are in the code itself :-) But in case anyone can find it useful: https://github…

Zero documentation? Do you expect potential users t9 figure it out by themselves?

Thanks for the feedback !

The evidently misguided assumption was that whoever uses it will need to tweak it anyhow, so might as well read it through. As I wrote - it’s very close to throwaway code.

Anyway, I decide to experiment with Claude also writing a README - the result doesn’t seem too terribly incorrect on the first squint, and hopefully gives a slightly more impression of what that thing was attempting to do. (Disclaimer: I didn’t test it much other than my use case, so YMMV on whether it works at all).

Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates

#75

Post by pdimitar »

ay wrote: post #74
pdimitar wrote: post #71

Earlier quoted context omitted.

Zero documentation? Do you expect potential users t9 figure it out by themselves?

Thanks for the feedback ! The evidently misguided assumption was that whoever uses it will need to tweak it anyhow, so might as well read it through. As I wrote - it’s very close to throwaway code. Anyway, I decide to experiment with Claude also writing a README - the result doesn’t seem too terribly incorrect on the first squint, and hopefully gives a slightly more impression of what that thing was attempting to do.…

That's much better, thanks.

> The evidently misguided assumption was that whoever uses it will need to tweak it anyhow, so might as well read it through. As I wrote - it’s very close to throwaway code.

Even if that assumption is true for part of the potential users, they would appreciate a starting point, you know.

Now I have bookmarked it and will check it out at one point.

If you ever figure you want to invest some more effort into it: try make it into an LSP server so it can integrate with LSP Code Actions directly.

Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates

#76

Post by pjmlp »

HKH2 wrote: post #26
xrisk wrote: post #7

Not sure about the nature of the generated code, but wouldn’t doing the equivalent of dynamic linking mitigate this problem?

How can you get around Rust's lack of a stable ABI?

The same way as in other ecosystems that lack it, compile everything with the same toolchain.

Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates

#77

Post by ay »

pdimitar wrote: post #75
ay wrote: post #74

Earlier quoted context omitted.

Thanks for the feedback ! The evidently misguided assumption was that whoever uses it will need to tweak it anyhow, so might as well read it through. As I wrote - it’s very close to throwaway code. Anyway, I decide to experiment with Claude also writing a README - the result doesn’t seem too terribly incorrect on the first squint, and hopefully gives a slightly more impression of what that thing was attempting to do.…

That's much better, thanks. > The evidently misguided assumption was that whoever uses it will need to tweak it anyhow, so might as well read it through. As I wrote - it’s very close to throwaway code. Even if that assumption is true for part of the potential users, they would appreciate a starting point, you know. Now I have bookmarked it and will check it out at one point. If you ever figure you want to invest some…

Thanks, hopefully it will be of any use and not too buggy ! (It was a classic “worked on my machine for my needs”, but I would be very suspicious of the autogenerated code being 100% bug-free).

I looked shortly at LSP but never had experience with it, and it looked very overwhelming… (and given that I generally use vi, it seemed like a bit too much overhead to also start using a different editor or learn integrations - which I looked at but they seemed a bit unsatisfying).

As a result this exercise got me into an entirely worse kind of shiny: writing my own TUI editor, with a function/type being the unit of editing rather than file. facepalm.

probably entirely worthless exercise, but it is a ton of fun and that is what matters for now ! :-)

Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates

#78

Post by swiftcoder »

steveklabnik wrote: post #53
swiftcoder wrote: post #28

Earlier quoted context omitted.

And Rust's module design makes this significantly more complicated than in superficially-similar languages. That's why splitting it into modules brings drastic improvements - you are effectively giving the compiler clear boundaries across which it doesn't need to propagate as much information.

Splitting it into crates , not modules.

Yeah, my bad. I do think this terminology drives some confusion, honestly, since neither "module" nor "crate" is a very portable term to other languages' compilation schemes.

Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates

#79

Post by pjmlp »

dathinab wrote: post #35
pjmlp wrote: post #25

Earlier quoted context omitted.

Except the Rust ecosystem lacks the solutions we have in C++ land to compile fast and have easy paralelisation of builds. Because C and C++ communities for historical reasons embrace binary libraries, and binary component frameworks like COM, so while in theory a full build from scratch takes similar time as Rust, in practice that isn't the case. Also note that D, a language as complex as C++, with three compilers, o…

> have easy paralelisation of builds. except the company had a straight forward solution to the problem and it's not always possible in C++ land either and COM isn't part of C/C++ but a microsoft specific extension which solves very different issues as it's for cross application communication while we here have compile time perf issues inside of a single library > binary libraries I'm not sure if you mean dynamic lin…

COM is one approach, among others like SOM, DCE, ... as means to write binary libraries in poliglot languages.

Turns out that it is mostly used on Windows since Vista days, as means to have an OOP based OS, with most libraries written in C++, and having a stable ABI for such components.

So while it isn't ISO C++, it is mostly used by and from C++. .NET land usually only reaches for COM when using Windows APIs.

Binary libraries, mean binary libraries, doesn't matter if static or dynamically linked.

My point is that Rust ecosystem does not use them, you always need to compile from source code the complete dependency tree after a git clone, some of them even multiple times due to different feature flags configurations.

Not so with most commercial C and C++ development, we enjoy having binary libraries for dependecies, after a git clone only the main code needs to be compiled from scratch.

Yes there are ways to kind of do with sccache, but it is additionally tooling, not something that apparently cargo will ever support.

Also if you watch recent talks from Microsoft regarding their Rust adoption, the lack of tooling support for binary libraries distribution is one of their pain points.

Re: Cutting down Rust compile times from 30 to 2 minutes with one thousand crates

#80

Post by pdimitar »

ay wrote: post #77
pdimitar wrote: post #75

Earlier quoted context omitted.

That's much better, thanks. > The evidently misguided assumption was that whoever uses it will need to tweak it anyhow, so might as well read it through. As I wrote - it’s very close to throwaway code. Even if that assumption is true for part of the potential users, they would appreciate a starting point, you know. Now I have bookmarked it and will check it out at one point. If you ever figure you want to invest some…

Thanks, hopefully it will be of any use and not too buggy ! (It was a classic “worked on my machine for my needs”, but I would be very suspicious of the autogenerated code being 100% bug-free). I looked shortly at LSP but never had experience with it, and it looked very overwhelming… (and given that I generally use vi, it seemed like a bit too much overhead to also start using a different editor or learn integrations…

Well, you can at least check out Zed and Helix first? Many people say they work perfectly for them and that they are simpler than both Emacs and [Neo]vim.

LSP Code Actions is super neat though. You can have a compiler error or a warning and when your cursor is positioned on it (inside the editor) you can invoke LSP Code Actions and have changes offered to you with a preview, then you can just agree to it and boom, it's done.

Obviously this might be too much work or too tedious for a hobby project, but it's good for you to know what's out there and how it's used. I don't use LSP Code Actions too often but find them invaluable when I do.

Post reply on HN