81–89 of 89 posts
Earlier quoted context omitted.
Splitting it into crates , not modules.
Yeah, my bad. I do think this terminology drives some confusion, honestly, since neither "module" nor "crate" is a very portable term to other languages' compilation schemes.
I also think it's kind of why people get confused by Rust's module system; they assume that it works like the module system of whatever language they're coming from. But they all work differently!
Earlier quoted context omitted.
codegen units is quite the same as the `-j` flag cargo has the `-j` flag and defaults it to #cpus (logical cpus), so it's by default using the most times most optimal choice there And this will parallelize the compilation of all "jobs", roughly like with make. Where a job is normally (oversimplified) compiling one code unit into one object file (.o). And cargo does that too. The problem is that where rust and C/C++ (…
> Hence why rust internally split one "semantic code unit" into multiple internal code units passed to LLM. And the same has happened in C and C++ land, albeit in the opposite direction, where multiple compilation units can be optimized together, i.e. LTO. See, e.g., GCC's -flto-partition option for selecting strategies for partitioning symbols for LTO. Also note that you can manually partition LTO in your Makefile b…
you also have that in rust to link different crates (and the internal splits for parallel compute) together
Earlier quoted context omitted.
> have easy paralelisation of builds. except the company had a straight forward solution to the problem and it's not always possible in C++ land either and COM isn't part of C/C++ but a microsoft specific extension which solves very different issues as it's for cross application communication while we here have compile time perf issues inside of a single library > binary libraries I'm not sure if you mean dynamic lin…
COM is one approach, among others like SOM, DCE, ... as means to write binary libraries in poliglot languages. Turns out that it is mostly used on Windows since Vista days, as means to have an OOP based OS, with most libraries written in C++, and having a stable ABI for such components. So while it isn't ISO C++, it is mostly used by and from C++. .NET land usually only reaches for COM when using Windows APIs. Binary…
mainly the concepts of
- how build steps are cached in on instance of the project
- how build steps are cached across projects of common dependencies
- linking to system libraries
- bundling dependencies
--
Lets first look at it from a POV of system dependencies vs. bundling dependencies:
For each dependency you either link them, making them a system dependencies (because you link against the on in you system and require systems to have them) or bundle them, doesn't matter if it's C or Rust. The problem with system dependencies is they don't just need API compatibility but also ABI compatibility and not everything with ABI compatibility is actually API compatible. Which is all nice and fun except a ton of thing highly useful (some would say required) do not work well (or at all) if you need ABI compatibility and there are tone of (potential security bugs) seemingly API compatible but not API compatible libraries has.
In general history has shown that making most dependencies system dependencies is a complete shit show not worth anyone time and money, especially if people start mixing versions which seem compatible but aren't leading to strange runtime bug aren't possible with supported builds but anyway somehow are your fault as library maintainer.
Which is why the huge majority of the software industry _gave up on them_ for anything where they aren't strictly needed.
Rust can produce and use system dependencies, using a C API and with some less officially supported way also using rlibs (i.e. the binary libraries rust produces when compiling a crate, so yes it's using binary libraries).
But mostly it's not worth bothering with it, in the same way the majority of the rest of the software ecosystem stopped doing it.
--
Then let's look at reusing builds i.e. caching.
By default rust does that, but only on the scope of the project. I.e. you build a project change something and then rebuild it and dependencies won't be build again (except if you need to rebuild them I come back to it later). To be clear this is _not_ incremental building, which is a feature to re-use build parts on a more granular level then crates.
If you want it to cache things across projects or with some company build server you can do so using 3rd party software, i.e. same situation as with C.
> most commercial C and C++ development
Committing binary build artifact to a source code repo is a huge anti pattern, a terrible way to have distributed build caches. Stuff like that can easily make your company fail security reviews or become classified as having acted negligently if sued for damaged (e.g. caused by a virus sneaked into your program).
Also please _never ever_ checkout a 3rd party open source project with any pre-build binary artifacts in it, it's a huge security threat.
So in C/C++ you also should use the additional tools.
> Rust ecosystem does not use them
as mentioned they produce rlibs which are binary libraries (or you could say binary libraries bundled with metadata, and stuff which is roughly like how C++ templates are handled wrt. binary libraries)
And yes the tooling for shipping pre-build rlibs could be better, and it probably will get better. It's not that it can't be done, just priorities have been elsewhere so far.
> even multiple times due to different feature flags configurations.
Features are strictly additive, so no that won't happen.
The only reason for them being build multiple times is different incompatible versions of the package (which from rust POV are two different dependencies altogether). And while that seems initially kinda dump (unnecessary binary size/build time) but I can't understate how much of a huge blessing this turned out to be.
> not something that apparently cargo will ever support.
yes and make doesn't support distributed build caches without including 3rd party tools either. But it doesn't matter as long as you can just pull in the 3rd party tools if you need them.
EDIT: Rust features is like using #if and similar in the C/C++ pre-processor, i.e. if they change you have to rebuild. Like in C/C++. Also even without a crate might have been only partially compiled before (e.g. only 1 of 3 functions) so if you start using the other parts they still need to be compiled which will look a lot like recompilation (and without the incremental build feature might be a rebuild).
The main issue here is: - in rust one semantic compilation unit is one crate - in C one semantic compilation unit is one file There are quite a bunch of benefits in the rust approach, but also drawbacks, like huge projects have to be split into multiple workspaces to maximize parallel building. Oversimplified the codegen-units setting tells the compiler into how many parts the compiler is allowed to split the a singl…
Rust has a great compromise between crate and file: module. I wonder why that's not the compilation unit?
- some subtleties related to (proc-)macros
- better optimizations (potentially, not always, sometimes not at all)
- how generics and compilation units interact (reduces the benefit of making each module a compilation unit)
- a lot of unclearity about how rust will develop in the future when this decision was made
Also when people speak about rust compiling slow and splitting helping it's most times related to better caching of repeated builds (unrelated to the incremental build feature) and not the specific issue here. But there is definitely potential to improve on it to make humongous single crates work better (like instead of just 16/256 internal splits you could factor in the crate size, maybe add a attribute to hint code unit splits etc.), but so far no one has deemed it important enough to invest their time into fixing it. I mean splitting crates is often easy so you do that once are good forever or at lest a long time.
> Given that we now fully utilize 128 threads or 64 cores for pretty much the entire compile time, we can do a back of the envelope calculation for how long it should take: 25 min / 128 = 12 sec (or maybe 24 sec since hyper-threads aren't real cores). Yet it takes 170s to compile everything. Amdahl’s Law would like to have a word.
Earlier quoted context omitted.
Thanks, hopefully it will be of any use and not too buggy ! (It was a classic “worked on my machine for my needs”, but I would be very suspicious of the autogenerated code being 100% bug-free). I looked shortly at LSP but never had experience with it, and it looked very overwhelming… (and given that I generally use vi, it seemed like a bit too much overhead to also start using a different editor or learn integrations…
Well, you can at least check out Zed and Helix first? Many people say they work perfectly for them and that they are simpler than both Emacs and [Neo]vim. LSP Code Actions is super neat though. You can have a compiler error or a warning and when your cursor is positioned on it (inside the editor) you can invoke LSP Code Actions and have changes offered to you with a preview, then you can just agree to it and boom, it…
It might be a bit closer to what I am after, so I will definitely give it a try even if as a source of inspiration for my reinvention of bicycle !
Thanks a lot !
Earlier quoted context omitted.
The same way as in other ecosystems that lack it, compile everything with the same toolchain.
And then you have to recompile everything every time you upgrade the toolchain.
Earlier quoted context omitted.
COM is one approach, among others like SOM, DCE, ... as means to write binary libraries in poliglot languages. Turns out that it is mostly used on Windows since Vista days, as means to have an OOP based OS, with most libraries written in C++, and having a stable ABI for such components. So while it isn't ISO C++, it is mostly used by and from C++. .NET land usually only reaches for COM when using Windows APIs. Binary…
you are mixing up use cases and concepts mainly the concepts of - how build steps are cached in on instance of the project - how build steps are cached across projects of common dependencies - linking to system libraries - bundling dependencies -- Lets first look at it from a POV of system dependencies vs. bundling dependencies: For each dependency you either link them, making them a system dependencies (because you…
it _has_ issues there, which can be improved on, people are working on it, just kinda slowly
but they aren't fundamental ones, and aren't because rust doesn't "embrace binary libraries"
the only thing AFIK rust misses out one is "rust ABI/API" not being suited for becoming a new system library binary interface. I.e. it doesn't provide what swift is providing for Apple. But, it can implement many of the existing system library binary interfaces reasonable fine (mainly C ABI, COM, etc.).