Does LLVM still take up much of the overall time spent by the Rust compiler? I was thinking of getting involved over there as the most effective way to make speed-ups happen.
Would it be possible to write a rust compiler that skips llvm entirely?
The Rust compiler is still getting faster
51–60 of 100 posts
Re: The Rust compiler is still getting faster
#52Earlier quoted context omitted.
Is there a blog post about Visual C++ being multi-threaded end-to-end ? (e.g. doing parsing, type-checking, etc. of a single TU in parallel using all cores ?) That would be super interesting to read because it is mainly a C++ compiler and some C++ features like two-phase lookup and macros make it quite hard to do things in parallel. You have to do things in a certain order but I suppose that if its query-based as wel…
As the documentation says: "The /MP (Build with Multiple Processes) compiler flag specifies the number of cl.exe processes that simultaneously compile the source files. The /cgthreads option specifies the number of threads used by each cl.exe process." I've never used /CGTHREADS but /MP is working well (except for the precompliled header file).
The documentation also says that /cgthreads is the number of threads used by the code generation and optimization passes of the compiler, not by the whole compiler.
A compiler needs to do a lot of stuff beyond code generation and optimizations (e.g. in debug builds optimizations are even often disabled). For C++ you have overload resolution, template argument deduction, template instantiation, constexpr evaluation... and well parsing, tokenization, type checking, static analysis (e.g. for warnings), etc.
Parallel optimizations and codegen is trivial when compared with an end-to-end multi-threaded compiler. All LLVM frontends for all programming languages do parallel optimizations and parallel codegen, it's only an LLVM option away. You enable it, and it's done.
> The /MP (Build with Multiple Processes)
This is one process per translation unit, each single translation unit is then compiled with a single thread until optimizations and codegen.
Often you need to finish compiling/linking some translation units before continuing.
The Rust compiler has pipelined compilation: compilation of a translation unit starts as soon as what this requires of its dependencies is already available, before its dependencies have finished compiling. It also compiles each translation unit itself using multiple threads end-to-end, so that if one of your translation units is bigger than the others, you can speed the compilation of that one by throwing more threads at it.
Re: The Rust compiler is still getting faster
#53Earlier quoted context omitted.
The thing is a) these structures change all the time and b) published APIs shouldn't change all the time. Which is why Erlang uses a public model and a private model and why most don't bother. Doesn't mean I wouldn't like it myself, but it is a genuine drag on continued compiler development.
There are business-logic data structures, and then there are formal data-structures from compiler theory. I only really want access to the latter. The nice thing about these formal data structures (or formal ADTs, I should say, though the papers themselves never tend to think of themselves as introducing an ADT) is that successive papers that find better algorithms, or make small changes to the data structure, pretty…
I don't think it's that clear.
If you'd like to do topological sorting you'd use one algorithm if that topological sorting is for a set of fixed tasks. Online topological sorting (that is - delta-based algorithms) used in databases would use a different algorithm and will also have different performance constraints.
I could see that in the long run we'll eventually figure out how to abstract all of the "complete/delta" stuff. I don't think we have yet figured this out yet.
Re: The Rust compiler is still getting faster
#54Earlier quoted context omitted.
The rust compiler team continues to set ambitious goals. Last I checked they wanted to extract much of the compiler front end into independent crates that can be reused by the Language Server. That's a pretty daunting refactoring, but the way they're going seems like it's achievable.
The one thing I’ve always wanted from a language runtime is for the data structures used by the compiler to be exposed in the stdlib. E.g. every compiler uses control flow graphs; so why doesn’t every language give me a batteries-included digraph ADT, that has every feature you’d need to implement a control flow graph, such that the compiler’s CFG is just a stdlib digraph instance? It’d be a great boon to writing you…
Salsa is an example of logic they are writing for the Rust compiler that is being kept in a generic crate for anyone to use
Re: The Rust compiler is still getting faster
#55Does LLVM still take up much of the overall time spent by the Rust compiler? I was thinking of getting involved over there as the most effective way to make speed-ups happen.
Would it be possible to write a rust compiler that skips llvm entirely?
Cranelift is being created as an alternative to LLVM, targeting a different set of requirements. Last I heard talk about Rust using it, the thought is that it would fit in well for fast debug builds but not be a good fit for optimized release builds.
See https://github.com/bjorn3/rustc_codegen_cranelift#not-yet-su... for progress
Re: The Rust compiler is still getting faster
#56Does LLVM still take up much of the overall time spent by the Rust compiler? I was thinking of getting involved over there as the most effective way to make speed-ups happen.
I remember reading that it does, but because the LLVM IR generated by rustc is verbose. If less IR was generated, LLVM would have less work to do.
I thought it was two factors (1) unoptimized IR and (2) large translation units (crate rather than file).
Re: The Rust compiler is still getting faster
#57Earlier quoted context omitted.
As the documentation says: "The /MP (Build with Multiple Processes) compiler flag specifies the number of cl.exe processes that simultaneously compile the source files. The /cgthreads option specifies the number of threads used by each cl.exe process." I've never used /CGTHREADS but /MP is working well (except for the precompliled header file).
That's for different source files though, right? Try having a 10Mb auto-generated C++ file, see how fast it compiles, regardless how powerful your computer is.
No need to autogenerate anything: expand all headerfiles into a .cpp file, and you easily end up with > 10Mb files.
Visual C++ pre-compiled headers is quite amazing and works really great, avoiding this almost completely.
Re: The Rust compiler is still getting faster
#58Does LLVM still take up much of the overall time spent by the Rust compiler? I was thinking of getting involved over there as the most effective way to make speed-ups happen.
Would it be possible to write a rust compiler that skips llvm entirely?
Of course it would be possible, but then you lose all the benefits that llvm has regarding target platforms / optimizations / ...
Re: The Rust compiler is still getting faster
#59Earlier quoted context omitted.
I want to make a distinction between three kinds of data structures. 1. Implementation-specific thing that has no formal equivalent. Structs of structs of structs, with business logic intermingled with the structure. Plenty of examples of this. I’m not suggesting anyone put these in the stdlib; that’d be silly. 2. ADT with no formalism behind it, that implements a particular set of behaviours “for best performance”,…
> It’s #3 that I would suggest is a good candidate for stdlib inclusion You started arguing that the Rust compiler should expose its internal data-structures, which it does, and somehow ended arguing that the Rust standard library should expose spatial data-structures for geometry processing. I have no idea how you got from one to the other, but writing a huge wall of text full of disorganized thoughts shows very lit…
These are not disorganized thoughts, and derefr is still arguing for the same thing as before.
Re: The Rust compiler is still getting faster
#60Faster compiler is nice, but you know what's faster? Not having to compile anything. I'm also looking forward to crates.io serving precompiled crates ( https://www.ncameron.org/blog/cargo-in-2019/ )
I'm curious to see what direction they take this.
Will they have a trusted compile farm, only supporting a subset of targets, or involve some kind of distributed trust model supporting whatever people use? Will it be greedily populated with a specific subset of targets / features or lazily populated based on combinations people actually use?