The Rust compiler is still getting faster • Hacker News Board

Re: The Rust compiler is still getting faster

#61

Post by masklinn » Thu, Jul 25, 2019, 1:36 PM UTC

nindalf wrote: post #9

howenterprisey wrote: post #6

Does LLVM still take up much of the overall time spent by the Rust compiler? I was thinking of getting involved over there as the most effective way to make speed-ups happen.

I remember reading that it does, but because the LLVM IR generated by rustc is verbose. If less IR was generated, LLVM would have less work to do.

I think that's one of the eventual purposes of MIR (by both optimising Rust upstream from the LLVM IR generation and having a much simpler language to generate IR for), but I don't know if there's been work towards that yet.

Re: The Rust compiler is still getting faster

#62

Post by maximilianburke » Thu, Jul 25, 2019, 1:56 PM UTC

pjmlp wrote: post #46

gpderetta wrote: post #28

Earlier quoted context omitted.

In my experience VC++ is order of magnitude slower than GCC or clang though. Might have been a bad build system though.

Only when incremental compiler, incremental linking and pre-compiled headers are disabled.

It's not just the compiler but also the system SDKs that play a significant role in the compile times.

In an apples-to-apples comparison (ie: same pre-processed code content) from circa 2015, Visual Studio's compilation was about the same speed as clang and faster than GCC, without the use of incremental compilation / incremental linking / PCH.

It's very easy to dig yourself into a hole when developing for Windows because Microsoft has historically had terrible discipline for minimizing the amount of code brought in through the Windows platform includes as well as the language headers. PCH's ease the penalty significantly but are a bitter pill to swallow if you're trying to cut down on the number of symbols exposed to each translation unit.

Re: The Rust compiler is still getting faster

#63

Post by barrkel » Thu, Jul 25, 2019, 2:02 PM UTC

derefr wrote: post #34

moomin wrote: post #22

Earlier quoted context omitted.

The thing is a) these structures change all the time and b) published APIs shouldn't change all the time. Which is why Erlang uses a public model and a private model and why most don't bother. Doesn't mean I wouldn't like it myself, but it is a genuine drag on continued compiler development.

There are business-logic data structures, and then there are formal data-structures from compiler theory. I only really want access to the latter. The nice thing about these formal data structures (or formal ADTs, I should say, though the papers themselves never tend to think of themselves as introducing an ADT) is that successive papers that find better algorithms, or make small changes to the data structure, pretty…

> should be run under the Microsoft philosophy: you don’t get to decide what your stable interfaces are

The philosophy only works because private APIs are hidden enough to reduce the effort required in maintaining backward compatibility. The fact that MS works (or rather, worked, it was much more vital in the Windows 95 transition) hard to maintain compatibility even on private dependencies doesn't mean that private doesn't mean anything.

I think a bigger (and certainly more visible) element of the MS philosophy is in maintaining compatibility for public, stable APIs. For example, look at how Windows 10 still has all these old Control Panel dialogs. That's not incidental compatibility with undocumented APIs; it's designed for extensibility, the maintenance of which holds the whole platform back to a certain degree.

I worked on the Delphi compiler and runtime library. Versioning was an exceedingly important concern. Patch releases couldn't break APIs because it could affect our partner market, if we affected the dependencies of third-party components, those third parties would need to re-release. The ecosystem only worked because there was a clear separation between public and private. You need private APIs because you need the flexibility to change things - if you can't change things, you must create everything perfectly first time, and that's just not possible.

You might want to get access to the compiler internals, but if you build something nifty with that access, and that nifty thing gets widespread use, you will hold back the entire ecosystem. You will be the cause of warts like Windows 10's 20 year old Control Panel dialogs.

Re: The Rust compiler is still getting faster

#64

Post by tkoskine » Thu, Jul 25, 2019, 2:09 PM UTC

fluffything wrote: post #3

The rust compiler now has features that few or none C and C++ compilers have: incremental compilation within a single translation unit, pipelined compilation, multi-threaded and lazy query-based compilation, ... Implementing each of these features have required whole program refactorings in a large-scale codebase performed by few individuals while hundreds of other developers where simultaneously evolving the softwar…

> ... the ability to refactor large-scale software without breaking anything ... and no low-level programming language except for Rust has it.

Ada has had the same characteristics since '83. Thanks to strict typing (among other features), Ada programmers have enjoyed "safe" code refactoring for decades.

But it is nice that new languages like Rust are finally picking similar ideas and design choices.

Re: The Rust compiler is still getting faster

#65

Post by losvedir » Thu, Jul 25, 2019, 2:12 PM UTC

fluffything wrote: post #48

derefr wrote: post #39

Earlier quoted context omitted.

I want to make a distinction between three kinds of data structures. 1. Implementation-specific thing that has no formal equivalent. Structs of structs of structs, with business logic intermingled with the structure. Plenty of examples of this. I’m not suggesting anyone put these in the stdlib; that’d be silly. 2. ADT with no formalism behind it, that implements a particular set of behaviours “for best performance”,…

> It’s #3 that I would suggest is a good candidate for stdlib inclusion You started arguing that the Rust compiler should expose its internal data-structures, which it does, and somehow ended arguing that the Rust standard library should expose spatial data-structures for geometry processing. I have no idea how you got from one to the other, but writing a huge wall of text full of disorganized thoughts shows very lit…

This is unnecessarily mean, and ironically you're the one showing little appreciation since you didn't even read derefr's initial post closely enough. They argued that the data structures (maps, graphs, etc) and algorithms used by the compiler could be included in the stdlib, not that the particular internal instantiations of them should be accessible. The whole point was the compiler uses a directed acyclic graph, so why isn't there a nice directed acyclic graph in the standard library?

Re: The Rust compiler is still getting faster

#66

Post by sfkdjf9j3j » Thu, Jul 25, 2019, 2:18 PM UTC

tkoskine wrote: post #64

fluffything wrote: post #3

The rust compiler now has features that few or none C and C++ compilers have: incremental compilation within a single translation unit, pipelined compilation, multi-threaded and lazy query-based compilation, ... Implementing each of these features have required whole program refactorings in a large-scale codebase performed by few individuals while hundreds of other developers where simultaneously evolving the softwar…

> ... the ability to refactor large-scale software without breaking anything ... and no low-level programming language except for Rust has it. Ada has had the same characteristics since '83. Thanks to strict typing (among other features), Ada programmers have enjoyed "safe" code refactoring for decades. But it is nice that new languages like Rust are finally picking similar ideas and design choices.

Ada languished for years with expensive, proprietary compilers and a community that largely ignored open source culture. Is it any surprise that Rust is having more success breaking into the modern mainstream software world?

Re: The Rust compiler is still getting faster

#67

Post by nn3 » Thu, Jul 25, 2019, 3:03 PM UTC

1. make a really slow compiler

2. slightly improve it over time

3. write blogs about it

4. win?

How about releasing software with reasonable performance in the first place? This was just discussed yesterday that everyone prefers faster software.

Re: The Rust compiler is still getting faster

#68

Post by pmoriarty » Thu, Jul 25, 2019, 3:10 PM UTC

I wish compiling the Rust compiler itself was faster. It literally takes days to compile on my old laptop.

Re: The Rust compiler is still getting faster

#69

Post by carlmr » Thu, Jul 25, 2019, 3:11 PM UTC

nn3 wrote: post #67

1. make a really slow compiler 2. slightly improve it over time 3. write blogs about it 4. win? How about releasing software with reasonable performance in the first place? This was just discussed yesterday that everyone prefers faster software.

For most software I would agree with you. Usually you have some CRUD application where people don't know how to index and use hashing.

But the Rust compiler is an amazing static analyzer. Running one on C++ to get the same level of memory safety will often take hours on sizeable projects.

Re: The Rust compiler is still getting faster

#70

Post by mrec » Thu, Jul 25, 2019, 3:24 PM UTC

fluffything wrote: post #3

The rust compiler now has features that few or none C and C++ compilers have: incremental compilation within a single translation unit, pipelined compilation, multi-threaded and lazy query-based compilation, ... Implementing each of these features have required whole program refactorings in a large-scale codebase performed by few individuals while hundreds of other developers where simultaneously evolving the softwar…

> Yet they do pay off in Rust because of my favourite Rust feature: the ability to refactor large-scale software without breaking anything

I think crater also deserves some credit here - given how much Rust source is in Cargo, it's very useful to be able to run a refactored compiler against all those packages and see which don't compile or start failing their unit tests.

[1] https://github.com/rust-lang/crater