Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zig's Incremental Compilation Internals

mlugg.co.uk

Post reply on HN
Print view View original

11–20 of 292 posts

Re: Zig's Incremental Compilation Internals

#11

Post by nh2 »

steveklabnik wrote: post #7

Zig's toolchain work is continually impressive. While I still don't plan to write software in it, given that I believe memory safety is table stakes, all of this stuff is very, very good. Before the incremental work, it was the toolchain and cross-compiler work. The toolchain stuff has continually been fantastic. I'm very curious to see what they come up with next! > Semantic analysis is the most difficult part of th…

10 years ago, I commented on the Rust issue for "Incremental recompilation", where it was suggested that Rust could at least adopt Haskell GHC's model of incrementality, which is currently file-level:

https://github.com/rust-lang/rust/issues/2369#issuecomment-1...

This would already help a lot.

I recommend anybody who's interested in incremental recompilation to read what GHC does, because the effort to achieve that is relatively low.

Of course there's always desire for more:

GHC currently needs to parse+typecheck+codegen a file before it can process other files that import it. Codegen is slow. Thus, there's currently demand split compilation into "stages", so that the next file can be typechecked after its imports have been just typechecked (not codegenned).

I would also enjoy if recompilation avoidance were to happen at the function level, not the file level.

Macro systems are a key language feature that can destroy incremental recompilation. In theory, Haskell is well set up for that, as its macro system (TemplateHaskell) is fully AST based and _theoretically_ could distinguish "fully pure" macros from side-effectful macros (such as splicing the current git commit in as a string literal). But the recompilation avoidance system does not currently exploit such differences.

Re: Zig's Incremental Compilation Internals

#12

Post by bpavuk »

hoppp wrote: post #4

The zig compiler can compile C so will it work with C also?

not quite! if your project is mixed C/Zig, then editing Zig would work but editing C would not. the Zig compiler caches Zig AIR but no information from C. plus, the C compilation can only really be done with LLVM and writing a self-hosted backend would be PITA for C

Re: Zig's Incremental Compilation Internals

#14

Post by jamiejquinn »

steveklabnik wrote: post #9
hoppp wrote: post #8

Earlier quoted context omitted.

The thing with rust is that you get safety with slow compilation, it's a tradeoff. Zig doesn't have the same safety guarantees, it's on the dev to use safe coding patterns, so the tradeoff for safety is discipline or experience.

Rust's safety checks have basically nothing to do with its slow compile times. This is something that sounds intuitive but is just completely incorrect. In particular, Rust made several good design decisions around this stuff that keeps those checks fast, like keeping checks local rather than being global.

That's interesting. Coming from C++ and Zig, the massive time "wasters" are metaprogramming features, i.e. Templates and comptime. Are Rust's macros the compile-time culprits?

Re: Zig's Incremental Compilation Internals

#15

Post by steveklabnik »

nh2 wrote: post #11
steveklabnik wrote: post #7

Zig's toolchain work is continually impressive. While I still don't plan to write software in it, given that I believe memory safety is table stakes, all of this stuff is very, very good. Before the incremental work, it was the toolchain and cross-compiler work. The toolchain stuff has continually been fantastic. I'm very curious to see what they come up with next! > Semantic analysis is the most difficult part of th…

10 years ago, I commented on the Rust issue for "Incremental recompilation", where it was suggested that Rust could at least adopt Haskell GHC's model of incrementality, which is currently file-level: https://github.com/rust-lang/rust/issues/2369#issuecomment-1... This would already help a lot. I recommend anybody who's interested in incremental recompilation to read what GHC does, because the effort to achieve that…

Just to be clear about it, Rust today does do some amount of incremental compilation, and there is more work being done to continue to make it moreso. It's just very difficult to re-architect such a large and heavily used codebase. People are putting in heroic amounts of effort to improve things.

An example that's being funded right now: https://rust-lang.github.io/rust-project-goals/2026/expansio...

Re: Zig's Incremental Compilation Internals

#16

Post by pron »

steveklabnik wrote: post #7

Zig's toolchain work is continually impressive. While I still don't plan to write software in it, given that I believe memory safety is table stakes, all of this stuff is very, very good. Before the incremental work, it was the toolchain and cross-compiler work. The toolchain stuff has continually been fantastic. I'm very curious to see what they come up with next! > Semantic analysis is the most difficult part of th…

> I believe memory safety is table stakes

I'm not sure what that means. Java lets you do many things programs may want to do in a memory-safe way but not everything. Rust lets you do fewer things than Java in a memory-safe way, but more things than Zig. Zig lets you do fewer things in a memory-safe way than Rust, but more than C. So among these four languages we already have four levels of memory safety, none of them is 100%, all of them give up something in exchange for what they offer, and different programmers have different preferences for the compromise they prefer, and even that preference is context-dependent. Which of those less-than-100% memory safety compromises is the table stakes? And given that all of these compromises require something that could be quite substantial, depending on the circumstance, in exchange, and consequently programmers with the highest level of knowledge and expertise choose every one of those four in different situations, to me it seems pretty obvious that none of these is "table stakes".

Re: Zig's Incremental Compilation Internals

#17

Post by steveklabnik »

jamiejquinn wrote: post #14
steveklabnik wrote: post #9

Earlier quoted context omitted.

Rust's safety checks have basically nothing to do with its slow compile times. This is something that sounds intuitive but is just completely incorrect. In particular, Rust made several good design decisions around this stuff that keeps those checks fast, like keeping checks local rather than being global.

That's interesting. Coming from C++ and Zig, the massive time "wasters" are metaprogramming features, i.e. Templates and comptime. Are Rust's macros the compile-time culprits?

Macros can be, but in part because they can produce new items (top level declarations, to sort of make the same handwave as the article does) and so that means you have to do macro expansion and stuff before you can even start to check some things, and similar issues. See the link I posted above for some details on a related issue.

There's also stuff around name resolution.

Proc macros are just an inherently very slow way to do what they do.

Because Rust commits to the traditional compilation model and pipleine (which I think is overall a good thing, or at least, a good thing to support), it does a lot of work that will eventually be thrown away. Consider this example: I have a library with a function foo that returns a simple 42. I have a binary which calls foo from that library and prints the result. Now imagine the library is a hundred thousand lines of unrelated code to what the binary needs, but is useful for other people. Because compilation works in the "produce libraries, produce binary, link them all together" style model, you have to compile the entire library with all of that code, when all you need is really one function. That intermediate work is useful, and I'm picking an example that's deliberately extreme, of course.

There's a bunch of stuff like this, and I do not have time to really say more than that right now. But yeah, monomorphized generics also produce a lot of compile time pressure too, in various ways.

Anyway I just also want to reiterate a few things: first of all, all of these decisions were made for good reasons, and there are pros to what Rust does and why. It's just that compile times suffer because of it. What I wish was that we had taken compile times into more consideration when deciding what to do and why in a more serious way. The same decisions might have been made, but at least it would have been known, rather than the situation now, where there's just a tremendous amount of work to try to optimize what exists, rather than having the freedom to maybe tweak some things to make that job way easier.

Re: Zig's Incremental Compilation Internals

#18

Post by amelius »

nh2 wrote: post #11
steveklabnik wrote: post #7

Zig's toolchain work is continually impressive. While I still don't plan to write software in it, given that I believe memory safety is table stakes, all of this stuff is very, very good. Before the incremental work, it was the toolchain and cross-compiler work. The toolchain stuff has continually been fantastic. I'm very curious to see what they come up with next! > Semantic analysis is the most difficult part of th…

10 years ago, I commented on the Rust issue for "Incremental recompilation", where it was suggested that Rust could at least adopt Haskell GHC's model of incrementality, which is currently file-level: https://github.com/rust-lang/rust/issues/2369#issuecomment-1... This would already help a lot. I recommend anybody who's interested in incremental recompilation to read what GHC does, because the effort to achieve that…

Can't we have a system where we trade some performance for quick incremental compilation?

We can always compile with full optimization just before shipping?

Re: Zig's Incremental Compilation Internals

#19

Post by saghm »

pron wrote: post #16
steveklabnik wrote: post #7

Zig's toolchain work is continually impressive. While I still don't plan to write software in it, given that I believe memory safety is table stakes, all of this stuff is very, very good. Before the incremental work, it was the toolchain and cross-compiler work. The toolchain stuff has continually been fantastic. I'm very curious to see what they come up with next! > Semantic analysis is the most difficult part of th…

> I believe memory safety is table stakes I'm not sure what that means. Java lets you do many things programs may want to do in a memory-safe way but not everything. Rust lets you do fewer things than Java in a memory-safe way, but more things than Zig. Zig lets you do fewer things in a memory-safe way than Rust, but more than C. So among these four languages we already have four levels of memory safety, none of them…

> I'm not sure what that means. Java lets you do many things in a memory-safe way but not everything. Rust lets you do fewer things than Java in a memory-safe way, but more things than Zig.

I don't think I agree with this framing. The question to me isn't "what can you do while being memory safe", it's "can you accidentally do something memory unsafe without noticing?" Rust and Java are the same here; you need to explicitly opt into using the language's mechanism for relaxing restrictions (Rust's `unsafe` blocks, Java's `Unsafe` class APIs), whereas from what I understand, neither Zig or C offers anything strict in that way.

Re: Zig's Incremental Compilation Internals

#20

Post by pjmlp »

amelius wrote: post #18
nh2 wrote: post #11

Earlier quoted context omitted.

10 years ago, I commented on the Rust issue for "Incremental recompilation", where it was suggested that Rust could at least adopt Haskell GHC's model of incrementality, which is currently file-level: https://github.com/rust-lang/rust/issues/2369#issuecomment-1... This would already help a lot. I recommend anybody who's interested in incremental recompilation to read what GHC does, because the effort to achieve that…

Can't we have a system where we trade some performance for quick incremental compilation? We can always compile with full optimization just before shipping?

Of course we can, C++ even REPL and hot reloading tools.

The main issue is that so far such tools haven't been a priority for Rust.

Post reply on HN