Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zig's Incremental Compilation Internals

mlugg.co.uk

Post reply on HN
Print view View original

21–30 of 292 posts

Re: Zig's Incremental Compilation Internals

#21

Post by steveklabnik »

pron wrote: post #16
steveklabnik wrote: post #7

Zig's toolchain work is continually impressive. While I still don't plan to write software in it, given that I believe memory safety is table stakes, all of this stuff is very, very good. Before the incremental work, it was the toolchain and cross-compiler work. The toolchain stuff has continually been fantastic. I'm very curious to see what they come up with next! > Semantic analysis is the most difficult part of th…

> I believe memory safety is table stakes I'm not sure what that means. Java lets you do many things programs may want to do in a memory-safe way but not everything. Rust lets you do fewer things than Java in a memory-safe way, but more things than Zig. Zig lets you do fewer things in a memory-safe way than Rust, but more than C. So among these four languages we already have four levels of memory safety, none of them…

I'll say the same thing I said to you as I said to Andrew, last time he and I talked about this: the way that everyone talks about memory safety (with maybe two exceptions, one okay (go) and one I dislike (fil-c)) is that "memory safe language" is about there being a clear delineation between what is memory safe and what is not, and that the unsafe aspect is a superset. Rust and Java both are memory safe, except where explicitly demarcated as not (unsafe in Rust, JNI or sun.misc.unsafe or whatever in Java). Zig and C have no such separation. When I (and others) talk about wanting memory safety, this is the important aspect of the design. This is what enables the "I know statically that a large part of the code is safe, and I also know where to check if something goes wrong" aspect of things.

Re: Zig's Incremental Compilation Internals

#22

Post by steveklabnik »

amelius wrote: post #18
nh2 wrote: post #11

Earlier quoted context omitted.

10 years ago, I commented on the Rust issue for "Incremental recompilation", where it was suggested that Rust could at least adopt Haskell GHC's model of incrementality, which is currently file-level: https://github.com/rust-lang/rust/issues/2369#issuecomment-1... This would already help a lot. I recommend anybody who's interested in incremental recompilation to read what GHC does, because the effort to achieve that…

Can't we have a system where we trade some performance for quick incremental compilation? We can always compile with full optimization just before shipping?

The article gestures at (and the author has made a comment in this thread about) how this is the case for Zig. You are right that there is tension here, and so that's exactly what you do: accept less performance for the gains in incremental, and then don't do incremental for final builds. It's a fine way to go about it, assuming that the lack of performance doesn't make the program unusuable. (Some people add some basic optimizations to their Rust debug builds, for example, because no optimizations is too painful to actually use.)

Re: Zig's Incremental Compilation Internals

#23

Post by pjmlp »

jamiejquinn wrote: post #14
steveklabnik wrote: post #9

Earlier quoted context omitted.

Rust's safety checks have basically nothing to do with its slow compile times. This is something that sounds intuitive but is just completely incorrect. In particular, Rust made several good design decisions around this stuff that keeps those checks fast, like keeping checks local rather than being global.

That's interesting. Coming from C++ and Zig, the massive time "wasters" are metaprogramming features, i.e. Templates and comptime. Are Rust's macros the compile-time culprits?

Yet that doesn't prevent C++ to have REPL and hot reloading tools, use binary libraries instead of compiling the world from scratch, all of which allow for a much better experience.

Re: Zig's Incremental Compilation Internals

#24

Post by saghm »

steveklabnik wrote: post #17
jamiejquinn wrote: post #14

Earlier quoted context omitted.

That's interesting. Coming from C++ and Zig, the massive time "wasters" are metaprogramming features, i.e. Templates and comptime. Are Rust's macros the compile-time culprits?

Macros can be, but in part because they can produce new items (top level declarations, to sort of make the same handwave as the article does) and so that means you have to do macro expansion and stuff before you can even start to check some things, and similar issues. See the link I posted above for some details on a related issue. There's also stuff around name resolution. Proc macros are just an inherently very slo…

> Because Rust commits to the traditional compilation model and pipleine (which I think is overall a good thing, or at least, a good thing to support), it does a lot of work that will eventually be thrown away. Consider this example: I have a library with a function foo that returns a simple 42. I have a binary which calls foo from that library and prints the result. Now imagine the library is a hundred thousand lines of unrelated code to what the binary needs, but is useful for other people. Because compilation works in the "produce libraries, produce binary, link them all together" style model, you have to compile the entire library with all of that code, when all you need is really one function. That intermediate work is useful, and I'm picking an example that's deliberately extreme, of course.

I feel extremely validated reading this! I've had a pet theory for a while that compile times would be drastically reduced if every single item in a crate were implicitly under a cargo feature flag and then they only got enabled if they were imported (and used in non-dead code). I know that making something like that work isn't anywhere close to as simple as I'm describing it, and there are probably a million edge cases, but I've long felt that the ergonomics of Cargo features basically making it too annoying to expose everything conditionally (and then transitively expose all of the features from all of the direct dependencies as well so that things depending on your library could also only conditionally enable them) is secretly the reason that people think Rust compile times are slow, and seeing someone who has way more direct knowledge than me of how all of it works under the hood give a similar take makes me more confident that I might have been on to something all along.

Re: Zig's Incremental Compilation Internals

#25

Post by pjmlp »

applfanboysbgon wrote: post #5

It disappoints me how unseriously the industry has taken compilation speed for so long. I'm glad to see Zig doing incredibly valuable, high-impact work here.

Yes, we get people amazed to rediscover Modula-2, Object Pascal compiler execution speed.

Re: Zig's Incremental Compilation Internals

#26

Post by vlovich123 »

pron wrote: post #16
steveklabnik wrote: post #7

Zig's toolchain work is continually impressive. While I still don't plan to write software in it, given that I believe memory safety is table stakes, all of this stuff is very, very good. Before the incremental work, it was the toolchain and cross-compiler work. The toolchain stuff has continually been fantastic. I'm very curious to see what they come up with next! > Semantic analysis is the most difficult part of th…

> I believe memory safety is table stakes I'm not sure what that means. Java lets you do many things programs may want to do in a memory-safe way but not everything. Rust lets you do fewer things than Java in a memory-safe way, but more things than Zig. Zig lets you do fewer things in a memory-safe way than Rust, but more than C. So among these four languages we already have four levels of memory safety, none of them…

Java and Rust have actually very similar memory safety profiles. Rust let's you within the language escape the memory safety requirements whereas Java does not but both are considered memory safe languages. Rust also enforces thread safety as well which Java does not, but the slower JVM memory model doesn't let race conditions become memory safety issues whereas Rust is lower-level like Zig/C/C++ and thus thread safety could be a memory safety issue.

Zig has an identical memory safety profile to C. It has facilities to make it easier to stay memory safe, but those facilities are basically equivalent to what you have in C++ and that's equivalent memory safety profile as C.

> So among these four languages we already have four levels of memory safety, none of them is 100%

No, you've pretended like there's four when really it's Java / Rust which are safe by default and Zig/C/C++ which are unsafe by default.

One effective metric to evaluate is memory safety per LoC. Rust is ~0.2 vulnerabilities per MLoC. Java is effectively 0. C and C++ both seem to be about 1,000 vulnerabilities per MLoC. Zig is too new and hasn't had any analysis done on it, but generously it's likely at least 10-100.

So the table stakes could be defined as 1 memory safety vulnerability per MLoC.

Re: Zig's Incremental Compilation Internals

#27

Post by steveklabnik »

saghm wrote: post #24
steveklabnik wrote: post #17

Earlier quoted context omitted.

Macros can be, but in part because they can produce new items (top level declarations, to sort of make the same handwave as the article does) and so that means you have to do macro expansion and stuff before you can even start to check some things, and similar issues. See the link I posted above for some details on a related issue. There's also stuff around name resolution. Proc macros are just an inherently very slo…

> Because Rust commits to the traditional compilation model and pipleine (which I think is overall a good thing, or at least, a good thing to support), it does a lot of work that will eventually be thrown away. Consider this example: I have a library with a function foo that returns a simple 42. I have a binary which calls foo from that library and prints the result. Now imagine the library is a hundred thousand line…

Yes, this idea is sort of similar, just like, more complicated in a sense than the clean design, which is what Zig does.

Incidentally, you might want to look at gc-sections, which Rust already does. As well as https://rust-lang.github.io/rust-project-goals/2025h2/relink... which is kinda related.

The sort of key here is understanding that "produce a library" means that every public item is "used" in the sense of "do we need to compile it." The real trick is to do demand-driven compilation starting from the actual final program's needs, and this is inherently at odds with the idea of producing standalone libraries and combining them into the final artifact.

Re: Zig's Incremental Compilation Internals

#28

Post by applfanboysbgon »

vlovich123 wrote: post #26
pron wrote: post #16

Earlier quoted context omitted.

> I believe memory safety is table stakes I'm not sure what that means. Java lets you do many things programs may want to do in a memory-safe way but not everything. Rust lets you do fewer things than Java in a memory-safe way, but more things than Zig. Zig lets you do fewer things in a memory-safe way than Rust, but more than C. So among these four languages we already have four levels of memory safety, none of them…

Java and Rust have actually very similar memory safety profiles. Rust let's you within the language escape the memory safety requirements whereas Java does not but both are considered memory safe languages. Rust also enforces thread safety as well which Java does not, but the slower JVM memory model doesn't let race conditions become memory safety issues whereas Rust is lower-level like Zig/C/C++ and thus thread safe…

> Rust is ~0.2 vulnerabilities per MLoC. Java is effectively 0. C and C++ both seem to be about 1,000 vulnerabilities per MLoC. Zig is too new and hasn't had any analysis done on it, but generously it's likely at least 10-100.

You have just described six orders of magnitude in your attempt to rebut pron pointing out the four languages have four levels of memory safety.

Re: Zig's Incremental Compilation Internals

#29

Post by mi_lk »

steveklabnik wrote: post #21
pron wrote: post #16

Earlier quoted context omitted.

> I believe memory safety is table stakes I'm not sure what that means. Java lets you do many things programs may want to do in a memory-safe way but not everything. Rust lets you do fewer things than Java in a memory-safe way, but more things than Zig. Zig lets you do fewer things in a memory-safe way than Rust, but more than C. So among these four languages we already have four levels of memory safety, none of them…

I'll say the same thing I said to you as I said to Andrew, last time he and I talked about this: the way that everyone talks about memory safety (with maybe two exceptions, one okay (go) and one I dislike (fil-c)) is that "memory safe language" is about there being a clear delineation between what is memory safe and what is not, and that the unsafe aspect is a superset. Rust and Java both are memory safe, except wher…

Would you mind sharing some thoughts about fil-c? AFAICT its claims mostly check out so besides implementation details (GC?) it seems directionally good.

Re: Zig's Incremental Compilation Internals

#30

Post by steveklabnik »

mi_lk wrote: post #29
steveklabnik wrote: post #21

Earlier quoted context omitted.

I'll say the same thing I said to you as I said to Andrew, last time he and I talked about this: the way that everyone talks about memory safety (with maybe two exceptions, one okay (go) and one I dislike (fil-c)) is that "memory safe language" is about there being a clear delineation between what is memory safe and what is not, and that the unsafe aspect is a superset. Rust and Java both are memory safe, except wher…

Would you mind sharing some thoughts about fil-c? AFAICT its claims mostly check out so besides implementation details (GC?) it seems directionally good.

I'll come back and say some more in a bit, but in short: I like the idea of fil-c, but I also have some issues. I think it's overall a good and interesting project, but with some caveats.
Post reply on HN