Live data from Hacker News

Zig's Incremental Compilation Internals

mlugg.co.uk

111–120 of 292 posts

Re: Zig's Incremental Compilation Internals

#111
post #16

Zig's toolchain work is continually impressive. While I still don't plan to write software in it, given that I believe memory safety is table stakes, all of this stuff is very, very good. Before the incremental work, it was the toolchain and cross-compiler work. The toolchain stuff has continually been fantastic. I'm very curious to see what they come up with next! > Semantic analysis is the most difficult part of th…

> I believe memory safety is table stakes I'm not sure what that means. Java lets you do many things programs may want to do in a memory-safe way but not everything. Rust lets you do fewer things than Java in a memory-safe way, but more things than Zig. Zig lets you do fewer things in a memory-safe way than Rust, but more than C. So among these four languages we already have four levels of memory safety, none of them…

safe Rust is actually more memory safe than Java, since it guards against data races (in Java data races are not UB, but they are still one of the worst kinds of bugs because it leads to logically impossible program states)

Also note that Java has unsafe, but doesn't have the culture of plainly stating safety invariants like Rust. The unsafe features of Java are less widely used, but when they are you rarely know if a Java library has unsafe internals for performance, and if they do, it may be hard to audit

Re: Zig's Incremental Compilation Internals

#113
post #68

Earlier quoted context omitted.

> I'd argue that the value of Rust is that it makes low-level viable for a lot of stuff that would otherwise require a lack of memory safety; a lot of it is stuff that might be written in a higher level language, but that's just because relatively few programs are impossible to write in higher level languages. Maybe, but I don't see making a low language viable for something it's not needed as offering much value. Lo…

I just wanna give my perspective since I came from high level languages and pretty much exclusively use Rust now, so perhaps I can articulate why I find value in the language. And my apologies if my input is not wanted, no need to respond if so. First of all I respect your point of view - I'm not a Rust absolutist, I think that garbage collected languages are a massive advantage for a lot of things and would never cr…

I get that perspective and I agree it has value, but for me, Rust is a jack of all trades but master of none, all while being one of the most complicated languages ever made and requires very long build times. So I agree it continues C++'s dream of being "one language for everything", but I think that dream is misguided, and that Rust suffers from most of the same problems as C++.

For low-level programs, I already said that Rust doesn't offer much safety for the things I reach low-level languages for (or, conversely, its safe subset doesn't offer the very control I'm after in such a language). Furthermore, the complexity and implicitness of the language make it harder for me to carefully understand the kind of subtle code I write in such programs. The long build times could mean I write fewer tests.

For high-level programs, Rust's safe subset is technically sufficient, but the problems are even worse (and exactly match C++'s): High level Rust code looks quite good and is easy to write, same as in C++, but the problems start with the maintenance and evolution. Small local changes - to a returned object's lifetime or thread-share ability, or between static and dynamic dispatch - require non-local changes. That's because low-level languages have low abstraction, i.e. the same contract covers fewer possible implementations. True, unlike C++, Rust tells you what things you need to change, but you still need to change them. That was the main problem we had with C++: the code looks great and it's very easy to write at first, but the maintenance and evolution costs - especially when the program is large and long-lived - get high and remain high forever. Furthermore, once a program grows large, it starts suffering from similar performance ovhearheads large C++ programs suffer from: you find yourself needing more dynamic dispatch, which is slow in Rust and C++; you find yourself needing more shared objects with different lifetimes, which are also slow in those languages, so the program isn't even particularly fast or scalable (sure it's faster than a JS or a Go program, but that doesn't say much). Java (or C#) which is aimed at optimising the performance of large programs, removes many of these overheads. Lastly, deep always-on observability/profiling isn't quite poor (it's better now with eBPF, but still a long ways away from what you get with Java or C#).

So yes, Rust and C++ are intended as "one language for everything", and Rust is probably somewhat better than C++, but your high level programs pay for the low level feature (i.e. suffer from the maintenance and performance costs of low-level languages), while your low-level programs pay for the high-level features (the complexity needed for implicitness and safety). So yes, you can do everything, but rarely as well as could be done, and while I see the value in getting expertise only in one language, 1. it's a language that requires a lot of expertise as its "multi-functionality" makes it very complicated, so much so that you could probably become an expert at two more specialised languages for not much more effort, and 2. I think that if you really need to write low-level code, e.g. you're writing a kernel or a hardware driver or a controller or a GC, then expertise in the domain dwarfs expertise in the language anyway (i.e. we're talking years of required experience until you're really good at it).

BUT I acknowledge that the weight I assign to these things is subjective, and I'm certain others reach the opposite conclusion through arguments that are no less reasonable than mine.

Re: Zig's Incremental Compilation Internals

#114
post #41

Earlier quoted context omitted.

> The real trick is to do demand-driven compilation starting from the actual final program's needs this is not so far off from hint-mostly-unused, no?

hint-mostly-unused defers codegen of a crate's functions until compiling a dependent crate where those functions are actually called. Therefore, unused functions will not need to be codegen'd. The downside is that functions which are called from multiple dependent crates will need to be codegen'd in each of their dependents, so this can increase compile times if the crate is not "mostly unused." So it's not quite as…

Can't you just cache them?

Re: Zig's Incremental Compilation Internals

#115
post #107
post #100

Earlier quoted context omitted.

> To me, those are also performance characteristics. Maybe my view on what constitutes "performance" is broader than average here. Yes, but they come with speed gains, so you can't say that you pay "performance overheads" when Java removes some of the performance overheads that programs in low-level languages and replaces them with others. You could similarly say that you pay performance overheads when going in the o…

> > and the experience of the large number of former C/C++ devs I've worked with after they learned Rust > And it's not my experience or a large number of C/C++ devs I work with. >> the only people I've talked to with that experience didn't really try to learn Rust and went in hoping that it wouldn't work for them > Then your exposure isn't wide enough. Or maybe your exposure is only to people who didn't give it a fa…

> or that you are writing programs that are doing things that would require unsafe literally everywhere

You don't need unsafe "literally everywhere" to run into issues. First, what matters most are the areas that are most subtle/tricky in your program. If in those areas Rust doesn't add much safety and makes things worse due to language complexity, that's a problem. Second, when you want low-level control, you might well want it in quite large swaths of the code. For example, one thing that low-level languages currently, in principle, do better than Java is arenas. But the whole point of arenas is that you want _all_ allocations in some large and elaborate call chain to go in the arena (and you'd like to enjoy both the standard library and 3rd party libraries). Rust doesn't make that easy (and neither does C++, for that matter).

> the more it seems like you just genuinely seem to think that you're too smart to accidentally write memory safety bugs, or that the memory safety bugs don't matter much

I don't see how you've reached that conclusion. I told you that for most programs I choose a language that is more memory-safe than Rust, and when I choose a language that's less memory-safe than Rust it's when Rust doesn't offer much safety, either.

See, this is exactly the thing I find so annoying in the Rust discourse. There's no doubt Rust significantly helps avoid memory safety issues (i.e. Rust => more men-safety) but that doesn't mean that caring about memory safety issues means preferring Rust (more mem-safety => Rust). One simply doesn't follow from the other because the logical implication is reversed.

Re: Zig's Incremental Compilation Internals

#116
post #58

Earlier quoted context omitted.

> Look at how many basic data structures (in the standard library or outside it) require unsafe features in Java vs Rust. This is a fundamental misunderstanding of how "unsafe" code relates to a platform's trusted computing base. Rust could move all of those unsafe data structures out of the standard library and into the compiler itself, thereby reducing the amount of occurrences of the string "unsafe" in the source,…

The point is not that those specific implementations use unsafe Rust but to illustrate that to write even basic data structures you need unsafe Rust.

That's just false. You can use `Arc` or even one of the safe GC crates available, and get semantics like Java with no `unsafe`.

Re: Zig's Incremental Compilation Internals

#117
This post is really interesting. As a member of the rust-analyzer team, I cannot avoid comparing it to the situation in Rust land. Rust famously has not less (or even more) sophisticated system for incremental compilation, yet its compilation is way slower. I attribute that to two main things:

- Language design. Zig was designed for fast and incremental compilation, Rust is just not. For instance, the post states that Zig has four properties (layout, type, value, body) that the compiler has to track for changes. Rust has much more, to the point that tracking them statically is just impossible, so the compiler uses a query system that tracks them dynamically, which adds overhead.

- Compiler implementation. Rust is much more complicated to compile than Zig, and rustc is both older and bigger (10x-20x LOC) than the Zig compiler, making changing it way harder.

Re: Zig's Incremental Compilation Internals

#118

Earlier quoted context omitted.

Yes, it is absolutely possible to build a language that uses Zig's compilation model and do borrow checking. Zig is not going to add a borrow checker though, so as a user, it's sort of a moot point.

no. let me be clearer; it is possible to intercept zigs ir from the compiler NOW (well, 15.2 proven) and have a third party package do borrow checking from the data that flow through, without changing zig (think "how miri works without changing rust"). this is not currently directly possible without changing the compiler (~ 50 loc), however the core team has indicated that exporting ir, the only change needed, will b…

I’m incredibly excited for this and have been looking forward to it for a while - I think new moves in IR will solve many issues with have with dynamic analysis of memory safety. The decompilation into other representations like Binary Ninjas IR have been a godsend to actually seeing what the hell Rust and Objective C do on the backend and understand actual cost the compiler makes to create memory fences.

Re: Zig's Incremental Compilation Internals

#119
post #113

Earlier quoted context omitted.

I just wanna give my perspective since I came from high level languages and pretty much exclusively use Rust now, so perhaps I can articulate why I find value in the language. And my apologies if my input is not wanted, no need to respond if so. First of all I respect your point of view - I'm not a Rust absolutist, I think that garbage collected languages are a massive advantage for a lot of things and would never cr…

I get that perspective and I agree it has value, but for me, Rust is a jack of all trades but master of none, all while being one of the most complicated languages ever made and requires very long build times. So I agree it continues C++'s dream of being "one language for everything", but I think that dream is misguided, and that Rust suffers from most of the same problems as C++. For low-level programs, I already sa…

I think it's also a matter of experience - someone used to writing code in unsafe low level languages has a different approach to solving problems and may find Rust gets in the way. I actually started with C++ and after writing a reasonable amount of it I found myself wondering why I had to keep track of lifetimes, nullability etc in my head when it was so easy to mess those up. I kind of discovered "why Rust" from first principles and from then on I was hooked.

I'm not sure I understand your point about dynamic dispatch being slow in Rust/C++ or shared objects? If you're targeting native (which I find important) neither Java or C# are going to be faster surely. Maybe if you're willing to run Java/C# JIT you might find some wins (skeptical it's faster across the board) but you also don't need dynamic dispatch in performance-critical areas. I rarely reach for a Box even in my high-level code.

I haven't worked on large Rust projects (> 500k loc) so I can't speak to the maintenance costs of that, but for me it doesn't matter (at least yet).

But we see even experienced professionals making mistakes with low level languages and I think it's worth considering if it's worth some of the cons you bring up to avoid those. Kind of reminds me of Carmack talking about static code analysis years ago:

https://archive.is/qC9a

> The more I push code through static analysis, the more I’m amazed that computers boot at all.

Re: Zig's Incremental Compilation Internals

#120

I just looked up a Hello World program from the Zig Wikipedia article: const std = @import("std"); const File = std.Io.File; pub fn main(init: std.process.Init) !void { _ = try File.stdout().writeStreamingAll(init.io, "Hello, World!\n"); } That's a lot to follow, just to output a plan-text message, especially after this line: "The primary goal of Zig is to be a better solution to the sorts of tasks that are currently…

Everything it's doing it clear and readable. It's just not as easy to write. It streams the bytes to stdout using the default IO interface, and it can fail.

Alternatively, here's a simpler version (prints to stderr).

    const std = @import("std");

    pub fn main() void {
        std.debug.print("Hello, world!\n", .{}); 
    }
In practice, you normally don't want to print messages to stdout. So the increased friction here actually pushes you in a better direction.
Post reply on HN