Live data from Hacker News

Zig's Incremental Compilation Internals

mlugg.co.uk

41–50 of 292 posts

Re: Zig's Incremental Compilation Internals

#41
post #24

Earlier quoted context omitted.

> Because Rust commits to the traditional compilation model and pipleine (which I think is overall a good thing, or at least, a good thing to support), it does a lot of work that will eventually be thrown away. Consider this example: I have a library with a function foo that returns a simple 42. I have a binary which calls foo from that library and prints the result. Now imagine the library is a hundred thousand line…

Yes, this idea is sort of similar, just like, more complicated in a sense than the clean design, which is what Zig does. Incidentally, you might want to look at gc-sections, which Rust already does. As well as https://rust-lang.github.io/rust-project-goals/2025h2/relink... which is kinda related. The sort of key here is understanding that "produce a library" means that every public item is "used" in the sense of "do…

> The real trick is to do demand-driven compilation starting from the actual final program's needs

this is not so far off from hint-mostly-unused, no?

Re: Zig's Incremental Compilation Internals

#42
post #8

Zig's toolchain work is continually impressive. While I still don't plan to write software in it, given that I believe memory safety is table stakes, all of this stuff is very, very good. Before the incremental work, it was the toolchain and cross-compiler work. The toolchain stuff has continually been fantastic. I'm very curious to see what they come up with next! > Semantic analysis is the most difficult part of th…

The thing with rust is that you get safety with slow compilation, it's a tradeoff. Zig doesn't have the same safety guarantees, it's on the dev to use safe coding patterns, so the tradeoff for safety is discipline or experience.

(tongue in cheek) It seems recent history has shown that zig can get you to working software faster, then you can port it to rust once you are acquired or find market fit?

Maybe at some point in the future zig could add a rust compilation target ( like with `-ofmt=c` )...

Re: Zig's Incremental Compilation Internals

#43
There is something that I don't fully understand about this design: why are they insisting on building a giant binary for debug builds that contains all of the code? From my perspective, a simpler approach is to generate many smaller shared libraries (perhaps at the file level) and link them in to the final binary. With this approach, the program binary would have a tiny text section and a (potentially long) list of shared libraries to load. But even with thousands of shared libraries to load, the resulting program binary would not be all that long and there would be no need for binary patching.

I understand that for a release mode a single giant binary may be desirable, but I am struggling to understand this design for debug builds. Moreover, while reading this article, I found myself wondering what happens if the main binary becomes corrupted. Maybe the user cancels compilation with ctrl+c while it is patching the binary. Even if they have a story for avoiding and/or detecting corruption, it is simpler to not patch in the first place and always generate a new main binary. Again, this is reasonable because the new binary is mainly just a list of shared libraries to link which will not take up much space and can be written to disk quickly. Moreover, this process can be done recursively, e.g. at the subdirectory level, so that during incremental linking a few quite small shared libraries may be produced rather than patching in the new code and writing cascading relocations.

Re: Zig's Incremental Compilation Internals

#44
post #29

Earlier quoted context omitted.

I'll say the same thing I said to you as I said to Andrew, last time he and I talked about this: the way that everyone talks about memory safety (with maybe two exceptions, one okay (go) and one I dislike (fil-c)) is that "memory safe language" is about there being a clear delineation between what is memory safe and what is not, and that the unsafe aspect is a superset. Rust and Java both are memory safe, except wher…

Would you mind sharing some thoughts about fil-c? AFAICT its claims mostly check out so besides implementation details (GC?) it seems directionally good.

Okay so: in general, as a rule of thumb: anything that makes stuff have more memory safety is good. And experiments towards that end are also good.

What I do not like, primarily comes down to how the project is talked about and marketed. First, because it promotes an "us vs them" mindset, instead of a "we're all trying to improve memory safety" mindset, and second, because in doing so, it also overstates its case.

These things are sort of intertwined. Let's talk about the overstatement first. Fil-c has its own definition of memory safety that is slightly different than others. For example, I saw this recently:

    #include 
    #include 
    #include 
    
    struct User {
        char name[8];
        int is_root;
    };
    
    int main(int argc, char*argv[]) {
        struct User* user = malloc(sizeof(struct User));
        strcpy(user->name, argv[1]);
        if (user->is_root) {
            printf("I am root!\n");
        } else {
            printf("I am not root :(\n");
        }
        return 0;
    }
This, when invoked with "012345678" passed in, will print "I am root!". In my understanding, this is deliberately allowed.

But beyond corners like this, fil-c's author will go on about "Rust has unsafe as a hatch, fil-c does not" while if you control-f for "zunsafe_" on https://fil-c.org/stdfil you get ... escape hatches.

The author regularly erases the difference between "traps at runtime" and "is prevented at compile time", which are legitimate tradeoffs where one or the other may be better depending on what you're doing. But they're presented as either equivalent, or one is superior, and I find this muddles the discourse. The performance issues also tie into this, "add a GC" is absolutely a valid way to handle these sorts of issues, but it is not the same thing as what Rust does. And that's okay! But presenting it as purely superior means that it's just hard to talk about.

Speaking of muddling the discourse, the author regularly trolls on X, providing tons of bad faith arguments and generally trying to rile up a "fil-c vs Rust" war that I think reduces our ability to talk about these differences in a calm, engineering focused context.

Finally, due to its design, fil-c is effectively Linux only. That's great for Linux, but many people also use other systems, and so it is not a meaningful option for them.

Anyway, after saying all that: I still think that it is a good project, and that it should exist and continue to be worked on. I just wish that the heat was turned down, and people could talk about the various approaches and their tradeoffs without turning it into a culture war.

Re: Zig's Incremental Compilation Internals

#45
post #39
post #31

Earlier quoted context omitted.

That framing may seem intellectually satisfying, but it's not useful in practice. Consider the extreme edge case of C: We can clearly mechanically delineate between the empty program and a non-empty one, we call the empty program safe and any program that isn't empty unsafe (i.e. C is memory-safe if you want to do nothing and not if you want to do anything). And so, we also have this property that in C you can't do a…

> That framing may seem intellectually satisfying, but it's not useful in practice. Honestly, that's exactly how I feel in reverse. The framing you gave is more intellectually interesting, but it doesn't help explain the actual real-world outcomes where in practice, Rust and Java both don't have much problem with unsafety, whereas C does, and at least from what I've heard, Zig does as well. > I reach for a low-level…

> Rust and Java both don't have much problem with unsafety, whereas C does, and at least from what I've heard, Zig does as well.

I'm not interested in the definition so much as I am in calling it "table stakes", and so the fact that these languages satisfy their promises is uninteresting in isolation. What matters is the value of their promises. The majority of Rust programs I see, I wouldn't have written in a low-level language, so the fact that it offers memory safety for the things I don't need it to do does nothing for me.

Now, clearly, Rust's originators didn't consider what Java offers (or at least what it offered 20 years ago when Rust was first conceived) to be table stakes or they wouldn't have wanted Rust. Java exacted some price in exchange for its memory safety that was unacceptable to Rust's originators and trumped its memory safety. But the same thing happens with Rust vs Zig. Rust exacts a heavy price for its memory safety, that - just as in Rust's case vs Java - is sometimes unacceptable. So I can't see how any of these could be "table stakes".

> I mean, sure, if you want to do things that are fundamentally not possible to validate because you think you're smart enough not to screw up, that's going to make Rust a tough sell.

What Rust can validate and what can fundamentally be validated are two very, very different things. Compared to what ATS can validate, what Rust can validate is almost indistinguishable from C. In Rust you have to do lots and lots of things that require you to be "smart enough not to screw up" that you could prove in ATS, and still no one (including Rust programmers) would say that what ATS offers is "table stakes" because, obviously, it comes at a high price that the people who choose Rust don't want to pay.

So clearly different languages offer different capabilities and charge a price for them. Sometimes the price is worth it and sometimes it isn't.

> so I'm distrustful of the claim that being smart and diligent is enough to prevent the sort of bugs that we're still dealing with after half a century of us learning how not to write C

But Java or Rust programs still suffer from a lot of bugs that ATS could eliminate, if you're willing to pay the price, and you're clearly unwilling. ATS programmers could say about Rust programmers what you say about C++ programmers. Clearly there's no universal table stakes here.

Re: Zig's Incremental Compilation Internals

#46
post #32

Earlier quoted context omitted.

I'll say the same thing I said to you as I said to Andrew, last time he and I talked about this: the way that everyone talks about memory safety (with maybe two exceptions, one okay (go) and one I dislike (fil-c)) is that "memory safe language" is about there being a clear delineation between what is memory safe and what is not, and that the unsafe aspect is a superset. Rust and Java both are memory safe, except wher…

So that could be a clear definition, but for it to be "table stakes" it needs to have some universal value and it doesn't (in fact, that very same definition could also classify even C as "memory safe"): https://news.ycombinator.com/item?id=49087458

I can say that something is "table stakes" for me without demanding that everyone else adhere to my values.

Re: Zig's Incremental Compilation Internals

#47
post #29

Earlier quoted context omitted.

Would you mind sharing some thoughts about fil-c? AFAICT its claims mostly check out so besides implementation details (GC?) it seems directionally good.

Okay so: in general, as a rule of thumb: anything that makes stuff have more memory safety is good. And experiments towards that end are also good. What I do not like, primarily comes down to how the project is talked about and marketed. First, because it promotes an "us vs them" mindset, instead of a "we're all trying to improve memory safety" mindset, and second, because in doing so, it also overstates its case. Th…

> it promotes an "us vs them" mindset, instead of a "we're all trying to improve memory safety" mindset

But you did the same thing when, on the spectrum that ranges from C to ATS, with Zig, Rust, and Java somewhere in the middle (though all closer to C than to ATS), you declared the exact compromise that Rust makes "table stakes"! [1]

Zig improves on C's memory safety when it comes to spatial safety, possibly the more impactful kind, so it, too, could be part of the "we're all trying to improve memory safety", yet you exclude it.

You're trying to draw some hard line that passes exactly between Rust and Zig on the C to ATS spectrum, and I'm trying to say that that line isn't there (your attempt at a definition of delineating safe and unsafe code also applies to C). Obviously, C, Zig, Rust, Java, and ATS all make very different tradeoffs, all of which may be more or less attractive to different people and in different circumstances, but there is no sharp line, at least not one that is meaningful enough to be "table stakes". Your personal inclinations place a premium on the things Rust offers and Zig doesn't while mine are the opposite, but I make no claim to universality.

I'm happy to accept that not everyone shares my aesthetics and can understand why some people prefer Rust, but those claims to or hints at universality annoy me (as they did when they were made by Haskellers, and I actually find Haskell's aesthetics quite pleasing), as they are simply unsupported. I've spent a lot of time studying formal methods and software correctness in general (https://pron.github.io) and if there's one thing we know in that field is that things are never that simple (and, bringing this back to this posts topic, even something like incremental compilation can contribute to program correctness).

(Now, you may argue that you're only talking about "memory safety" and not correctness in general, but what gives memory safety value is that violations are causes of many dangerous vulnerabilities; but once, say, Java eliminates all of them, 100% of bugs/vulnerability - which are still numerous - will be caused by other problems, all potentially avoidable with ATS, so why isn't ATS table stakes? Of course, the answer is cost, but all the languages on the spectrum differ in their costs.)

[1]: I assume that you meant Rust's compromise, because you implied that Zig doesn't pass that bar but Rust does.

Re: Zig's Incremental Compilation Internals

#48
post #12

Earlier quoted context omitted.

not quite! if your project is mixed C/Zig, then editing Zig would work but editing C would not. the Zig compiler caches Zig AIR but no information from C. plus, the C compilation can only really be done with LLVM and writing a self-hosted backend would be PITA for C

Not quite true, there is already a capable C compiler written in Zig (Aro/arocc), and a plan to transition to it for C compilation: https://codeberg.org/ziglang/translate-c Using this native (written in Zig) C compiler to translate C source into Zig source as a part of the build, would presumably lend itself trivially to all the incremental logic in TFA, as updating C would update the generated Zig, and the increment…

> a plan to transition to it for C compilation

That's not planned AFAIK (see https://github.com/ziglang/zig/issues/16269). `translate-c` is really only intended for header translation, not C source code.

See https://github.com/ziglang/zig/issues/20875 for the (not fully fleshed out yet) plans around C compilation.

Re: Zig's Incremental Compilation Internals

#49
post #38

Earlier quoted context omitted.

Yes, this idea is sort of similar, just like, more complicated in a sense than the clean design, which is what Zig does. Incidentally, you might want to look at gc-sections, which Rust already does. As well as https://rust-lang.github.io/rust-project-goals/2025h2/relink... which is kinda related. The sort of key here is understanding that "produce a library" means that every public item is "used" in the sense of "do…

Makes sense! I think what's most surprising to me about the library compilation model is that for the most part, it doesn't seem like it's actually that useful to how Rust does things by default. Without something like sccache, the intermediate libraries aren't going to be reused across all of my projects, so unless I'm compiling multiple binaries in the same project, I'm not really getting much out of having the ent…

I think that there is a general desire to work with the system, and since that is how other systems languages have generally done it, doing it the same way feels good. You don't necessarily want to innovate on everything all at once.

It is true that for various reasons, Rust can't take as much advantage as say, C can.

> I'm guessing this is related to what you mean by potentially being able to do something differently if there were more time before 1.0?

I mean more traditional language design things, but sure, this too.

Re: Zig's Incremental Compilation Internals

#50
post #8

Earlier quoted context omitted.

The thing with rust is that you get safety with slow compilation, it's a tradeoff. Zig doesn't have the same safety guarantees, it's on the dev to use safe coding patterns, so the tradeoff for safety is discipline or experience.

(tongue in cheek) It seems recent history has shown that zig can get you to working software faster, then you can port it to rust once you are acquired or find market fit? Maybe at some point in the future zig could add a rust compilation target ( like with `-ofmt=c` )...

I know this is mostly a joke but I have seriously wondered if “no hidden behavior” made it easier to port from Zig. Like, regardless of how easy or successful the port was in general, I think Zig’s explicitness may have worked in its favor.
Post reply on HN