Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zig's Incremental Compilation Internals

mlugg.co.uk

Post reply on HN
Print view View original

281–290 of 292 posts

Re: Zig's Incremental Compilation Internals

#281

Post by uecker »

dwattttt wrote: post #265
uecker wrote: post #254

Earlier quoted context omitted.

It certainly works in my C programs. And Fil-C demos imply that this also works for a lot of other complex real world programs, i.e. a basic Linux distribution with libreoffice on top... So yes, I would say this just works. I agree that catching this at compile-time is better, and to some degree this also works: https://godbolt.org/z/sse74vK9o Rust does not offer compile-time guarantee that an out-of-bounds access do…

Agreed on the last point. I'm very much in favour of developing an analysis that can assert the absence of an abort, I just can't see how an approach like Fil-C can achieve that. It can guarantee a potential memory corruption event will be caught at runtime, but at that point I have very limited options. To be clear, I prefer aborting safely to corrupting memory. But I prefer "issue detected at development time" sign…

I think we agree on this. Fil-C does not ensure this. Showing the absence of run-time errors at compile-time is a hard problem, and the most realistic solution in general is via model checkers which I think would be the ideal solution, but we would need better tooling. Dependently types languages are also interesting.

In the context of existing tooling and C, there is also a lot of potential in using the optimizer to show safety properties. This works quite well to show the absence of signed overflow issues and null-pointer correctness, but still less well for bounds checking, e.g. see here for some preliminary experiments: https://uecker.codeberg.page/2026-05-22.html

Re: Zig's Incremental Compilation Internals

#282

Post by nh2 »

panstromek wrote: post #163
nh2 wrote: post #11

Earlier quoted context omitted.

10 years ago, I commented on the Rust issue for "Incremental recompilation", where it was suggested that Rust could at least adopt Haskell GHC's model of incrementality, which is currently file-level: https://github.com/rust-lang/rust/issues/2369#issuecomment-1... This would already help a lot. I recommend anybody who's interested in incremental recompilation to read what GHC does, because the effort to achieve that…

> GHC currently needs to parse+typecheck+codegen a file before it can process other files that import it. Codegen is slow. Thus, there's currently demand split compilation into "stages", so that the next file can be typechecked after its imports have been just typechecked (not codegenned). > I would also enjoy if recompilation avoidance were to happen at the function level, not the file level. This sounds like Rust i…

Can you point at what you mean?

If you changed a function implementation in the libc crate, not changing that function's signature, how much codegen would happen in downstream packages?

The maximally recompilation-avoiding effect would be: Only that one function gets codegenned. Everything else just gets relinked into their final executable or .so.

Re: Zig's Incremental Compilation Internals

#283

Post by pron »

duped wrote: post #277
pron wrote: post #260

Earlier quoted context omitted.

They find it immensely useful in practice only when the safe subset is useful. In C, you could say that the same definition of memory-safety exists, only the safe subset is empty, and in that case people don't find the fact that C could be described as memory-safe in that way useful at all. And that's exactly my point. The safe subset of Rust doesn't sufficiently cover the very things that I choose a low-level langua…

You don't find it useful that use after free, double free, uninitialized memory, and many kinds of race conditions are just impossible in code that compiles? What exactly is the low level subset you feel like you can't use?

> You don't find it useful that use after free, double free, uninitialized memory, and many kinds of race conditions are just impossible in code that compiles?

Sure, but memory safety by itself is not the reason to pick Rust because other languages do memory safety even better.

> What exactly is the low level subset you feel like you can't use?

The things that make me want to use a low-level language in the first place, such as controlling exactly when memory is allocated and freed and where exactly objects are placed in memory.

When I don't need to do these things and all I want is memory safety and performance, then I already have better options.

Re: Zig's Incremental Compilation Internals

#284

Post by pron »

vlovich123 wrote: post #274
pron wrote: post #271

Earlier quoted context omitted.

> This is incorrect. You can write those data structures in safe Rust just as easily as you can in Java. You'd write them using the safe primitives that the Rust stdlib provides to you, just like how Java does it. Such collections could often be made to have superior performance by using unsafe Rust, which is why the collections in the stdlib use unsafe code internally, but it's an optimization, not a requirement. Of…

> Of course you can, but the reason you don't is that their performance would be quite bad Not very different from Java or C#. Worse than zig/c/c++ but that’s because those are the equivalent of using use in rust.

> Not very different from Java or C#

It will be significantly worse than Java, at least in some important situations (don't know about C#). I've been programming in C++ for many, many years, and I find it increasingly hard to even match Java's performance, even when writing unsafe code, especially when programs get larger and/or more concurrent (very broadly speaking, Java's performance is about that of C++ - in some situations it's worse and in others is better, but the same general vicinity; after all, the JVM was designed to address some of the performance issues that certain classes of C++ programs suffer from).

Performance is not the reason to use a low-level language in many domains, and in those domains, if you want super-high performance and safety, there are better and more popular alternatives already. You use a low-level language when you need the things low-level languages do best, but you also expect performance that is more-or-less the same as the high-performance safe alternatives.

Re: Zig's Incremental Compilation Internals

#285

Post by [deleted] »

josephg wrote: post #125
norman784 wrote: post #34

Earlier quoted context omitted.

Not OP, but AFAIK one big issue with FIL-C it does the checks at runtime, adding overhead, don't quote me on this, but IIRC is around 20% slower.

2x slower and 4x less memory efficient were the numbers I heard a year or so ago. Reaching within 20% of full native performance while using a garbage collector sounds too good to be true. Do you have any actual benchmarks?

[deleted]

Re: Zig's Incremental Compilation Internals

#286

Post by panstromek »

nh2 wrote: post #282
panstromek wrote: post #163

Earlier quoted context omitted.

> GHC currently needs to parse+typecheck+codegen a file before it can process other files that import it. Codegen is slow. Thus, there's currently demand split compilation into "stages", so that the next file can be typechecked after its imports have been just typechecked (not codegenned). > I would also enjoy if recompilation avoidance were to happen at the function level, not the file level. This sounds like Rust i…

Can you point at what you mean? If you changed a function implementation in the libc crate, not changing that function's signature, how much codegen would happen in downstream packages? The maximally recompilation-avoiding effect would be: Only that one function gets codegenned. Everything else just gets relinked into their final executable or .so.

This is difficult to answer, because these systems exhibit somewhat chaotic behaviour, and it gets even more complicated cross-crate. I was mostly talking about single crate scenario.

Since you mentioned libc, the likely answer is that nothing gets codegened in downstream crates. But this is only because libc functions are usually not generic or `#[inline]`. Changes to generic or inline functions can dirty downstream codegen units where the function was called. Inside `libc`, the change will trigger recompilation of at least one codegen unit, depending on how the function is used inside libc itself. Single crate is split into 256 units in incremental mode.

Nevertheless, even if the codegen is needed just for the `libc` crate, `libc` will dirty its metadata, which means that downstream crates will still need to recompile the initial steps before incremental kicks in (which is roughly parsing, macro expansion and name resolution). After that, the query system just returns cached results for all the subsequent steps.

There's some work going towards skipping the rustc invocation altogether in those cases (usually referred to as "Relink don't Rebuild" proposal), because even just loading the dependency graph and figuring out that you don't have to do anything can take quite bit of time for larger programs.

Re: Zig's Incremental Compilation Internals

#287

Post by boomlinde »

dlcarrier wrote: post #270
boomlinde wrote: post #202

Earlier quoted context omitted.

The typical Hello World implementation tends to reveal very little about the language, because their print/println/printf/whatever implementations have failure modes that are either impossible to handle or easily ignored (e.g. panicking, throwing exceptions or returning error codes which you can implicitly ignore without compilation error) which they frequently use to effectively hide the complexity inherent to the p…

It's really easy to make a C hello world program that forwards the success of printf: #include int main(void) { return printf("hello, world\n"); } I just tested it in a Bash shell, and it works great, only adding a single word, with clear functionality, to the example.

This program will always give a non-zero return code. This is unconventional if not straight up wrong, if the goal is for main to indicate whether it was successful in printing.

Re: Zig's Incremental Compilation Internals

#288

Post by throwaway2037 »

pron wrote: post #257
throwaway2037 wrote: post #220

Earlier quoted context omitted.

> Java lets you do many things programs may want to do in a memory-safe way but not everything. Can you give an example of Java lets you do that is not memory-safe? Honestly, I don't really count the crazy off-heap tricks that some people use. It's almost like writing a Python library in pure C, intead of Python, then complaining that Python allows you to do non-memory-safe things.

With FFM you can do unsafe things, such as call native code and directly manipulate memory read from or written to by native code.

Eh, I don't really count FFM. Would you have said JNI before Java 22? FFM is little more than a convenient wrapper around the same that can be dome with JNI.

Re: Zig's Incremental Compilation Internals

#289

Post by orf »

pizlonator wrote: post #210
mirashii wrote: post #85

Earlier quoted context omitted.

I generally agree with all of this, but I'll add a few additional remarks. Because it's come up a bunch lately, I decided to do a bunch of code review/audit of the Fil-C codebase, and I'd say while it's got a lot of good bones, there's a long way to go to being a foundation I'd be ready to build on. I've reported a few UAF's upstream, and I've got a few PRs I'll add on, but if it only took me a day or two to find som…

First of all, it’s incredible that on a HN thread about a language that isn’t C, there are 46 mentions of Fil-C! You guys are obsessed! I make bold claims because they hold water. - You can at worst corrupt only the capability you’re pointing to. - intra object overflows are almost never useful for memory corruption exploits unless they let you corrupt a pointer, and Fil-C prevents that from being useful because you…

> the zunsafe api is basically unused. One library uses it (OpenSSL) for good reasons

Wait, so there are escape hatches? But… you’ve repeatedly said, many times, that there are zero escape hatches?

And now here you’re saying not only that there only are escape hatches, but there’s a good reason to use them?

Damn. Misrepresenting `unsafe{}` whilst saying your language is better because there are no escape hatches and no need for escape hatches is like… 80% of your online personality.

When can we expect the website to be updated to remove the misleading claims?

Re: Zig's Incremental Compilation Internals

#290

Post by pizlonator »

orf wrote: post #289
pizlonator wrote: post #210

Earlier quoted context omitted.

First of all, it’s incredible that on a HN thread about a language that isn’t C, there are 46 mentions of Fil-C! You guys are obsessed! I make bold claims because they hold water. - You can at worst corrupt only the capability you’re pointing to. - intra object overflows are almost never useful for memory corruption exploits unless they let you corrupt a pointer, and Fil-C prevents that from being useful because you…

> the zunsafe api is basically unused. One library uses it (OpenSSL) for good reasons Wait, so there are escape hatches? But… you’ve repeatedly said, many times, that there are zero escape hatches? And now here you’re saying not only that there only are escape hatches, but there’s a good reason to use them? Damn. Misrepresenting `unsafe{}` whilst saying your language is better because there are no escape hatches and…

From the second paragraph of fil-c.org:

"Fil-C has no unsafe statement and only limited FFI to unsafe code."

`zunsafe_call` is a weird thing to get hung up on as an "escape hatch", considering it's just a super limited form of FFI, intentionally designed so that it's only usable for OpenSSL's use case.

> Misrepresenting `unsafe{}`

`unsafe` lets you write Rust code that violates any reasonable definition of memory safety (including Rust's definition or my definition), and it's widely used.

Post reply on HN