Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Comparison of C/POSIX standard library implementations for Linux

etalabs.net

Post reply on HN
Print view View original

51–58 of 58 posts

Re: Comparison of C/POSIX standard library implementations for Linux

#51

Post by pizlonator »

cryptonector wrote: post #50
pizlonator wrote: post #49

Earlier quoted context omitted.

Yeah YOLO. I needed a fun term to refer to the C that isn’t Fil-C. I call it Yolo-C. Hence yololand - the part of the Fil-C process that contains a bit of Yolo-C code for the Fil-C runtime.

Thanks. I went looking and saw this in the Fil-C manifesto: > It's even possible to allocate memory using malloc from within a signal handler (which is necessary because Fil-C heap-allocates stack allocations). Hmm, really? All stack allocations are heap-allocated? Doesn't that make Fil-C super slow? Is there no way to do stack allocation? Or did I misread what you meant by 'stack allocations'?

It’s a GC allocation, not a traditional malloc allocation. So slower than stack allocation but substantially faster than a malloc call.

And that GC allocation only happens if the compiler can’t prove that it’s nonescaping. The overwhelming majority of what look like stack allocations in C are proved nonescaping.

Consequently, while Fil-C does have overheads, this isn’t the one I worry about.

Re: Comparison of C/POSIX standard library implementations for Linux

#52

Post by cryptonector »

pizlonator wrote: post #51
cryptonector wrote: post #50

Earlier quoted context omitted.

Thanks. I went looking and saw this in the Fil-C manifesto: > It's even possible to allocate memory using malloc from within a signal handler (which is necessary because Fil-C heap-allocates stack allocations). Hmm, really? All stack allocations are heap-allocated? Doesn't that make Fil-C super slow? Is there no way to do stack allocation? Or did I misread what you meant by 'stack allocations'?

It’s a GC allocation, not a traditional malloc allocation. So slower than stack allocation but substantially faster than a malloc call. And that GC allocation only happens if the compiler can’t prove that it’s nonescaping. The overwhelming majority of what look like stack allocations in C are proved nonescaping. Consequently, while Fil-C does have overheads, this isn’t the one I worry about.

I see! Thanks for that answer. I'm sure I'll have lots of questions, like these:

You say you don't have to instrument malloc(), but somehow you must learn of the allocation size. How?

Are aliasing bugs detected?

I assume that Fil-C is a whole-program-only option. That is, that you can't mix libraries not compiled with Fil-C and ones compiled with Fil-C. Is that right?

So one might want a whole distro built with Fil-C.

How much are you living with Fil-C? How painful is it, performance-wise?

BTW, I think your approach is remarkable and remarkably interesting. Of course, to some degree this just highlights how bad C (and C++) is (are) at being memory-safe.

Re: Comparison of C/POSIX standard library implementations for Linux

#53

Post by pizlonator »

cryptonector wrote: post #52
pizlonator wrote: post #51

Earlier quoted context omitted.

It’s a GC allocation, not a traditional malloc allocation. So slower than stack allocation but substantially faster than a malloc call. And that GC allocation only happens if the compiler can’t prove that it’s nonescaping. The overwhelming majority of what look like stack allocations in C are proved nonescaping. Consequently, while Fil-C does have overheads, this isn’t the one I worry about.

I see! Thanks for that answer. I'm sure I'll have lots of questions, like these: You say you don't have to instrument malloc(), but somehow you must learn of the allocation size. How? Are aliasing bugs detected? I assume that Fil-C is a whole-program-only option. That is, that you can't mix libraries not compiled with Fil-C and ones compiled with Fil-C. Is that right? So one might want a whole distro built with Fil-C…

Malloc is just a wrapper for zgc_alloc and passes the size through. "Not instrumenting malloc" just means that the compiler doesn't have to detect that you're calling malloc and treat it specially (this is important as many past attempts to make C memory safe did require malloc instrumentation, which meant that if you called malloc via a wrapper, those implementations would just break; Fil-C handles that just fine).

Not sure exactly what you mean by aliasing bugs. I'm assuming strict aliasing violations. Fil-C allows a limited and safe set of strict aliasing optimizations, which end up having the effect of loads/stores moving according to a memory model that is weaker than maybe you'd want. So, Fil-C doesn't detect those. Like in any clang-based compiler, Fil-C allows you to pass `-fno-strict-aliasing` if you don't want those optimizations.

That's right, you have to go all in on Fil-C. All libs have to be compiled with Fil-C. That said, separate compilation of those modules and libraries just works. Dynamic linking just works. So long as everything is Fil-C.

Yes you could build a distro that is 100% Fil-C. I think that's possible today. I just haven't had the time to do that.

All of the software I've ported to Fil-C is fast enough to be usable. You don't notice the perf "problem" unless you deliberately benchmark compute workloads (which I do - I have a large and ever-growing benchmark suite). I wrote up my thoughts about this in a recent twitter discussion: https://x.com/filpizlo/status/1920848334429810751

A bunch of us PL implementers have long "joked" that the only thing unsafe about C are the implementations of C. The language itself is fine. Fil-C sort of proves that joke true.

Re: Comparison of C/POSIX standard library implementations for Linux

#54

Post by cryptonector »

pizlonator wrote: post #53
cryptonector wrote: post #52

Earlier quoted context omitted.

I see! Thanks for that answer. I'm sure I'll have lots of questions, like these: You say you don't have to instrument malloc(), but somehow you must learn of the allocation size. How? Are aliasing bugs detected? I assume that Fil-C is a whole-program-only option. That is, that you can't mix libraries not compiled with Fil-C and ones compiled with Fil-C. Is that right? So one might want a whole distro built with Fil-C…

Malloc is just a wrapper for zgc_alloc and passes the size through. "Not instrumenting malloc" just means that the compiler doesn't have to detect that you're calling malloc and treat it specially (this is important as many past attempts to make C memory safe did require malloc instrumentation, which meant that if you called malloc via a wrapper, those implementations would just break; Fil-C handles that just fine).…

> Not sure exactly what you mean by aliasing bugs.

I meant that if the same allocation were accessed as different kinds of objects, as if through a union, ... I guess what I really meant to ask is: does Fil-C know the types of objects being pointed to by a pointer, and therefore also the number of elements in arrays?

Re: Comparison of C/POSIX standard library implementations for Linux

#55

Post by pizlonator »

cryptonector wrote: post #54
pizlonator wrote: post #53

Earlier quoted context omitted.

Malloc is just a wrapper for zgc_alloc and passes the size through. "Not instrumenting malloc" just means that the compiler doesn't have to detect that you're calling malloc and treat it specially (this is important as many past attempts to make C memory safe did require malloc instrumentation, which meant that if you called malloc via a wrapper, those implementations would just break; Fil-C handles that just fine).…

> Not sure exactly what you mean by aliasing bugs. I meant that if the same allocation were accessed as different kinds of objects, as if through a union, ... I guess what I really meant to ask is: does Fil-C know the types of objects being pointed to by a pointer, and therefore also the number of elements in arrays?

It’s a dynamically typed capability system.

So, if you store a pointer to a location in memory and then load from that location using pointer type, then you get the capability that was last stored. But if the thing stored at the location was an integer, you get an invalid capability.

So Fil-C’s “type” for an object is ever evolving. The memory returned from malloc will be nothing but invalid capabilities for each pointer width word in that allocation but as soon as you store pointers to it then the locations you stored those pointers to will be understood as being pointer locations. This makes unions and weird pointer casts just work. But you can ever type confuse an int with a pointer, or different pointer types, in a manner that would let you violate the capability model (ie achieve the kind of weird state where you can access any memory you like).

Lots of tricks under the hood to make this thread safe and not too expensive.

Re: Comparison of C/POSIX standard library implementations for Linux

#56

Post by flohofwoe »

snickerer wrote: post #3

Fun libc comparison by the author of musl. My getaway is: glibc is bloated but fast. Quite unexpected combination. Am I right?

Choice still matters IMHO. E.g. a very small but slow malloc/free may be preferable if your code only allocates infrequently. Also linking musl statically avoids the whole glibc dll version mess, admittedly only useful for cmdline tools though.

Re: Comparison of C/POSIX standard library implementations for Linux

#57

Post by cryptonector »

pizlonator wrote: post #55
cryptonector wrote: post #54

Earlier quoted context omitted.

> Not sure exactly what you mean by aliasing bugs. I meant that if the same allocation were accessed as different kinds of objects, as if through a union, ... I guess what I really meant to ask is: does Fil-C know the types of objects being pointed to by a pointer, and therefore also the number of elements in arrays?

It’s a dynamically typed capability system. So, if you store a pointer to a location in memory and then load from that location using pointer type, then you get the capability that was last stored. But if the thing stored at the location was an integer, you get an invalid capability. So Fil-C’s “type” for an object is ever evolving. The memory returned from malloc will be nothing but invalid capabilities for each poi…

So Fil-C has two types: pointers, and everything else. Clever.

Re: Comparison of C/POSIX standard library implementations for Linux

#58

Post by GabrielTFS »

ape4 wrote: post #45
kstrauser wrote: post #4

Earlier quoted context omitted.

It’s not shocking. More complex implementations using more sophisticated algorithms can be faster. That’s not always true, but it often is. For example, look at some of the string search algorithms used by things like ripgrep. They’re way more complex than just looping across the input and matching character by character, and they pay off. Something like glibc has had decades to swap in complex, fast code for simple-…

Yeah look at even strlen() https://github.com/lattera/glibc/blob/master/string/strlen.c

That's the generic implementation - it's not used on most popular architectures (I think the most popular architecture it's used on would be RISC-V or MIPS) because they all have architecture-specific implementations. The implementation running on the average (x86) computer is likely to be https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86... (if you have AVX512), https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86... (if you have AVX2 and not AVX512) or https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86... (if you have neither AVX2 nor AVX512 - rather rare these days)
Post reply on HN