Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does C have the best file API

maurycyz.com

Post reply on HN
Print view View original

151–160 of 169 posts

Re: Why does C have the best file API

#151

Post by nagaiaida »

ajross wrote: post #132
nickelpro wrote: post #24

Earlier quoted context omitted.

If we're going to be pedantic, mmap is a syscall. It happens that the C version is standardized by POSIX. The underlying syscall doesn't use the C ABI, you need to wrap it to use it from C in the same way you need to wrap it to use it from any language, which is exactly what glibc and friends do. Moral of the story is mmap belongs to the platform, not the language.

No, that's too far down the pedantry rabbit hole. "mmap()" is quite literally a C function in the 4.2BSD libc. It happens to wrap a system call of the same name, but to claim that they are different when they arrived in the same software and were written by the same author at the same time is straining the argument past the breaking point. You now have a "C Erasure Polemic" and not a clarifying comment. If you take a…

and if i directly do an mmap syscall on linux from a freestanding forth that doesn't go through libc for anything? sure, c unfortunately defines how i have say, pass a string, but that's effectively an arbitrary calling convention at that point; there's no c runtime on the calling side so it's not particularly useful to contend that what i'm using is a c api.

or perhaps mmap is incontrovertibly a c function on platforms where libc wrappers are the sole stable interface to the kernel but something else entirely on linux?

Re: Why does C have the best file API

#153

Post by lionkor »

zadikian wrote: post #146
lionkor wrote: post #107

Earlier quoted context omitted.

That seems highly unlikely. Let's assume that all compilers use the exact same padding in C structs, that all architectures use the same alignment, and that endianness is made up, that types are the same size across 64 and 32 bit platforms, and also pretend that pointers inside a struct will work fine when sent across the network; the question remains still: Why? Is THIS your bottleneck? Will a couple memcpy() operat…

I guess to not have to set up protobuf or asn1. Those preconditions of both platforms using the same padding and endianness aren't that hard to satisfy if you own it all. But do you really have such a complex struct where everything inside is fixed-size? I wouldn't be surprised if it happens, but this isn't so general-purpose like the article suggests.

There are at least 10 steps between protobuf and casting a struct to a char*.

Re: Why does C have the best file API

#154

Post by jcelerier »

hyc_symas wrote: post #144
api wrote: post #113

Earlier quoted context omitted.

ARM is usually bi-endian, and almost always run in little endian mode. All Apple ARM is LE. Not sure about Android but I’d guess it’s the same. I don’t think I’ve ever seen BE ARM in the wild. Big endian is as far as I know extinct for larger mainstream CPUs. Power still exists but is on life support. MIPS and Sparc are dead. M68k is dead. X86 has always been LE. RISC-V is LE. It’s not an arbitrary choice. Little end…

Network data and most serialization formats are big endian because it's easiest to shift bits in and out of a shift register onto a serial comm channel in that order. If you used little endian, the shifter on output would have to operate in reverse direction relative to the shifter on input, which just causes stupid inconsistency headaches.

Isn't the issue with shift registers related to endianness at the bit level, while the discourse above is about endianness at the byte level? Both are pretty much entirely separate problems

Re: Why does C have the best file API

#155

Post by ajross »

nagaiaida wrote: post #151
ajross wrote: post #132

Earlier quoted context omitted.

No, that's too far down the pedantry rabbit hole. "mmap()" is quite literally a C function in the 4.2BSD libc. It happens to wrap a system call of the same name, but to claim that they are different when they arrived in the same software and were written by the same author at the same time is straining the argument past the breaking point. You now have a "C Erasure Polemic" and not a clarifying comment. If you take a…

and if i directly do an mmap syscall on linux from a freestanding forth that doesn't go through libc for anything? sure, c unfortunately defines how i have say, pass a string, but that's effectively an arbitrary calling convention at that point; there's no c runtime on the calling side so it's not particularly useful to contend that what i'm using is a c api. or perhaps mmap is incontrovertibly a c function on platfo…

> and if i directly do an mmap syscall on linux from a freestanding forth

... mmap() remains a system call to a C kernel designed for use from the C library in C programs, and you're running what amounts to an emulator.

The fact that you can imagine[1] an environment where that might not be the case doesn't mean that it isn't the case in the real world.

Your argument appears to be one of Personal Liberty: de facto truths don't matter because you can just make your own. This is sort of a software variant of a Sovereign Citizen, I think.

[1] Can you even link a "freestanding forth" with an mmap() binding on any Unix that doesn't live above the libc implementation? I mean, absent everything else it would have to open code all the flag constants, whose values change between systems. This appears to be a completely fictitious runtime you've invented, which if anything sits as evidence in my favor and not yours.

Re: Why does C have the best file API

#156

Post by nickelpro »

hyc_symas wrote: post #145
nickelpro wrote: post #19

Earlier quoted context omitted.

mmap is also relatively slow (compared to modern solutions, io_uring and friends), and immensely painful for error handling. It's simple, I'll give it that.

On BSD, read() was already implemented in the kernel by page-faulting in the desired pages of the file, to then be copied into the user-supplied buffer. So from the first time mmap was ever implemented, it was always the fastest input mechanism. (First deployed implementation was in SunOS btw, 4.2BSD specified and documented it but didn't implement it.) Anyway there's no magic to get data off a device into memory fas…

mmap is slow because stalling on page faults is slow. Your process stalls and sits around doing nothing instead of processing data you've read already. You can google the benchmarks if you like. io_uring wasn't built just for kicks.

https://www.bitflux.ai/blog/memory-is-slow-part2/

Re: Why does C have the best file API

#157

Post by direwolf20 »

gopalv wrote: post #64
amluto wrote: post #55

I can’t entirely tell what the article’s point is. It seems to be trying to say that many languages can mmap bytes , but: > (as far as I'm aware) C is the only language that lets you specify a binary format and just use it. I assume they mean: struct foo { fields; }; foo *data = mmap(…); And yes, C is one of relatively few languages that let you do this without complaint, because it’s a terrible idea . And C doesn’t…

> correspond to a binary format in accordance with the C ABI on your particular system. We're so deep in this hole that people are fixing this on a CPU with silicon. The Graviton team made a little-endian version of ARM just to allow lazy code like this to migrate away from Intel chips without having to rewrite struct unpacking (& also IBM with the ppc64le). Early in my career, I spent a lot of my time reading Java b…

ARM has always been little-endian. Some were configurable endian.

And it's not a hole. We're not about to spend 100 cycles parsing a decimal string that could have been a little-endian binary number, just because you feel a dependency on a certain endianness is architecturally impure. Know what else is architecturally impure? Making binary machines handle decimal.

Re: Why does C have the best file API

#158

Post by zephen »

amluto wrote: post #124
jklowden wrote: post #120

Earlier quoted context omitted.

It’s a terribly useful idea. FTFY. The program you used to leave your comment, and the libraries it used, were loaded into memory via mmap(2) prior to execution. To use protobuf or whatever, you use mmap. The only reason mmap isn’t more generally useful is the dearth of general-use binary on-disk formats such as ELF. We could build more memory-mapped applications if we had better library support for them. But we don’…

Entire libraries are a weird sort of exception. They fundamentally target a specific architecture, and all the nonportable or version dependent data structures are self describing in the sense that the code that accesses them are shipped along with the data. And if you load library A that references library B’s data and you change B’s data format but forget to update A, you crash horribly. Similarly, if you modify a…

> Entire libraries are a weird sort of exception.

Not really. The entire point of the article is that there are a lot of problem domains where data stays on a single machine, or at least a single type of machine.

Re: Why does C have the best file API

#160

Post by nagaiaida »

ajross wrote: post #155
nagaiaida wrote: post #151

Earlier quoted context omitted.

and if i directly do an mmap syscall on linux from a freestanding forth that doesn't go through libc for anything? sure, c unfortunately defines how i have say, pass a string, but that's effectively an arbitrary calling convention at that point; there's no c runtime on the calling side so it's not particularly useful to contend that what i'm using is a c api. or perhaps mmap is incontrovertibly a c function on platfo…

> and if i directly do an mmap syscall on linux from a freestanding forth ... mmap() remains a system call to a C kernel designed for use from the C library in C programs, and you're running what amounts to an emulator. The fact that you can imagine[1] an environment where that might not be the case doesn't mean that it isn't the case in the real world. Your argument appears to be one of Personal Liberty: de facto tr…

?

i'm not so much imagining an environment per se¹ as describing one i've already written, so i'm not entirely sure where any of this is coming from. if you care to have some additional assurance this isn't somehow an elaborate rhetorical trap, a previous comment about forth tail call elimination with a bit of demonstrative assembly is presumably only a short scroll down my profile. ctrl-f for cmov if you want to find it quickly. as i recall, it came up for similar reasons then because people often make similar incorrect generalizations about lots of things that implicitly sit atop a c runtime in their minds. that said, you're the first one to call me a sovcit before asking any clarifying questions so at least there's some new pizzazz there.

i was clear that i was talking specifically about linux precisely because this isn't something one can do portably for exactly the reasons you're describing (which, yes, makes porting things built like this off of linux before the point you've built up enough to be able to go through libc annoying and ad hoc at the very least).

the fact remains that i can, right now, non-theoretically, on a well supported common unixlike os, and entirely unrelated to whatever weird crusade you seem to have invented to stand in for my side of this discussion, link a pile of assembly with -static -nolibc, fire up the repl, and mmap files into memory as i please with nary a bit of c on the userspace side.

as i originally said, i'm happy to consider linux a weird exception to the point you're making in a wider context since this isn't something you can do portably, but there still are entirely useful things one can do today with mmap that involve zero userspace c code on a widely supported platform.

edit: lol forgot to even get to this part. i'm also somewhat curious what you mean with this bit: "you're running what amounts to an emulator." perhaps i'm not firing on all cylinders today but i fail to see how it's useful to characterize performing bare syscalls from assembly (or something more high-level built out of assembly legos) as an emulator in any way, but i'm open to having missed some interesting nuance there.

¹ unless you mean trivially (seeing as this is code i imagined and then proceeded to write) in which case i suppose i agree

Post reply on HN