Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does C have the best file API

maurycyz.com

Post reply on HN
Print view View original

121–130 of 169 posts

Re: Why does C have the best file API

#121

Post by groundzeros2015 »

Dwedit wrote: post #12

Using mmap means that you need to be able to handle memory access exceptions when a disk read or write fails. Examples of disk access that fails includes reading from a file on a Wifi network drive, a USB device with a cable that suddenly loses its connection when the cable is jiggled, or even a removable USB drive where all disk reads fail after it sees one bad sector. If you're not prepared to handle a memory acces…

> file on a Wifi network drive,

I would simply not mmap this.

> If you're not prepared to handle a memory access exception when you access the mapped file, don't use mmap.

fread can fail too. I don't know why you would be prepared for one and not the other.

Re: Why does C have the best file API

#122

Post by amluto »

api wrote: post #113
gopalv wrote: post #64

Earlier quoted context omitted.

> correspond to a binary format in accordance with the C ABI on your particular system. We're so deep in this hole that people are fixing this on a CPU with silicon. The Graviton team made a little-endian version of ARM just to allow lazy code like this to migrate away from Intel chips without having to rewrite struct unpacking (& also IBM with the ppc64le). Early in my career, I spent a lot of my time reading Java b…

ARM is usually bi-endian, and almost always run in little endian mode. All Apple ARM is LE. Not sure about Android but I’d guess it’s the same. I don’t think I’ve ever seen BE ARM in the wild. Big endian is as far as I know extinct for larger mainstream CPUs. Power still exists but is on life support. MIPS and Sparc are dead. M68k is dead. X86 has always been LE. RISC-V is LE. It’s not an arbitrary choice. Little end…

> Little endian is superior because you can cast between integer types without pointer arithmetic

I’ve heard this one several times and it never really made sense. Is the argument that y you can do:

    short s;
    long *p = (long*)&s;
Or vice versa and it kind of works under some circumstances?

Re: Why does C have the best file API

#123

Post by Dwedit »

groundzeros2015 wrote: post #121
Dwedit wrote: post #12

Using mmap means that you need to be able to handle memory access exceptions when a disk read or write fails. Examples of disk access that fails includes reading from a file on a Wifi network drive, a USB device with a cable that suddenly loses its connection when the cable is jiggled, or even a removable USB drive where all disk reads fail after it sees one bad sector. If you're not prepared to handle a memory acces…

> file on a Wifi network drive, I would simply not mmap this. > If you're not prepared to handle a memory access exception when you access the mapped file, don't use mmap. fread can fail too. I don't know why you would be prepared for one and not the other.

Because you're way deep down the call stack in some function that happened to take in a pointer, far far away from the code that opened the file.

Re: Why does C have the best file API

#124

Post by amluto »

jklowden wrote: post #120
amluto wrote: post #55

I can’t entirely tell what the article’s point is. It seems to be trying to say that many languages can mmap bytes , but: > (as far as I'm aware) C is the only language that lets you specify a binary format and just use it. I assume they mean: struct foo { fields; }; foo *data = mmap(…); And yes, C is one of relatively few languages that let you do this without complaint, because it’s a terrible idea . And C doesn’t…

It’s a terribly useful idea. FTFY. The program you used to leave your comment, and the libraries it used, were loaded into memory via mmap(2) prior to execution. To use protobuf or whatever, you use mmap. The only reason mmap isn’t more generally useful is the dearth of general-use binary on-disk formats such as ELF. We could build more memory-mapped applications if we had better library support for them. But we don’…

Entire libraries are a weird sort of exception. They fundamentally target a specific architecture, and all the nonportable or version dependent data structures are self describing in the sense that the code that accesses them are shipped along with the data.

And if you load library A that references library B’s data and you change B’s data format but forget to update A, you crash horribly. Similarly, if you modify a shared library while it’s in use (your OS and/or your linker may try to avoid this), you can easily crash any process that has it mapped.

Re: Why does C have the best file API

#125

Post by groundzeros2015 »

nobody mentioning the "file system as a noSQL database comment". I found most of the friction when using bash/unix style tools when I tried to put everything in structured files that needed parsing. Once you see folder/files as the structure these tools work great.

It's also interesting to me that many noSQL starts from the assumption that relations are too complex, and that trees are preferred.

Re: Why does C have the best file API

#126

Post by elehack »

amluto wrote: post #122
api wrote: post #113

Earlier quoted context omitted.

ARM is usually bi-endian, and almost always run in little endian mode. All Apple ARM is LE. Not sure about Android but I’d guess it’s the same. I don’t think I’ve ever seen BE ARM in the wild. Big endian is as far as I know extinct for larger mainstream CPUs. Power still exists but is on life support. MIPS and Sparc are dead. M68k is dead. X86 has always been LE. RISC-V is LE. It’s not an arbitrary choice. Little end…

> Little endian is superior because you can cast between integer types without pointer arithmetic I’ve heard this one several times and it never really made sense. Is the argument that y you can do: short s; long *p = (long*)&s; Or vice versa and it kind of works under some circumstances?

Yes. In little-endian, the difference between short and long at a specific address is how many bytes you read from that address. In big-endian, to cast a long to a short, you have to jump forward 6 bytes to get to the 2 least-significant bytes.

Re: Why does C have the best file API

#127

Post by amluto »

nly wrote: post #92
ddtaylor wrote: post #79

Earlier quoted context omitted.

C allows most of this, whereas C++ doesn't allow pointer aliasing without a compiler flag, tricks and problems. I agree you can certainly just use bytes of the correct sizes, but often to get the coverage you need for the data structure you end up writing some form of wrapper or fixup code, which is still easier and gives you the control versus most of the protobuf like stuff that introduces a lot of complexity and t…

__attribute__((may_alias, packed)) right on the struct.

Check your generated code. Most compilers assume that packed also means unaligned and will generate unaligned load and store sequences, which are large, slow, and may lose whatever atomicity properties they might have had.

Re: Why does C have the best file API

#128

Post by qalmakka »

benj111 wrote: post #119
qalmakka wrote: post #88

Earlier quoted context omitted.

> C is the natural language for exposing Unix system calls No, C is the language _designed_ to write UNIX. Unix is older than C, C was designed to write it and that's why all UNIX APIs follow C conventions. It's obvious that when you design something for a system it will have its best APIs in the language the system is written in. C has also multiple weird and quirky APIs that suck, especially in the ISO C libc.

If I write an OS in Basic, surely the 'natural' language for exposing the system calls is Basic? Yes Unix predates C. But at this point in time 50+ years down the road, where the majority on nix users don't use anything that ever contained that code, and the minority use a nix that has been thoroughly ship of Theseused, Unix is to all intents and purposes a C operating system.

> If I write an OS in Basic, surely the 'natural' language for exposing the system calls is Basic?

For that specific OS, that would probably be the case? I think every API is bound to reflect the specific constraints of the language it has been written in. What I was trying to clarify was that UNIX and C are intertwined in an especially deep way, more than basically other OS that doesn't have a UNIX API, because both were born and written alongside each other, so some Unix APIs rely on C-specific behaviour and quirks and some C features were born and designed around the same historical context UNIX was born

Re: Why does C have the best file API

#129

Post by groundzeros2015 »

Dwedit wrote: post #123
groundzeros2015 wrote: post #121

Earlier quoted context omitted.

> file on a Wifi network drive, I would simply not mmap this. > If you're not prepared to handle a memory access exception when you access the mapped file, don't use mmap. fread can fail too. I don't know why you would be prepared for one and not the other.

Because you're way deep down the call stack in some function that happened to take in a pointer, far far away from the code that opened the file.

If that's your program design then fread is not a substitute. Because you would need to pass in the FILE* pointer to all those calls.

And what are you hoping to do in those call stacks when you find an error? Can any of that logic hope to do anything useful if it can't access this data? Let the OS handle this. crash your program and restart.

Re: Why does C have the best file API

#130

Post by ajross »

qalmakka wrote: post #88
ajross wrote: post #20

Earlier quoted context omitted.

I think this is sort of missing the point, though. Yes, mmap() is in POSIX[1] in the sense of "where is it specified". But mmap() was implemented in C because C is the natural language for exposing Unix system calls and mmap() is a syscall provided by the OS. And this is true up and down the stack. Best language for integrating with low level kernel networking (sockopts, routing, etc...)? C. Best language for async I…

> C is the natural language for exposing Unix system calls No, C is the language _designed_ to write UNIX. Unix is older than C, C was designed to write it and that's why all UNIX APIs follow C conventions. It's obvious that when you design something for a system it will have its best APIs in the language the system is written in. C has also multiple weird and quirky APIs that suck, especially in the ISO C libc.

>> C is the natural language for exposing Unix system calls

> No, C is the language _designed_ to write UNIX. [...]

This is one of those hilarious situations where internet discussion goes off the rails. Everything you wrote, to the last word, would carry the same meaning and the same benefit to the discussion had you written "Yes" instead of "No" as the first word.

Literally you're agreeing with me, but phrasing it as a disagreement only because you feel I left something out.

Post reply on HN