Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does C have the best file API

maurycyz.com

Post reply on HN
Print view View original

61–70 of 169 posts

Re: Why does C have the best file API

#61

Post by amluto »

Negitivefrags wrote: post #58
amluto wrote: post #55

I can’t entirely tell what the article’s point is. It seems to be trying to say that many languages can mmap bytes , but: > (as far as I'm aware) C is the only language that lets you specify a binary format and just use it. I assume they mean: struct foo { fields; }; foo *data = mmap(…); And yes, C is one of relatively few languages that let you do this without complaint, because it’s a terrible idea . And C doesn’t…

Why is it such a terrible idea? No need to add complexity, dependancies and reduced performance by using these libraries.

Lots of reasons:

The code is not portable between architectures.

You can’t actually define your data structure. You can pretend with your compiler’s version of “pack” with regrettable results.

You probably have multiple kinds of undefined behavior.

Dealing with compatibility between versions of your software is awkward at best.

You might not even get amazing performance. mmap is not a panacea. Page faults and TLB flushing are not free.

You can’t use any sort of advanced data types — you get exactly what C gives you.

Forget about enforcing any sort of invariant at the language level.

Re: Why does C have the best file API

#62

Post by jraph »

Negitivefrags wrote: post #58
amluto wrote: post #55

I can’t entirely tell what the article’s point is. It seems to be trying to say that many languages can mmap bytes , but: > (as far as I'm aware) C is the only language that lets you specify a binary format and just use it. I assume they mean: struct foo { fields; }; foo *data = mmap(…); And yes, C is one of relatively few languages that let you do this without complaint, because it’s a terrible idea . And C doesn’t…

Why is it such a terrible idea? No need to add complexity, dependancies and reduced performance by using these libraries.

Because a struct might not serialize the same way from a CPU architecture to another.

The sizes of ints, the byte order and the padding can be different for instance.

Re: Why does C have the best file API

#63

Post by usefulcat »

kelnos wrote: post #49
justmedep wrote: post #22

Earlier quoted context omitted.

You can even mmap a socket on some systems (iOS and macOS via GCD). But doing that is super fragile. Socket errors are swallowed. My interpretation always was the mmap should only be used for immutable and local files. You may still run into issues with those type of files but it’s very unlikely.

mmap is also good for passing shared memory around. (You still need to be careful, of course.)

It’s also great for when you have a lot of data on local storage, and a lot of different processes that need to access the same subset of that data concurrently.

Without mmap, every process ends up caching its own private copy of that data in memory (think fopen, fread, etc). With mmap, every process accesses the same cached copy of that data directly from the FS cache.

Granted this is a rather specific use case, but for this case it makes a huge difference.

Re: Why does C have the best file API

#64

Post by gopalv »

amluto wrote: post #55

I can’t entirely tell what the article’s point is. It seems to be trying to say that many languages can mmap bytes , but: > (as far as I'm aware) C is the only language that lets you specify a binary format and just use it. I assume they mean: struct foo { fields; }; foo *data = mmap(…); And yes, C is one of relatively few languages that let you do this without complaint, because it’s a terrible idea . And C doesn’t…

> correspond to a binary format in accordance with the C ABI on your particular system.

We're so deep in this hole that people are fixing this on a CPU with silicon.

The Graviton team made a little-endian version of ARM just to allow lazy code like this to migrate away from Intel chips without having to rewrite struct unpacking (& also IBM with the ppc64le).

Early in my career, I spent a lot of my time reading Java bytecode into little endian to match all the bytecode interpreter enums I had & completely hating how 0xCAFEBABE would literally say BE BA FE CA (jokingly referred as "be bull shit") in a (gdb) x views.

Re: Why does C have the best file API

#66

Post by Negitivefrags »

amluto wrote: post #61
Negitivefrags wrote: post #58

Earlier quoted context omitted.

Why is it such a terrible idea? No need to add complexity, dependancies and reduced performance by using these libraries.

Lots of reasons: The code is not portable between architectures. You can’t actually define your data structure. You can pretend with your compiler’s version of “pack” with regrettable results. You probably have multiple kinds of undefined behavior. Dealing with compatibility between versions of your software is awkward at best. You might not even get amazing performance. mmap is not a panacea. Page faults and TLB flu…

I've written a lot of code using that method, and never had any portability issues. You use types with number of bits in them.

Hell, I've slung C structs across the network between 3 CPU architectures. And I didn't even use htons!

Maybe it's not portable to some ancient architecture, but none that I have experienced.

If there is undefined behavior, it's certainly never been a problem either.

And I've seen a lot of talk about TLB shootdown, so I tried to reproduce those problems but even with over 32 threads, mmap was still faster than fread into memory in the tests I ran.

Look, obviously there are use cases for libraries like that, but a lot of the time you just need something simple, and writing some structs to disk can go a long way.

Re: Why does C have the best file API

#67

Post by teo_zero »

What a bizarre conclusion to draw! It's like saying that cars are the best means of transportation because you can travel to the Grand Canyon in them and the Grand Canyon is the best landscape in the world, and yes you could use other means to get there, but cars are what everybody's using.

If the real goal of TFA was to praise C's ability to reinterpret a chunk of memory (possibly mapped to a file) as another datatype, it would have been more effective to do so using C functions and not OS-specific system calls. For example:

  FILE *f = fopen(...);
  uint32_t *numbers;
  fread(numbers, ..., f);

  access numbers[...]

  frwite(numbers, ..., f);
  fclose(f);

Re: Why does C have the best file API

#68

Post by wahern »

zahlman wrote: post #14

Aside from what https://news.ycombinator.com/item?id=47210893 said, mmap() is a low-level design that makes it easier to work with files that don't fit in memory and fundamentally represent a single homogeneous array of some structure. But it turns out that files commonly do fit in memory (nowadays you commonly have on the order of ~100x as much disk as memory, but millions of files); and you very often want to read…

> But it turns out that files commonly do fit in memory

The difference between slurping a file into malloc'd memory and just mmap'ing it is that the latter doesn't use up anonymous memory. Under memory pressure, the mmap'd file can just be evicted and transparently reloaded later, whereas if it was copied into anonymous memory it either needs to be copied out to swap or, if there's not enough swap (e.g. if swap is disabled), the OOM killer will be invoked to shoot down some (often innocent) process.

If you need an entire file loaded into your address space, and you don't have to worry about the file being modified (e.g. have to deal with SIGBUS if the file is truncated), then mmap'ing the file is being a good citizen in terms of wisely using system resources. On a system like Linux that aggressively buffers file data, there likely won't be a performance difference if your system memory usage assumptions are correct, though you can use madvise & friends to hint to the kernel. If your assumptions are wrong, then you get graceful performance degradation (back pressure, effectively) rather than breaking things.

Are you tired of bloated software slowing your systems to a crawl because most developers and application processes think they're special snowflakes that will have a machine all to themselves? Be part of the solution, not part of the problem.

Re: Why does C have the best file API

#69

Post by m-schuetz »

teo_zero wrote: post #67

What a bizarre conclusion to draw! It's like saying that cars are the best means of transportation because you can travel to the Grand Canyon in them and the Grand Canyon is the best landscape in the world, and yes you could use other means to get there, but cars are what everybody's using. If the real goal of TFA was to praise C's ability to reinterpret a chunk of memory (possibly mapped to a file) as another dataty…

This is way more cumbersome than mmap if you need to out-of-core process the file in non-sequential patterns. Way way more cumbersome, since you need to deal with intermediate staging buffers, and reuse them if you actually want to be fast. mmap, on the other hand, is absolutely trivial to use, like any regular buffer pointer. And at least on windows, the mmap counterpart can be faster when processing the file with multiple threads, compared to fread.

But I agree that it's a bizarre article since mmap is not a C standard, and relies on platform-dependend Operating System APIs.

Re: Why does C have the best file API

#70

Post by namibj »

nixon_why69 wrote: post #52
nickelpro wrote: post #19

Earlier quoted context omitted.

mmap is also relatively slow (compared to modern solutions, io_uring and friends), and immensely painful for error handling. It's simple, I'll give it that.

Page faults are slower than being deliberate about your I/O but mapped memory is no faster or slower than "normal" memory, its the same mechanism.

Nah, usually can't have huge pages. Almost certainly can't have giant pages. Can't even fit all L3$ capacity into the L2 TLB if done via 4k pages...
Post reply on HN