Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does C have the best file API

maurycyz.com

Post reply on HN
Print view View original

141–150 of 169 posts

Re: Why does C have the best file API

#141

Post by groundzeros2015 »

zadikian wrote: post #134
groundzeros2015 wrote: post #125

nobody mentioning the "file system as a noSQL database comment". I found most of the friction when using bash/unix style tools when I tried to put everything in structured files that needed parsing. Once you see folder/files as the structure these tools work great. It's also interesting to me that many noSQL starts from the assumption that relations are too complex, and that trees are preferred.

NoSQL was more of a convenience thing during that trend. You might have some bag of nested attributes that maps perfectly fine onto relations, but it's cumbersome for someone who just wants to load it all into an object, edit, then resave. People used to use ORMs to get around that, then NoSQL became popular, and now you can just use jsonb in SQL while still maintaining relations for other things. This doesn't say mu…

Yes, and I’m suggesting to try the file system if you think a tree is convenient.

Re: Why does C have the best file API

#142

Post by fulafel »

xmcqdpt2 wrote: post #114
fc417fc802 wrote: post #97

Earlier quoted context omitted.

Well at least in this case the timing won't be arbitrary. Execution will have blocked waiting on the read and you will (AFAIK) receive the signal promptly in this case. Since the code in question was doing IO that you knew could fail handling the situation can be as simple as setting a flag from within the signal handler. I'm unclear what would happen in the event you had configured the mask to force SIGBUS to a diff…

> Since the code in question was doing IO that you knew could fail handling the situation can be as simple as setting a flag from within the signal handler. If you are using mmap like malloc (as the article does) you don't necessarily know that you are "reading" from disk. You may have passed the disk-backed pointers to other code. The fact that malloc and mmap return the same type of values is what makes mmap in C s…

Yes, and for writing (the example is read-write) it's of course yet another kettle of fish. The error might never get reported at all. Or you might get a SIGBUS (at least with sparse files).

Re: Why does C have the best file API

#143

Post by nitwit005 »

okanat wrote: post #32

I guess the author didn't use that many other programming languages or OSes. You can do the same even in garbage collected languages like Java and C# and on Windows too. https://docs.oracle.com/javase/8/docs/api/java/nio/MappedByt... https://learn.microsoft.com/en-us/dotnet/api/system.io.memor... https://learn.microsoft.com/en-us/windows/win32/memory/creat... Memory mapping is very common.

I'd be careful though, as they all have quirks due to how tricky it is handling mmap faults. The Java API mentions both unique garbage collection behavior and throwing unspecified exceptions at unspecified times.

Re: Why does C have the best file API

#144

Post by hyc_symas »

api wrote: post #113
gopalv wrote: post #64

Earlier quoted context omitted.

> correspond to a binary format in accordance with the C ABI on your particular system. We're so deep in this hole that people are fixing this on a CPU with silicon. The Graviton team made a little-endian version of ARM just to allow lazy code like this to migrate away from Intel chips without having to rewrite struct unpacking (& also IBM with the ppc64le). Early in my career, I spent a lot of my time reading Java b…

ARM is usually bi-endian, and almost always run in little endian mode. All Apple ARM is LE. Not sure about Android but I’d guess it’s the same. I don’t think I’ve ever seen BE ARM in the wild. Big endian is as far as I know extinct for larger mainstream CPUs. Power still exists but is on life support. MIPS and Sparc are dead. M68k is dead. X86 has always been LE. RISC-V is LE. It’s not an arbitrary choice. Little end…

Network data and most serialization formats are big endian because it's easiest to shift bits in and out of a shift register onto a serial comm channel in that order. If you used little endian, the shifter on output would have to operate in reverse direction relative to the shifter on input, which just causes stupid inconsistency headaches.

Re: Why does C have the best file API

#145

Post by hyc_symas »

nickelpro wrote: post #19
zahlman wrote: post #14

Aside from what https://news.ycombinator.com/item?id=47210893 said, mmap() is a low-level design that makes it easier to work with files that don't fit in memory and fundamentally represent a single homogeneous array of some structure. But it turns out that files commonly do fit in memory (nowadays you commonly have on the order of ~100x as much disk as memory, but millions of files); and you very often want to read…

mmap is also relatively slow (compared to modern solutions, io_uring and friends), and immensely painful for error handling. It's simple, I'll give it that.

On BSD, read() was already implemented in the kernel by page-faulting in the desired pages of the file, to then be copied into the user-supplied buffer. So from the first time mmap was ever implemented, it was always the fastest input mechanism. (First deployed implementation was in SunOS btw, 4.2BSD specified and documented it but didn't implement it.) Anyway there's no magic to get data off a device into memory faster, io_uring just lets you hide the delay in some other thread's time.

Re: Why does C have the best file API

#146

Post by zadikian »

lionkor wrote: post #107
Negitivefrags wrote: post #66

Earlier quoted context omitted.

I've written a lot of code using that method, and never had any portability issues. You use types with number of bits in them. Hell, I've slung C structs across the network between 3 CPU architectures. And I didn't even use htons! Maybe it's not portable to some ancient architecture, but none that I have experienced. If there is undefined behavior, it's certainly never been a problem either. And I've seen a lot of ta…

That seems highly unlikely. Let's assume that all compilers use the exact same padding in C structs, that all architectures use the same alignment, and that endianness is made up, that types are the same size across 64 and 32 bit platforms, and also pretend that pointers inside a struct will work fine when sent across the network; the question remains still: Why? Is THIS your bottleneck? Will a couple memcpy() operat…

I guess to not have to set up protobuf or asn1. Those preconditions of both platforms using the same padding and endianness aren't that hard to satisfy if you own it all.

But do you really have such a complex struct where everything inside is fixed-size? I wouldn't be surprised if it happens, but this isn't so general-purpose like the article suggests.

Re: Why does C have the best file API

#147

Post by pjmlp »

benj111 wrote: post #140
pjmlp wrote: post #118

Earlier quoted context omitted.

As proven by many languages without native support for plain old goto, it isn't really required when proper structured programming constructs are available, even if it happens to be a goto under the hood, managed by the compiler.

My point is it's bad debating style. 'Everyone knows C is bad for all kinds of reasons ergo, even when someone presents their own actual experience, I can respond with a refrain that sounds good' Not using goto because you've heard it's always bad is the same kind of thing. Yes it has issues, but that isn't a reason to brush anyone off that have actual valid uses for it.

Since I am coding since 1986, lets say I have plenty of experience with goto in various places myself.

Re: Why does C have the best file API

#148

Post by jim33442 »

bregma wrote: post #108
jim33442 wrote: post #76

You can't do dynamic memory with that, right? Not without a custom malloc implementation. So it's not all that comparable to pickle.

Most modern C runtime libraries on POSIX OSes implement malloc() and friends using the mmap() system call.

I mean in your C code that's using mmap, I know you can do this with a custom malloc impl, but idk if there's a built-in way:

     int len = 1000;
     int file = open("numbers.void", O_RDWR | O_CREAT, 0600);
     ftruncate(file, len);
     void\* buf = mmap(NULL, len, 
  PROT_READ | PROT_WRITE, MAP_SHARED,
  file, 0);

     // Treat mmapped buffer as a heap
     initialize_heap(buf);
     // Manage dynamically-sized array on disk
     int* my_array = malloc(sizeof(int) * 8, buf);
     my_array = realloc(my_array, sizeof(int) * 16, buf);
     free(my_array, buf);
Protobuf, Python pickle, etc can all handle dynamic memory that gets flattened when you want to serialize.

Re: Why does C have the best file API

#149

Post by jasomill »

lxgr wrote: post #136
Dwedit wrote: post #12

Using mmap means that you need to be able to handle memory access exceptions when a disk read or write fails. Examples of disk access that fails includes reading from a file on a Wifi network drive, a USB device with a cable that suddenly loses its connection when the cable is jiggled, or even a removable USB drive where all disk reads fail after it sees one bad sector. If you're not prepared to handle a memory acces…

Do these really ever result in access failures instead of just hangs? How are they surfaced to processes? In my experience, all these things just cause whatever process is memory mapping to freeze up horribly and make me regret ever using a network file system or external hard drive.

Depends on the implementation.

Most I/O calls return errors when reads or writes fail, but NFS, for example, would traditionally block on network errors by default — you probably don't want your entire lab full of diskless workstations to kernel panic every time there's a transient network glitch.

You also have the issue of multiple levels of caching and when and how to report delayed errors to programs that don't explicitly use mechanisms like fsync.

Re: Why does C have the best file API

#150

Post by yjftsjthsd-h »

gopalv wrote: post #64
amluto wrote: post #55

I can’t entirely tell what the article’s point is. It seems to be trying to say that many languages can mmap bytes , but: > (as far as I'm aware) C is the only language that lets you specify a binary format and just use it. I assume they mean: struct foo { fields; }; foo *data = mmap(…); And yes, C is one of relatively few languages that let you do this without complaint, because it’s a terrible idea . And C doesn’t…

> correspond to a binary format in accordance with the C ABI on your particular system. We're so deep in this hole that people are fixing this on a CPU with silicon. The Graviton team made a little-endian version of ARM just to allow lazy code like this to migrate away from Intel chips without having to rewrite struct unpacking (& also IBM with the ppc64le). Early in my career, I spent a lot of my time reading Java b…

> The Graviton team made a little-endian version of ARM just to allow lazy code like this to migrate away from Intel chips without having to rewrite struct unpacking

No? Most ARM is little endian.

Post reply on HN