Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does C have the best file API

maurycyz.com

Post reply on HN
Print view View original

71–80 of 169 posts

Re: Why does C have the best file API

#71

Post by zadikian »

dvt wrote: post #57
amluto wrote: post #55

I can’t entirely tell what the article’s point is. It seems to be trying to say that many languages can mmap bytes , but: > (as far as I'm aware) C is the only language that lets you specify a binary format and just use it. I assume they mean: struct foo { fields; }; foo *data = mmap(…); And yes, C is one of relatively few languages that let you do this without complaint, because it’s a terrible idea . And C doesn’t…

Had the same thought. Also confused at the backhanded compliment that pickle got: > Just look at Python's pickle: it's a completely insecure serialization format. Loading a file can cause code execution even if you just wanted some numbers... but still very widely used because it fits the mix-code-and-data model of python. Like, are they saying it's bad? Are they saying it's good? I don't even get it. While I was rea…

The article is saying it's good, or at least good enough. I don't necessarily agree with the rest of the article.

Re: Why does C have the best file API

#72

Post by wahern »

gopalv wrote: post #64
amluto wrote: post #55

I can’t entirely tell what the article’s point is. It seems to be trying to say that many languages can mmap bytes , but: > (as far as I'm aware) C is the only language that lets you specify a binary format and just use it. I assume they mean: struct foo { fields; }; foo *data = mmap(…); And yes, C is one of relatively few languages that let you do this without complaint, because it’s a terrible idea . And C doesn’t…

> correspond to a binary format in accordance with the C ABI on your particular system. We're so deep in this hole that people are fixing this on a CPU with silicon. The Graviton team made a little-endian version of ARM just to allow lazy code like this to migrate away from Intel chips without having to rewrite struct unpacking (& also IBM with the ppc64le). Early in my career, I spent a lot of my time reading Java b…

GCC supports specifying endianness of structs and unions: https://gcc.gnu.org/onlinedocs/gcc-15.2.0/gcc/Common-Type-At...

I'm not sure how useful it is, though it was only added 10 years ago with GCC 6.1 (recent'ish in the world of arcane features like this, and only just about now something you could reasonably rely upon existing in all enterprise environments), so it seems some people thought it would still be useful.

Re: Why does C have the best file API

#73

Post by pmontra »

I think that I open files in very few cases in my job. I read and write PDF, xlsx, csv, yaml and I write docx. Those have their own formats and we use them to communicate with other apps or with users. Everything else goes in a PostgreSQL database or in sqlite3 because of many reasons and among them because of interoperability with other apps and ease of human inspection. A custom file format could be OK for data that only that app must use or for performance reasons, if you know how to make data retrieval performant.

Re: Why does C have the best file API

#74

Post by raincole »

At first glance, it's a quite weird article. But at the bottom:

> This simply isn't true on memory constrained systems — and with 100 GB files — every system is memory constrained.

I suppose the author might have a point in the context of making apps that constantly need to process 100GB files? I personally never have to deal with 100GB files so I am no one to judge if the rest of the article makes sense.

Re: Why does C have the best file API

#75

Post by neonsunset »

okanat wrote: post #32

I guess the author didn't use that many other programming languages or OSes. You can do the same even in garbage collected languages like Java and C# and on Windows too. https://docs.oracle.com/javase/8/docs/api/java/nio/MappedByt... https://learn.microsoft.com/en-us/dotnet/api/system.io.memor... https://learn.microsoft.com/en-us/windows/win32/memory/creat... Memory mapping is very common.

[dead]

Re: Why does C have the best file API

#77

Post by jim33442 »

raincole wrote: post #74

At first glance, it's a quite weird article. But at the bottom: > This simply isn't true on memory constrained systems — and with 100 GB files — every system is memory constrained. I suppose the author might have a point in the context of making apps that constantly need to process 100GB files? I personally never have to deal with 100GB files so I am no one to judge if the rest of the article makes sense.

Yeah, the article makes more sense when you assume it's for that use case, but still not really sure when this case comes up. I've dealt with plenty of 100GB files, but they were coming from outside as CSVs or sqlite DBs or something. This one is, your C program is going to generate a 100GB file to use later, and you also don't need a DB.

Re: Why does C have the best file API

#78

Post by [deleted] »

nickelpro wrote: post #24
ajross wrote: post #20

Earlier quoted context omitted.

I think this is sort of missing the point, though. Yes, mmap() is in POSIX[1] in the sense of "where is it specified". But mmap() was implemented in C because C is the natural language for exposing Unix system calls and mmap() is a syscall provided by the OS. And this is true up and down the stack. Best language for integrating with low level kernel networking (sockopts, routing, etc...)? C. Best language for async I…

If we're going to be pedantic, mmap is a syscall. It happens that the C version is standardized by POSIX. The underlying syscall doesn't use the C ABI, you need to wrap it to use it from C in the same way you need to wrap it to use it from any language, which is exactly what glibc and friends do. Moral of the story is mmap belongs to the platform, not the language.

[deleted]

Re: Why does C have the best file API

#79

Post by ddtaylor »

Negitivefrags wrote: post #66
amluto wrote: post #61

Earlier quoted context omitted.

Lots of reasons: The code is not portable between architectures. You can’t actually define your data structure. You can pretend with your compiler’s version of “pack” with regrettable results. You probably have multiple kinds of undefined behavior. Dealing with compatibility between versions of your software is awkward at best. You might not even get amazing performance. mmap is not a panacea. Page faults and TLB flu…

I've written a lot of code using that method, and never had any portability issues. You use types with number of bits in them. Hell, I've slung C structs across the network between 3 CPU architectures. And I didn't even use htons! Maybe it's not portable to some ancient architecture, but none that I have experienced. If there is undefined behavior, it's certainly never been a problem either. And I've seen a lot of ta…

C allows most of this, whereas C++ doesn't allow pointer aliasing without a compiler flag, tricks and problems.

I agree you can certainly just use bytes of the correct sizes, but often to get the coverage you need for the data structure you end up writing some form of wrapper or fixup code, which is still easier and gives you the control versus most of the protobuf like stuff that introduces a lot of complexity and tons of code.

Re: Why does C have the best file API

#80

Post by fulafel »

tremon wrote: post #46
Dwedit wrote: post #12

Using mmap means that you need to be able to handle memory access exceptions when a disk read or write fails. Examples of disk access that fails includes reading from a file on a Wifi network drive, a USB device with a cable that suddenly loses its connection when the cable is jiggled, or even a removable USB drive where all disk reads fail after it sees one bad sector. If you're not prepared to handle a memory acces…

C doesn't have exceptions, do you mean signals? If not, I don't see how that is that any different from having to handle I/O errors from write() and/or open() calls.

It's very different since at random points of your program your signal handler is caleld asynchronously, and you can only do a very limited signal-safe things there, and the flow of control in your i/o, logic etc code has no idea it's happening.

tldr; it's very different.

Post reply on HN