Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does C have the best file API

maurycyz.com

Post reply on HN
Print view View original

21–30 of 169 posts

Re: Why does C have the best file API

#21

Post by alanfranz »

Well...

I'm not sure what the author really wants to say. mmap is available in many languages (e.g. Python) on Linux (and many other *nix I suppose). C provides you with raw memory access, so using mmap is sort-of-convenient for this use case.

But if you use Python then, yes, you'll need a bytearray, because Python doesn't give you raw access to such memory - and I'm not sure you'd want to mmap a PyObject anyway?

Then, writing and reading this kind of raw memory can be kind of dangerous and non-portable - I'm not really sure that the pickle analogy even makes sense. I very much suppose (I've never tried) that if you mmap-read malicious data in C, a vulnerability would be _quite_ easy to exploit.

Re: Why does C have the best file API

#22

Post by justmedep »

Dwedit wrote: post #12

Using mmap means that you need to be able to handle memory access exceptions when a disk read or write fails. Examples of disk access that fails includes reading from a file on a Wifi network drive, a USB device with a cable that suddenly loses its connection when the cable is jiggled, or even a removable USB drive where all disk reads fail after it sees one bad sector. If you're not prepared to handle a memory acces…

You can even mmap a socket on some systems (iOS and macOS via GCD). But doing that is super fragile. Socket errors are swallowed.

My interpretation always was the mmap should only be used for immutable and local files. You may still run into issues with those type of files but it’s very unlikely.

Re: Why does C have the best file API

#23

Post by actionfromafar »

userbinator wrote: post #18

It still works if the file doesn't fit in RAM No it doesn't. If you have a file that's 2^36 bytes and your address space is only 2^32, it won't work. On a related digression, I've seen so many cases of programs that could've handled infinitely long input in constant space instead implemented as some form of "read the whole input into memory", which unnecessarily puts a limit on the input length.

You can mmap with offset, for that case. Just FYI in anyone thought it was a hard limit.

Re: Why does C have the best file API

#24

Post by nickelpro »

ajross wrote: post #20
seba_dos1 wrote: post #3

mmap is not a C feature, but POSIX. There are C platforms that don't provide mmap, and on those that do you can use mmap from other languages (there's mmap module in the Python's standard library, for example).

I think this is sort of missing the point, though. Yes, mmap() is in POSIX[1] in the sense of "where is it specified". But mmap() was implemented in C because C is the natural language for exposing Unix system calls and mmap() is a syscall provided by the OS. And this is true up and down the stack. Best language for integrating with low level kernel networking (sockopts, routing, etc...)? C. Best language for async I…

If we're going to be pedantic, mmap is a syscall. It happens that the C version is standardized by POSIX.

The underlying syscall doesn't use the C ABI, you need to wrap it to use it from C in the same way you need to wrap it to use it from any language, which is exactly what glibc and friends do.

Moral of the story is mmap belongs to the platform, not the language.

Re: Why does C have the best file API

#25

Post by a-dub »

nickelpro wrote: post #24
ajross wrote: post #20

Earlier quoted context omitted.

I think this is sort of missing the point, though. Yes, mmap() is in POSIX[1] in the sense of "where is it specified". But mmap() was implemented in C because C is the natural language for exposing Unix system calls and mmap() is a syscall provided by the OS. And this is true up and down the stack. Best language for integrating with low level kernel networking (sockopts, routing, etc...)? C. Best language for async I…

If we're going to be pedantic, mmap is a syscall. It happens that the C version is standardized by POSIX. The underlying syscall doesn't use the C ABI, you need to wrap it to use it from C in the same way you need to wrap it to use it from any language, which is exactly what glibc and friends do. Moral of the story is mmap belongs to the platform, not the language.

it also appears in operating systems that aren't written in c. i see it as an operating system feature, categorically.

Re: Why does C have the best file API

#27

Post by bvrmn »

alanfranz wrote: post #21

Well... I'm not sure what the author really wants to say. mmap is available in many languages (e.g. Python) on Linux (and many other *nix I suppose). C provides you with raw memory access, so using mmap is sort-of-convenient for this use case. But if you use Python then, yes, you'll need a bytearray, because Python doesn't give you raw access to such memory - and I'm not sure you'd want to mmap a PyObject anyway? The…

Actually in Python you could recast (zerocopy) bytearray as other primitive C type or even any other structure using ctypes module.

Re: Why does C have the best file API

#29

Post by projektfu »

ajross wrote: post #20
seba_dos1 wrote: post #3

mmap is not a C feature, but POSIX. There are C platforms that don't provide mmap, and on those that do you can use mmap from other languages (there's mmap module in the Python's standard library, for example).

I think this is sort of missing the point, though. Yes, mmap() is in POSIX[1] in the sense of "where is it specified". But mmap() was implemented in C because C is the natural language for exposing Unix system calls and mmap() is a syscall provided by the OS. And this is true up and down the stack. Best language for integrating with low level kernel networking (sockopts, routing, etc...)? C. Best language for async I…

Why does Ada have the best file API?

https://github.com/AdaCore/florist/blob/master/libsrc/posix-...

Post reply on HN