Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does C have the best file API

maurycyz.com

Post reply on HN
Print view View original

41–50 of 169 posts

Re: Why does C have the best file API

#41

Post by jiggawatts »

userbinator wrote: post #18

It still works if the file doesn't fit in RAM No it doesn't. If you have a file that's 2^36 bytes and your address space is only 2^32, it won't work. On a related digression, I've seen so many cases of programs that could've handled infinitely long input in constant space instead implemented as some form of "read the whole input into memory", which unnecessarily puts a limit on the input length.

All memory map APIs support moveable “windows” or views into files that are much larger than either physical memory or the virtual address space.

I’ve seen otherwise competent developers use compile time flags to bypass memmap on 32-bit systems even though this always worked! I dealt with database engines in the 1990s that used memmap for files tens of gigabytes in size.

Re: Why does C have the best file API

#42

Post by [deleted] »

ajross wrote: post #20
seba_dos1 wrote: post #3

mmap is not a C feature, but POSIX. There are C platforms that don't provide mmap, and on those that do you can use mmap from other languages (there's mmap module in the Python's standard library, for example).

I think this is sort of missing the point, though. Yes, mmap() is in POSIX[1] in the sense of "where is it specified". But mmap() was implemented in C because C is the natural language for exposing Unix system calls and mmap() is a syscall provided by the OS. And this is true up and down the stack. Best language for integrating with low level kernel networking (sockopts, routing, etc...)? C. Best language for async I…

[deleted]

Re: Why does C have the best file API

#43

Post by TZubiri »

Here's a negative signal I'm seeing often:

When a developer that usually consumes the language starts critiquing the language.

I could go on as to why it's a bad signal, psychologically, but let's just say that empirically it usually doesn't come from a good place, it's more like a developer raising the stakes of their application failing and blaming the language.

Sure one out of a thousand devs might be the next Linus Torvalds and develop the next Rust. But the odds aren't great.

Re: Why does C have the best file API

#44

Post by Const-me »

SvenL wrote: post #31
Const-me wrote: post #8

I think C# standard library is better. You can do same unsafe code as in C, SafeBuffer.AcquirePointer method then directly access the memory. Or you can do safer and slightly slower by calling Read or Write methods of MemoryMappedViewAccessor. All these methods are in the standard library, i.e. they work on all platforms. The C code is specific to POSIX; Windows supports memory mapped files too but the APIs are quite…

I think you don’t need to be unsafe, they have normal API for it. https://learn.microsoft.com/en-us/dotnet/standard/io/memory-...

Indeed, but these normal APIs have runtime costs for bounds checking. For some use cases, unsafe can be better. For instance, last time I used a memory-mapped file was for a large immutable Bloom filter. I knew the file should be exactly 4GB, validated that in the constructor, then when testing 12 bits from random location of the mapped file on each query, I opted for unsafe codes.

Re: Why does C have the best file API

#45

Post by racingmars »

okanat wrote: post #34
alanfranz wrote: post #21

Well... I'm not sure what the author really wants to say. mmap is available in many languages (e.g. Python) on Linux (and many other *nix I suppose). C provides you with raw memory access, so using mmap is sort-of-convenient for this use case. But if you use Python then, yes, you'll need a bytearray, because Python doesn't give you raw access to such memory - and I'm not sure you'd want to mmap a PyObject anyway? The…

Creating memory mapped files is a very common OS feature since 90s. Many high level languages have it as OS agnostic POSIX or not.

> very common OS feature since 90s

And if you want to go farther back, even if it wasn't called "mmap" or a specific function you had to invoke -- there were operating systems that used a "single-level store" (notably MULTICS and IBM's AS/400..err OS/400... err i5 OS... err today IBM i [seriously, IBM, pick a name and stick with it]) where the interface to disk storage on the platform is that the entire disk storage/filesystem is always mapped into the same address space as the rest of your process's memory. Memory-mapped files were basically the only interface there was, and the operating system "magically" persisted certain areas of your memory to permanent storage.

Re: Why does C have the best file API

#46

Post by tremon »

Dwedit wrote: post #12

Using mmap means that you need to be able to handle memory access exceptions when a disk read or write fails. Examples of disk access that fails includes reading from a file on a Wifi network drive, a USB device with a cable that suddenly loses its connection when the cable is jiggled, or even a removable USB drive where all disk reads fail after it sees one bad sector. If you're not prepared to handle a memory acces…

C doesn't have exceptions, do you mean signals? If not, I don't see how that is that any different from having to handle I/O errors from write() and/or open() calls.

Re: Why does C have the best file API

#47

Post by lights0123 »

tremon wrote: post #46
Dwedit wrote: post #12

Using mmap means that you need to be able to handle memory access exceptions when a disk read or write fails. Examples of disk access that fails includes reading from a file on a Wifi network drive, a USB device with a cable that suddenly loses its connection when the cable is jiggled, or even a removable USB drive where all disk reads fail after it sees one bad sector. If you're not prepared to handle a memory acces…

C doesn't have exceptions, do you mean signals? If not, I don't see how that is that any different from having to handle I/O errors from write() and/or open() calls.

Yes, it’s the SIGBUS signal.

Re: Why does C have the best file API

#48

Post by dataflow »

dekhn wrote: post #33
userbinator wrote: post #18

It still works if the file doesn't fit in RAM No it doesn't. If you have a file that's 2^36 bytes and your address space is only 2^32, it won't work. On a related digression, I've seen so many cases of programs that could've handled infinitely long input in constant space instead implemented as some form of "read the whole input into memory", which unnecessarily puts a limit on the input length.

Address space size and RAM are two different things.

What they said is correct regardless of that though?

Re: Why does C have the best file API

#49

Post by kelnos »

justmedep wrote: post #22
Dwedit wrote: post #12

Using mmap means that you need to be able to handle memory access exceptions when a disk read or write fails. Examples of disk access that fails includes reading from a file on a Wifi network drive, a USB device with a cable that suddenly loses its connection when the cable is jiggled, or even a removable USB drive where all disk reads fail after it sees one bad sector. If you're not prepared to handle a memory acces…

You can even mmap a socket on some systems (iOS and macOS via GCD). But doing that is super fragile. Socket errors are swallowed. My interpretation always was the mmap should only be used for immutable and local files. You may still run into issues with those type of files but it’s very unlikely.

mmap is also good for passing shared memory around.

(You still need to be careful, of course.)

Re: Why does C have the best file API

#50

Post by kelnos »

mmap() is useful for some narrow use-cases, I think, but error-handling is a huge pain. I don't want to have to deal with SIGBUS in my application.

I agree that the model of mmap() is amazing, though: being able to treat a file as just a range of bytes in memory, while letting the OS handle the fussy details, is incredibly useful. (It's just that the OS doesn't handle all of the fussy details, and that's a pain.)

Post reply on HN