Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are you sure you want to use MMAP in your database management system? [pdf]

db.cs.cmu.edu

Post reply on HN
Print view View original

121–130 of 137 posts

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#121

Post by CyberDildonics »

jstimpfle wrote: post #116
CyberDildonics wrote: post #115

Earlier quoted context omitted.

Files being persistent on storage has nothing to do with communicating through shared memory. It isn't necessary and it doesn't interfere if it's there. It is completely orthogonal, I don't know why it would ever be a part of the conversation when talking about direct reading and writing to the same memory.

> Files being persistent on storage has nothing to do with communicating through shared memory. Files (whether persistent or not) have not really anything to do with communication through shared memory. In the implementation of an API like shm_open(), the VFS (virtual filesystem) is simply the address space and lookup mechanism that an operating system like Linux happens to use in order to find the memory that should…

system like Linux happens to use in order to find the memory that should be shared.

Right. Is there some other mechanism to coordinate mapping the same memory between processes? That's all I ever asked.

Sure it does interfere. By backing memory needlessly with a persistent file, you're causing disk I/O from the loading and flushing (that can't really be controlled) and potentially bad performance.

That is orthogonal, since once you have the memory mapped into both processes you can use atomics for lock free IPC. That's the whole thing. It doesn't matter what the OS does or doesn't do in the background, atomically reading and writing to memory is unaffected.

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#122

Post by jstimpfle »

CyberDildonics wrote: post #120
pritambaral wrote: post #119

Earlier quoted context omitted.

> These are still memory mapping files using file paths ... No, they're not. The entire purpose of MAP_ANONYMOUS is to avoid using files. Sources: 1. The Linux Kernel source code [1], where it comes with the code comment: "don't use a file". 2. The glibc source code [2], where it comes with the same code comment: "Don't use a file". 3. The Linux man-pages project documentation of mmap [3], where it is documented thus…

This just goes back to the same question - what do two processes use to map the same memory into their memory space if it isn't a path to a file? I'm not saying there isn't anything, I'm just seeing an extreme avoidance to an actual answer. The other guy went down a rabbit hole of syncing that memory to storage, which has nothing to do with anything.

I'm starting to think you're even more confused than I had assumed. You were literally given a reasonable possible answer to your question multiple times (MAP_ANONYMOUS). And if there wasn't a big confusion you wouldn't be asking these questions in the first place because you could just make up your own answer.

I'm also left uncertain if you're assuming Linux and not talking about it. At least your objections to general statements are weirdly specific, while you never clarify the context (e.g. what OS you're talking about), and you seem to assume that there couldn't be other ways of achieving stuff. There seems to be a weird lack of understanding of the basics in your comments.

At the core, everything you need to share memory is that the participating processes agree about the (physical) address range of that memory (e.g. a 64-bit starting address and a 64-bit size). You could literally hardcode a physical address range, map this range to arbitrary (and possibly different) virtual address ranges in each of the processes, and start communicating through that shared memory. Note that the mappings are stored in the RAM and CPU, it has nothing at all to do with any files or filepaths.

And this whole discussion is completely pointless anyway because it started of YOU misunderstanding what I meant by "file-backed memory", which is not my fault at all. The term is completely unambiguous, it means (as opposed to POSIX SHM / MAP_ANONYMOUS / whatever) page cache memory that gets synced to an underlying file on a filesystem.

Please stop questioning and start experimenting and understanding what we're saying. We know what we're talking about. You don't.

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#123

Post by jstimpfle »

CyberDildonics wrote: post #121
jstimpfle wrote: post #116

Earlier quoted context omitted.

> Files being persistent on storage has nothing to do with communicating through shared memory. Files (whether persistent or not) have not really anything to do with communication through shared memory. In the implementation of an API like shm_open(), the VFS (virtual filesystem) is simply the address space and lookup mechanism that an operating system like Linux happens to use in order to find the memory that should…

system like Linux happens to use in order to find the memory that should be shared. Right. Is there some other mechanism to coordinate mapping the same memory between processes? That's all I ever asked. Sure it does interfere. By backing memory needlessly with a persistent file, you're causing disk I/O from the loading and flushing (that can't really be controlled) and potentially bad performance. That is orthogonal,…

Why are you constantly ignoring what I'm writing?

> It doesn't matter what the OS does or doesn't do in the background, atomically reading and writing to memory is unaffected.

That's not true. If this thing is file backed there is usually no guarantee that the page of virtual memory (i.e. a page of the file data) you're accessing is present in physical memory. You'll cause page faults and data transfers to/from disk. This can delay the execution of an atomic read or write potentially infinitely, or even cause a "crash" of some kind if the disk transfer fails.

You can avoid the page faulting part of this if you somehow pin the memory. Which is completely ridiculous given that all you ever wanted is anonymous memory. I've looked up a website that seems to explain this better (but I haven't checked it deeply). Maybe it helps: https://eric-lo.gitbook.io/memory-mapped-io/pin-the-page

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#124

Post by pritambaral »

CyberDildonics wrote: post #120
pritambaral wrote: post #119

Earlier quoted context omitted.

> These are still memory mapping files using file paths ... No, they're not. The entire purpose of MAP_ANONYMOUS is to avoid using files. Sources: 1. The Linux Kernel source code [1], where it comes with the code comment: "don't use a file". 2. The glibc source code [2], where it comes with the same code comment: "Don't use a file". 3. The Linux man-pages project documentation of mmap [3], where it is documented thus…

This just goes back to the same question - what do two processes use to map the same memory into their memory space if it isn't a path to a file? I'm not saying there isn't anything, I'm just seeing an extreme avoidance to an actual answer. The other guy went down a rabbit hole of syncing that memory to storage, which has nothing to do with anything.

Please just RTFM on `mmap(... MAP_SHARED | MAP_ANONYMOUS, -1, 0);`. There is no file path involved.

That's all I can say to you now, apart from "this is not StackOverflow".

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#125

Post by CyberDildonics »

jstimpfle wrote: post #122
CyberDildonics wrote: post #120

Earlier quoted context omitted.

This just goes back to the same question - what do two processes use to map the same memory into their memory space if it isn't a path to a file? I'm not saying there isn't anything, I'm just seeing an extreme avoidance to an actual answer. The other guy went down a rabbit hole of syncing that memory to storage, which has nothing to do with anything.

I'm starting to think you're even more confused than I had assumed. You were literally given a reasonable possible answer to your question multiple times (MAP_ANONYMOUS). And if there wasn't a big confusion you wouldn't be asking these questions in the first place because you could just make up your own answer. I'm also left uncertain if you're assuming Linux and not talking about it. At least your objections to gene…

"MAP_ANONYMOUS|MAP_SHARED mapped memory can only be accessed by the process which does that mmap() call or its child processes. There is no way for another process to map the same memory because that memory can not be referred to from elsewhere since it is anonymous."

https://stackoverflow.com/questions/4991533/sharing-memory-b...

misunderstanding what I meant by "file-backed memory"

No, it started by talking about using atomics for lock free interprocess communication, something MAP_ANONYMOUS can't do.

You hallucinated writing to storage as being part of this, didn't explain yourself and are getting upset about it. Atomic instructions that manipulate memory is orthogonal to what the OS does is the background. No one would think an operation on the order of nanoseconds has anything to do with writing permanent storage.

clarify the context (e.g. what OS you're talking about)

This thread is about mmap - it says it in the title.

it has nothing at all to do with any files or filepaths.

Two processes need some way to map the same memory and they do it through file paths.

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#126

Post by pritambaral »

CyberDildonics wrote: post #125
jstimpfle wrote: post #122

Earlier quoted context omitted.

I'm starting to think you're even more confused than I had assumed. You were literally given a reasonable possible answer to your question multiple times (MAP_ANONYMOUS). And if there wasn't a big confusion you wouldn't be asking these questions in the first place because you could just make up your own answer. I'm also left uncertain if you're assuming Linux and not talking about it. At least your objections to gene…

"MAP_ANONYMOUS|MAP_SHARED mapped memory can only be accessed by the process which does that mmap() call or its child processes. There is no way for another process to map the same memory because that memory can not be referred to from elsewhere since it is anonymous." https://stackoverflow.com/questions/4991533/sharing-memory-b... misunderstanding what I meant by "file-backed memory" No, it started by talking about u…

> ... they do it through file paths.

>> ... or its child processes

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#127

Post by CyberDildonics »

jstimpfle wrote: post #123
CyberDildonics wrote: post #121

Earlier quoted context omitted.

system like Linux happens to use in order to find the memory that should be shared. Right. Is there some other mechanism to coordinate mapping the same memory between processes? That's all I ever asked. Sure it does interfere. By backing memory needlessly with a persistent file, you're causing disk I/O from the loading and flushing (that can't really be controlled) and potentially bad performance. That is orthogonal,…

Why are you constantly ignoring what I'm writing? > It doesn't matter what the OS does or doesn't do in the background, atomically reading and writing to memory is unaffected. That's not true. If this thing is file backed there is usually no guarantee that the page of virtual memory (i.e. a page of the file data) you're accessing is present in physical memory. You'll cause page faults and data transfers to/from disk.…

This can delay the execution of an atomic read or write

You can play "what if" all you want if you don't know what else running, but this was always about lock free interprocess communication, which is not broken by a page fault or process suspension.

An atomic instruction by design will do everything it needs to when the instruction runs.

Saying the OS can ultimately control the execution of a process is a nonsense cop out to try to skew away from the original point.

all you ever wanted is anonymous memory

This is local to a process tree and does not work for interprocess communication.

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#128

Post by [deleted] »

CyberDildonics wrote: post #125
jstimpfle wrote: post #122

Earlier quoted context omitted.

I'm starting to think you're even more confused than I had assumed. You were literally given a reasonable possible answer to your question multiple times (MAP_ANONYMOUS). And if there wasn't a big confusion you wouldn't be asking these questions in the first place because you could just make up your own answer. I'm also left uncertain if you're assuming Linux and not talking about it. At least your objections to gene…

"MAP_ANONYMOUS|MAP_SHARED mapped memory can only be accessed by the process which does that mmap() call or its child processes. There is no way for another process to map the same memory because that memory can not be referred to from elsewhere since it is anonymous." https://stackoverflow.com/questions/4991533/sharing-memory-b... misunderstanding what I meant by "file-backed memory" No, it started by talking about u…

[deleted]

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#129

Post by jstimpfle »

CyberDildonics wrote: post #125
jstimpfle wrote: post #122

Earlier quoted context omitted.

I'm starting to think you're even more confused than I had assumed. You were literally given a reasonable possible answer to your question multiple times (MAP_ANONYMOUS). And if there wasn't a big confusion you wouldn't be asking these questions in the first place because you could just make up your own answer. I'm also left uncertain if you're assuming Linux and not talking about it. At least your objections to gene…

"MAP_ANONYMOUS|MAP_SHARED mapped memory can only be accessed by the process which does that mmap() call or its child processes. There is no way for another process to map the same memory because that memory can not be referred to from elsewhere since it is anonymous." https://stackoverflow.com/questions/4991533/sharing-memory-b... misunderstanding what I meant by "file-backed memory" No, it started by talking about u…

> something MAP_ANONYMOUS can't do.

> No one would think an operation on the order of nanoseconds has anything to do with writing permanent storage.

I literally just explained to you why with a file-backed mapping it is not nanoseconds but potentially an infinite time: https://news.ycombinator.com/item?id=29977672

> it has nothing at all to do with any files or filepaths.

I literally just explained to you that is doesn't have to be filepaths (and even with shm_open() POSIX standard, pedantically IT IS NOT A FILEPATH).

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#130

Post by jstimpfle »

pritambaral wrote: post #126
CyberDildonics wrote: post #125

Earlier quoted context omitted.

"MAP_ANONYMOUS|MAP_SHARED mapped memory can only be accessed by the process which does that mmap() call or its child processes. There is no way for another process to map the same memory because that memory can not be referred to from elsewhere since it is anonymous." https://stackoverflow.com/questions/4991533/sharing-memory-b... misunderstanding what I meant by "file-backed memory" No, it started by talking about u…

> ... they do it through file paths. >> ... or its child processes

... or by just knowing a fixed physical address.

Or, by whatever convention. The possibilities are infinite.

Post reply on HN