Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are you sure you want to use MMAP in your database management system? [pdf]

db.cs.cmu.edu

Post reply on HN
Print view View original

111–120 of 137 posts

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#111

Post by [deleted] »

CyberDildonics wrote: post #110
jstimpfle wrote: post #109

Earlier quoted context omitted.

> If you didn't want to waste time, you would have explained what you meant or asked questions. You clearly haven't done your homework, because I did. > You conflated files with disks on your own. No one did that for you. I did not really conflate this. It is just conventional but imprecise terminology, and everyone who gets into such a discussion (especially when starting personal attacks) is expected to know to be…

You said There is no need to back this memory with files Then you wouldn't explain it and eventually admit that you do need to have a file path to give to another process, but only after I asked you to show what you meant multiple times.

[deleted]

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#112

Post by [deleted] »

CyberDildonics wrote: post #110
jstimpfle wrote: post #109

Earlier quoted context omitted.

> If you didn't want to waste time, you would have explained what you meant or asked questions. You clearly haven't done your homework, because I did. > You conflated files with disks on your own. No one did that for you. I did not really conflate this. It is just conventional but imprecise terminology, and everyone who gets into such a discussion (especially when starting personal attacks) is expected to know to be…

You said There is no need to back this memory with files Then you wouldn't explain it and eventually admit that you do need to have a file path to give to another process, but only after I asked you to show what you meant multiple times.

[deleted]

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#113

Post by jstimpfle »

CyberDildonics wrote: post #110
jstimpfle wrote: post #109

Earlier quoted context omitted.

> If you didn't want to waste time, you would have explained what you meant or asked questions. You clearly haven't done your homework, because I did. > You conflated files with disks on your own. No one did that for you. I did not really conflate this. It is just conventional but imprecise terminology, and everyone who gets into such a discussion (especially when starting personal attacks) is expected to know to be…

You said There is no need to back this memory with files Then you wouldn't explain it and eventually admit that you do need to have a file path to give to another process, but only after I asked you to show what you meant multiple times.

> There is no need to back this memory with files

And there isn't. It seems you just don't really understand virtual memory, and don't want to acknowledge what everyone else understands by "file-backed memory". And given that I find it courageous how stubborn you are, as well as starting personal attacks.

> Then you wouldn't explain it and eventually admit that you do need to have a file path

Need to have a file path IN WHICH ENVIRONMENT, IN WHICH CONTEXT??? Could YOU please clarify. We can easily make a simple OS which doesn't have "files" but does have processes that can share memory using virtual memory technology.

Shared memory IPC is fundamentally not about files, and you were even shown a way to setup shared memory mappings between Linux processes using normal userland API entirely without the use of files or file paths - with the restriction that the mappings have to be inherited (fork()).

How someone, even with no real understanding of the topic, could not at the latest at https://news.ycombinator.com/item?id=29947339 acknowledge that I was being perfectly clear that I was talking about persistent files (I literally said on a hard drive), is beyond me. I should have stopped this discussion at that point.

Now get off my lawn.

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#114

Post by [deleted] »

CyberDildonics wrote: post #110
jstimpfle wrote: post #109

Earlier quoted context omitted.

> If you didn't want to waste time, you would have explained what you meant or asked questions. You clearly haven't done your homework, because I did. > You conflated files with disks on your own. No one did that for you. I did not really conflate this. It is just conventional but imprecise terminology, and everyone who gets into such a discussion (especially when starting personal attacks) is expected to know to be…

You said There is no need to back this memory with files Then you wouldn't explain it and eventually admit that you do need to have a file path to give to another process, but only after I asked you to show what you meant multiple times.

[deleted]

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#115

Post by CyberDildonics »

jstimpfle wrote: post #113
CyberDildonics wrote: post #110

Earlier quoted context omitted.

You said There is no need to back this memory with files Then you wouldn't explain it and eventually admit that you do need to have a file path to give to another process, but only after I asked you to show what you meant multiple times.

> There is no need to back this memory with files And there isn't. It seems you just don't really understand virtual memory, and don't want to acknowledge what everyone else understands by "file-backed memory". And given that I find it courageous how stubborn you are, as well as starting personal attacks. > Then you wouldn't explain it and eventually admit that you do need to have a file path Need to have a file path…

Files being persistent on storage has nothing to do with communicating through shared memory. It isn't necessary and it doesn't interfere if it's there. It is completely orthogonal, I don't know why it would ever be a part of the conversation when talking about direct reading and writing to the same memory.

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#116

Post by jstimpfle »

CyberDildonics wrote: post #115
jstimpfle wrote: post #113

Earlier quoted context omitted.

> There is no need to back this memory with files And there isn't. It seems you just don't really understand virtual memory, and don't want to acknowledge what everyone else understands by "file-backed memory". And given that I find it courageous how stubborn you are, as well as starting personal attacks. > Then you wouldn't explain it and eventually admit that you do need to have a file path Need to have a file path…

Files being persistent on storage has nothing to do with communicating through shared memory. It isn't necessary and it doesn't interfere if it's there. It is completely orthogonal, I don't know why it would ever be a part of the conversation when talking about direct reading and writing to the same memory.

> Files being persistent on storage has nothing to do with communicating through shared memory.

Files (whether persistent or not) have not really anything to do with communication through shared memory. In the implementation of an API like shm_open(), the VFS (virtual filesystem) is simply the address space and lookup mechanism that an operating system like Linux happens to use in order to find the memory that should be shared.

> It isn't necessary and it doesn't interfere if it's there.

Sure it does interfere. By backing memory needlessly with a persistent file, you're causing disk I/O from the loading and flushing (that can't really be controlled) and potentially bad performance.

Also, as explained, if you use a persistent file to track the synchronization state, the synchronization state won't be reset when the communicating processes die unexpectedly, and this might be problematic.

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#117

Post by sprachspiel »

jnwatson wrote: post #85
jzwinck wrote: post #61

Earlier quoted context omitted.

Can you share a couple examples of those parameters?

This describes the relevant kernel variables: https://synapse.docs.vertex.link/en/latest/synapse/devguides...

These settings control writing back modified pages. The experiments in the paper are read-only. With writes the situation is even worse than shown in the paper (though tuning these settings may help a bit).

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#119

Post by pritambaral »

CyberDildonics wrote: post #106
pritambaral wrote: post #103

Earlier quoted context omitted.

> How do you have two processes writing to the same place in memory without memory mapping a file? In increasing order of modernity: System V SHM, POSIX SHM, and `mmap(... MAP_SHARED | MAP_ANONYMOUS ...)` (e.g., on Linux).

These are still memory mapping files using file paths and returning file descriptors as far as I know, which makes sense because you have to have something coordinated between the two processes.

> These are still memory mapping files using file paths ...

No, they're not. The entire purpose of MAP_ANONYMOUS is to avoid using files.

Sources:

1. The Linux Kernel source code [1], where it comes with the code comment: "don't use a file".

2. The glibc source code [2], where it comes with the same code comment: "Don't use a file".

3. The Linux man-pages project documentation of mmap [3], where it is documented thus: "The mapping is not backed by any file; its contents are initialized to zero. The fd argument is ignored"

Similarly for SHM, but if you still don't get the point about MAP_ANONYMOUS, I doubt you'll get it for SHM either.

> ... and returning file descriptors

A socket is a file descriptor. An epoll handle is a file descriptor. On modern Linux kernels, a pid handle is a file descriptor. None of them are backed by "files".

> ... because you have to have something coordinated between the two processes.

FDs are not the only things processes can share, even if you go back to the venerable, original Unices, so I don't see what you mean.

----

[1] https://elixir.bootlin.com/linux/0.99.14r/source/include/lin... (the first version it was released in) to https://elixir.bootlin.com/linux/v5.16.1/source/include/uapi... (the latest version)

[2] https://code.woboq.org/userspace/glibc/sysdeps/unix/sysv/lin...

[3] https://man7.org/linux/man-pages/man2/mmap.2.html

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#120

Post by CyberDildonics »

pritambaral wrote: post #119
CyberDildonics wrote: post #106

Earlier quoted context omitted.

These are still memory mapping files using file paths and returning file descriptors as far as I know, which makes sense because you have to have something coordinated between the two processes.

> These are still memory mapping files using file paths ... No, they're not. The entire purpose of MAP_ANONYMOUS is to avoid using files. Sources: 1. The Linux Kernel source code [1], where it comes with the code comment: "don't use a file". 2. The glibc source code [2], where it comes with the same code comment: "Don't use a file". 3. The Linux man-pages project documentation of mmap [3], where it is documented thus…

This just goes back to the same question - what do two processes use to map the same memory into their memory space if it isn't a path to a file?

I'm not saying there isn't anything, I'm just seeing an extreme avoidance to an actual answer. The other guy went down a rabbit hole of syncing that memory to storage, which has nothing to do with anything.

Post reply on HN