Live data from Hacker News

Are you sure you want to use MMAP in your database management system? [pdf]

db.cs.cmu.edu

81–90 of 137 posts

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#81

Earlier quoted context omitted.

Why do you need lockless atomic updates to a file-backed memory area? Genuinely curious.

Because it allows you to do lock free memory based interprocess communication, which can be extremely fast.

There is no need for file-backed memory to do that.

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#83
post #77

Earlier quoted context omitted.

The current URL just redirects back to that "superficial-accessible 'home page'" anyway (probably as a substitute for 404 handling, I'd guess); if the intent is to link directly to the paper/PDF, you probably want https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf But I agree with the other person; if people really won't read any further from the homepage, then I highly doubt they'd read any further than the h…

They must have changed https://db.cs.cmu.edu/papers/2022/p13-crotty.pdf to make it redirect. I've changed it again. IMO you guys (as well as whoever did that) are underestimating the difference in how the two different kinds of submission affect resulting discussion. I did offer to change the top URL to point to the video, if they felt that was more important, but never heard back.

How many hours had it been between them making that change and you updating the URL again, though? If linking to just the PDF v. a homepage with the PDF + presentation + code would affect the resulting discussion, it's probably fair to say that the resulting discussion has already been affected (this particular conversation notwithstanding), no?

Even that aside, if the authors care so much about what people see first that they actively set a particular URL to redirect to their preference (EDIT: and have explicitly stated that preference in these comments, assuming "apavlo" is Andy Pavlo: https://news.ycombinator.com/item?id=29939332), should that preference not be respected?

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#84
post #57

Earlier quoted context omitted.

This makes me wonder whether there would be value in an OS that is also a DBMS (or vice versa). In other words, if the DBMS has total control over the hardware, perhaps performance can be maximized without too much additional complexity.

This is a bad idea from the 1960s: IBM TPF, MUMPS, Pick. As soon as the hardware changes it becomes slower and more complicated.

That was back when hardware was changing to a significant degree, though. Nowadays, there ain't really much that's new about hardware today v. hardware from 10 or 20 years ago - hence operating systems / filesystems being able to remain mostly stable instead of suffering from the exact same problem.

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#85
post #61

I have a great deal of experience in running very large memory-mapped databases using LMDB. The default Linux settings dealing with memory mapped files are pretty horrible. The observed poor performance is directly related to not configuring several very important kernel parameters.

Can you share a couple examples of those parameters?

This describes the relevant kernel variables:

https://synapse.docs.vertex.link/en/latest/synapse/devguides...

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#86
post #2

RavenDB's response to this paper: https://ayende.com/blog/196161-C/re-are-you-sure-you-want-to...

Thank you for these counter-arguments. It's good to have them to make up your own mind, especially when recognized experts use a mocking tone "you will no dare think the contrary".

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#87

Earlier quoted context omitted.

Their last sentence literally said "data races which would be common when using read() and write() without locks."

Which is not literally what I said.

They said you get a data race if you use read() and write() without file locks.

You asked "Are you saying that without mmap() there will be data races??".

No, they are saying you get a data race if you use read() and write() without file locks.

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#88

Earlier quoted context omitted.

Because it allows you to do lock free memory based interprocess communication, which can be extremely fast.

There is no need for file-backed memory to do that.

Sounds good, what is your solution and why didn't you explain it in your first reply?

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#89

Earlier quoted context omitted.

Which is not literally what I said.

They said you get a data race if you use read() and write() without file locks. You asked "Are you saying that without mmap() there will be data races??". No, they are saying you get a data race if you use read() and write() without file locks.

> No, they are saying you get a data race if you use read() and write() without file locks.

Since you seem to understand better and my questions never made sense, maybe you can explain now why would we do that?

Re: Are you sure you want to use MMAP in your database management system? [pdf]

#90

Earlier quoted context omitted.

There is no need for file-backed memory to do that.

Sounds good, what is your solution and why didn't you explain it in your first reply?

What is my solution to what? To database I/O? I guess that's what the article is about...
Post reply on HN