Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie

news.ycombinator.com

Post reply on HN
Print view View original

21–28 of 28 posts

Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie

#21

Post by hyc_symas »

otterley wrote: post #20
hyc_symas wrote: post #19

Earlier quoted context omitted.

Wrong, LMDB fully supports multiprocess concurrency as well as DBs multiple orders of magnitude larger than RAM. Wherever you got your info from is dead wrong. Among embedded key/value stores, only LMDB and BerkeleyDB support multiprocess access. RocksDB, LevelDB, etc. are all single process.

My mistake. Doesn’t it have a global lock though? Also, even if LMDB supports databases larger than RAM, that’s it doesn’t mean it’s a good idea to have a working set that exceeds that size. Unless you’re claiming it’s scan resistant?

It has a single writer transaction mutex, yes. But it's a process-shared mutex, so it will serialize write transactions across an arbitrary number of processes. And of course, read transactions are completely lockfree/waitfree across arbitrarily many processes.

As for working set size, that is always merely the height of the B+tree. Scans won't change that. It will always be far more efficient than any other DB under the same conditions.

Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie

#22

Post by otterley »

hyc_symas wrote: post #21
otterley wrote: post #20

Earlier quoted context omitted.

My mistake. Doesn’t it have a global lock though? Also, even if LMDB supports databases larger than RAM, that’s it doesn’t mean it’s a good idea to have a working set that exceeds that size. Unless you’re claiming it’s scan resistant?

It has a single writer transaction mutex, yes. But it's a process-shared mutex, so it will serialize write transactions across an arbitrary number of processes. And of course, read transactions are completely lockfree/waitfree across arbitrarily many processes. As for working set size, that is always merely the height of the B+tree. Scans won't change that. It will always be far more efficient than any other DB under…

> As for working set size, that is always merely the height of the B+tree.

This statement makes no sense to me. Are you using a different definition of "working set" than the rest of us? A working set size is application and access pattern dependent.

> It will always be far more efficient than any other DB under the same conditions

That depends on how broadly or narrowly one defines "same conditions" :-)

Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie

#23

Post by hyc_symas »

otterley wrote: post #22
hyc_symas wrote: post #21

Earlier quoted context omitted.

It has a single writer transaction mutex, yes. But it's a process-shared mutex, so it will serialize write transactions across an arbitrary number of processes. And of course, read transactions are completely lockfree/waitfree across arbitrarily many processes. As for working set size, that is always merely the height of the B+tree. Scans won't change that. It will always be far more efficient than any other DB under…

> As for working set size, that is always merely the height of the B+tree. This statement makes no sense to me. Are you using a different definition of "working set" than the rest of us? A working set size is application and access pattern dependent. > It will always be far more efficient than any other DB under the same conditions That depends on how broadly or narrowly one defines "same conditions" :-)

Identical hardware, same RAM size, same data volume.

Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie

#24

Post by otterley »

hyc_symas wrote: post #23
otterley wrote: post #22

Earlier quoted context omitted.

> As for working set size, that is always merely the height of the B+tree. This statement makes no sense to me. Are you using a different definition of "working set" than the rest of us? A working set size is application and access pattern dependent. > It will always be far more efficient than any other DB under the same conditions That depends on how broadly or narrowly one defines "same conditions" :-)

Identical hardware, same RAM size, same data volume.

That’s a bold claim. Are you saying that LMDB outperforms every other database on the same hardware, regardless of access pattern? And if so, is there proof of this?

Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie

#25

Post by hyc_symas »

otterley wrote: post #24
hyc_symas wrote: post #23

Earlier quoted context omitted.

Identical hardware, same RAM size, same data volume.

That’s a bold claim. Are you saying that LMDB outperforms every other database on the same hardware, regardless of access pattern? And if so, is there proof of this?

Plenty of proof. http://www.lmdb.tech/bench/

Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie

#26

Post by otterley »

hyc_symas wrote: post #25
otterley wrote: post #24

Earlier quoted context omitted.

That’s a bold claim. Are you saying that LMDB outperforms every other database on the same hardware, regardless of access pattern? And if so, is there proof of this?

Plenty of proof. http://www.lmdb.tech/bench/

Since the first question of my two-part inquiry not explicitly answered in the affirmative: To be absolutely clear, you are claiming, in writing, that LMDB outperforms every other database there is, regardless of access pattern, using the same hardware?

Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie

#27

Post by hyc_symas »

otterley wrote: post #26
hyc_symas wrote: post #25

Earlier quoted context omitted.

Plenty of proof. http://www.lmdb.tech/bench/

Since the first question of my two-part inquiry not explicitly answered in the affirmative: To be absolutely clear, you are claiming, in writing, that LMDB outperforms every other database there is, regardless of access pattern, using the same hardware?

Not every.

LMDB is optimized for read-heavy workloads. I make no particular claims about write-heavy workloads.

Because it's so efficient, it can retain more useful data in-memory than other DBs for a given RAM size. For DBs much larger than RAM it will get more useful work done with the available RAM than other DBs. You can examine the benchmark reports linked above, they provide not just the data but also the analysis of why the results are as they are.

Re: Ask HN: Scheduling stateful nodes when MMAP makes memory accounting a lie

#28

Post by hyc_symas »

hyc_symas wrote: post #25
otterley wrote: post #24

Earlier quoted context omitted.

That’s a bold claim. Are you saying that LMDB outperforms every other database on the same hardware, regardless of access pattern? And if so, is there proof of this?

Plenty of proof. http://www.lmdb.tech/bench/

You don't have to take my word for it. Plenty of other developers know. https://www.youtube.com/watch?v=CfiQ0h4bGWM
Post reply on HN