Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TurboKV: Insanely fast Rust key-value store

github.com

Post reply on HN
Print view View original

61–70 of 87 posts

Re: TurboKV: Insanely fast Rust key-value store

#61

Post by airstrike »

DarmokTanagra wrote: post #45
insanitybit wrote: post #43

Earlier quoted context omitted.

The same reason as ever. Not everyone wants to use the same runtime.

Sure, but now if my rust application isn't using tokio I have to include it as a new dependency because the author of this lib decided to use it as his async runtime? I'm not trying to be pedantic, but this split over async runtimes was what originally turned me off of rust years ago and it still seems to be an issue.

The author of this library can leave the async library choice up to the user.

iced does that quite well.

Re: TurboKV: Insanely fast Rust key-value store

#62

Post by re-thc »

deepsun wrote: post #23
dangoodmanUT wrote: post #6

> DbOptions::durable() > Appended to the WAL without a per-write sync So… it’s not durable? Durable doesn’t mean “survives a process restart”, it means “durably saved to persistent storage”. For example, this “durable” mode wouldn’t survive power loss.

Are we back to MongoDB -- no fsync() but webscale speed?

Agent scale!

Re: TurboKV: Insanely fast Rust key-value store

#63

Post by muvlon »

stingraycharles wrote: post #7
dangoodmanUT wrote: post #6

> DbOptions::durable() > Appended to the WAL without a per-write sync So… it’s not durable? Durable doesn’t mean “survives a process restart”, it means “durably saved to persistent storage”. For example, this “durable” mode wouldn’t survive power loss.

Yeah this should be benchmarked against other systems that have flush() disabled. mmap is nice but it doesn’t support durable semantics in the way that we usually mean with databases. if a write is acknowledged it should not be forgotten, which is not what this is.

As of a couple years ago, mmap actually has a MAP_SYNC flag that makes it durable in the DB sense. The caveat is that it requires DAX on the file and so comes with a whole bunch of restrictions w.r.t. filesystem, storage media and even CPU architecture.

Re: TurboKV: Insanely fast Rust key-value store

#65

Post by KAdot »

The benchmark is setup to test 80 MiB dataset on a machine with 32 GiB RAM, which doesn't represent a typical database workload. How does the key-value store perform on larger than RAM datasets? MMAP is fast when the dataset fits in memory, but it can slow to a crawl when it doesn't, especially if the workload is mostly random point lookups.

Re: TurboKV: Insanely fast Rust key-value store

#66

Post by otabdeveloper4 »

riffraff wrote: post #27
otabdeveloper4 wrote: post #22

Earlier quoted context omitted.

A file-backed hashtable isn't really a DB.

Well, dbm (database manager) is basically that and has been called that for almost 50 years

dbm isn't a database either, and people who call it that are simply wrong.

Re: TurboKV: Insanely fast Rust key-value store

#70

Post by vlovich123 »

DarmokTanagra wrote: post #45
insanitybit wrote: post #43

Earlier quoted context omitted.

The same reason as ever. Not everyone wants to use the same runtime.

Sure, but now if my rust application isn't using tokio I have to include it as a new dependency because the author of this lib decided to use it as his async runtime? I'm not trying to be pedantic, but this split over async runtimes was what originally turned me off of rust years ago and it still seems to be an issue.

Because Tokio, while great for lots of things (ie default decent performance with ease of use), is a performance bottleneck if you want extremely high performance.

For example, the best this DB can do is ~3.7Mkeys/s for non durable writes and 162k/s for durable. I have an equivalent DB that’s always durable and does 30M/s* (for 8 byte entries) because it doesn’t use Tokio among other things. For this dataset it would be saturating the disk I/O no problem so I would expect it to be running ~7-14M/s depending on how fast your SSD is (~2-4x faster than non durable mode and 40-80x faster than its “durable”)

* it was running at 70-100mhz at one point but the challenge is keeping the hot path at ~10ns as you add features and other things.

Post reply on HN