Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TurboKV: Insanely fast Rust key-value store

github.com

Post reply on HN
Print view View original

31–40 of 90 posts

Re: TurboKV: Insanely fast Rust key-value store

#31

Post by deathanatos »

rgbimbochamp wrote: post #8
stingraycharles wrote: post #7

Earlier quoted context omitted.

Yeah this should be benchmarked against other systems that have flush() disabled. mmap is nice but it doesn’t support durable semantics in the way that we usually mean with databases. if a write is acknowledged it should not be forgotten, which is not what this is.

You're right, that mode provides process crash recovery, not power-loss durability. The benchmark compares it against fjall’s equivalent buffered-WAL mode.

Word choice matters. Defaults matter. People will go "well it says durable right here" and while arguably, yes, they should RTFM, it would still be great if tool-builders did not set the shotgun's default state to State::AT_FOOT. It would be nice if every paragraph of technical writing that I have to do need not be burdened by a thousand asterisks of "durable in this context means something other than durable".

Re: TurboKV: Insanely fast Rust key-value store

#34

Post by Sesse__ »

nine_k wrote: post #2

I suppose the insane speed is due to this: > TurboKV's persisted Bloom-filter format uses hardware AES. Also, built-in LZ4 compression. I would expect SIMD to be used for scans.

For any complex system, there's never one single trick or design choice that makes it fast. It's always a large amount of engineering (or exaggerations, of course).

Re: TurboKV: Insanely fast Rust key-value store

#36

Post by carlmr »

deathanatos wrote: post #31
rgbimbochamp wrote: post #8

Earlier quoted context omitted.

You're right, that mode provides process crash recovery, not power-loss durability. The benchmark compares it against fjall’s equivalent buffered-WAL mode.

Word choice matters. Defaults matter. People will go "well it says durable right here" and while arguably, yes, they should RTFM, it would still be great if tool-builders did not set the shotgun's default state to State::AT_FOOT. It would be nice if every paragraph of technical writing that I have to do need not be burdened by a thousand asterisks of "durable in this context means something other than durable".

>arguably, yes, they should RTFM

Agreed. Good design is when the things do what you expect them to do without reading the manual, don't reuse wording with other meaning in the wrong way. That way if you do encounter nee wording, you know you should read the manual.

Re: TurboKV: Insanely fast Rust key-value store

#37

Post by ncruces »

yawniek wrote: post #28
medv wrote: post #20

Every programmer eventually creates own db: https://github.com/antonmedv/medb

Building dbs is lots of fun. I recently did a few. Some in rust and some in go. And its really interesting.

Agreed. I went the route of building on top of something else. Sill lots of fun.

https://github.com/ncruces/go-sqlite3

And I'm still having fun.

https://github.com/ncruces/go-sqlite3/pull/421

Re: TurboKV: Insanely fast Rust key-value store

#39

Post by imtringued »

taneq wrote: post #10
bestouff wrote: post #9

Earlier quoted context omitted.

I said elsewhere this doesn't survive a power loss.

While it’s important to make this explicit, at what point do we just assume a high-reliability UPS is table stakes? Of course, if you need SIL2 type reliability then you need to assume any given hardware component can spontaneously combust and become a total loss, at which point the data loss caused by a power cut is a rounding error.

Not sure how you managed to do it but you got it completely backwards.

If someone demands that the database should use fsync and only respond with success once the write finished, it is not some arbitrarily high reliability demand that needs to be implemented using reliable hardware. In fact, the entire point of implementing the power loss protection in software is so that you don't need perfectly reliable hardware. The power loss event turns into a downtime event which is often completely acceptable.

The requirement to have infallible hardware only emerged because the software refused to do its job. Infallible hardware is not a requirement decided by the user, it's a requirement decided by the developer of TurboKV to intentionally restrict his software to exclusively operate in a reliable hardware environment.

The fact that the user specified durability of the KV store during power loss does not make the user obsessed over hardware reliability, the software shifted the burden onto the hardware and forced the user to deal with this mess.

I don't know how exactly TurboKV works so let's talk about a hypothetical software instead.

Let's say the software cannot survive a power loss event and just corrupts the database. If the user wants to operate the software, he is forced by the software to operate it in an infallible environment where power loss can never occur. Based on how the software was designed, power loss is a catastrophic event. The SIL2 type reliability you're talking about only makes sense in contexts with catastrophic events.

So how it went is that the user made a reasonable demand with bounded reliability: "please survive power loss with durable writes" and the author says, sure just run the software on a SIL2 type reliability hardware environment.

It's not the user who blew up the hardware requirements.

Post reply on HN