Live data from Hacker News

TurboKV: Insanely fast Rust key-value store

github.com

21–30 of 89 posts

Re: TurboKV: Insanely fast Rust key-value store

#21
post #10
post #9

Earlier quoted context omitted.

I said elsewhere this doesn't survive a power loss.

While it’s important to make this explicit, at what point do we just assume a high-reliability UPS is table stakes? Of course, if you need SIL2 type reliability then you need to assume any given hardware component can spontaneously combust and become a total loss, at which point the data loss caused by a power cut is a rounding error.

> While it’s important to make this explicit, at what point do we just assume a high-reliability UPS is table stakes?

Several years after they become commercially available?

My experience with small UPSes is they tend to cook the batteries and you don't find out until they switch the load and the battery doesn't hold up.

Large facility UPSes tend to do better, but automatic transfer switches have a tendancy to fail ocassionally. If you're hosted in many locations, it's not unusual to have a couple ATS failures per decade.

All that said, unexpected power loss is certainly one reason that writes may be lost, but OSes crash too. Disk firmware can also crash, but if thst bricks the disk, writes in progress don't really matter. Sometimes cabling fails. Or you get a uncorrectable ECC error (which will typically cause an OS panic... unless you're running a very fancy OS, but if it's in dirty disk backed page, even a fancy OS wouldn't save you)

Plenty of applications don't need or want to pay the cost for full commit to disk, but calling something durable when it's not committed to disk is inaccurate.

And that's before we get into the whole thing where the OS and the disk like to return success when things haven't quite finished.

Re: TurboKV: Insanely fast Rust key-value store

#23

> DbOptions::durable() > Appended to the WAL without a per-write sync So… it’s not durable? Durable doesn’t mean “survives a process restart”, it means “durably saved to persistent storage”. For example, this “durable” mode wouldn’t survive power loss.

Are we back to MongoDB -- no fsync() but webscale speed?

Re: TurboKV: Insanely fast Rust key-value store

#24
post #9

Earlier quoted context omitted.

Those help but the main write speed gain is the WAL, that uses preallocated mmap segments to avoid a write(2) per durable mutation while preserving crash recovery. AES hashing mainly helps Bloom filter point lookups and LZ4 mainly helps SSTable I/O. Scans benefit indirectly, but don’t yet use a custom SIMD merge loop.

I said elsewhere this doesn't survive a power loss.

paranoid() does survive power loss.

Re: TurboKV: Insanely fast Rust key-value store

#25

Earlier quoted context omitted.

You're right, that mode provides process crash recovery, not power-loss durability. The benchmark compares it against fjall’s equivalent buffered-WAL mode.

If that's your design constraint, couldn't you speed it up by getting rid of the WAL?

You'd lose durability against process crashes.

If your system has a reasonable tolerance for power failure (multi-az multi-cloud), this can provide much better throughput

Re: TurboKV: Insanely fast Rust key-value store

#26

Earlier quoted context omitted.

If that's your design constraint, couldn't you speed it up by getting rid of the WAL?

You'd lose durability against process crashes. If your system has a reasonable tolerance for power failure (multi-az multi-cloud), this can provide much better throughput

Indeed, a common enough pattern for etcd is to run it backed by a RAMdisk and have multi-az availability + periodic backups + tolerance at a business level to be OK losing some recent data.

Re: TurboKV: Insanely fast Rust key-value store

#29
post #23

> DbOptions::durable() > Appended to the WAL without a per-write sync So… it’s not durable? Durable doesn’t mean “survives a process restart”, it means “durably saved to persistent storage”. For example, this “durable” mode wouldn’t survive power loss.

Are we back to MongoDB -- no fsync() but webscale speed?

Nope - https://github.com/kingroryg/turbokv#durability-presets

Re: TurboKV: Insanely fast Rust key-value store

#30

> DbOptions::durable() > Appended to the WAL without a per-write sync So… it’s not durable? Durable doesn’t mean “survives a process restart”, it means “durably saved to persistent storage”. For example, this “durable” mode wouldn’t survive power loss.

Pretty much... paranoid() seems to be the real durable() which isn't a great look for a database project.

Being able to recover a db without corruption beyound losing the last few writes is a pretty useful feature, and buys a lot of performance, but it would be better to label that clearly, as a reasonable expectation on the durable() preset would be for it to be Durable.

Post reply on HN