TurboKV: Insanely fast Rust key-value store
1–10 of 89 posts
Re: TurboKV: Insanely fast Rust key-value store
#2I suppose the insane speed is due to this:
> TurboKV's persisted Bloom-filter format uses hardware AES.
Also, built-in LZ4 compression.
I would expect SIMD to be used for scans.
Re: TurboKV: Insanely fast Rust key-value store
#3I suppose the insane speed is due to this: > TurboKV's persisted Bloom-filter format uses hardware AES. Also, built-in LZ4 compression. I would expect SIMD to be used for scans.
Those help but the main write speed gain is the WAL, that uses preallocated mmap segments to avoid a write(2) per durable mutation while preserving crash recovery.
AES hashing mainly helps Bloom filter point lookups and LZ4 mainly helps SSTable I/O. Scans benefit indirectly, but don’t yet use a custom SIMD merge loop.
Re: TurboKV: Insanely fast Rust key-value store
#4I suppose the insane speed is due to this: > TurboKV's persisted Bloom-filter format uses hardware AES. Also, built-in LZ4 compression. I would expect SIMD to be used for scans.
Those help but the main write speed gain is the WAL, that uses preallocated mmap segments to avoid a write(2) per durable mutation while preserving crash recovery. AES hashing mainly helps Bloom filter point lookups and LZ4 mainly helps SSTable I/O. Scans benefit indirectly, but don’t yet use a custom SIMD merge loop.
Iirc that’s how badger handles the WAL as well
Re: TurboKV: Insanely fast Rust key-value store
#5I suppose the insane speed is due to this: > TurboKV's persisted Bloom-filter format uses hardware AES. Also, built-in LZ4 compression. I would expect SIMD to be used for scans.
I assume this is for hashing. I've seen several hashing algorithms turn to hardware AES instructions before, but I haven't seen any evidence that this technique outperforms state-of-the-art hashes like RapidHash (https://github.com/Nicoshev/rapidhash) in either quality or speed.
Re: TurboKV: Insanely fast Rust key-value store
#6> DbOptions::durable()
> Appended to the WAL without a per-write sync
So… it’s not durable? Durable doesn’t mean “survives a process restart”, it means “durably saved to persistent storage”. For example, this “durable” mode wouldn’t survive power loss.
Re: TurboKV: Insanely fast Rust key-value store
#7> DbOptions::durable() > Appended to the WAL without a per-write sync So… it’s not durable? Durable doesn’t mean “survives a process restart”, it means “durably saved to persistent storage”. For example, this “durable” mode wouldn’t survive power loss.
Yeah this should be benchmarked against other systems that have flush() disabled.
mmap is nice but it doesn’t support durable semantics in the way that we usually mean with databases.
if a write is acknowledged it should not be forgotten, which is not what this is.
Re: TurboKV: Insanely fast Rust key-value store
#8> DbOptions::durable() > Appended to the WAL without a per-write sync So… it’s not durable? Durable doesn’t mean “survives a process restart”, it means “durably saved to persistent storage”. For example, this “durable” mode wouldn’t survive power loss.
Yeah this should be benchmarked against other systems that have flush() disabled. mmap is nice but it doesn’t support durable semantics in the way that we usually mean with databases. if a write is acknowledged it should not be forgotten, which is not what this is.
You're right, that mode provides process crash recovery, not power-loss durability. The benchmark compares it against fjall’s equivalent buffered-WAL mode.
Re: TurboKV: Insanely fast Rust key-value store
#9I suppose the insane speed is due to this: > TurboKV's persisted Bloom-filter format uses hardware AES. Also, built-in LZ4 compression. I would expect SIMD to be used for scans.
Those help but the main write speed gain is the WAL, that uses preallocated mmap segments to avoid a write(2) per durable mutation while preserving crash recovery. AES hashing mainly helps Bloom filter point lookups and LZ4 mainly helps SSTable I/O. Scans benefit indirectly, but don’t yet use a custom SIMD merge loop.
I said elsewhere this doesn't survive a power loss.
Re: TurboKV: Insanely fast Rust key-value store
#10Earlier quoted context omitted.
Those help but the main write speed gain is the WAL, that uses preallocated mmap segments to avoid a write(2) per durable mutation while preserving crash recovery. AES hashing mainly helps Bloom filter point lookups and LZ4 mainly helps SSTable I/O. Scans benefit indirectly, but don’t yet use a custom SIMD merge loop.
I said elsewhere this doesn't survive a power loss.
While it’s important to make this explicit, at what point do we just assume a high-reliability UPS is table stakes?
Of course, if you need SIL2 type reliability then you need to assume any given hardware component can spontaneously combust and become a total loss, at which point the data loss caused by a power cut is a rounding error.