How RocksDB Works
artem.krylysov.com
How RocksDB Works
1–10 of 71 posts
Re: How RocksDB Works
#2RocksDB is awesome, though don't use it with regular glibc malloc because it can cause extreme memory fragmentation. Use jemalloc, tcmalloc, or mimalloc: basically any other advance malloc libraries that can effectively reuse memory.
Re: How RocksDB Works
#3Congratulations to the author for a remarkably clear, easy-to-follow, and informative post. Excellent technical writing!
Re: How RocksDB Works
#4I am looking for optimal storage engine(KV) which can store operational telemetry (temporarily) at source node. As we know, operational telemetry is generated frequently and need to merge similar operations frequently (little compaction). Once it reaches good amount of size (100mb), we can transfer it to dedicated time series database engines through various mechanisms. I am struggling to find a fast, write heavy, memory optimal storage for this.
RocksDB seems to fit few boxes but there could be much better solution as we don't need deletes/range scans sort of operations.
Any suggestions?
Re: How RocksDB Works
#5Well written article--clarification on how Meta uses it though. It is not Tao it is ZippyDb:
https://engineering.fb.com/2021/08/06/core-data/zippydb/
Re: How RocksDB Works
#6I am looking for optimal storage engine(KV) which can store operational telemetry (temporarily) at source node. As we know, operational telemetry is generated frequently and need to merge similar operations frequently (little compaction). Once it reaches good amount of size (100mb), we can transfer it to dedicated time series database engines through various mechanisms. I am struggling to find a fast, write heavy, me…
Any reason you can't shove it into Kafka?
Re: How RocksDB Works
#7I am looking for optimal storage engine(KV) which can store operational telemetry (temporarily) at source node. As we know, operational telemetry is generated frequently and need to merge similar operations frequently (little compaction). Once it reaches good amount of size (100mb), we can transfer it to dedicated time series database engines through various mechanisms. I am struggling to find a fast, write heavy, me…
Deletes (tombstones) shouldn't really get in your way if you don't need them. Similarly, range scans just come for free from SSTables being sorted. Archiving RocksDB SSTable files can be a decent strategy.
One thing to pay attention to is if your telemetry data is indexed by timestamp (i.e. you're writing to the keyspace in order), the compaction of immutable SSTables layers could be wasteful? Although, the author's nice example of non-overlapping SSTables key ranges suggests there may be minimal write amplification here too.
Re: How RocksDB Works
#8I am looking for optimal storage engine(KV) which can store operational telemetry (temporarily) at source node. As we know, operational telemetry is generated frequently and need to merge similar operations frequently (little compaction). Once it reaches good amount of size (100mb), we can transfer it to dedicated time series database engines through various mechanisms. I am struggling to find a fast, write heavy, me…
Any reason you can't shove it into Kafka?
[deleted]
Re: How RocksDB Works
#9RocksDB is an amazing piece of engineering that deserve to be more known.
It is battle tested. It does one job and does it well.
I have used it in the past as a middleware database taking an average of 2-3k req/sec with over 400 GB of data stored. It works like a charm.
If I had a single reproach to do to it, it would be around the instrumentation. It is not that straightforward to get proper metrics and reporting of the internals.
Re: How RocksDB Works
#10Congratulations to the author for a remarkably clear, easy-to-follow, and informative post. Excellent technical writing!
Agreed. This is the best explanation of Log-Structured Merge Trees I've seen. I finally feel confident that I understand the concept.