61–70 of 71 posts
Earlier quoted context omitted.
I am lucky enough to have worked on all three of these systems (TAO, ZippyDB, and currently MySQL) so can shed some light here. Both MySQL and ZippyDB are datastores that use RocksDB under the hood, in a slightly different way and with different querying capabilities exposed to the end user. ZippyDB uses it exclusively, but MySQL uses both the traditional InnoDB and RocksDB (MyRocks). TAO is in memory graph database,…
/wave
Well written article--clarification on how Meta uses it though. It is not Tao it is ZippyDb: https://engineering.fb.com/2021/08/06/core-data/zippydb/
Earlier quoted context omitted.
Any reason you can't shove it into Kafka?
Too many network calls. Technically it's feasible, operationally it's expensive for Telemetry usecase. Ex: Imagine we are capturing API telemetry. If there are 1000 API calls per minute per node, then we will end up somewhere 1000*10 calls per minute to Kafka. It's not efficient.
Batching sends under the covers to reduce network round trips is all baked in.
This is also one of the things that most existing telemetry clients handle for you ie batching telemetry in memory and shipping it out on an interval, so there's a great deal of existing work you can draw from if not outright copy.
> To find a specific key, we could use binary search on the SST file blocks. I don't think it's possible except when both key and value are fixed size (which is not the case in the example shown).
How does flushing in a background process work if it says that it's an embeddable database that's in your application? It says there is no external process so how is there a background process that performs compaction and flushing?
It's a thread: https://artem.krylysov.com/blog/2023/04/19/how-rocksdb-works... > RocksDB runs a dedicated background thread that persists immutable memtables to disk. They are using "process" to mean "mechanism", something that happens, not a literal OS process. I agree that it's a bit confusing to use the word both ways.
Earlier quoted context omitted.
> That is you write fewer hardware blocks to the flash chips with a well tuned rocksdb. What would you consider a well-tuned rocksdb? My understanding is that, due to level-based compaction, there is always a decent amount of write amplification that is unavoidable -- i.e. for one modification to eventually end up in the bottommost level (e.g. L6), it would need to be (re-)written to disk 5 or 6 times. That's quite h…
All those writes happen in big blocks with other updates, so the number of writes per update is still relatively low. Compare that to e.g. postgres where a single update causes quite a few page writes that probably don't get batched up (add new row version in heap, update old row version in heap, maybe update indexes, vacuum old tuple in heap, maybe vacuum indexes).
If you write (say) 1000 small, contiguous rows in a single transaction, the B-tree wins. The B-tree writes a few full leaf pages and then updates just enough interior pages to point to them. The LSM-tree writes the rows to the first level, then later will write them again to the second level, then to the third layer, then... N times for N levels. The data also has to be read N-1 times.
At (say) 100,000 contiguous rows in a single transaction, the classic LSM-tree is still poor, but a modern LSM-tree with multiple files in the bulk level, with special handling of bulk loads to bypass the levels and write directly to the logical-middle in the bulk level, is similar I/O performance to the B-tree, so it catches up. I think RocksDB has this but you have to request it explicitly, rather than it detecting when to do so automatically. Probably with lower layout discontiguity if it works by writing to files or large contiguous zones. But if we are comparing LSM-tree with modification for bulk loads, we may as well compare with B-tree with a similar modification, which can also use a zone strategy to reduce discontiguity on bulk loads.
Well written article--clarification on how Meta uses it though. It is not Tao it is ZippyDb: https://engineering.fb.com/2021/08/06/core-data/zippydb/
Author here. > Well written article Thanks! > It is not Tao it is ZippyDb I don't work for Meta, so might have made a mistake. There is an old blog post[1] about Tao and there is a recent paper[2] mentioning that the graph database is powered by MyRocks, which runs on RocksDB. [1]: https://engineering.fb.com/2013/06/25/core-data/tao-the-powe... [2]: https://www.vldb.org/pvldb/vol13/p3217-matsunobu.pdf
Well written article--clarification on how Meta uses it though. It is not Tao it is ZippyDb: https://engineering.fb.com/2021/08/06/core-data/zippydb/
rocksdb is used in many more systems, but the storage layer behind TAO (UDB) is using MyRocks for quite a few years by now - migration was done in 2016 or so - https://engineering.fb.com/2016/08/31/core-data/myrocks-a-sp...
Earlier quoted context omitted.
And then there's the SSD controller, which yet again tries to maintain the illusion of efficiently supporting random I/O. For software that tries to get the best possible performance of the physical media, avoiding the OS and controller overhead, it's definitely important to maintain data in sequential blocks. In fact, NVMe ZNS [0], doesn't even allow you to do random writes. [0] - https://zonedstorage.io/docs/introd…
At this point, we are in logs three levels deep—we have a database with a log-structured merge tree, we have a journaling filesystem like ext4, and we have an SSD controller which uses log structures internally for wear leveling. I recommend the paper, “Don’t stack your Log on my Log”: https://www.usenix.org/system/files/conference/inflow14/infl... Basically, the beautiful thing about log structures is that they work…
I am looking for optimal storage engine(KV) which can store operational telemetry (temporarily) at source node. As we know, operational telemetry is generated frequently and need to merge similar operations frequently (little compaction). Once it reaches good amount of size (100mb), we can transfer it to dedicated time series database engines through various mechanisms. I am struggling to find a fast, write heavy, me…