Earlier quoted context omitted.
Interesting, they used to be the largest ZFS user. Hard to Google for it without getting AI slop on it, but apparently they built their own stack in 2019. Not sure I like their solution, "Meta-data is persisted in RocksDB databases using a proprietary KV store called QuarkDB." Unless QuarkDB has magically removed RocksDB's amazing ability to corrupt and lose data frequently, this whole thing sounds like a bad idea. A…
There's at least 3 copies of all the data and possibly more spread around the world at different Tier1 and Tier2 centers. The T1 and T2 centers are very likely using different filesystems and everything is checksummed pretty much at every major step (data generation, before and after any network transfers, before analysis, etc).
Re: CERN bids farewell to the LHC and enters Long Shutdown 3
#111The problem with RocksDB is Facebook uses it, and has a similar system that replicates data across the world... and still faces unrecoverable data loss because of RocksDB.