I run a file sharing / content delivery platform called pixeldrain: https://pixeldrain.com The system serves 4 PB of data to 60 million visitors per month. I have served 30 PB and 700 million file views since I started tracking usage somewhere in 2018. I'll go from front to back: - Most of the frontend is plain HTML, CSS and JS. I have started transitioning some pages to Svelte. I like this framework for its speed an…
Curious to learn more about CockroachDB's problems you had. They market themselves as a turn-key solution to globally distributed databases, it's worrying that you had a problem with that very use case. Thanks for the additional details!
Cockroachdb slices up its tables in 16 MiB shards. These shards are distributed over three nodes, one master and two slaves. The master gets chosen with the raft protocol. The master is the only server which can write to the shard. For a write to happen all three replicas need to be synchronized, they do this by locking the entire shard whenever a query comes in that writes to a row in the shard.
The problem is that if the latency between the master and the slave node is too large, you are severely limited in how many queries can be executed. For example if you have one shard in the US and one in Europe there might be a latency of 150 - 200ms between them. This means you can only execute 5 queries per second. Any more requests that come in are queued. After a while the queue gets so long that incoming queries have to be dropped.
This implementation is great for consistency, but if you frequently need to run update queries on your rows it's not the right solution. You need eventual consistency like Scylla and Cassandra provide.