Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How does database sharding work?

planetscale.com

Post reply on HN
Print view View original

11–20 of 113 posts

Re: How does database sharding work?

#11

Post by AdieuToLogic »

paulryanrogers wrote: post #6
dikei wrote: post #3

One thing I notice is the over-usage of sharding, especially hash-based, might turn your Relational Database into just another key-value store, with consistency constraints moving into application code, and you lose many advantages of a traditional RDBMS

What's the alternative as things get too big? Sharding by date? By client? How do you prevent hotspots?

Pick a a stable, guaranteed-to-exist, shard key (composite or atomic properties) and use that. If composite, order properties used in most-to-least distinct value distribution.

Re: How does database sharding work?

#12

Post by brightball »

samsquire wrote: post #4

Thanks for this article planetscale. I've been enamoured with sharding recently but more for multithreaded performance. I want multimaster postgres. I started trying to write a postgres synchronizer, by sorting every data by row and column and hashing the data of every column and row, then doing a rolling hash of the data. In theory, two databases can synchronize by sending the final hash of their data and then doing…

You need Citus

Re: How does database sharding work?

#13

Post by setr »

AdieuToLogic wrote: post #8
samsquire wrote: post #4

Thanks for this article planetscale. I've been enamoured with sharding recently but more for multithreaded performance. I want multimaster postgres. I started trying to write a postgres synchronizer, by sorting every data by row and column and hashing the data of every column and row, then doing a rolling hash of the data. In theory, two databases can synchronize by sending the final hash of their data and then doing…

> One problem I've not worked out is how to decide which database is the winning database without having to change application queries. I believe this is a case of the "Two Generals' Problem"[0], which implies that there is no provably correct solution to achieve this. > If you synchronize two multimaster Postgres databases that have had independent writes to different sections, how do you identify which database is…

> You can't without a quorum[1] and even that does not guarantee success.

Isn’t split-brain largely resolved by the rules:

1. Must have a strict majority of the total group agree to commit

2. Total should be an odd number

Re: How does database sharding work?

#14

Post by dilyevsky »

AdieuToLogic wrote: post #11
paulryanrogers wrote: post #6

Earlier quoted context omitted.

What's the alternative as things get too big? Sharding by date? By client? How do you prevent hotspots?

Pick a a stable, guaranteed-to-exist, shard key (composite or atomic properties) and use that. If composite, order properties used in most-to-least distinct value distribution.

Not so easy at scale because there’s additional requirement for it to also not be susceptible to hotspotting

Re: How does database sharding work?

#15

Post by charcircuit »

dikei wrote: post #3

One thing I notice is the over-usage of sharding, especially hash-based, might turn your Relational Database into just another key-value store, with consistency constraints moving into application code, and you lose many advantages of a traditional RDBMS

Yes, this is where caching layers come in. Now your cache acts as the RDBMS.

Re: How does database sharding work?

#16

Post by dehrmann »

dikei wrote: post #3

One thing I notice is the over-usage of sharding, especially hash-based, might turn your Relational Database into just another key-value store, with consistency constraints moving into application code, and you lose many advantages of a traditional RDBMS

Online use cases always(?) scale out to key-value stores. Offline use cases almost always scale to distributed, date-partitioned columnar stores.

Re: How does database sharding work?

#17

Post by preseinger »

AdieuToLogic wrote: post #10
preseinger wrote: post #7

Earlier quoted context omitted.

the only real answer here is to shard by customer optimistically, you can try to shard by read use case, but that's never gonna be stable over time if you need a true multi-tenant system you can only shard by individual entity and move all of the composition logic to the next layer up, there's no way to cheat

> the only real answer here is to shard by customer No. Pick a a stable, guaranteed-to-exist, shard key (composite or atomic properties) and use that. If composite, order properties used in most-to-least distinct value distribution. > if you need a true multi-tenant system you can only shard by individual entity and move all of the composition logic to the next layer up, there's no way to cheat This is incorrect. Sha…

> No. Pick a a stable, guaranteed-to-exist, shard key (composite or atomic properties) and use that. If composite, order properties used in most-to-least distinct value distribution.

what??

if you shard users by user ID and orders by order ID, then a query that joins a bunch of user orders in the same tenant namespace will spread across multiple user shards and multiple order shards

value distribution doesn't really have any impact here

(shard keys are also guaranteed to exist by definition, not clear what you mean by that)

if you don't care that specific queries cross sharding boundaries, okay, then no problem, but in that case sharding is not solving the problem that we are talking about here

> Sharding and multi-tenancy are orthogonal concepts.

sharding and multi-tenancy are only orthogonal if you don't care that a single tenant can have information on multiple shards

Re: How does database sharding work?

#18

Post by phamilton »

A favorite resource: https://learn.microsoft.com/en-us/azure/architecture/pattern...

Microsofts Azure Cloud Patterns is some of the best documentation out there. It's not Azure centric. It focuses on why you may want to do something and describes the techniques that are commonly used to solve it. It's pretty great.

Re: How does database sharding work?

#19

Post by beebmam »

phamilton wrote: post #18

A favorite resource: https://learn.microsoft.com/en-us/azure/architecture/pattern... Microsofts Azure Cloud Patterns is some of the best documentation out there. It's not Azure centric. It focuses on why you may want to do something and describes the techniques that are commonly used to solve it. It's pretty great.

I also couldn't recommend this higher. If you are designing a new application, read these docs. You will almost certainly learn something deeply useful that you'll carry me with you for the rest of your career.

Re: How does database sharding work?

#20

Post by anamexis »

setr wrote: post #13
AdieuToLogic wrote: post #8

Earlier quoted context omitted.

> One problem I've not worked out is how to decide which database is the winning database without having to change application queries. I believe this is a case of the "Two Generals' Problem"[0], which implies that there is no provably correct solution to achieve this. > If you synchronize two multimaster Postgres databases that have had independent writes to different sections, how do you identify which database is…

> You can't without a quorum[1] and even that does not guarantee success. Isn’t split-brain largely resolved by the rules: 1. Must have a strict majority of the total group agree to commit 2. Total should be an odd number

Only if there can only be one simultaneous split.
Post reply on HN