Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How does database sharding work?

planetscale.com

Post reply on HN
Print view View original

61–70 of 113 posts

Re: How does database sharding work?

#61

Post by dikei »

dalyons wrote: post #58
dikei wrote: post #49

Earlier quoted context omitted.

Pass a certain point, you ought to think about whether to keep using a RDBMS as a K-V store or switch to a real distributed K-V store like Cassandra, ScyllaDB, DynamoDB and the like About hot spots, it has always been an issue with K-V stores, and the only real solution is a good key design, though there are some tricks: * Use a uniformly distributed but deterministic key prefix. For example, instead of using raw use…

Or just change to user_uuid (or similar non int pk)

Sure, unless your sharding/partitioning keys need to be derivable from the business keys.

Re: How does database sharding work?

#62

Post by 411111111111111 »

vinay_ys wrote: post #56
dagss wrote: post #44

Earlier quoted context omitted.

Would love to be able to use Spanner.. What do you suggest one does if one has to run on Azure though? PS Azure Cosmos is not a real product, it is a beta toy that Microsoft just has made expensive enough that people think it cannot possibly be as bad as it is..

You could consider one of these three – 1. yugabytedb (postgres compatible) 2. cockroachdb (postgres compatible) 3. tidb (mysql compatible) You could self-host these on azure or subscribe to each of their managed service offering. Of course, sticker price for managed will be more expensive than self-hosted enterprise product which itself will be more expensive than the self-hosted, community-supported, open-source op…

It always triggers me to read "postgres compatible" when they've entirely skipped features such as database triggers

They've just selected a few features from postgres which they're compatible in.

Re: How does database sharding work?

#63

Post by cj »

avinassh wrote: post #46
dagss wrote: post #44

Earlier quoted context omitted.

Would love to be able to use Spanner.. What do you suggest one does if one has to run on Azure though? PS Azure Cosmos is not a real product, it is a beta toy that Microsoft just has made expensive enough that people think it cannot possibly be as bad as it is..

> PS Azure Cosmos is not a real product, it is a beta toy that Microsoft just has made expensive enough that people think it cannot possibly be as bad as it is.. what kind of problems you ran into with Cosmos?

It’s supposed to be MongoDB compatible (if I’m not mistaken).

Last I checked, that “compatibility” run at least a couple major version behind the real MongoDB.

To be fair, the same is probably also true with AWS’s DocumentDB

Re: How does database sharding work?

#64

Post by anecdotal1 »

411111111111111 wrote: post #62
vinay_ys wrote: post #56

Earlier quoted context omitted.

You could consider one of these three – 1. yugabytedb (postgres compatible) 2. cockroachdb (postgres compatible) 3. tidb (mysql compatible) You could self-host these on azure or subscribe to each of their managed service offering. Of course, sticker price for managed will be more expensive than self-hosted enterprise product which itself will be more expensive than the self-hosted, community-supported, open-source op…

It always triggers me to read "postgres compatible" when they've entirely skipped features such as database triggers They've just selected a few features from postgres which they're compatible in.

[dead]

Re: How does database sharding work?

#65

Post by bpicolo »

grogers wrote: post #40
GeneralMayhem wrote: post #34

> I suppose the more fundamental question is: why are you not using a database that does sharding for you? Over the past few years the so-called “serverless” database has gotten a lot more traction. Starting with the infamous Spanner paper, many have been thinking about how running a distributed system should be native to the database itself, the foremost of which has been CockroachDB. You can even run cloud Spanner…

All the "auto-sharding" DBs have their own quirks, especially around hot key throughput. You often end up having to add "sharding bits" to the beginning of your keys to get enough throughput. The size of one partition is usually tiny compared to one partition of an RDBMS too. So it ends up being not nearly the panacea that it would seem to be.

Software side sharding keys seem significantly simpler than managing sharding infrastructure. That's the beauty of memcached right?

Re: How does database sharding work?

#66

Post by bpicolo »

anamexis wrote: post #20
setr wrote: post #13

Earlier quoted context omitted.

> You can't without a quorum[1] and even that does not guarantee success. Isn’t split-brain largely resolved by the rules: 1. Must have a strict majority of the total group agree to commit 2. Total should be an odd number

Only if there can only be one simultaneous split.

If all nodes lose traffic to each other (which is plausible with bad config changes), good luck!

Re: How does database sharding work?

#67

Post by alexvoda »

411111111111111 wrote: post #62
vinay_ys wrote: post #56

Earlier quoted context omitted.

You could consider one of these three – 1. yugabytedb (postgres compatible) 2. cockroachdb (postgres compatible) 3. tidb (mysql compatible) You could self-host these on azure or subscribe to each of their managed service offering. Of course, sticker price for managed will be more expensive than self-hosted enterprise product which itself will be more expensive than the self-hosted, community-supported, open-source op…

It always triggers me to read "postgres compatible" when they've entirely skipped features such as database triggers They've just selected a few features from postgres which they're compatible in.

I agree. I guess what they mean is you can use the existing adapter for your language.

The PostgreSQL organisation should create some trademarked badges to properly label the levels of compatibility other products offer.

Re: How does database sharding work?

#68

Post by bsaul »

mamcx wrote: post #29

And the best sharding? Doing multi-tenant. Sharding at table level is very complex and expensive. Fully give a single DB per tenant is very practical, and the reasons to do sharding (like reporting) is where you do the other fancy things (like ship events in to kafka, etc, etc). Also, most issues of scalability are dominated by a few tenants that consume most resources, and distribute the loads is more easy per-tenan…

I've been doing that as much as possible. however you're still left with availability issues, such as replicating / failover management etc. Which really are orthogonal issues to tenancy.

How do you manage that ?

Re: How does database sharding work?

#69

Post by vlovich123 »

Is the hash based approach scalable? A list operation would require contacting every single shard which seems outrageously expensive. It seems like you want some locality but I’m not aware of locality-preserving hash functions (seems like a contradiction but maybe people here have encountered one?)

Re: How does database sharding work?

#70

Post by bsaul »

GeneralMayhem wrote: post #34

> I suppose the more fundamental question is: why are you not using a database that does sharding for you? Over the past few years the so-called “serverless” database has gotten a lot more traction. Starting with the infamous Spanner paper, many have been thinking about how running a distributed system should be native to the database itself, the foremost of which has been CockroachDB. You can even run cloud Spanner…

How does cockroach compares in terms of performance to manual sharded databases ?

My intuition is that a properly sharded database will perform faster-or-same as a non-sharded one in all scenarios. Whereas automatically-sharded database will actually perform worst until you start reaching critical traffic that a single instance won't handle no matter what.

Am i wrong ?

Post reply on HN