Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How does database sharding work?

planetscale.com

Post reply on HN
Print view View original

81–90 of 113 posts

Re: How does database sharding work?

#81

Post by samlambert »

billythemaniam wrote: post #77
samlambert wrote: post #73

Earlier quoted context omitted.

> (TiDB is what Vitess is trying to be) Not true. Vitess is far more proven.

I'm not sure what "more proven" exactly means. If you mean, it is not as well known and used in US? Yes that's true because the core developers are Chinese. If you mean, it hasn't been used at the same scale in production? False, just completely false. At my current job, we have reached a scale and use case that requires either manually sharding the database or using a distributed one. 20 TB including indexes, 10-20…

20TB of data and 10k/s of transactions is not a large cluster. There are Vitess cutters with petabytes of data and 100s of millions of users doing millions of QPS.

The largest Vitess cluster we know of runs on 10s of thousands of servers. So "False, just completely false" is dramatic.

Slack is powered by Vitess https://slack.engineering/scaling-datastores-at-slack-with-v... Twitter runs Vitess https://blog.twitter.com/engineering/en_us/topics/infrastruc...

I am sorry you had a buggy experience running Vitess yourself but you just wrong about the size of these production workloads.

Re: How does database sharding work?

#82

Post by crazygringo »

bsaul wrote: post #70
GeneralMayhem wrote: post #34

> I suppose the more fundamental question is: why are you not using a database that does sharding for you? Over the past few years the so-called “serverless” database has gotten a lot more traction. Starting with the infamous Spanner paper, many have been thinking about how running a distributed system should be native to the database itself, the foremost of which has been CockroachDB. You can even run cloud Spanner…

How does cockroach compares in terms of performance to manual sharded databases ? My intuition is that a properly sharded database will perform faster-or-same as a non-sharded one in all scenarios. Whereas automatically-sharded database will actually perform worst until you start reaching critical traffic that a single instance won't handle no matter what. Am i wrong ?

> My intuition is that a properly sharded database will perform faster-or-same as a non-sharded one in all scenarios.

No, because as soon as you have to JOIN across shards performance plummets. I mean, you can't JOIN across shards so you have to simulate that client-side which is very slow.

The secret to performant manual sharding is to figure out a way to minimize JOINs across shards, and that the ones you have to do are very small. Or to start replicating data which quickly becomes its own nightmare of consistency.

Re: How does database sharding work?

#83

Post by billythemaniam »

samlambert wrote: post #81
billythemaniam wrote: post #77

Earlier quoted context omitted.

I'm not sure what "more proven" exactly means. If you mean, it is not as well known and used in US? Yes that's true because the core developers are Chinese. If you mean, it hasn't been used at the same scale in production? False, just completely false. At my current job, we have reached a scale and use case that requires either manually sharding the database or using a distributed one. 20 TB including indexes, 10-20…

20TB of data and 10k/s of transactions is not a large cluster. There are Vitess cutters with petabytes of data and 100s of millions of users doing millions of QPS. The largest Vitess cluster we know of runs on 10s of thousands of servers. So "False, just completely false" is dramatic. Slack is powered by Vitess https://slack.engineering/scaling-datastores-at-slack-with-v... Twitter runs Vitess https://blog.twitter.co…

I didn't claim that I am dealing with a gigantic workload (total QPS is a lot higher btw). Rather I claimed we are at the point where a distributed database is a good idea if the business wants to continue to scale without major database trouble. I gave the specs so others can understand the parameters of the evaluation. If Vitess didn't perform well at that scale, I can't imagine the pain at Slack and Twitter scale. I don't envy them at all.

Re: How does database sharding work?

#84

Post by samlambert »

billythemaniam wrote: post #83
samlambert wrote: post #81

Earlier quoted context omitted.

20TB of data and 10k/s of transactions is not a large cluster. There are Vitess cutters with petabytes of data and 100s of millions of users doing millions of QPS. The largest Vitess cluster we know of runs on 10s of thousands of servers. So "False, just completely false" is dramatic. Slack is powered by Vitess https://slack.engineering/scaling-datastores-at-slack-with-v... Twitter runs Vitess https://blog.twitter.co…

I didn't claim that I am dealing with a gigantic workload (total QPS is a lot higher btw). Rather I claimed we are at the point where a distributed database is a good idea if the business wants to continue to scale without major database trouble. I gave the specs so others can understand the parameters of the evaluation. If Vitess didn't perform well at that scale, I can't imagine the pain at Slack and Twitter scale.…

https://twitter.com/stewart/status/1252385972014088194

They are pretty happy!

Different people find different things difficult. Don't feel bad.

Re: How does database sharding work?

#85

Post by billythemaniam »

samlambert wrote: post #84
billythemaniam wrote: post #83

Earlier quoted context omitted.

I didn't claim that I am dealing with a gigantic workload (total QPS is a lot higher btw). Rather I claimed we are at the point where a distributed database is a good idea if the business wants to continue to scale without major database trouble. I gave the specs so others can understand the parameters of the evaluation. If Vitess didn't perform well at that scale, I can't imagine the pain at Slack and Twitter scale.…

https://twitter.com/stewart/status/1252385972014088194 They are pretty happy! Different people find different things difficult. Don't feel bad.

I don't. I am very happy to not use Vitess and more than willing to recommend your competitors. Don't feel bad.

Re: How does database sharding work?

#86

Post by dalyons »

varsketiz wrote: post #42
GeneralMayhem wrote: post #34

> I suppose the more fundamental question is: why are you not using a database that does sharding for you? Over the past few years the so-called “serverless” database has gotten a lot more traction. Starting with the infamous Spanner paper, many have been thinking about how running a distributed system should be native to the database itself, the foremost of which has been CockroachDB. You can even run cloud Spanner…

> In a world where things like Dynamo/Cassandra or Spanner/Cockroach exist, manually-sharded DB solutions are pretty much entirely obsolete. Far from it. Plenty of companies that are now running mysql or postgress will shard manually when they will need to scale.

It can still be an obsolete practice even if some continue to choose to do it.

Re: How does database sharding work?

#87

Post by arjunnarayan »

bsaul wrote: post #70
GeneralMayhem wrote: post #34

> I suppose the more fundamental question is: why are you not using a database that does sharding for you? Over the past few years the so-called “serverless” database has gotten a lot more traction. Starting with the infamous Spanner paper, many have been thinking about how running a distributed system should be native to the database itself, the foremost of which has been CockroachDB. You can even run cloud Spanner…

How does cockroach compares in terms of performance to manual sharded databases ? My intuition is that a properly sharded database will perform faster-or-same as a non-sharded one in all scenarios. Whereas automatically-sharded database will actually perform worst until you start reaching critical traffic that a single instance won't handle no matter what. Am i wrong ?

[disclosure, former cockroachdb engineer]

you can get expected "single shard" performance in CockroachDB by manually splitting the shards (called "ranges" in CockroachDB) along the lines of the expected single shard queries (what you call a "properly shared database"). This is easy to do with a single SQL command. (This is what we do today; we use CockroachDB for strongly consistent metadata).

The difference between CockroachDB and a manually sharded database is that when you _do_ have to perform some cross-shard transactions (which you inevitably have to do at some point), in CockroachDB you can execute them (with a reasonable performance penalty) with strong consistency and 2PC between the shards, whereas in your manually sharded database... good luck! Hope you implement 2PC correctly.

Re: How does database sharding work?

#88

Post by pradeepchhetri »

samlambert wrote: post #81
billythemaniam wrote: post #77

Earlier quoted context omitted.

I'm not sure what "more proven" exactly means. If you mean, it is not as well known and used in US? Yes that's true because the core developers are Chinese. If you mean, it hasn't been used at the same scale in production? False, just completely false. At my current job, we have reached a scale and use case that requires either manually sharding the database or using a distributed one. 20 TB including indexes, 10-20…

20TB of data and 10k/s of transactions is not a large cluster. There are Vitess cutters with petabytes of data and 100s of millions of users doing millions of QPS. The largest Vitess cluster we know of runs on 10s of thousands of servers. So "False, just completely false" is dramatic. Slack is powered by Vitess https://slack.engineering/scaling-datastores-at-slack-with-v... Twitter runs Vitess https://blog.twitter.co…

> There are Vitess cutters with petabytes of data.

Is it GitHub? Where can I find more details about them? It will be interesting to read more about it.

Re: How does database sharding work?

#89

Post by BiteCode_dev »

GeneralMayhem wrote: post #34

> I suppose the more fundamental question is: why are you not using a database that does sharding for you? Over the past few years the so-called “serverless” database has gotten a lot more traction. Starting with the infamous Spanner paper, many have been thinking about how running a distributed system should be native to the database itself, the foremost of which has been CockroachDB. You can even run cloud Spanner…

If it's one more man-week of pain a year, but the spanner solution add a additional cost that is 10 times that, there is an incentive for the employer to keep the painful solution.

That and lock in, open source ecosystem and so on.

It's not a black and white situation.

Re: How does database sharding work?

#90

Post by game_the0ry »

GeneralMayhem wrote: post #34

> I suppose the more fundamental question is: why are you not using a database that does sharding for you? Over the past few years the so-called “serverless” database has gotten a lot more traction. Starting with the infamous Spanner paper, many have been thinking about how running a distributed system should be native to the database itself, the foremost of which has been CockroachDB. You can even run cloud Spanner…

I just have to say - thanks for your response.

I am a full stack dev but I end up doing a lot of front end and rarely have to touch SQL. Posts like yours help me to realize there's still a lot I do not know while still being informative. So thanks <3

Post reply on HN