Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How does database sharding work?

planetscale.com

Post reply on HN
Print view View original

51–60 of 113 posts

Re: How does database sharding work?

#51

Post by yawgmoth »

dagss wrote: post #44
GeneralMayhem wrote: post #34

> I suppose the more fundamental question is: why are you not using a database that does sharding for you? Over the past few years the so-called “serverless” database has gotten a lot more traction. Starting with the infamous Spanner paper, many have been thinking about how running a distributed system should be native to the database itself, the foremost of which has been CockroachDB. You can even run cloud Spanner…

Would love to be able to use Spanner.. What do you suggest one does if one has to run on Azure though? PS Azure Cosmos is not a real product, it is a beta toy that Microsoft just has made expensive enough that people think it cannot possibly be as bad as it is..

Recently joined an Azure shop so new to Cosmos, but it's fairly abstract as a product. It seems to be a front for different database types like Mongo or Postgres or SQL Server with some additional features such as subscriptions.

Would be interested to hear your challenges.

Re: How does database sharding work?

#52

Post by yawgmoth »

revskill wrote: post #30
phamilton wrote: post #18

A favorite resource: https://learn.microsoft.com/en-us/azure/architecture/pattern... Microsofts Azure Cloud Patterns is some of the best documentation out there. It's not Azure centric. It focuses on why you may want to do something and describes the techniques that are commonly used to solve it. It's pretty great.

The main issue is, i can't stand C# or OOP syntax to illustrate patterns. Why class here ? For God sake, please use simple functions to prove the points.

They might supply Python examples if you ask. A lot of Microsoft documentation has Python as well as C# these days.

Re: How does database sharding work?

#53

Post by KaiserPro »

GeneralMayhem wrote: post #34

> I suppose the more fundamental question is: why are you not using a database that does sharding for you? Over the past few years the so-called “serverless” database has gotten a lot more traction. Starting with the infamous Spanner paper, many have been thinking about how running a distributed system should be native to the database itself, the foremost of which has been CockroachDB. You can even run cloud Spanner…

> In a world where things like Dynamo/Cassandra or Spanner/Cockroach exist, manually-sharded DB solutions are pretty much entirely obsolete.

Not really. Cassandra is a write optimised, slow, network and IO heavy beast thats a pain in the arse to administer. We replaced a 6 node m4.4xlarge with a single db.m4.2xlarge. on postgres.

You need to pick your DB to match your data, not the otherway around.

Re: How does database sharding work?

#54

Post by elsadek »

I haven't heard again about DB sharding technique since prior 2010. This technique was used to separate, in a given table, most involved fields from those less used by using different server for each shard. With the rise of memory database like Redis, sharding was abondonned.

Re: How does database sharding work?

#55

Post by rvba »

> For Amazon, that means the orders table and the products table containing the products in the orders table need to be physically colocated.

Isnt this some wrong simplification? Product number can stay the same, but be a different thing over time. Product 12345 can be a book on first January 2023, while a comic book on first March 2023 . The idea is that product versions can change over time. (What is mostly abused by scammers who farm fake reviews and bait and switch products).

So probably they have some product history table, or save the product information in the order.

Also sharding in big data environment is when your data does not fit into Excel anymore, so you have 20 Acess files (1GB database limit in Access).

Re: How does database sharding work?

#56

Post by vinay_ys »

dagss wrote: post #44
GeneralMayhem wrote: post #34

> I suppose the more fundamental question is: why are you not using a database that does sharding for you? Over the past few years the so-called “serverless” database has gotten a lot more traction. Starting with the infamous Spanner paper, many have been thinking about how running a distributed system should be native to the database itself, the foremost of which has been CockroachDB. You can even run cloud Spanner…

Would love to be able to use Spanner.. What do you suggest one does if one has to run on Azure though? PS Azure Cosmos is not a real product, it is a beta toy that Microsoft just has made expensive enough that people think it cannot possibly be as bad as it is..

You could consider one of these three –

1. yugabytedb (postgres compatible)

2. cockroachdb (postgres compatible)

3. tidb (mysql compatible)

You could self-host these on azure or subscribe to each of their managed service offering.

Of course, sticker price for managed will be more expensive than self-hosted enterprise product which itself will be more expensive than the self-hosted, community-supported, open-source option (which seems to be a real option with YugabyteDB).

Re: How does database sharding work?

#57

Post by dalyons »

alexvoda wrote: post #50
branko_d wrote: post #47

Earlier quoted context omitted.

> Consider a beefy AF bare metal SQL server, see how far that gets you. Quite far, possibly to millions of users. Case in point: StackOverflow runs on SQL Server.

Actually it can vary "from not that" far to "all the way" (StackOverflow). It depends very much on the schema and the type of workload. But in general people really underestimate how far an RDBMS can take you.

Yes and no - I think people also underestimate how miserable running a single giant RDBMS can be. Everything gets hard and dangerous - backup/restores, upgrades, online migrations & schema changes. Adding the wrong index can easily brings the whole thing down. They are complicated beasts that get very fragile at the large end (size+throughput)

Re: How does database sharding work?

#58

Post by dalyons »

dikei wrote: post #49
paulryanrogers wrote: post #6

Earlier quoted context omitted.

What's the alternative as things get too big? Sharding by date? By client? How do you prevent hotspots?

Pass a certain point, you ought to think about whether to keep using a RDBMS as a K-V store or switch to a real distributed K-V store like Cassandra, ScyllaDB, DynamoDB and the like About hot spots, it has always been an issue with K-V stores, and the only real solution is a good key design, though there are some tricks: * Use a uniformly distributed but deterministic key prefix. For example, instead of using raw use…

Or just change to user_uuid (or similar non int pk)

Re: How does database sharding work?

#59

Post by sgtfrankieboy »

vinay_ys wrote: post #56
dagss wrote: post #44

Earlier quoted context omitted.

Would love to be able to use Spanner.. What do you suggest one does if one has to run on Azure though? PS Azure Cosmos is not a real product, it is a beta toy that Microsoft just has made expensive enough that people think it cannot possibly be as bad as it is..

You could consider one of these three – 1. yugabytedb (postgres compatible) 2. cockroachdb (postgres compatible) 3. tidb (mysql compatible) You could self-host these on azure or subscribe to each of their managed service offering. Of course, sticker price for managed will be more expensive than self-hosted enterprise product which itself will be more expensive than the self-hosted, community-supported, open-source op…

It's also a real option with CockroachDB, unless you need regional sharding. Which a lot don't actually need anyways.

Re: How does database sharding work?

#60

Post by billythemaniam »

KaiserPro wrote: post #53
GeneralMayhem wrote: post #34

> I suppose the more fundamental question is: why are you not using a database that does sharding for you? Over the past few years the so-called “serverless” database has gotten a lot more traction. Starting with the infamous Spanner paper, many have been thinking about how running a distributed system should be native to the database itself, the foremost of which has been CockroachDB. You can even run cloud Spanner…

> In a world where things like Dynamo/Cassandra or Spanner/Cockroach exist, manually-sharded DB solutions are pretty much entirely obsolete. Not really. Cassandra is a write optimised, slow, network and IO heavy beast thats a pain in the arse to administer. We replaced a 6 node m4.4xlarge with a single db.m4.2xlarge. on postgres. You need to pick your DB to match your data, not the otherway around.

Spanner and Cassandra really shouldn't be in the same sentence. They are optimized for very different use cases. The "obsolete" part of that quote does apply to Spanner, TiDB, and CockroachDB in my experience. I haven't used Yugabyte, but the other sharded databases, including Vitess (TiDB is what Vitess is trying to be), actually make life harder.
Post reply on HN