Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Principles of Sharding for Relational Databases

citusdata.com

Post reply on HN
Print view View original

41–49 of 49 posts

Re: Principles of Sharding for Relational Databases

#41

Post by ozgune »

freels wrote: post #21
brianwawok wrote: post #12

Earlier quoted context omitted.

P.S. Google Spanner and FaunaDB both shard. They can call it something else. But unless every node has all data on it, it is sharded.

It is true that Spanner and FaunaDB partition a cluster's dataset across multiple nodes but it's handled transparently by the database. Whenever I've heard the term "sharding" it's usually in reference to the application-level sharding described in the article. Partitioning the dataset isn't really novel these days (Cassandra, Riak, Mongo et al do the same of course), but what is a significant difference is that both…

(Ozgun from Citus Data)

> Whenever I've heard the term "sharding" it's usually in reference to the application-level sharding described in the article.

I wanted to drop a quick clarification note here. In the article, I used the term "sharding" to refer to both application and database level sharding.

For anyone that's looking at sharding as an option for scaling, we're always happy to chat and help point you in the right direction. My email's ozgun @ citusdata.com

If you're looking databases that come with built-in sharding, I'd definitely check out Citus (then again, I'm biased): https://www.citusdata.com/

Re: Principles of Sharding for Relational Databases

#42

Post by qaq »

AznHisoka wrote: post #4

I find the "you don't want to shard" camp quite annoying. Of course, I don't want to shard! Who does?! It adds complexity, both implementation-wise and operational. But if you got 5 TB of data, that needs to be in a SSD drive, then please tell me how I can get that into 1 single physical database.

did you loose a 0 somewhere? Even something crappy like RDS will be able to help you with 5TB database. you could do more than 10 on i3.16xlarge or if you are ok running on own/leased hardware you could 20-30 for PG or 100+ for commercial DBs.

Re: Principles of Sharding for Relational Databases

#43

Post by kabdib »

AznHisoka wrote: post #4

I find the "you don't want to shard" camp quite annoying. Of course, I don't want to shard! Who does?! It adds complexity, both implementation-wise and operational. But if you got 5 TB of data, that needs to be in a SSD drive, then please tell me how I can get that into 1 single physical database.

Um, we run many databases of 20-30TB, some well over 100TB. We use SQL Server, and it just allocates more files. It's not zero touch, but with the right storage technology it's not bad, either.

Re: Principles of Sharding for Relational Databases

#44

Post by dcosson »

Interesting that sharding by customer for a sass business is the example of the best use of sharding. That can also go very wrong - what if you get a huge customer that's as big as everyone else combined? You're effectively maxed out at 2 shards.

Definitely depends on the workload, but often the "micro service" approach (whether or not it's a true micro service in its own runtime) of sharding just one type of data/small set of related tables that you can shard by a primary key or user id or something seems like the only reasonable option for sharding. If your data is becoming unwieldy there's often a bottleneck data set that's bigger than everything else so you don't necessarily have to share everything all at once.

Re: Principles of Sharding for Relational Databases

#45

Post by gjjrfcbugxbhf »

user5994461 wrote: post #36
sigstoat wrote: post #34

Earlier quoted context omitted.

if you're insisting on using the cheapest drives on amazon, you probably can't fit 5TB of data into a small room worth of computers. if you're a reasonable person, you buy 6 1TB samsung SSDs and stuff them into a single 2U case and you're done.

And you're gonna cry when the RAID0/JBOD fails and you lose all your data. Let's not pretend there is anything reasonable in this setup.

Presumably your data is (1) mirrored to another similar server ready to replace in case of failure and (2) also regularly backed up to two off-site locations...

Re: Principles of Sharding for Relational Databases

#46

Post by CamperBob2 »

dboreham wrote: post #19
megamindbrian wrote: post #13

I laugh every time I read that word.

Pottery, right?

The term comes from Ultima Online, one of the first graphical MMORPGs that gained mass-market acceptance. The rationale behind the architecture that Origin used for geographical load balancing was that the independent copy of the game world that resided on each server represented a "shard" of the shattered gem of Mondain the Wizard.

http://www.uoguide.com/Mondain

Re: Principles of Sharding for Relational Databases

#47

Post by zaptheimpaler »

freels wrote: post #21
brianwawok wrote: post #12

Earlier quoted context omitted.

P.S. Google Spanner and FaunaDB both shard. They can call it something else. But unless every node has all data on it, it is sharded.

It is true that Spanner and FaunaDB partition a cluster's dataset across multiple nodes but it's handled transparently by the database. Whenever I've heard the term "sharding" it's usually in reference to the application-level sharding described in the article. Partitioning the dataset isn't really novel these days (Cassandra, Riak, Mongo et al do the same of course), but what is a significant difference is that both…

Abstractions are leaky. As soon as you run into a query that runs fast when the data is on one shard and slow when its not, you now need to know about sharding.

Re: Principles of Sharding for Relational Databases

#48

Post by dboreham »

CamperBob2 wrote: post #46
dboreham wrote: post #19

Earlier quoted context omitted.

Pottery, right?

The term comes from Ultima Online, one of the first graphical MMORPGs that gained mass-market acceptance. The rationale behind the architecture that Origin used for geographical load balancing was that the independent copy of the game world that resided on each server represented a "shard" of the shattered gem of Mondain the Wizard. http://www.uoguide.com/Mondain

Well yes I knew that. But the point is that people had been distributing their load over a set of identical resources for decades before that, usually calling it "Partitioning" or some other word that wasn't "Shard". This is an often repeated patter in the field unfortunately : new people invent something that they don't realize already existed and make up a new name.

Re: Principles of Sharding for Relational Databases

#49

Post by CamperBob2 »

dboreham wrote: post #48
CamperBob2 wrote: post #46

Earlier quoted context omitted.

The term comes from Ultima Online, one of the first graphical MMORPGs that gained mass-market acceptance. The rationale behind the architecture that Origin used for geographical load balancing was that the independent copy of the game world that resided on each server represented a "shard" of the shattered gem of Mondain the Wizard. http://www.uoguide.com/Mondain

Well yes I knew that. But the point is that people had been distributing their load over a set of identical resources for decades before that, usually calling it "Partitioning" or some other word that wasn't "Shard". This is an often repeated patter in the field unfortunately : new people invent something that they don't realize already existed and make up a new name.

Sure, but they weren't trying to invent a technical term -- it just caught on for some reason.

It turns out that if your game backstory reads too much like an MCSE study guide, your game won't sell very well.

Post reply on HN