Live data from Hacker News

Principles of Sharding for Relational Databases

citusdata.com

31–40 of 49 posts

Re: Principles of Sharding for Relational Databases

#31

I find the "you don't want to shard" camp quite annoying. Of course, I don't want to shard! Who does?! It adds complexity, both implementation-wise and operational. But if you got 5 TB of data, that needs to be in a SSD drive, then please tell me how I can get that into 1 single physical database.

http://www.fixstars.com/en/ssd/ ? if all you need is a lot of data in a single database, there's basically nothing except for money between you and your goal. JBODs full of SSDs coming into a single machine via SAS will get you into petabytes, just with commodity hardware you can order from amazon. i'm expect IBM could sell you a mainframe that'll do it for whatever capacity you care to name.

The thing is that 5TB of company data cannot reasonably be kept in JBOD on the cheapest drives you could find on Amazon.

Re: Principles of Sharding for Relational Databases

#32

Earlier quoted context omitted.

http://www.fixstars.com/en/ssd/ ? if all you need is a lot of data in a single database, there's basically nothing except for money between you and your goal. JBODs full of SSDs coming into a single machine via SAS will get you into petabytes, just with commodity hardware you can order from amazon. i'm expect IBM could sell you a mainframe that'll do it for whatever capacity you care to name.

The thing is that 5TB of company data cannot reasonably be kept in JBOD on the cheapest drives you could find on Amazon.

Why do you want the cheapest drives? Do you value your application data?

Re: Principles of Sharding for Relational Databases

#33
post #8

Earlier quoted context omitted.

Timeframes for sharding projects vary quite a bit. If you have a B2B database, we find that sharding projects usually take between one to eight weeks of engineering (not clock) time. Most take two to three weeks. A good way to tell is by looking at your database schema. If you have a dozen tables, you'll likely migrate with one week's of effort. If your database has 250+ tables, then you'll take about eight weeks. Wh…

I think you're understating how tough it can be. There are applications that * are mature and complex * with 100s of tables * serving millions of users * have to be broken into multiple micro-services * have developer resource constraints So you're easily looking at a 1-2 year project, not 1-8 weeks. You've also ignored some of the complexities, such as resharding (moving data between shards), which may significantly…

Also, when architecturing for shards, you must take into account availability.

Having several shards can lower the availability of your application if it cannot handle the absence of a shard.

For example if you have 99.9% availability on your individual DBs, and if you split it up into 10 shards, availability will drop to 99% (8 hours VS 3 days of downtime a year).

To handle that, you need to add replication and automatic fail-overs, adding even more complexity.

Re: Principles of Sharding for Relational Databases

#34

Earlier quoted context omitted.

http://www.fixstars.com/en/ssd/ ? if all you need is a lot of data in a single database, there's basically nothing except for money between you and your goal. JBODs full of SSDs coming into a single machine via SAS will get you into petabytes, just with commodity hardware you can order from amazon. i'm expect IBM could sell you a mainframe that'll do it for whatever capacity you care to name.

The thing is that 5TB of company data cannot reasonably be kept in JBOD on the cheapest drives you could find on Amazon.

if you're insisting on using the cheapest drives on amazon, you probably can't fit 5TB of data into a small room worth of computers.

if you're a reasonable person, you buy 6 1TB samsung SSDs and stuff them into a single 2U case and you're done.

Re: Principles of Sharding for Relational Databases

#35
post #29

Earlier quoted context omitted.

You can _easily_ buy a box with 60+TB of SSD... http://www.dell.com/en-us/work/shop/povw/poweredge-r930 Some of us do need to shard for sure though (I have multi petabyte data sets).

Storing 60+TB of data is different than searching and doing complex computation on 60TB of data. Also, operations on a such huge data set can be really painful. Think how to backup a DB like that safely, or how to update the engine. Some slides (little old, 2014) about a huge postgres instance serving as a backend for leboncoin.fr (main classified advertising website in France). https://fr.slideshare.net/jlb666/pgday…

Absolutely. Queries on 60TB of data can certainly merit more than one box. Hell, queries on 1TB of data can merit more than one box.

I have a workload that runs close to 1.2million TPS for hours at a time and needs less than 100 millisecond response times at the 99th percentile. That uses more than 1 box and sits (replicated) in RAM.

However, 5TB of data really _isn't_ that much on modern SSD's. You can fit a sizable chunk of that in RAM on a decent server, so you probably _don't_ need more than one box.

I have 5TB of data that needs to sit on an SSD is, to be honest, a really poor performance metric. If you are genuinely specing out hardware and a database a better statement would be:

"I have 5TB of relational data, with a pareto distribution for access, at a peak of 100K TPS". Then we can start talking about what solves the problem.

Re: Principles of Sharding for Relational Databases

#36

Earlier quoted context omitted.

The thing is that 5TB of company data cannot reasonably be kept in JBOD on the cheapest drives you could find on Amazon.

if you're insisting on using the cheapest drives on amazon, you probably can't fit 5TB of data into a small room worth of computers. if you're a reasonable person, you buy 6 1TB samsung SSDs and stuff them into a single 2U case and you're done.

And you're gonna cry when the RAID0/JBOD fails and you lose all your data.

Let's not pretend there is anything reasonable in this setup.

Re: Principles of Sharding for Relational Databases

#37
post #8

Earlier quoted context omitted.

Timeframes for sharding projects vary quite a bit. If you have a B2B database, we find that sharding projects usually take between one to eight weeks of engineering (not clock) time. Most take two to three weeks. A good way to tell is by looking at your database schema. If you have a dozen tables, you'll likely migrate with one week's of effort. If your database has 250+ tables, then you'll take about eight weeks. Wh…

I think you're understating how tough it can be. There are applications that * are mature and complex * with 100s of tables * serving millions of users * have to be broken into multiple micro-services * have developer resource constraints So you're easily looking at a 1-2 year project, not 1-8 weeks. You've also ignored some of the complexities, such as resharding (moving data between shards), which may significantly…

I used to think similarly several years ago. I now think differently for the following reasons:

* Citus and other technologies can now provide features that do a lot of the heavy lifting. Some examples are resharding, shard rebalancing, and the high availability features mentioned below.

* My estimates are for B2B (multi-tenant) apps. For those apps, we found that the steps you need to take in re-modeling your data and changing your app are fairly similar. At Citus, we used to shy away when we saw 200-300 tables. These days, complex apps and schemas have become commonplace.

* We saw dozens of complex B2B databases migrate in similar time frames. Yes, some took longer - I'm in the tech business and always an optimist. :)

I also don't want to generalize without knowing more about your setup. If you drop me a line at ozgun @ citusdata.com, happy to chat more!

Re: Principles of Sharding for Relational Databases

#39
post #8

Earlier quoted context omitted.

Timeframes for sharding projects vary quite a bit. If you have a B2B database, we find that sharding projects usually take between one to eight weeks of engineering (not clock) time. Most take two to three weeks. A good way to tell is by looking at your database schema. If you have a dozen tables, you'll likely migrate with one week's of effort. If your database has 250+ tables, then you'll take about eight weeks. Wh…

I think you're understating how tough it can be. There are applications that * are mature and complex * with 100s of tables * serving millions of users * have to be broken into multiple micro-services * have developer resource constraints So you're easily looking at a 1-2 year project, not 1-8 weeks. You've also ignored some of the complexities, such as resharding (moving data between shards), which may significantly…

At Prosperworks we offer a CRM which integrates closely with G Suite applications like Gmail and Calendar.

We consider our app to be maturing if not mature. It is certainly complex - we integrate with dozens of partners and external APIs. We have 80 tables and 300k LoC of Rails code which is runs on several TB of PostgresSQL data. We have not broken our app into multiple micro-services. Like everybody, we always feel that our developer resources are constrained.

Our data model is very similar to the CRM example in Ozgun's article: _mostly_ we have a master customer table and a wide halo of tables which are directly or transitively associated with customers. We called this the "company sharding domain". Since we allow one user to be associated with multiple accounts, we shard our user table independently: there is a smaller halo of tables in the "user sharding domain". And we have a handful of global tables for content and feature configuration in the "unsharded domain".

We kicked off our migration project from unsharded Postgres to sharded CitusCloud in early Q4 2016. We had one dev work on it solid for one quarter updating our code to be shard-ready. Then another 1.5 devs joined for a month in the final build up to the physical migration. We migrated in late Feb 2017, then consumed perhaps another 3 dev-months on follow-up activities like stamping out some distributed queries which we had unwisely neglected and updating our internal process for our brave new world.

Two years ago at another company I was tech lead on a migration of two much larger Mongo collections to sharded Mongo. That was a larger PHP application which was organized partly into microservices. That effort had a similar cost: as I recall I spent one quarter and two other devs spent about one month, and there were some post-migration follow-up costs as well.

I am confident that real world applications of significant complexity can be migrated from unsharded to sharded storage with a level of effort less than 1 year. I admit that 8 weeks feels fast but I'm sure I could have done it if we had been willing to tie up more devs.

Why were these efforts easier than 2 years? Because we didn't have to build the sharding solution itself - those came off the shelf from some great partners (shout outs to CitusData and mLabs). We just had to update our applications to be shard-smart and coordinate a sometimes complicated physical migration, derisking, and cutover process.

That said, I can imagine the work growing slowly but linearly in the number of tables, and quickly but linearly in the number of micro-services.

Re: Principles of Sharding for Relational Databases

#40
post #6
post #2

Hey everyone, it's Ozgun. When I first wrote this blog post, it was much longer. Based on initial feedback, I edited out parts of it to keep the post focused. If you have any questions that aren't covered in the post, happy to answer them here!

Thats funny, because I was dying for it to be longer. I felt like the post was just an introduction. I'd love to see a part 2 with a more detailed description that touches more of the implementation of a sharding plan. For me a major question I have as I consider sharding is what my application code will look like. Let's say I have a query like: 'select products.name from vendor inner join products on vendor.id = pro…

Reading through your comment again, I realize I completely missed the mark on your question.

If you use Citus, you don't have to make any changes in your application. You just need to remodel your data and define your tables' sharding column(s). Citus will take care of the rest. [1]

In other words, your app thinks it's talking to Postgres. Behind the covers, Citus shards the tables, routes and parallelizes queries. Citus also provides transactions, joins, and foreign keys in a distributed environment.

[1] Almost. Over the past two years, we've been adding features to make app integration seamless. With our upcoming release, we'll get there: https://github.com/citusdata/citus/issues/595

Post reply on HN