Principles of Sharding for Relational Databases
citusdata.com
Principles of Sharding for Relational Databases
1–10 of 49 posts
Re: Principles of Sharding for Relational Databases
#2Hey everyone, it's Ozgun. When I first wrote this blog post, it was much longer. Based on initial feedback, I edited out parts of it to keep the post focused.
If you have any questions that aren't covered in the post, happy to answer them here!
Re: Principles of Sharding for Relational Databases
#3Hey everyone, it's Ozgun. When I first wrote this blog post, it was much longer. Based on initial feedback, I edited out parts of it to keep the post focused. If you have any questions that aren't covered in the post, happy to answer them here!
Thanks for writing the post. Sharding is something I’m consudering at my current job.
How long do these sharding projects usually take? Do you know of any posts that break down the steps in more detail?
Re: Principles of Sharding for Relational Databases
#4I find the "you don't want to shard" camp quite annoying. Of course, I don't want to shard! Who does?! It adds complexity, both implementation-wise and operational.
But if you got 5 TB of data, that needs to be in a SSD drive, then please tell me how I can get that into 1 single physical database.
Re: Principles of Sharding for Relational Databases
#5I find the "you don't want to shard" camp quite annoying. Of course, I don't want to shard! Who does?! It adds complexity, both implementation-wise and operational. But if you got 5 TB of data, that needs to be in a SSD drive, then please tell me how I can get that into 1 single physical database.
You get one of these
https://petapixel.com/2015/08/15/samsung-16tb-ssd-is-the-wor...
Re: Principles of Sharding for Relational Databases
#6Hey everyone, it's Ozgun. When I first wrote this blog post, it was much longer. Based on initial feedback, I edited out parts of it to keep the post focused. If you have any questions that aren't covered in the post, happy to answer them here!
Thats funny, because I was dying for it to be longer. I felt like the post was just an introduction. I'd love to see a part 2 with a more detailed description that touches more of the implementation of a sharding plan.
For me a major question I have as I consider sharding is what my application code will look like. Let's say I have a query like:
'select products.name from vendor inner join products on vendor.id = products.vendor where vendor.location = "USA"'
If I shard such that there are many products table (1 per vendor), what would my query look like?
Re: Principles of Sharding for Relational Databases
#7I find the "you don't want to shard" camp quite annoying. Of course, I don't want to shard! Who does?! It adds complexity, both implementation-wise and operational. But if you got 5 TB of data, that needs to be in a SSD drive, then please tell me how I can get that into 1 single physical database.
There is a new generation of relational databases that are native to multi-node operation, and don't require sharding. I'm speaking of tech like Google Spanner and my employer, FaunaDB.
Now you don't have to shard. More info on how we accomplish distributed transactions. https://fauna.com/blog/distributed-consistency-at-scale-span...
Re: Principles of Sharding for Relational Databases
#8Hey everyone, it's Ozgun. When I first wrote this blog post, it was much longer. Based on initial feedback, I edited out parts of it to keep the post focused. If you have any questions that aren't covered in the post, happy to answer them here!
Thanks for writing the post. Sharding is something I’m consudering at my current job. How long do these sharding projects usually take? Do you know of any posts that break down the steps in more detail?
Timeframes for sharding projects vary quite a bit. If you have a B2B database, we find that sharding projects usually take between one to eight weeks of engineering (not clock) time. Most take two to three weeks.
A good way to tell is by looking at your database schema. If you have a dozen tables, you'll likely migrate with one week's of effort. If your database has 250+ tables, then you'll take about eight weeks.
When you're looking to shard your B2B database, you usually need to take the following steps:
1. Find tables that don't have a customer / tenant column, and add that column. Change primary and foreign key definitions to include this column. (You'll have a few tables that can't have a customer column, and these will be reference tables)
2. Backfill data to tables that don't didn't have customer_id / tenant_id
3. Change your application to talk to this new model. For Rails/Django, we have libraries available that make the app changes simpler (100-150 lines). For example: https://github.com/citusdata/activerecord-multi-tenant
4. Migrate your data over to a distributed database. Fortunately, online data migrations are starting to become possible with logical decoding in Postgres.
If you have a B2C app, these estimates and steps will be different. In particular, you'll need to figure out how many dimensions (columns) are central to your application. From there on, you'll need to separate out the data and shard each data group separately.
Re: Principles of Sharding for Relational Databases
#9Hey everyone, it's Ozgun. When I first wrote this blog post, it was much longer. Based on initial feedback, I edited out parts of it to keep the post focused. If you have any questions that aren't covered in the post, happy to answer them here!
Thats funny, because I was dying for it to be longer. I felt like the post was just an introduction. I'd love to see a part 2 with a more detailed description that touches more of the implementation of a sharding plan. For me a major question I have as I consider sharding is what my application code will look like. Let's say I have a query like: 'select products.name from vendor inner join products on vendor.id = pro…
Same here.
To me it read like just a basic introductory post to a longer series.
Re: Principles of Sharding for Relational Databases
#10I find the "you don't want to shard" camp quite annoying. Of course, I don't want to shard! Who does?! It adds complexity, both implementation-wise and operational. But if you got 5 TB of data, that needs to be in a SSD drive, then please tell me how I can get that into 1 single physical database.
[deleted]