11–20 of 49 posts
I find the "you don't want to shard" camp quite annoying. Of course, I don't want to shard! Who does?! It adds complexity, both implementation-wise and operational. But if you got 5 TB of data, that needs to be in a SSD drive, then please tell me how I can get that into 1 single physical database.
I find the "you don't want to shard" camp quite annoying. Of course, I don't want to shard! Who does?! It adds complexity, both implementation-wise and operational. But if you got 5 TB of data, that needs to be in a SSD drive, then please tell me how I can get that into 1 single physical database.
There is a new generation of relational databases that are native to multi-node operation, and don't require sharding. I'm speaking of tech like Google Spanner and my employer, FaunaDB. Now you don't have to shard. More info on how we accomplish distributed transactions. https://fauna.com/blog/distributed-consistency-at-scale-span...
Hey everyone, it's Ozgun. When I first wrote this blog post, it was much longer. Based on initial feedback, I edited out parts of it to keep the post focused. If you have any questions that aren't covered in the post, happy to answer them here!
Thats funny, because I was dying for it to be longer. I felt like the post was just an introduction. I'd love to see a part 2 with a more detailed description that touches more of the implementation of a sharding plan. For me a major question I have as I consider sharding is what my application code will look like. Let's say I have a query like: 'select products.name from vendor inner join products on vendor.id = pro…
If I understood your example query, your application serves vendors and each vendor has different products. Is that correct?
You can approach this sharding question in one of two ways.
1. Merge different product tables into one large product table and add a vendor column
2. Model product tables as "reference tables". This will replicate the product tables to all nodes in the cluster
Without knowing more about your application / table schemas, I'd recommend the first approach. I'd also be happy to chat more if you drop us a line.
Hey everyone, it's Ozgun. When I first wrote this blog post, it was much longer. Based on initial feedback, I edited out parts of it to keep the post focused. If you have any questions that aren't covered in the post, happy to answer them here!
Thats funny, because I was dying for it to be longer. I felt like the post was just an introduction. I'd love to see a part 2 with a more detailed description that touches more of the implementation of a sharding plan. For me a major question I have as I consider sharding is what my application code will look like. Let's say I have a query like: 'select products.name from vendor inner join products on vendor.id = pro…
If that's too much work, then an easy preliminary step is to add the abstraction layer in your application code. That gets you most of the benefits of a proxy for the purpose of having clean application logic, and makes it easy to switch over later, but is less powerful and feature complete.
I find the "you don't want to shard" camp quite annoying. Of course, I don't want to shard! Who does?! It adds complexity, both implementation-wise and operational. But if you got 5 TB of data, that needs to be in a SSD drive, then please tell me how I can get that into 1 single physical database.
Sorry if this is obvious, but why not use multiple drives?
I find the "you don't want to shard" camp quite annoying. Of course, I don't want to shard! Who does?! It adds complexity, both implementation-wise and operational. But if you got 5 TB of data, that needs to be in a SSD drive, then please tell me how I can get that into 1 single physical database.
Sorry if this is obvious, but why not use multiple drives?
or did you mean just use a large capacity RAID setup? that will probably work fine for a lot of situations but it can expensive and introduce more latency for certain types of operations (but that might not matter, depends on context).
I find the "you don't want to shard" camp quite annoying. Of course, I don't want to shard! Who does?! It adds complexity, both implementation-wise and operational. But if you got 5 TB of data, that needs to be in a SSD drive, then please tell me how I can get that into 1 single physical database.
There is a new generation of relational databases that are native to multi-node operation, and don't require sharding. I'm speaking of tech like Google Spanner and my employer, FaunaDB. Now you don't have to shard. More info on how we accomplish distributed transactions. https://fauna.com/blog/distributed-consistency-at-scale-span...
I laugh every time I read that word.
I find the "you don't want to shard" camp quite annoying. Of course, I don't want to shard! Who does?! It adds complexity, both implementation-wise and operational. But if you got 5 TB of data, that needs to be in a SSD drive, then please tell me how I can get that into 1 single physical database.
http://www.dell.com/en-us/work/shop/povw/poweredge-r930
Some of us do need to shard for sure though (I have multi petabyte data sets).