Is this the guy that's going to reddit?
Yes
Sharding Pinterest: How we scaled our MySQL fleet
11–20 of 87 posts
Re: Sharding Pinterest: How we scaled our MySQL fleet
#12This looks like a big hack to compensate for using the wrong tool. Cassandra would have been a better solution IMO. With Cassandra, you can set replication factors, speed up the writes, and automatically shard the data without having to manage your own "mapping tables".
Cassandra was immature when this work was started (late 2011). The team was also much more familiar with the ins and outs of operating mysql.
Technical debt is not an excuse to use the wrong tool. This is how you end up having a hammer.
Cassandra was mature by late 2011 and we were using it in production back then (billions of records).
Re: Sharding Pinterest: How we scaled our MySQL fleet
#13This looks like a big hack to compensate for using the wrong tool. Cassandra would have been a better solution IMO. With Cassandra, you can set replication factors, speed up the writes, and automatically shard the data without having to manage your own "mapping tables".
Cassandra was immature when this work was started (late 2011). The team was also much more familiar with the ins and outs of operating mysql.
There's also no transactions in Cassandra, although that's questionable useful given that the unidirectional map may be on a different machine.
Re: Sharding Pinterest: How we scaled our MySQL fleet
#14This looks like a big hack to compensate for using the wrong tool. Cassandra would have been a better solution IMO. With Cassandra, you can set replication factors, speed up the writes, and automatically shard the data without having to manage your own "mapping tables".
Cassandra was immature when this work was started (late 2011). The team was also much more familiar with the ins and outs of operating mysql.
[deleted]
Re: Sharding Pinterest: How we scaled our MySQL fleet
#15Earlier quoted context omitted.
Cassandra was immature when this work was started (late 2011). The team was also much more familiar with the ins and outs of operating mysql.
Technical debt is not an excuse to use the wrong tool. This is how you end up having a hammer. Cassandra was mature by late 2011 and we were using it in production back then (billions of records).
The mapping tables are used to store relationships between objects, not to define shard locations. The shard mapping is encoded in the object id.
In any case, I would have a knee jerk reaction to not trust any database system that hit 1.0 a month before I wanted to start using it, especially for mission critical core data.
Re: Sharding Pinterest: How we scaled our MySQL fleet
#16Can somebody help me understand this conversion? Shard ID = (241294492511762325 >> 46) & 0xFFFF = 3429 Type ID = (241294492511762325 >> 36) & 0x3FF = 1 Local ID = (241294492511762325 >> 0) & 0xFFFFFFFFF = 7075733
bitwise operations.
241294492511762325 >> 46 means shift the binary form of number which is
0000001101011001010000000001000000000000011010111111011110010101
0000000000000000000000000000000000000000000000000000110101100101
& 0xFFFF is doing a AND operation of the bits.
0000110101100101
1111111111111111
----------------
0000110101100101
Re: Sharding Pinterest: How we scaled our MySQL fleet
#17Earlier quoted context omitted.
Cassandra was immature when this work was started (late 2011). The team was also much more familiar with the ins and outs of operating mysql.
Technical debt is not an excuse to use the wrong tool. This is how you end up having a hammer. Cassandra was mature by late 2011 and we were using it in production back then (billions of records).
Technical debt doesn't necessarily arise from using the wrong (a.k.a pretty similar but with a few features less than what you need) tool. In my experience, it's more likely that technical debt arises from trying to use thew newest tool (which looks to be a silver bullet) rather than trusting your own experiences. It's surprising to me that people are willing to move to a completely different technology rather than extend and repurpose what your engineers are already familiar with.
Re: Sharding Pinterest: How we scaled our MySQL fleet
#18Earlier quoted context omitted.
Technical debt is not an excuse to use the wrong tool. This is how you end up having a hammer. Cassandra was mature by late 2011 and we were using it in production back then (billions of records).
The mapping tables are used to store relationships between objects, not to define shard locations. The shard mapping is encoded in the object id. In any case, I would have a knee jerk reaction to not trust any database system that hit 1.0 a month before I wanted to start using it, especially for mission critical core data.
Doubly so when everything is catching fire around you from how overloaded it all is.
Re: Sharding Pinterest: How we scaled our MySQL fleet
#19Nice article and good explanations.
But I still wonder why some people store blobs in a database rather then simple files on a file-system!?
Re: Sharding Pinterest: How we scaled our MySQL fleet
#20Earlier quoted context omitted.
Cassandra was immature when this work was started (late 2011). The team was also much more familiar with the ins and outs of operating mysql.
Technical debt is not an excuse to use the wrong tool. This is how you end up having a hammer. Cassandra was mature by late 2011 and we were using it in production back then (billions of records).
But if you also need to use MySql anyway, Cassandra would be the "wrong Tool". Because you would have to maintain two DBMS instead of one. You want to minimize complexity and this can mean implementing your own Sharding.