21–30 of 53 posts
Holy god. The post mentions "We evaluated a few different NoSQL solutions, but (...)". I couldn't even imagine having to _think_ about this sort of thing; CouchDB makes this an absolute no-brainer. I mean, the act of creating a document assigns it a UUID in the database _by_ _default_. Or, do you want to fetch a UUID before assigning it to any data? localhost:5984/_uuids Want to fetch 10? localhost:5984/_uuids?count=…
Earlier quoted context omitted.
Ah so you don't actually have 1000 (or whatever) schemas set up right off the bat? Or I might be misunderstanding, and you're saying that you're just mapping which physical server has the schemas
We set-up the schemas ahead of time, and then each lookup is (with, say, 1000 schemas): user_id % 1000 -> schema ID schema ID -> database ID then select FROM schemaID.tablename etc on that particular database.
The schemas are definitely a neat way to handle this though so you dont have to worry about table names.
How to do sharding: 1. Define your keyspace in bits. 2. Shard your keyspace in CIDR notation 3. Resharding always splits a keyspace in half eg. 0.0.0.0/0 becomes 0.0.0.0/1 and 128.0.0.0/1 4. Store your shard keyspace in DNS with SRV/TXT records 5. Assign IDs randomly This gives a couple interesting properties, for replication odds are very high that your corresponding server has a shard of similar size and if not its…
Luckily for the rest of us, there is not just one way to do sharding (which in the original version of your post you assertively referred to as "how to do sharding right").
It depends on the individual architecture and the tradeoffs people consider important. That means you don't get to have an undeserved but legal monopoly holding the entire scalable web hostage, just a part of it. Nevertheless: you should be ashamed of yourself.
Holy god. The post mentions "We evaluated a few different NoSQL solutions, but (...)". I couldn't even imagine having to _think_ about this sort of thing; CouchDB makes this an absolute no-brainer. I mean, the act of creating a document assigns it a UUID in the database _by_ _default_. Or, do you want to fetch a UUID before assigning it to any data? localhost:5984/_uuids Want to fetch 10? localhost:5984/_uuids?count=…
Requirement number one was that IDs be sortable by time. CouchDB UUIDs are random.
We're generating more GUIDs than ever with this system and those boxes are more or less idle on every metric. They're right in that we don't meet their time-ordered requirement, but I just wanted to say that writing (or reading) is not a bottleneck.
What if you have two tables with the same autoincrement value being updated at the same millisecond by two users with the same UserID%NumShards?
Or is there some relationship between physical and logical shards that makes this impossible?
Cool technology, good explanation. Legitimate questions below. What you're describing (uploading photos + storing metadata) sounds like something which Facebook has tech talked at length about at multiple venues. Their solution was to use distributed FS for images (such as HDFS, though FB uses their internal "Haystack") and then use HBase for the metadata. To be honest, your solution while it works now, looks like a…
We're on EC2, which has its set of limitations but means we can run a 10 million + user system with two-and-a-half engineers (and no ops team / overhead). So while we hear about more and more folks using SSDs in their DBs, it's not an option in our near-term future.
For SQL vs HBase/Haystack, we don't really have to worry about the photo storage itself, since S3 handles all of it. The data we shard out is more suited to an RDBMS, and since we're way more familiar with that world than with HBase and similar, it was the choice that let us make the most progress in a short time with a small team. Hope that's a helpful description of how we thought about it.
I work at Flickr and I see they mentioned Flickr's ticket server idea, (ab)using MySQL's autoincrement and "REPLACE INTO" trick and mentioned that a con was the write bottleneck. We're generating more GUIDs than ever with this system and those boxes are more or less idle on every metric. They're right in that we don't meet their time-ordered requirement, but I just wanted to say that writing (or reading) is not a bot…
I work at Flickr and I see they mentioned Flickr's ticket server idea, (ab)using MySQL's autoincrement and "REPLACE INTO" trick and mentioned that a con was the write bottleneck. We're generating more GUIDs than ever with this system and those boxes are more or less idle on every metric. They're right in that we don't meet their time-ordered requirement, but I just wanted to say that writing (or reading) is not a bot…
Thanks for the info, I updated the post to mention that the write bottleneck isn't an issue at Flickr.
I like that you guys are using your database's stock features to accomplish this. Most of the time you don't need complicated systems to get things done. Reducing that mental overload of YET another system is huge.
Holy god. The post mentions "We evaluated a few different NoSQL solutions, but (...)". I couldn't even imagine having to _think_ about this sort of thing; CouchDB makes this an absolute no-brainer. I mean, the act of creating a document assigns it a UUID in the database _by_ _default_. Or, do you want to fetch a UUID before assigning it to any data? localhost:5984/_uuids Want to fetch 10? localhost:5984/_uuids?count=…
Also, we'd still have to write the middleware to assign data to shards and fetch data from shards in our system (unless we used something like BigCouch), so having a more tried-and-tested solution that we already understood well was more appealing.