Neat.
You could avoid having to create a per-schema id function by passing the shard id and the sequence oid as params to the function, so you'd have default public.next_id(5,'insta5.table_id_seq'::regclass)
Sharding and IDs at Instagram
41–50 of 53 posts
Re: Sharding and IDs at Instagram
#42I think it's a mistake to tie the shard id into the object id. Shard id should be derived from the object id dynamically based on the placement of the object among the shards. If the shards grow or shrink or object migrated, a different shard id is generated, but the object id doesn't have to change. Edit: I like how constructive criticism got downvoted. Thanks for discouraging technical discussion.
Right on. The shard id could be derived from the last few bits of the object id, leaving space for additional entropy in the object id itself.
Edit: you don't need an object id / shard id mapping. Each time you need to locate a shard for an object, you take the last few bits of the object id modulo the number of shards to get your shard id.
Re: Sharding and IDs at Instagram
#43This seems fairly neat. Is it possible to do something like this with MySQL?
(author here) I think this should also be possible with MySQL's stored functions, but one huge benefit to PostgreSQL is the schema/tablespace feature, since it means all our logical shards all live inside one database (you could do something similar with MySQL, but it would mean multiple databases or prefixing table names with the shard ID).
PostgreSQL schemas are very similar to MySQL databases in functionality. In fact, in MySQL you can use SCHEMA in all places you can use the term database, ie. CREATE SCHEMA foo; instead of CREATE DATABASE foo;
Re: Sharding and IDs at Instagram
#44We use a solution similar to snowflake at Formspring:
https://github.com/formspring/flake
In the 18 months this system has been running, we never experienced any issue.
Re: Sharding and IDs at Instagram
#45Cool technology, good explanation. Legitimate questions below. What you're describing (uploading photos + storing metadata) sounds like something which Facebook has tech talked at length about at multiple venues. Their solution was to use distributed FS for images (such as HDFS, though FB uses their internal "Haystack") and then use HBase for the metadata. To be honest, your solution while it works now, looks like a…
Valid questions! We're on EC2, which has its set of limitations but means we can run a 10 million + user system with two-and-a-half engineers (and no ops team / overhead). So while we hear about more and more folks using SSDs in their DBs, it's not an option in our near-term future. For SQL vs HBase/Haystack, we don't really have to worry about the photo storage itself, since S3 handles all of it. The data we shard o…
From what I am understanding from the post is that you are actually sharding other tables like photos, likes, comments but not the main users table, right?
So, if one day you want to shard the users table, it will render all current sharding useless, right?
Re: Sharding and IDs at Instagram
#46I just love how startups keep reinventing the wheel
to solve problems that haven't been problems for decades.
Come on, 25 pics + 90 likes per second... that almost like... [wait for it]... nothing.
I'm pretty sure you got your number wrong.
Re: Sharding and IDs at Instagram
#47I just love how startups keep reinventing the wheel to solve problems that haven't been problems for decades. Come on, 25 pics + 90 likes per second... that almost like... [wait for it]... nothing. I'm pretty sure you got your number wrong.
Almost forgot:
Sharding 101
"Always shard via directory..."
Re: Sharding and IDs at Instagram
#48Back in 2003 I was a developer on a team at Microsoft that was responsible for building a new storage backend for all the MSN Messenger and Hotmail contact lists. This store had to hold data for close to 300 million user accounts, which we sharded out to a few hundred SQL server databases. Our sharding system consisted of a database with a single table with a row for each user that mapped their 128 bit guid user ID to the ID of their assigned shard. New user creation involved generating a new guid and inserting into this table. Each read operation involved a select from this table.
The database ran on a machine with enough RAM to let SQL Server do its thing and cache almost the whole table in memory. At the time I left the team in 2005, it was executing over 25,000 requests per second, with an average latency of under 3ms. Pretty sure that on modern hardware it would handle much, much more.
Re: Sharding and IDs at Instagram
#49Hi Mike, excellent article! One question about sharding, do you find that it reduces High Availability overall, as your uptime depends on the vagaries of additional database servers??
Re: Sharding and IDs at Instagram
#50One nit about snowflake in your article (i'm the author of snowflake)– the zookeeper integration is optional and only used for sanity checking the configuration (that the worker ids are distinct, etc).