Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keeping Instagram up with over a million new users in twelve hours

instagram-engineering.tumblr.com

Post reply on HN
Print view View original

41–50 of 54 posts

Re: Keeping Instagram up with over a million new users in twelve hours

#41

Post by nupark2 »

mikeyk wrote: post #29
statictype wrote: post #27

Isn't there a risk with EBS snapshots that the snapshot of a live instance could have been taken while your db engine was in the middle of a transaction and leave the data in the newly spun instance in an inconsistent state? Is it that EBS snapshots are engineered to prevent this? Or just that it's not likely to happen in practice?

Yes, there is--we take all of our snapshots from a slave, and we stop the slave before taking a snapshot, then XFS-freeze all drives, then take the snapshot, to ensure it's consistent.

Are EBS snapshots not block-level atomic? In theory you should get a PITR image without stopping anything, assuming that:

1) The file system correctly orders or journals operations (I'm not familiar with XFS, but this is the case with FFS2/FreeBSD, ZFS, ext3/4 journaling, etc).

2) The database system correctly orders or journals operations, and properly fsync(s) to disk (which postgreSQL does)

Of course, there's no harm to an abundance of caution with something like this.

Re: Keeping Instagram up with over a million new users in twelve hours

#42

Post by mikeyk »

nupark2 wrote: post #41
mikeyk wrote: post #29

Earlier quoted context omitted.

Yes, there is--we take all of our snapshots from a slave, and we stop the slave before taking a snapshot, then XFS-freeze all drives, then take the snapshot, to ensure it's consistent.

Are EBS snapshots not block-level atomic? In theory you should get a PITR image without stopping anything, assuming that: 1) The file system correctly orders or journals operations (I'm not familiar with XFS, but this is the case with FFS2/FreeBSD, ZFS, ext3/4 journaling, etc). 2) The database system correctly orders or journals operations, and properly fsync(s) to disk (which postgreSQL does) Of course, there's no h…

They are, but we software-RAID our EBS drives to get better write throughput, and we put the Write-Ahead Logs (WALs) on a different RAID from the main database, so when you have both of those going on, you need something else to atomically snapshot our PG databases.

Re: Keeping Instagram up with over a million new users in twelve hours

#44

Post by mikeyk »

ab wrote: post #35
sciurus wrote: post #6

A slight tangent, since I saw that instagram are using both Graphite and Munin- Collectd just added a plugin to send metrics to Graphite. You might want to try it for tracking your machine stats over time. http://collectd.org/wiki/index.php/Plugin:Write_Graphite http://collectd.org/

Along the same lines, are you doing anything special with munin to make it fast? We've had performance issues with the RRDs and graph generation that led us to pipe metrics to graphite with collectd.

We've had to split munin across three masters (by machine role) because the graphing job was just locking on IO. Munin 2.0 moved over to all-dynamic CGI graphing, but I haven't gotten the chance to play with it yet.

Re: Keeping Instagram up with over a million new users in twelve hours

#45

Post by mikeyk »

cdelsolar wrote: post #19
mikeyk wrote: post #16

Earlier quoted context omitted.

We have a variety of Redis machines, some of them are in a consistent hash ring (the ones we're using for caching); some are using modulo-based hashing (the ones where losing data on adding more machines isn't an option), and some are just single-node installs.

How do you handle write replication? I haven't found any good document on how to do this (meaning, if the Redis master goes down, a slave should be promoted to master immediately).

You can hypothetically use something like hearbeatd to do it; we run every Redis master with an attached slave and manually failover for now.

For a small team like ours, we prefer solutions that are easy to reason about and get back into a healthy state (it would take one server deploy to point all appservers at new Redis master), rather than fully automated failover and the "fun" split-brain issues that ensue. Of course that may change as we build out our Ops team, etc.

Re: Keeping Instagram up with over a million new users in twelve hours

#46

Post by peterwwillis »

pkaler wrote: post #36
0xbadcafebee wrote: post #23

Why use Graphite instead of Ganglia? Ganglia uses RRDs. It's been around forever, it's fairly low on resource use, it's fast, and you can generate custom graphs like with Graphite. I actually ended up doing some graphs with google charts and ganglia last time I messed with it. (Also, nobody has really simple tools to tell you which of your 3,000 cluster nodes has red flags in real time and spit them into a fire-fight…

Sorry but this is not 'ideal', this is Capacity Planning 101. If you're launching a new product which you expect to be very popular, take your peak traffic and double or quadruple it and build out infrastructure to handle it ahead of time. I thought this was the whole point of the "cloud"? Add a metric shit-ton of resources for a planned peak and dial it down after. Paul is nice so we are nice. Last time I checked, I…

I'm sorry that my comments come off as harsh, but the original line struck me as so completely basic it's like something you would tell someone who had never worked in IT. They clarified later that they had tried to plan ahead but came up a little short, which I can understand; no estimation is perfect.

I have no idea how many users Sportsline had but it was a bunch. Peaks of 64k hits per second on the dynamic layer, up to 8 gigabits sustained traffic in one datacenter... it was pretty ugly on firefighting days. I don't mean to poop on them, but if they're as big as they seem to be I hold them to a higher standard than a 6 month old start-up fresh out of college.

I agree it's hard. The fact that they were able to handle the traffic they did with only a small amount of downtime is a testament to the fact that they did have their shit together (as well they should with the number of users they had already).

Re: Keeping Instagram up with over a million new users in twelve hours

#47

Post by Sujan »

camwest wrote: post #43
jurre wrote: post #38

Very interesting read, but doesn't New Relic do all these things for you? Maybe it's not possible to use with their setup?

I'm interested in comparing statsd to a commercial product like New Relic as well. -C

Statsd and NewRelic are very different.

NewRelic gives you mainly a predefined set of metrics, where you just have to install the agent to get them. Then. There's an additional module where you can send your own set of metrics and display them.

Statsd on the contrary is 'only' a tool to collect and then display metrics. You have to define everything you eant to measure yourself (or use plugins to your app).

So these two are definitely related, but better used for different (although overlapping) jobs.

Post reply on HN