Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keeping Instagram up with over a million new users in twelve hours

instagram-engineering.tumblr.com

Post reply on HN
Print view View original

31–40 of 54 posts

Re: Keeping Instagram up with over a million new users in twelve hours

#31

Post by mikeyk »

lenn0x wrote: post #26

What kind of instances are you guys running for Redis/memcached? I am a bit surprised on the numbers here, but to be fair I don't do much in the virtualization world. With low cpu overhead, it sounds like you might be saturating the number of interrupts on the network card if it's not a bandwidth issue. Memcache can usually push 100-300k/s on an 8-core Westmere (could go higher if you removed the big lock). Redis on…

Those are really useful numbers--I think a lot of it can be chalked up to virtualization, but we should definitely explore more around IRQ pinning for queues. Any good starting points / reading, are you mostly using taskset?

Re: Keeping Instagram up with over a million new users in twelve hours

#34

Post by lenn0x »

mikeyk wrote: post #31
lenn0x wrote: post #26

What kind of instances are you guys running for Redis/memcached? I am a bit surprised on the numbers here, but to be fair I don't do much in the virtualization world. With low cpu overhead, it sounds like you might be saturating the number of interrupts on the network card if it's not a bandwidth issue. Memcache can usually push 100-300k/s on an 8-core Westmere (could go higher if you removed the big lock). Redis on…

Those are really useful numbers--I think a lot of it can be chalked up to virtualization, but we should definitely explore more around IRQ pinning for queues. Any good starting points / reading, are you mostly using taskset?

Taskset is fine for the process pinning. Don't forget about hyperthreading, you want to try to keep each thread on each hardware thread. IRQ pinning, see an example script I have:

http://chrisgoffinet.com/pin_network.sh

This will set queues 0-7 to specific smp affinity slots.

Re: Keeping Instagram up with over a million new users in twelve hours

#35

Post by ab »

sciurus wrote: post #6

A slight tangent, since I saw that instagram are using both Graphite and Munin- Collectd just added a plugin to send metrics to Graphite. You might want to try it for tracking your machine stats over time. http://collectd.org/wiki/index.php/Plugin:Write_Graphite http://collectd.org/

Along the same lines, are you doing anything special with munin to make it fast? We've had performance issues with the RRDs and graph generation that led us to pipe metrics to graphite with collectd.

Re: Keeping Instagram up with over a million new users in twelve hours

#36

Post by pkaler »

0xbadcafebee wrote: post #23

Why use Graphite instead of Ganglia? Ganglia uses RRDs. It's been around forever, it's fairly low on resource use, it's fast, and you can generate custom graphs like with Graphite. I actually ended up doing some graphs with google charts and ganglia last time I messed with it. (Also, nobody has really simple tools to tell you which of your 3,000 cluster nodes has red flags in real time and spit them into a fire-fight…

Sorry but this is not 'ideal', this is Capacity Planning 101. If you're launching a new product which you expect to be very popular, take your peak traffic and double or quadruple it and build out infrastructure to handle it ahead of time. I thought this was the whole point of the "cloud"? Add a metric shit-ton of resources for a planned peak and dial it down after.

Paul is nice so we are nice.

Last time I checked, I haven't built a service with +20mm users. I Googled you. I don't think you have built a service with +20mm users.

Programming is hard. Scaling is harder.

Let's have some empathy here. I bet the Instagram team has parents and siblings and significant others and friends that they haven't seen in a while. I bet they have responsibilities that they have neglected to keep the service up. I'd rather not poop on their head when they are trying to scale their service by millions of users.

This stuff is hard. Leaving a comment on a news aggregation service is easy.

Re: Keeping Instagram up with over a million new users in twelve hours

#39

Post by canop_fr »

8ig8 wrote: post #32

> We use the counters to track everything from number of signups per second. Per second... It must be quite a moment when you reach this point.

To put it in perspective : there are less than 5 human births per seconds.

So if they want to keep their counter greater than that for a long time, they'll probably have to extend their market beyond humans.

Post reply on HN