Stack Overflow: How we upgrade a live data center

#1

Post by Nick-Craver » Thu, Mar 05, 2015, 3:03 PM UTC

Stack Overflow: How we upgrade a live data center
blog.serverfault.com

Re: Stack Overflow: How we upgrade a live data center

#2

Post by 3princip » Thu, Mar 05, 2015, 3:45 PM UTC

I love reading about StackOverflow, particularly their infrastructure.

The site has been a useful resource for so many years, and it works so well. It was a joy to discover that it all ran on like two racks worth of servers, and still does. Having seen corporate intranet portals, with maybe a thousand daily active users, running on excessive* hardware (needlessly, of course), it's like a breath of fresh air.

*EDIT: Removed hyperbole. Not more hardware, but too much nonetheless.

Re: Stack Overflow: How we upgrade a live data center

#3

Post by GeorgeBeech » Thu, Mar 05, 2015, 3:52 PM UTC

3princip wrote: post #2

I love reading about StackOverflow, particularly their infrastructure. The site has been a useful resource for so many years, and it works so well. It was a joy to discover that it all ran on like two racks worth of servers, and still does. Having seen corporate intranet portals, with maybe a thousand daily active users, running on excessive* hardware (needlessly, of course), it's like a breath of fresh air. *EDIT: R…

We take a lot of pride in how much we get out of each piece of hardware. You don't need to have 1000 servers to run a large site, just the mindset of performance first.

Re: Stack Overflow: How we upgrade a live data center

#4

Post by [deleted] » Thu, Mar 05, 2015, 4:00 PM UTC

GeorgeBeech wrote: post #3

3princip wrote: post #2

I love reading about StackOverflow, particularly their infrastructure. The site has been a useful resource for so many years, and it works so well. It was a joy to discover that it all ran on like two racks worth of servers, and still does. Having seen corporate intranet portals, with maybe a thousand daily active users, running on excessive* hardware (needlessly, of course), it's like a breath of fresh air. *EDIT: R…

We take a lot of pride in how much we get out of each piece of hardware. You don't need to have 1000 servers to run a large site, just the mindset of performance first.

[deleted]

Re: Stack Overflow: How we upgrade a live data center

#5

Post by xtrumanx » Thu, Mar 05, 2015, 4:14 PM UTC

> We learned that accidentally sticking a server with nothing but naked IIS into rotation is really bad. Sorry about that one.

Wait, does that mean what I think it means? Did someone get an IIS splash page when visiting SO?

I was going to say they should live-blog during their next upgrade but it they did it on twitter which is awesome.

Re: Stack Overflow: How we upgrade a live data center

#6

Post by aalear » Thu, Mar 05, 2015, 4:18 PM UTC

xtrumanx wrote: post #5

> We learned that accidentally sticking a server with nothing but naked IIS into rotation is really bad. Sorry about that one. Wait, does that mean what I think it means? Did someone get an IIS splash page when visiting SO? I was going to say they should live-blog during their next upgrade but it they did it on twitter which is awesome.

Basically, yes. This happened: https://twitter.com/atimmer10/status/560568418945236995.

Re: Stack Overflow: How we upgrade a live data center

#7

Post by GeorgeBeech » Thu, Mar 05, 2015, 4:20 PM UTC

xtrumanx wrote: post #5

> We learned that accidentally sticking a server with nothing but naked IIS into rotation is really bad. Sorry about that one. Wait, does that mean what I think it means? Did someone get an IIS splash page when visiting SO? I was going to say they should live-blog during their next upgrade but it they did it on twitter which is awesome.

We generally pick a hashtag on twitter and do it that way. We've also done Hangouts On Air for non-hardware in the past which have been pretty fun as well.

Re: Stack Overflow: How we upgrade a live data center

#8

Post by import » Thu, Mar 05, 2015, 4:48 PM UTC

I'm really wondering how SO uses Redis.

For example, how failure scenarios handled and what is the role of slaves.

Re: Stack Overflow: How we upgrade a live data center

#9

Post by daigoba66 » Thu, Mar 05, 2015, 5:12 PM UTC

import wrote: post #8

I'm really wondering how SO uses Redis. For example, how failure scenarios handled and what is the role of slaves.

I've asked, and gotten an answer: http://meta.stackexchange.com/questions/248446/how-does-stac...

Other discussions:

http://meta.stackexchange.com/questions/69164/does-stack-ove...

http://meta.stackexchange.com/questions/110320/stack-overflo...

Re: Stack Overflow: How we upgrade a live data center

#10

Post by Nick-Craver » Thu, Mar 05, 2015, 5:15 PM UTC

import wrote: post #8

I'm really wondering how SO uses Redis. For example, how failure scenarios handled and what is the role of slaves.

We have a hot slave at all times for all instances. In the event of a failure the second slave will kick in. All applications are already connected to both servers via the StackExchange.Redis (https://github.com/StackExchange/StackExchange.Redis) library. The mechanisms for failover, etc. are built in there. We use this library via the dashboard (pictured at the bottom of the post) in Opserver (https://github.com/opserver/Opserver) to do quick swapping of master/slave, etc. We can do this during the day without anyone noticing.

Slaves are not just for backups - they also serve as pub/sub mechanisms. Every publish propagates to slaves. Since we use pub/sub for things like web sockets, we can easily move that entire concurrent connection load to another data center with a simple DNS change. Yep, we've tested this - it worked well.

It is of course noting: redis just doesn't fail. We had one out of memory fail for the Q&A web sites when forking and that's it. It has been rock solid here.