Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stack Overflow: How we upgrade a live data center

blog.serverfault.com

Post reply on HN
Print view View original

31–40 of 74 posts

Re: Stack Overflow: How we upgrade a live data center

#31

Post by Nick-Craver »

bagels wrote: post #29
xtrumanx wrote: post #5

> We learned that accidentally sticking a server with nothing but naked IIS into rotation is really bad. Sorry about that one. Wait, does that mean what I think it means? Did someone get an IIS splash page when visiting SO? I was going to say they should live-blog during their next upgrade but it they did it on twitter which is awesome.

Why is their load balancer not configured to take servers like that out of rotation?

It normally would, except the base IIS page is a proper 200 to all requests. The load balancer removes all servers having issues - from HAProxy's point of view things were dandy.

As always, we are looking how to adjust our deployment process to prevent it from happening in the future.

Re: Stack Overflow: How we upgrade a live data center

#32

Post by GeorgeBeech »

rasur wrote: post #28
Nick-Craver wrote: post #26

Earlier quoted context omitted.

Thanks! We honestly do this for us and my mild cabling OCD, but if we can help anyone by sharing the details then it's worth it.

I'm curious to know what your perceptions of the FX2's are, since it had been racked up. (EDIT: Over and above what's in the article.. Because they look quite nice, is why I'm asking - we haven't got any.. yet! Still performing well?)

They are nice, we honestly haven't had to do much with them after we got them setup since they are VM hosts.

The most annoying part was that the Force10 OS that the IOA's run is slightly annoying to work with when you are used to working on Cisco gear. I'd almost rather people stopped doing "it's close to Cicso" and did their own thing because the connotative dissonance is jarring when something is close but not quite what you expect.

Re: Stack Overflow: How we upgrade a live data center

#33

Post by ak217 »

jimwalsh wrote: post #21
jmccree wrote: post #15

Perhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.

Actually, occasionally you do need to think about it. Reference the various times AWS emails customers about unplanned outages that need to occur because of hardware issues/patching/etc. Cloud is great, but there str still plenty of reasons to run your own datacenter. Yes for many startups it might not make sense, but at a certain sized company/application it can easily make sense.

AWS just announced a few days ago that their latest Xen patch will be deployed through a live update to their hypervisor kernel, and that going forward they expect patches like this to be rolled out live.

The real upside of AWS is that they have relentlessly pursued and killed off reasons for you to care about things like this. They've eliminated points of failure in their infrastructure and given operators a wealth of tools to ensure their apps stay up through any update or event (AZ-affine ELBs and autoscaling groups, single-IP ELBs, continuous improvements to EBS and S3, etc.) Given the scale of their infrastructure in us-east-1, it's now also highly unlikely that any customer will manage to overload it on their own.

Re: Stack Overflow: How we upgrade a live data center

#34

Post by GeorgeBeech »

rdoherty wrote: post #20
jmccree wrote: post #15

Perhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.

I agree. I read this post and was shocked at the amount of planning, process, man-hours, hardware issues and other problems that come with hardware. I've worked at places with ~500 EC2 machines in a dozen autoscaling groups across 3 AZs with many ELBs, databases, SQS queues and other AWS infrastructure and never had to deal with anything like this when upgrading. Upgrading hardware in EC2 is as simple as changing a l…

> How many ops employees would you need for a fleet of 500 servers in a datacenter? We managed it all with 4 people with AWS.

I'd say our goal is to keep growing and serving more content without _needing_ 500 servers in a data center. We are doing pretty well at that so far. We'll see what happens in the future.

Re: Stack Overflow: How we upgrade a live data center

#35

Post by _cudgel »

3princip wrote: post #2

I love reading about StackOverflow, particularly their infrastructure. The site has been a useful resource for so many years, and it works so well. It was a joy to discover that it all ran on like two racks worth of servers, and still does. Having seen corporate intranet portals, with maybe a thousand daily active users, running on excessive* hardware (needlessly, of course), it's like a breath of fresh air. *EDIT: R…

Likewise, I always find these write-ups fascinating, well written, well planned, etc. I really appreciate the extra effort involved in making all this public!

Re: Stack Overflow: How we upgrade a live data center

#36

Post by Nick-Craver »

ak217 wrote: post #33
jimwalsh wrote: post #21

Earlier quoted context omitted.

Actually, occasionally you do need to think about it. Reference the various times AWS emails customers about unplanned outages that need to occur because of hardware issues/patching/etc. Cloud is great, but there str still plenty of reasons to run your own datacenter. Yes for many startups it might not make sense, but at a certain sized company/application it can easily make sense.

AWS just announced a few days ago that their latest Xen patch will be deployed through a live update to their hypervisor kernel, and that going forward they expect patches like this to be rolled out live. The real upside of AWS is that they have relentlessly pursued and killed off reasons for you to care about things like this. They've eliminated points of failure in their infrastructure and given operators a wealth…

I can't resist reminding you that 1 command from a sysadmin routing traffic to the wrong network was the cause of the last major outage there :)

They are getting much better, as all providers are. They're still just not a fit for many people because of performance requirements that are either impossible or too costly to meet on that type of infrastructure.

I've always said this: the cloud isn't a good fit for us; do what works for you.

Re: Stack Overflow: How we upgrade a live data center

#37

Post by brongondwana »

dsp1234 wrote: post #23
rdoherty wrote: post #20

Earlier quoted context omitted.

I agree. I read this post and was shocked at the amount of planning, process, man-hours, hardware issues and other problems that come with hardware. I've worked at places with ~500 EC2 machines in a dozen autoscaling groups across 3 AZs with many ELBs, databases, SQS queues and other AWS infrastructure and never had to deal with anything like this when upgrading. Upgrading hardware in EC2 is as simple as changing a l…

How many ops employees would you need for a fleet of 500 servers in a datacenter? We managed it all with 4 people with AWS. This could be a false dichotomy. Just because a service with AWS uses so many servers doesn't mean a more monolithic system would need as many. We did talks with one of our competitors (before they were a competitor). We mentioned that we ran our infrastructure on 4 large VM hosts (with a light…

We run about a hundred servers (soon to be lots more) with a part time staff of 3 (as in, we all do dev work most of the time). It used to be mostly me for ages, but we got big enough that I got promoted out of most of the day-to-day stuff.

All own hardware, and having just had a reboot on Softlayer's schedule to fix the Xen issue for a separate project we're running on their gear - being able to schedule your own maintenance windows is so much nicer. We spend less time dealing with problems on our own hardware than we do dealing with cloud providers having issues.

Re: Stack Overflow: How we upgrade a live data center

#38

Post by MrBuddyCasino »

bri3d wrote: post #22
rdoherty wrote: post #20

Earlier quoted context omitted.

I agree. I read this post and was shocked at the amount of planning, process, man-hours, hardware issues and other problems that come with hardware. I've worked at places with ~500 EC2 machines in a dozen autoscaling groups across 3 AZs with many ELBs, databases, SQS queues and other AWS infrastructure and never had to deal with anything like this when upgrading. Upgrading hardware in EC2 is as simple as changing a l…

The Stack Exchange philosophy is that because they can buy truly mega hardware (each one of those two blade chassis they bought has 72 cores and 1.4TB of RAM, remember!), they don't need those 500 servers to start with. Plus the hardware is an asset and you get to depreciate it. Everywhere I've ever worked we've had the "big spreadsheet" of projected cloud costs, projected ops costs, and hardware costs. In general th…

Can you elaborate? I thought the answer to that question was to scale up if you can, because its much simpler and therefore cheaper. Similar to how you don't give up ACID unless the scale you're working at doesn't permit it anymore.

Re: Stack Overflow: How we upgrade a live data center

#39

Post by rasur »

GeorgeBeech wrote: post #32
rasur wrote: post #28

Earlier quoted context omitted.

I'm curious to know what your perceptions of the FX2's are, since it had been racked up. (EDIT: Over and above what's in the article.. Because they look quite nice, is why I'm asking - we haven't got any.. yet! Still performing well?)

They are nice, we honestly haven't had to do much with them after we got them setup since they are VM hosts. The most annoying part was that the Force10 OS that the IOA's run is slightly annoying to work with when you are used to working on Cisco gear. I'd almost rather people stopped doing "it's close to Cicso" and did their own thing because the connotative dissonance is jarring when something is close but not quit…

We've got 2 M1000e's, but they're slowly being retired and TBH they're in a 'do-less not do-more' situation (mostly!), so I haven't honestly had much experience past run-of-the-mill blade config, so that particular issue happily hasn't really bitten me :)

But thanks (to both you and Nick) for responding, the FX2s look interesting enough that I might have to see about an eval unit :)

Re: Stack Overflow: How we upgrade a live data center

#40

Post by clogston »

Nick-Craver wrote: post #31
bagels wrote: post #29

Earlier quoted context omitted.

Why is their load balancer not configured to take servers like that out of rotation?

It normally would, except the base IIS page is a proper 200 to all requests. The load balancer removes all servers having issues - from HAProxy's point of view things were dandy. As always, we are looking how to adjust our deployment process to prevent it from happening in the future.

Everyone I know big enough to run a load balancer has had this happen.

The next step in evolution is to have your script look for specific text on a page, which will change 24 months later and have all the perfectly good servers pulled out of rotation.

Post reply on HN