Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stack Overflow: How we upgrade a live data center

blog.serverfault.com

Post reply on HN
Print view View original

11–20 of 74 posts

Re: Stack Overflow: How we upgrade a live data center

#11

Post by gbrayut »

GeorgeBeech wrote: post #7
xtrumanx wrote: post #5

> We learned that accidentally sticking a server with nothing but naked IIS into rotation is really bad. Sorry about that one. Wait, does that mean what I think it means? Did someone get an IIS splash page when visiting SO? I was going to say they should live-blog during their next upgrade but it they did it on twitter which is awesome.

We generally pick a hashtag on twitter and do it that way. We've also done Hangouts On Air for non-hardware in the past which have been pretty fun as well.

If by fun you mean a 4 hour maintenance window turning into 8 hours ending well after midnight :-P

Re: Stack Overflow: How we upgrade a live data center

#12

Post by GeorgeBeech »

gbrayut wrote: post #11
GeorgeBeech wrote: post #7

Earlier quoted context omitted.

We generally pick a hashtag on twitter and do it that way. We've also done Hangouts On Air for non-hardware in the past which have been pretty fun as well.

If by fun you mean a 4 hour maintenance window turning into 8 hours ending well after midnight :-P

I never claimed to be sane

Re: Stack Overflow: How we upgrade a live data center

#13

Post by cagenut »

Woah, the datacenter you guys moved to is across the street from my apartment. Just a heads up, I considered it for a project myself but ruled it out because... well this is the back corner of that building: http://40.media.tumblr.com/tumblr_lqnh988WRS1qzpdb2o4_1280.j...

Re: Stack Overflow: How we upgrade a live data center

#14

Post by xtrumanx »

aalear wrote: post #6
xtrumanx wrote: post #5

> We learned that accidentally sticking a server with nothing but naked IIS into rotation is really bad. Sorry about that one. Wait, does that mean what I think it means? Did someone get an IIS splash page when visiting SO? I was going to say they should live-blog during their next upgrade but it they did it on twitter which is awesome.

Basically, yes. This happened: https://twitter.com/atimmer10/status/560568418945236995 .

How long did it take to figure out it occurred? Do you have any stats on how many people hit that machine?

Would be hilarious if someone who went to SO to figure out why the IIS splash was showing instead of his app happened to hit that page.

Re: Stack Overflow: How we upgrade a live data center

#16

Post by GeorgeBeech »

cagenut wrote: post #13

Woah, the datacenter you guys moved to is across the street from my apartment. Just a heads up, I considered it for a project myself but ruled it out because... well this is the back corner of that building: http://40.media.tumblr.com/tumblr_lqnh988WRS1qzpdb2o4_1280.j...

That's nothing: https://gigaom.com/2012/10/31/how-good-prep-and-a-bucket-bri...

We where hosted at that Peer 1 facility during Sandy. Also, if the water makes it up to the 16th floor it's time to give up anyway ... no one will care at that point :-D

Re: Stack Overflow: How we upgrade a live data center

#17

Post by cagenut »

GeorgeBeech wrote: post #16
cagenut wrote: post #13

Woah, the datacenter you guys moved to is across the street from my apartment. Just a heads up, I considered it for a project myself but ruled it out because... well this is the back corner of that building: http://40.media.tumblr.com/tumblr_lqnh988WRS1qzpdb2o4_1280.j...

That's nothing: https://gigaom.com/2012/10/31/how-good-prep-and-a-bucket-bri... We where hosted at that Peer 1 facility during Sandy. Also, if the water makes it up to the 16th floor it's time to give up anyway ... no one will care at that point :-D

yea we got hit hard by it too (33 whitehall), thats why we went shopping for a second location in jersey. we wound up in newark at 165halsey.

Re: Stack Overflow: How we upgrade a live data center

#18

Post by Nick-Craver »

jmccree wrote: post #15

Perhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.

The cloud is awesome if you have zero interest whatsoever in hardware. It's not without trade-offs (nor is the other direction), and too many for us - but if it works for you then great I say.

We obviously feel very different, and are just doing what works best for us.

Re: Stack Overflow: How we upgrade a live data center

#19

Post by GeorgeBeech »

xtrumanx wrote: post #14
aalear wrote: post #6

Earlier quoted context omitted.

Basically, yes. This happened: https://twitter.com/atimmer10/status/560568418945236995 .

How long did it take to figure out it occurred? Do you have any stats on how many people hit that machine? Would be hilarious if someone who went to SO to figure out why the IIS splash was showing instead of his app happened to hit that page.

It took about 20 minutes to notice and fix. The Twitter alerting system let us know really quick.

Re: Stack Overflow: How we upgrade a live data center

#20

Post by rdoherty »

jmccree wrote: post #15

Perhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.

I agree. I read this post and was shocked at the amount of planning, process, man-hours, hardware issues and other problems that come with hardware. I've worked at places with ~500 EC2 machines in a dozen autoscaling groups across 3 AZs with many ELBs, databases, SQS queues and other AWS infrastructure and never had to deal with anything like this when upgrading.

Upgrading hardware in EC2 is as simple as changing a launch configuration and updating an auto-scaling group. Maybe an hour of my time to update configs, verify and deploy. Updating something like a database or caching servers is more work for sure, but with 0 time needed to get to the DC, unpack, rack and configure servers you do save time with 'the cloud'.

I get that you do pay more for EC2 instances, especially if you keep hardware for 4 years. But AWS prices drop every year or two along with (generally) faster versions of software so your overall costs do drop.

How many ops employees would you need for a fleet of 500 servers in a datacenter? We managed it all with 4 people with AWS.

Post reply on HN