Stack Overflow: How we upgrade a live data center • Hacker News Board

Re: Stack Overflow: How we upgrade a live data center

#21

Post by jimwalsh » Thu, Mar 05, 2015, 6:32 PM UTC

jmccree wrote: post #15

Perhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.

Actually, occasionally you do need to think about it. Reference the various times AWS emails customers about unplanned outages that need to occur because of hardware issues/patching/etc.

Cloud is great, but there str still plenty of reasons to run your own datacenter. Yes for many startups it might not make sense, but at a certain sized company/application it can easily make sense.

Re: Stack Overflow: How we upgrade a live data center

#22

Post by bri3d » Thu, Mar 05, 2015, 6:40 PM UTC

rdoherty wrote: post #20

jmccree wrote: post #15

Perhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.

I agree. I read this post and was shocked at the amount of planning, process, man-hours, hardware issues and other problems that come with hardware. I've worked at places with ~500 EC2 machines in a dozen autoscaling groups across 3 AZs with many ELBs, databases, SQS queues and other AWS infrastructure and never had to deal with anything like this when upgrading. Upgrading hardware in EC2 is as simple as changing a l…

The Stack Exchange philosophy is that because they can buy truly mega hardware (each one of those two blade chassis they bought has 72 cores and 1.4TB of RAM, remember!), they don't need those 500 servers to start with. Plus the hardware is an asset and you get to depreciate it.

Everywhere I've ever worked we've had the "big spreadsheet" of projected cloud costs, projected ops costs, and hardware costs. In general the "scale horizontally" philosophy will favor the cloud while the "scale vertical" philosophy still seems to favor owned hardware in local datacenters. Which is superior is a crazy, long-standing debate with no clear answer.

Re: Stack Overflow: How we upgrade a live data center

#23

Post by dsp1234 » Thu, Mar 05, 2015, 6:52 PM UTC

rdoherty wrote: post #20

jmccree wrote: post #15

Perhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.

I agree. I read this post and was shocked at the amount of planning, process, man-hours, hardware issues and other problems that come with hardware. I've worked at places with ~500 EC2 machines in a dozen autoscaling groups across 3 AZs with many ELBs, databases, SQS queues and other AWS infrastructure and never had to deal with anything like this when upgrading. Upgrading hardware in EC2 is as simple as changing a l…

How many ops employees would you need for a fleet of 500 servers in a datacenter? We managed it all with 4 people with AWS.

This could be a false dichotomy. Just because a service with AWS uses so many servers doesn't mean a more monolithic system would need as many.

We did talks with one of our competitors (before they were a competitor). We mentioned that we ran our infrastructure on 4 large VM hosts (with a light density of 3-4 VMs per host). They were shocked. They were currently running over a hundred EC2 instances, with the relevant satellite services. They literally could not believe that we could provide a comparable service without reliance on something like AWS.

It's amazing what an be done with the right knowledge. In our case, myself and one of my coworkers maintain our infrastructure at something like 4-6 hours per week total (mostly patching, reviewing logs, etc). We both have previous networking, hardware, and software experience. When we do major upgrades (about every 2 years), it takes one of us about a week to source the hardware, get it loaded in to the rack, and turned on. Then we migrate guests over and we're done. This doesn't even get in to the cost savings on running on our own hardware vs AWS pricing.

Re: Stack Overflow: How we upgrade a live data center

#24

Post by chupy » Thu, Mar 05, 2015, 7:25 PM UTC

rdoherty wrote: post #20

jmccree wrote: post #15

Perhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.

I agree. I read this post and was shocked at the amount of planning, process, man-hours, hardware issues and other problems that come with hardware. I've worked at places with ~500 EC2 machines in a dozen autoscaling groups across 3 AZs with many ELBs, databases, SQS queues and other AWS infrastructure and never had to deal with anything like this when upgrading. Upgrading hardware in EC2 is as simple as changing a l…

Where I am currently working, thanks to good automation procedures there are only 3 people that are managing 4 datacenters on 3 continents with over 1000 virtual machines and also a couple of hundred physical servers. Those 3 people are 1 linux sysadmin, one network guy and one vmware guy. None of them works fulltime on maintaining the infrastructure just on patching/upgrading/installing new systems and that's 1/2 days a week at most. I have now finished the plan for the migration of two datacenters and that process takes about 2 months with shipping/networking/configuring/installing machines.

I really don't understand the obsession of getting rid of ops / hardware guys and relying on Amazon/Google/CoolCloudProvider to handle everything.

Re: Stack Overflow: How we upgrade a live data center

#25

Post by rasur » Thu, Mar 05, 2015, 7:32 PM UTC

That's a really nice cabling job. I've inherited 2 DC's with just-awful-enough cabling that I'd be exceptionally happy to set up new racks again and migrate, just to sort the bloody cables out

Sigh, don't think it's gonna happen.

As a customer of Dell, it was interesting to see someone elses insight on managing a dell farm.

So neat! I am mildly envious! ;)

Re: Stack Overflow: How we upgrade a live data center

#26

Post by Nick-Craver » Thu, Mar 05, 2015, 7:42 PM UTC

rasur wrote: post #25

That's a really nice cabling job. I've inherited 2 DC's with just-awful-enough cabling that I'd be exceptionally happy to set up new racks again and migrate, just to sort the bloody cables out Sigh, don't think it's gonna happen. As a customer of Dell, it was interesting to see someone elses insight on managing a dell farm. So neat! I am mildly envious! ;)

Thanks! We honestly do this for us and my mild cabling OCD, but if we can help anyone by sharing the details then it's worth it.

Re: Stack Overflow: How we upgrade a live data center

#27

Post by nasalgoat » Thu, Mar 05, 2015, 7:47 PM UTC

rdoherty wrote: post #20

jmccree wrote: post #15

Perhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.

I agree. I read this post and was shocked at the amount of planning, process, man-hours, hardware issues and other problems that come with hardware. I've worked at places with ~500 EC2 machines in a dozen autoscaling groups across 3 AZs with many ELBs, databases, SQS queues and other AWS infrastructure and never had to deal with anything like this when upgrading. Upgrading hardware in EC2 is as simple as changing a l…

At my previous company, I built a dedicated hardware system that consistently delivered a sub-second response time at a cost of about a third of what it would cost to host on AWS.

After I left, the new CTO who replaced me migrated it Rackspace's cloud offering. I don't know the costs involved, but now the site averages an 8s load time.

You can't really beat the raw performance for the price of dedicated equipment.

Re: Stack Overflow: How we upgrade a live data center

#28

Post by rasur » Thu, Mar 05, 2015, 7:59 PM UTC

Nick-Craver wrote: post #26

rasur wrote: post #25

That's a really nice cabling job. I've inherited 2 DC's with just-awful-enough cabling that I'd be exceptionally happy to set up new racks again and migrate, just to sort the bloody cables out Sigh, don't think it's gonna happen. As a customer of Dell, it was interesting to see someone elses insight on managing a dell farm. So neat! I am mildly envious! ;)

Thanks! We honestly do this for us and my mild cabling OCD, but if we can help anyone by sharing the details then it's worth it.

I'm curious to know what your perceptions of the FX2's are, since it had been racked up.

(EDIT: Over and above what's in the article.. Because they look quite nice, is why I'm asking - we haven't got any.. yet! Still performing well?)

Re: Stack Overflow: How we upgrade a live data center

#29

Post by bagels » Thu, Mar 05, 2015, 8:05 PM UTC

xtrumanx wrote: post #5

> We learned that accidentally sticking a server with nothing but naked IIS into rotation is really bad. Sorry about that one. Wait, does that mean what I think it means? Did someone get an IIS splash page when visiting SO? I was going to say they should live-blog during their next upgrade but it they did it on twitter which is awesome.

Why is their load balancer not configured to take servers like that out of rotation?

Re: Stack Overflow: How we upgrade a live data center

#30

Post by Nick-Craver » Thu, Mar 05, 2015, 8:29 PM UTC

rasur wrote: post #28

Nick-Craver wrote: post #26

Earlier quoted context omitted.

Thanks! We honestly do this for us and my mild cabling OCD, but if we can help anyone by sharing the details then it's worth it.

I'm curious to know what your perceptions of the FX2's are, since it had been racked up. (EDIT: Over and above what's in the article.. Because they look quite nice, is why I'm asking - we haven't got any.. yet! Still performing well?)

I haven't had nearly as much experience as George and Shane have with them after initial spin up. I'll get one of those guys to chime in here.