Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stack Overflow: How we upgrade a live data center

blog.serverfault.com

Post reply on HN
Print view View original

41–50 of 74 posts

Re: Stack Overflow: How we upgrade a live data center

#41

Post by duggan »

Nick-Craver wrote: post #36
ak217 wrote: post #33

Earlier quoted context omitted.

AWS just announced a few days ago that their latest Xen patch will be deployed through a live update to their hypervisor kernel, and that going forward they expect patches like this to be rolled out live. The real upside of AWS is that they have relentlessly pursued and killed off reasons for you to care about things like this. They've eliminated points of failure in their infrastructure and given operators a wealth…

I can't resist reminding you that 1 command from a sysadmin routing traffic to the wrong network was the cause of the last major outage there :) They are getting much better, as all providers are. They're still just not a fit for many people because of performance requirements that are either impossible or too costly to meet on that type of infrastructure. I've always said this: the cloud isn't a good fit for us ; do…

> I can't resist reminding you that 1 command from a sysadmin routing traffic to the wrong network was the cause of the last major outage there :)

If you're going to bring that up, I can't resist reminding you of that time you had poor sysadmins running up and down stairs with buckets of fuel to keep your servers running[1].

> I've always said this: the cloud isn't a good fit for us; do what works for you.

There are costs. Frankly, it appears Stack Exchange prefers to lay those on its people rather than its purse.

[1]: http://blog.stackoverflow.com/2012/11/se-podcast-36-we-got-h...

Re: Stack Overflow: How we upgrade a live data center

#43

Post by balls187 »

Nick-Craver wrote: post #18
jmccree wrote: post #15

Perhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.

The cloud is awesome if you have zero interest whatsoever in hardware. It's not without trade-offs (nor is the other direction), and too many for us - but if it works for you then great I say. We obviously feel very different, and are just doing what works best for us.

> We obviously feel very different, and are just doing what works best for us.

And doing it very well, if I may say so.

Re: Stack Overflow: How we upgrade a live data center

#44

Post by [deleted] »

bri3d wrote: post #22
rdoherty wrote: post #20

Earlier quoted context omitted.

I agree. I read this post and was shocked at the amount of planning, process, man-hours, hardware issues and other problems that come with hardware. I've worked at places with ~500 EC2 machines in a dozen autoscaling groups across 3 AZs with many ELBs, databases, SQS queues and other AWS infrastructure and never had to deal with anything like this when upgrading. Upgrading hardware in EC2 is as simple as changing a l…

The Stack Exchange philosophy is that because they can buy truly mega hardware (each one of those two blade chassis they bought has 72 cores and 1.4TB of RAM, remember!), they don't need those 500 servers to start with. Plus the hardware is an asset and you get to depreciate it. Everywhere I've ever worked we've had the "big spreadsheet" of projected cloud costs, projected ops costs, and hardware costs. In general th…

[deleted]

Re: Stack Overflow: How we upgrade a live data center

#45

Post by thirsteh »

MrBuddyCasino wrote: post #38
bri3d wrote: post #22

Earlier quoted context omitted.

The Stack Exchange philosophy is that because they can buy truly mega hardware (each one of those two blade chassis they bought has 72 cores and 1.4TB of RAM, remember!), they don't need those 500 servers to start with. Plus the hardware is an asset and you get to depreciate it. Everywhere I've ever worked we've had the "big spreadsheet" of projected cloud costs, projected ops costs, and hardware costs. In general th…

Can you elaborate? I thought the answer to that question was to scale up if you can, because its much simpler and therefore cheaper. Similar to how you don't give up ACID unless the scale you're working at doesn't permit it anymore.

> I thought the answer to that question was to scale up if you can, because its much simpler and therefore cheaper

✱cough✱ SQL Server licensing fees ✱cough✱

Re: Stack Overflow: How we upgrade a live data center

#46

Post by bri3d »

MrBuddyCasino wrote: post #38
bri3d wrote: post #22

Earlier quoted context omitted.

The Stack Exchange philosophy is that because they can buy truly mega hardware (each one of those two blade chassis they bought has 72 cores and 1.4TB of RAM, remember!), they don't need those 500 servers to start with. Plus the hardware is an asset and you get to depreciate it. Everywhere I've ever worked we've had the "big spreadsheet" of projected cloud costs, projected ops costs, and hardware costs. In general th…

Can you elaborate? I thought the answer to that question was to scale up if you can, because its much simpler and therefore cheaper. Similar to how you don't give up ACID unless the scale you're working at doesn't permit it anymore.

There's never really a "one size fits all" answer, which is why it's a long-running debate and depends heavily on the product.

Scaling horizontally can let you use smaller, cheaper hardware on average and burst to higher capacity more easily if you need to, at the expense of a lot of complexity. It also (done right, which is rare) tends to gain you a greater degree of fault tolerance, since hardware instances become rapidly-replaceable commodities.

Most web apps have spiky but relatively predictable load. For example, a typical enterprise SaaS startup gets more traffic during work hours than on weekends. For these companies the complexity of developing a horizontally scaled architecture can be offset by the decreased cost of buying really big machines for peak load and then scaling back to a couple small instances for periods of below-average load.

That's (ostensibly) why AWS exists in the first place: Amazon had to buy a lot of peak capacity for Black Friday and Christmas and found it going unused the rest of the year. They never meant to sell their excess capacity, but they realized the tools that they built to dynamically scale their infrastructure were valuable to others.

Plus, a lot of work is offline data analytics, ETL, and so on. It's very cost effective to scale these workloads horizontally on-demand - spin up extra workers to run your reporting each hour/night and keep costs down the rest of the time when you don't need the capacity.

On the flip side, companies like Stack Exchange and Basecamp have high, relatively stable traffic worldwide. For companies like this it makes more sense to scale vertically - if they were in the cloud, they would never scale down or shut down their instances anyway.

Personally, I agree that horizontal scalability is oversold and most people can, indeed, scale up instead of out. However, a lot of plenty smart people disagree with me and have valid reasons to scale horizontally, too.

Re: Stack Overflow: How we upgrade a live data center

#47

Post by thethimble »

balls187 wrote: post #42
jmccree wrote: post #15

Perhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.

Are you at the same scale as Stack Exchange?

That's hardly an argument. Netflix, consuming 33% of the nation's bandwidth is a great counterexample.

In terms of scale, GoGuardian (the company I co-founded) has passed Stack Exchange. Articles like this make me so happy to be on AWS. Delegating this work to AWS allows us to focus on the product instead of the hardware that it runs on.

Re: Stack Overflow: How we upgrade a live data center

#48

Post by bri3d »

thirsteh wrote: post #45
MrBuddyCasino wrote: post #38

Earlier quoted context omitted.

Can you elaborate? I thought the answer to that question was to scale up if you can, because its much simpler and therefore cheaper. Similar to how you don't give up ACID unless the scale you're working at doesn't permit it anymore.

> I thought the answer to that question was to scale up if you can, because its much simpler and therefore cheaper ✱cough✱ SQL Server licensing fees ✱cough✱

That's a good point that I overlooked in my post - I'm sure this is a huge consideration for Stack Exchange.

For what it's worth Basecamp also evangelize the "scale up" approach and they're on an open-source stack.

Re: Stack Overflow: How we upgrade a live data center

#49

Post by jwatte »

bri3d wrote: post #22
rdoherty wrote: post #20

Earlier quoted context omitted.

I agree. I read this post and was shocked at the amount of planning, process, man-hours, hardware issues and other problems that come with hardware. I've worked at places with ~500 EC2 machines in a dozen autoscaling groups across 3 AZs with many ELBs, databases, SQS queues and other AWS infrastructure and never had to deal with anything like this when upgrading. Upgrading hardware in EC2 is as simple as changing a l…

The Stack Exchange philosophy is that because they can buy truly mega hardware (each one of those two blade chassis they bought has 72 cores and 1.4TB of RAM, remember!), they don't need those 500 servers to start with. Plus the hardware is an asset and you get to depreciate it. Everywhere I've ever worked we've had the "big spreadsheet" of projected cloud costs, projected ops costs, and hardware costs. In general th…

The biggest cost of using amazon isn't the hardware, it's the markup on traffic (if you are a dynamic site.)

Re: Stack Overflow: How we upgrade a live data center

#50

Post by [deleted] »

thethimble wrote: post #47
balls187 wrote: post #42

Earlier quoted context omitted.

Are you at the same scale as Stack Exchange?

That's hardly an argument. Netflix, consuming 33% of the nation's bandwidth is a great counterexample. In terms of scale, GoGuardian (the company I co-founded) has passed Stack Exchange. Articles like this make me so happy to be on AWS. Delegating this work to AWS allows us to focus on the product instead of the hardware that it runs on.

[deleted]
Post reply on HN