Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stack Overflow: How we upgrade a live data center

blog.serverfault.com

Post reply on HN
Print view View original

51–60 of 74 posts

Re: Stack Overflow: How we upgrade a live data center

#51

Post by balls187 »

thethimble wrote: post #47
balls187 wrote: post #42

Earlier quoted context omitted.

Are you at the same scale as Stack Exchange?

That's hardly an argument. Netflix, consuming 33% of the nation's bandwidth is a great counterexample. In terms of scale, GoGuardian (the company I co-founded) has passed Stack Exchange. Articles like this make me so happy to be on AWS. Delegating this work to AWS allows us to focus on the product instead of the hardware that it runs on.

> That's hardly an argument. Netflix, consuming 33% of the nation's bandwidth is a great counterexample.

Actually it isn't.

Can AWS handle the scale of StackOverflow?

Absolutely. Netflix is a GREAT example of the scale AWS can handle.

However, can netflix migrate their entire system off of AWS without careful planning (which the parent of my comment was scoffing at)?

I'm a huge proponent of AWS, but I wouldn't be so naive to say that AWS allows me to be oblivious of hardware and infrastructure costs.

> In terms of scale, GoGuardian (the company I co-founded) has passed Stack Exchange.

How so? StackOverflow is #56 in the world in Internet Traffic Rank (alexa.com).

Re: Stack Overflow: How we upgrade a live data center

#52

Post by deeviant »

rdoherty wrote: post #20
jmccree wrote: post #15

Perhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.

I agree. I read this post and was shocked at the amount of planning, process, man-hours, hardware issues and other problems that come with hardware. I've worked at places with ~500 EC2 machines in a dozen autoscaling groups across 3 AZs with many ELBs, databases, SQS queues and other AWS infrastructure and never had to deal with anything like this when upgrading. Upgrading hardware in EC2 is as simple as changing a l…

I worked for a large scada company. We collected large amounts of data from thousands of large industrial installations.

One day we got a new VP who came from a well known firm who was a "cloud expert". He moved (nearly) all of our infrastructure to AWS, after producing untold amounts of spreadsheets/power points expressing how much cheaper/better/faster it's going to be.

Long story short, it was 4x more expensive as running it in house. By the time they went back to our own infrastructure, most of the internal sysops(including "The Glue" guy) had moved on and much of the old internal hardware was re-purposed or gone. It was a fiasco that they still have no fully recovered from yet.

I would be very careful in characterizing AWS as the solution for every large scale computer infrastructure problem.

Conversely, I have had excellent experiences with AWS in my current job, although we still have a rather large HPC cluster internally which would never make sense to move to AWS.

Re: Stack Overflow: How we upgrade a live data center

#53

Post by KyleBrandt »

duggan wrote: post #41
Nick-Craver wrote: post #36

Earlier quoted context omitted.

I can't resist reminding you that 1 command from a sysadmin routing traffic to the wrong network was the cause of the last major outage there :) They are getting much better, as all providers are. They're still just not a fit for many people because of performance requirements that are either impossible or too costly to meet on that type of infrastructure. I've always said this: the cloud isn't a good fit for us ; do…

> I can't resist reminding you that 1 command from a sysadmin routing traffic to the wrong network was the cause of the last major outage there :) If you're going to bring that up, I can't resist reminding you of that time you had poor sysadmins running up and down stairs with buckets of fuel to keep your servers running[1]. > I've always said this: the cloud isn't a good fit for us; do what works for you. There are…

That wasn't stack exchange. We failed over to our secondary data center during sandy.

Re: Stack Overflow: How we upgrade a live data center

#55

Post by beachstartup »

Nick-Craver wrote: post #18
jmccree wrote: post #15

Perhaps off topic, but this post just reminds me so much of why I love AWS EC2, etc. Not ever having to think about hardware again is wonderful.

The cloud is awesome if you have zero interest whatsoever in hardware. It's not without trade-offs (nor is the other direction), and too many for us - but if it works for you then great I say. We obviously feel very different, and are just doing what works best for us.

most of the public cloud cheer-leading is just people rationalizing to themselves what an awesome decision they made deploying on aws onto a billion tiny instances or whatever. and for lots of folks, it probably is pretty awesome.

however, i've seen very, very few people compare actual before/after $ figures on hn. when it comes time to show your cards everyone gets cold feet, either because they 1. don't have any idea because they aren't the ones paying for it or 2. don't have a baseline for comparison and are just paying whatever amazon asks or 3. it ended up costing 2-4x as much on amazon

when your bill is $5k/month 2-4x isn't that big of a deal. when it's $100k+ a month, it becomes a really big deal.

Re: Stack Overflow: How we upgrade a live data center

#58

Post by jschwartzi »

cagenut wrote: post #13

Woah, the datacenter you guys moved to is across the street from my apartment. Just a heads up, I considered it for a project myself but ruled it out because... well this is the back corner of that building: http://40.media.tumblr.com/tumblr_lqnh988WRS1qzpdb2o4_1280.j...

It's odd to see bollards and speed bumps in a swimming pool that big.

Re: Stack Overflow: How we upgrade a live data center

#59

Post by Nick-Craver »

duggan wrote: post #41
Nick-Craver wrote: post #36

Earlier quoted context omitted.

I can't resist reminding you that 1 command from a sysadmin routing traffic to the wrong network was the cause of the last major outage there :) They are getting much better, as all providers are. They're still just not a fit for many people because of performance requirements that are either impossible or too costly to meet on that type of infrastructure. I've always said this: the cloud isn't a good fit for us ; do…

> I can't resist reminding you that 1 command from a sysadmin routing traffic to the wrong network was the cause of the last major outage there :) If you're going to bring that up, I can't resist reminding you of that time you had poor sysadmins running up and down stairs with buckets of fuel to keep your servers running[1]. > I've always said this: the cloud isn't a good fit for us; do what works for you. There are…

As Kyle says, we were helping our sister company Fog Creek keep their servers online (as well as other people in that facility like Squarespace) because we cared. Our traffic was not being served from that data center and in fact we shut down most of our servers during that to conserve generator fuel. Our traffic was flowing just fine from Oregon. A decision Kyle and I made the night before when concluding they would probably shut down power to lower manhattan in preparation for flooding.

When your neighbor's house is on fire you don't argue over the price of the hose. You help. Our remote people that couldn't come help in person also helped them replicate their entire network in AWS as a backup plan.

I don't usually post pissed off comments, but you're dead wrong here and intetionally or not demeaning a good company and good people whom, because they cared, came to help in a time of emergency. I take it you weren't in New York during Sandy; it looked like a post-apocalyptic war zone afterwards.

TL;DR - You don't know what you're talking about.

Re: Stack Overflow: How we upgrade a live data center

#60

Post by chx »

bri3d wrote: post #46
MrBuddyCasino wrote: post #38

Earlier quoted context omitted.

Can you elaborate? I thought the answer to that question was to scale up if you can, because its much simpler and therefore cheaper. Similar to how you don't give up ACID unless the scale you're working at doesn't permit it anymore.

There's never really a "one size fits all" answer, which is why it's a long-running debate and depends heavily on the product. Scaling horizontally can let you use smaller, cheaper hardware on average and burst to higher capacity more easily if you need to, at the expense of a lot of complexity. It also (done right, which is rare) tends to gain you a greater degree of fault tolerance, since hardware instances become…

> a typical enterprise SaaS startup gets more traffic during work hours than on weekends.

You still need to budget what you can get if renting dedicated hardware vs. renting virtual machines. For eg. a Dual Xeon X5670 machine w/ 96GB RAM and 4x480GB SSD can be had for $249 per month (just something random I found for demo purposes). Even if you do a reserved instance for a year on EC2, you can get a m3.2xlarge for this kind of money, and that's only 30GB RAM and 2x80GB SSD.

It might worth it to rent this sort of iron instead of spinning up and down EC2 instances especially if you can reasonably buy a large enough machine to cut a lot of headaches arising from distributed computing. The right tool for the job.

Owning hardware is again a different bag of hurt.

Post reply on HN