Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon EC2 currently down. Affecting Heroku, Reddit, Others

status.aws.amazon.com

Post reply on HN
Print view View original

241–250 of 311 posts

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#241

Post by gfodor »

w_t_payne wrote: post #108
gfodor wrote: post #80

Earlier quoted context omitted.

If you go through the pains of architecting your system to span multiple AZs, or you avoid using EBS, then you probably dodge most of the EC2 outages. (Remains to be seen if that is the case here.) That said, I don't think most people think using the cloud means that downtime is a thing of the past. I think the more attractive proposition is when hardware breaks, or meteors hit the datacenter, etc, it is their proble…

Yeah. Even with this, they still do better than I would. My record: misconfigured air-conditioning unit alarm leading to servers being baked at high temperature over a weekend, leading to much wailing and gnashing of teeth. I now know to be really careful to set up air conditioning units properly, but what other lessons am I still waiting to learn? The main lesson that I took from this is that I should stick to what…

Yeah this is another important point. Part of the cost of AWS is also a bit of an insurance policy against you physically breaking your servers :)

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#242

Post by piggity »

fragsworth wrote: post #195
donrhummy wrote: post #185

Earlier quoted context omitted.

I don't understand why anyone's site is only in one datacenter. i thought the point of AWS was that it was distributed with fault tolerance? Why don't they distribute all the sites/apps across all their centers?

It takes development/engineering resources, and additional hardware resources to make your architecture more fault-tolerant and to maintain this fault-tolerance over long periods of time. Weigh this against the estimated costs of your application going down occasionally. It's really only economical for the largest applications (Netflix, etc.) to build these systems.

The other issue is that you can have redundant services, but when the control plane goes down - you are screwed.

Every day I have to build basic redundancy into my applications I wish that we could just go with a service provider (like Rackspace / Contegix) that offered more redundancy at the hardware level.

I know the cloud is awesome and all, but having to assume your disks will disappear, fail, go slow at random uncontrollable times is expensive to design around.

If you don't have an elastic load, then the cloud elasticity is pointless - and is ultimately an anchor around your infrastructure.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#243

Post by gazarsgo »

fragsworth wrote: post #195
donrhummy wrote: post #185

Earlier quoted context omitted.

I don't understand why anyone's site is only in one datacenter. i thought the point of AWS was that it was distributed with fault tolerance? Why don't they distribute all the sites/apps across all their centers?

It takes development/engineering resources, and additional hardware resources to make your architecture more fault-tolerant and to maintain this fault-tolerance over long periods of time. Weigh this against the estimated costs of your application going down occasionally. It's really only economical for the largest applications (Netflix, etc.) to build these systems.

disagree. the only area it really hurts the wallet is multi-AZ on your RDS, because it doubles your cost no matter what and RDS is toughest to scale horizontally. The upside is if you scale your data layer horizontally you don't need to use RDS anymore.

two c1.medium, which are very nice for webservers, are enough to host >1M pageviews a month (wordpress, not much caching) and cost around $120/mo each, effective $97/mo if you prepay for 12months at a time via reserved instances.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#244

Post by Angostura »

xenophanes wrote: post #231
Angostura wrote: post #154

Earlier quoted context omitted.

It was a joke. I made the same joke earlier today. No-one is seriously going to believe this.

He knows you were joking. It's a bad joke. "It's a joke" is not a magic bullet that means you can do no wrong. Poisonous ideas spread as jokes. That is one of the ways they spread. A person thinking well about the issue wouldn't find the joke funny because it doesn't make sense. The joke relies on some poisonous, bad thinking to be understood. It has bad assumptions, and a bad way of looking at the world, built in.

Let me explain to you why it is a funny joke. It is funny because it involves Amazon undertaking massive technical measures, with huge reputational damage in order to try to kill a story which is primarily not spreading via Amazon-hosted sites anyway.

It's akin to a man with athlete's foot deciding to remedy it by discharging a shotgun into his leg.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#245

Post by 31reasons »

toomuchtodo wrote: post #201
pg wrote: post #192

Convenient that we're too backward to use AWS. That means everyone can at least talk about it here when AWS is down.

Funny thing is, the last couple interviews I've had in Chicago and Silicon Valley, I actually get points when explaining caution is necessary when using AWS for production. A magic bullet it isn't.

Whats the alternative? Building your own is certainly not.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#246

Post by forgotusername »

ukd1 wrote: post #224
rdl wrote: post #215

Earlier quoted context omitted.

You can buy some seriously big boxes, and easily split off a lot of services onto multiple boxes. The big problem with the "single big box" strategy is being able to do upgrades -- I see hn go down frequently for 5-10 min at a time in the middle of the night, which I assume is upgrades/reboots. The happy medium is probably splitting database (master/slave at least) and cdn (if needed) and some other services (AAA? lo…

hn doesn't have a database capable of master/slave as such...so I think this will be harder if it ever becomes popular enough. I don't think it gets enough traffic it's ever likely to exceed what you can fit in a single box, from what I know.

Couldn't help but do a little digging..

In the first 99 comments of this page, average comment text size is 231 bytes. Counting all comments in articles on the front page right now, there's 1678 of them, making somewhere around 388kb of comments for the past 12 hours.

So for safety's sake round that to 1mb/day and multiply by site age (5 years).

That gets us 1825mb, projecting forward it's difficult to imagine a time when a single recent SSD on a machine with even average RAM wouldn't be able to handle all of HN's traffic needs. Considering the recent beefy Micron P320h and its 785kIOP/sec, that could serve the entire comment history of Hacker News to the present day once every 2 seconds, assuming it wasn't already occupying a teensy Even if Arc became a burden, a decent NAS box, gigabit Ethernet, and a few front end servers would probably take the site well into the future. Assuming exponential growth, Hacker News comments would max out a 512GB SSD sometime around 2020, or 2021 assuming gzip bought a final doubling.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#247

Post by johnrob »

toomuchtodo wrote: post #201
pg wrote: post #192

Convenient that we're too backward to use AWS. That means everyone can at least talk about it here when AWS is down.

Funny thing is, the last couple interviews I've had in Chicago and Silicon Valley, I actually get points when explaining caution is necessary when using AWS for production. A magic bullet it isn't.

Downtime is inevitable for most companies. The only question is how much work you want to put in to achieve the standard up time.

Re: Amazon EC2 currently down. Affecting Heroku, Reddit, Others

#248

Post by pg »

Splines wrote: post #200
pg wrote: post #192

Convenient that we're too backward to use AWS. That means everyone can at least talk about it here when AWS is down.

I'm not familiar with HN's technical stack (other than arc), how has it scaled as the community grew over the years?

Barely.

Nothing has changed in the stack. Robert has discovered and eliminated a series of bottlenecks, causing performance to oscillate about tolerable. Finding bottlenecks is not trivial, because Arc has zilch in the way of profiling, but fortunately Robert is good at this sort of thing.

Post reply on HN