Live data from Hacker News

Amazon Web Services are down

status.aws.amazon.com

281–290 of 346 posts

Re: Amazon Web Services are down

#281

http://venuetastic.com/ - feel bad for these guys. They launched yesterday and down today because of AWS. Murphy's law in practice.

Wow. I can only imagine the intense frustration the site owner must be feeling right about now. Makes you really stop and question the whole "cloud" based service. Or at least should make you realize you need fall-backs other than the cloud service itself.

Re: Amazon Web Services are down

#282

Earlier quoted context omitted.

That hurts. But you're right, we've had a lot of issues. I think the reason this is news is because it is a massive Amazon failure.

I can't imagine how frustrating the jobs of the Reddit admins must be.

Is admin supposed to be plural? I mean, do they really have multiple system admins now? I ask, only because I know people have been coming and going recently.

Frankly, for the size of the site, they do really, really well for the limited resources they have.

Re: Amazon Web Services are down

#283
Why is ELB not mentioned at all on the Service Health Dashboard?

We're experiencing problems with two of our ELBs, one indicating instance health as out of service, reporting "a transient error occurred". Another, new LB (what we hoped would replace the first problematic LB), reports: "instance registration is still in progress".

A support issue with Amazon indicated that it was related to the ongoing issues and to monitor the Service Health Dashboard. But, as I mentioned before, ELB isn't mentioned at all.

Re: Amazon Web Services are down

#284
post #202
post #69

A couple of hours into the failure, and no sign of coverage on Techcrunch (they're posting "business" stories though). It shows how detached Techcrunch has become from the startup world. Edit: I tweeted their European editor about it and he's posted a story up now.

Perhaps this isn't really news. These days it's normal.

It's ugly, but true enough. You don't have to like it to acknowledge it. It's just another cloud outage bringing down one or more high profile sites. It's a "dog bites man" story.

Re: Amazon Web Services are down

#285
post #222

Some quotes regarding how Netflix handled this without interruptions: "Netflix showed some increased latency, internal alarms went off but hasn't had a service outage." [1] "Netflix is deployed in three zones, sized to lose one and keep going. Cheaper than cost of being down." [2] [1] https://twitter.com/adrianco/status/61075904847282177 [2] https://twitter.com/adrianco/status/61076362680745984

Here's the 24h latency data on EC2 east, west, eu, apac: http://dl.dropbox.com/u/1898990/EC2-multiple-zones-24h.png

Last 60 minutes comparison data: http://dl.dropbox.com/u/1898990/EC2-multiple-zones-60m.png

time in GMT.

A study we (Cedexis) did in January comparing multiple ec2 zones and other cloud providers: (pdf) http://dl.dropbox.com/u/1898990/76-marty-kagan.pdf

Re: Amazon Web Services are down

#287
post #236

Earlier quoted context omitted.

http://cache-scale.appspot.com/c/www.reddit.com/ That's a cache of it. I really wish that the admins at reddit would implement something like this themselves, then link to it when downtime like this happens.

do you have any stats of this app in terms of total data stored, bandwidth required per day, requests per second?

I don't, unfortunately, because I didn't write it :(

Re: Amazon Web Services are down

#289
post #81
post #8

Earlier quoted context omitted.

Apparently most of their problems are caused by bad EBS writes/performance, or at least so they said a few weeks ago after some particularly bad downtime. It looks like EBS will randomly decide to switch to a few bps of performance from time to time. I would use Amazon for my startup, but these issues really make it hard to justify.

EC2 instances only have one network interface. The public IP address you have pointing to your instance is a DNAT done somewhere further up the chain. If you get a large network load to your instance - say, a DDoS attack - you can find you no longer have enough network capacity to talk to your EBS disks. This is what happened to Bitbucket in 2009: http://blog.bitbucket.org/2009/10/04/on-our-extended-downtim...

This doesn't appear to be the issue here, though. valisystem's link mentions that it wasn't an interface issue, EBS is just shit, apparently.

Re: Amazon Web Services are down

#290
post #236

Earlier quoted context omitted.

http://cache-scale.appspot.com/c/www.reddit.com/ That's a cache of it. I really wish that the admins at reddit would implement something like this themselves, then link to it when downtime like this happens.

They do have a read-only mode, don't they? I'm not sure why they don't enable read-only mode when things like this happen. It may be that Amazon's service being down forbids this. I dunno.

They have a read-only mode for "free" which is their akamai cache that the unlogged in users see.
Post reply on HN