Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 646 posts

Re: Tell HN: AWS appears to be down again

#181

Post by [deleted] »

christophilus wrote: post #147
BossingAround wrote: post #89

Earlier quoted context omitted.

Does your server at home handle similar traffic to that of US-East-1 since you're comparing uptime? Simiarly, my laptop, if I keep it plugged in the wall, and enable httpd on localhost, will surely have better uptime than any of the top clouds. I'd bet that it'd have 100% uptime if I plugged in a UPS and cared for traffic on my local network only.

Most people don't need to handle the traffic of US-East-1. They just need a single, simple, mostly reliable server. But they're often told, "Don't do that. It's too hard, and irresponsible, and what if you get a spike in traffic, and what if you need to add 5 new servers, and security is really hard." In reality, most people don't need to scale. An occasional spike in traffic is a nuisance, but not the end of the wor…

[deleted]

Re: Tell HN: AWS appears to be down again

#182

Post by saxonww »

bobviolier wrote: post #174

Seems unlogical that this is just a single region in a single US region We are having issues pulling images from public.ecr.aws from an EU region.

I don't know what's still true, but at one point us-east-1 seemed more critical than other regions because there were some things that had to be there. One thing that comes to mind is ACM certificates used with things like API Gateway (probably Cloudfront), they had to be in us-east-1 no matter where the rest of your infrastructure was.

So it's not shocking to me that something going down in us-east-1 could have impact on other regions.

Re: Tell HN: AWS appears to be down again

#183

Post by cassianoleal »

dijit wrote: post #134
MatteoFrigo wrote: post #70

Earlier quoted context omitted.

I don't know about AWS, but both Google Cloud and Oracle Cloud maintain at least a high level history of past outages. See https://status.cloud.google.com/summary and https://ocistatus.oraclecloud.com/history

Given the hilariously awful reputation of the AWS status page I would hazard a guess that such a page would also be incredibly inaccurate. If you can’t even admit you’re having an issue how can you keep an accurate record?

Similar with GCP. We had a pretty bad outage once where the status page was showing all green. Google informed us that because the actual issue was further down the stack and didn't trigger any internal SLOs the status didn't get an update. It took them hours to acknowledge and fix it.

Re: Tell HN: AWS appears to be down again

#184

Post by dijit »

ryanbrunner wrote: post #165
christophilus wrote: post #140

Earlier quoted context omitted.

They're not saying Heroku is low end. They're saying, "I'm tired of hearing that it's irresponsible to run your own servers." At least, that's what I understood.

Any place I've worked at that managed their own servers (to be fair, the last time I worked at a place like that was 2010) definitely had more protracted downtimes than AWS - it just felt not as bad because we were in control of the situation, but at the end of the day that didn't get us up any faster. Another side benefit of being with AWS is when you do have an outage, a lot of other people have outages, and so you…

I guess it's extremely dependent on an org to org basis.

I worked at a company that's bread and butter was online services (e-commerce SaaS platform, similar to Netsuite) and we had significantly fewer outages than AWS had.

But we had redundancies built in to most things, I'm not saying it was perfect but it worked.

The major difference might be that almost nobody is willing to spend 20% of what they spend on AWS/GCP to have a self-hosted solution.

The reason "cloud is so expensive" is because they're essentially telling you what the price will be and even if they only spend 40% of that on actual hardware and operations: it's more than most companies would invest in themselves.

This is absurd, of course, but it's absolutely true.

Re: Tell HN: AWS appears to be down again

#185

Post by falcolas »

pxue wrote: post #161
api wrote: post #99

Earlier quoted context omitted.

Nobody ever got fired for using AWS.

maybe except a team at google? ;)

I have a story from only a few years ago where the finance section, and a good portion of management, of Google had no idea how poor their GAE solution was for uptime, until they tried to do business critical work using software that was hosted on GAE.

Uptime improved rather dramatically after that.

Re: Tell HN: AWS appears to be down again

#186

Post by tapan_jk »

AQuantized wrote: post #149
alecbz wrote: post #138

Earlier quoted context omitted.

I wonder to what extent this actually becomes less of a problem the more people use AWS. At this point AWS being down just feels like "the internet is down", it's hard for customers to be too mad at any company being down when all their competitors are too. Though I guess there's still probably just lost revenue that could be captured by having better uptime, even if your competitors are down.

This seems like an interesting pendulum swing where the few companies not reliant on AWS could capture significant enough revenue by maintaining uptime during a potential busy season outage.

Agree. Now viable alternatives exist. The nextgen cloud providers will learn from the weaknesses of incumbents and innovate.

Re: Tell HN: AWS appears to be down again

#187

Post by dijit »

cassianoleal wrote: post #183
dijit wrote: post #134

Earlier quoted context omitted.

Given the hilariously awful reputation of the AWS status page I would hazard a guess that such a page would also be incredibly inaccurate. If you can’t even admit you’re having an issue how can you keep an accurate record?

Similar with GCP. We had a pretty bad outage once where the status page was showing all green. Google informed us that because the actual issue was further down the stack and didn't trigger any internal SLOs the status didn't get an update. It took them hours to acknowledge and fix it.

Assuming you have a support contract the rep should send out a post-mortem page.

This is what happens when we've been affected by outages (even without involving support).

Re: Tell HN: AWS appears to be down again

#188

Post by taf2 »

GrumpyNl wrote: post #166
aledalgrande wrote: post #133

If you haven't seen yet, news is it was a power loss: > 5:01 AM PST We can confirm a loss of power within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. This is affecting availability and connectivity to EC2 instances that are part of the affected data center within the affected Availability Zone. We are also experiencing elevated RunInstance API error rates for launches wi…

How come they dont have power backups?

it was not a total power loss. out of 40 instances we had running at the time of the incident only 5 of our instances appeared to be lost to the power outage. the bigger issue for us was ec2 api to stop/start these instances appeared to be unavailable (but probably due to the rack these instances were in having no power). The other issue that was impactful to us was that many of the remaining running instances in the zone had intermittent connectivity out to the internet. Additionally, the incident was made worse by many of our supporting vendors being impacted as well...

IMO it was handled rather well and fast by AWS... not saying we shouldn't beat them up (for a discount) but being honest this wasn't that bad.

Re: Tell HN: AWS appears to be down again

#189

Post by benedikt »

crescentfresh wrote: post #109
dolibasija wrote: post #3

One of our EC2 instances in us-east-1c is unavailable and stuck in "stopping" state after a force stop. Interestingly enough, EC2 instances in us-east-1b don't seem to be affected. The console is throwing errors from time to time. As usual no information on AWS status page.

The affected zone is use1-az4. Whatever that maps to (1a, 1b, 1c) is different per customer.

you can find out which zone is mapped to use1-az4 for your account with awscli:

    aws ec2 describe-availability-zones | jq -r '.AvailabilityZones[] | select(.ZoneId == "use1-az4") | .ZoneName'

Re: Tell HN: AWS appears to be down again

#190

Post by [deleted] »

christophilus wrote: post #140
henriquez wrote: post #52

Earlier quoted context omitted.

Heroku isn’t “low end,” it’s a PaaS built on top of AWS. So you’re really just hearing about another AWS outage lol

They're not saying Heroku is low end. They're saying, "I'm tired of hearing that it's irresponsible to run your own servers." At least, that's what I understood.

[deleted]
Post reply on HN