Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: AWS appears to be down again

news.ycombinator.com

Post reply on HN
Print view View original

191–200 of 646 posts

Re: Tell HN: AWS appears to be down again

#191

Post by jacob019 »

throwaway81523 wrote: post #27

Ok, enough AWS outages to say I'm tired of hearing about low end stuff being flaky.

Right. I've had an excellent experince with Vultr for the last couple years, for about 1/10th the cost of AWS. I use other small VPS providers as well. I run my own small business and I need to keep costs down to stay competitive. I used to use AWS more but the bill always creeps up to inappropriate levels. AWS billing is insulting, oh you forgot to renew your reserved instance? That's going to be double this month. I still use cloudfront, route 53, and a few of the smallest instances for mail servers and asterisk though. It's foolish to go all in with AWS, or with anything really.

Re: Tell HN: AWS appears to be down again

#192

Post by throwanem »

jeremyjh wrote: post #158
omh2 wrote: post #153

Earlier quoted context omitted.

AWS doesn't follow their own advice about hosting multi-regional. When us-east-1 is sufficiently borked the management API and IAM services in all regions tend to go down with it. Static infrastructures usually avoid the fallout, but anyone dependent on the API or otherwise dynamically created resources often get caught in the blast regardless of region

I didn't hear any reports of that happening in the most recent outage. The console was inoperable but you could work around using regional console host names.

It wasn't reliable. I heard of many more who weren't able to get in that way than who were, and was in the former category myself.

We didn't take any downtime, but if anything had gone wrong there would have been nothing we could do about it until IAM came back up.

Re: Tell HN: AWS appears to be down again

#193

Post by ryanbrunner »

christophilus wrote: post #147
BossingAround wrote: post #89

Earlier quoted context omitted.

Does your server at home handle similar traffic to that of US-East-1 since you're comparing uptime? Simiarly, my laptop, if I keep it plugged in the wall, and enable httpd on localhost, will surely have better uptime than any of the top clouds. I'd bet that it'd have 100% uptime if I plugged in a UPS and cared for traffic on my local network only.

Most people don't need to handle the traffic of US-East-1. They just need a single, simple, mostly reliable server. But they're often told, "Don't do that. It's too hard, and irresponsible, and what if you get a spike in traffic, and what if you need to add 5 new servers, and security is really hard." In reality, most people don't need to scale. An occasional spike in traffic is a nuisance, but not the end of the wor…

I think most people on here are coming from the perspective of startups, which scale out of a single server setup pretty quickly. At a bare minimum, most will have dedicated purpose-built servers like Redis or a DB, and often there's separate background workers, or a load balancer with a couple of web servers.

When your server requirements get into needing 5-6 servers (not at all atypical for a startup in their first year of being launched), running your own stuff becomes more of a challenge pretty quickly. Factor in 2-3x growth a year, and the challenges just mount.

Re: Tell HN: AWS appears to be down again

#194

Post by [deleted] »

RONROC wrote: post #154

The prevailing wisdom throughout the last couple of years was: “ditch your on-prem infrastructure and migrate to a major cloud provider” And its starting to seem like it could be something like: “ditch your on-prem infrastructure and spin up your own managed cloud” This is probably untenable for larger orgs where convenience gets the blank check treatment, but for smaller operations that can’t realize that value at s…

[deleted]

Re: Tell HN: AWS appears to be down again

#195

Post by darkwater »

potas wrote: post #13

Slack seems to have some issues because of that - I'm not sure if anyone is receiving messages, as it became completely silent for the last 15 minutes or so.

I fail to understand how a big player like Slack can be impacted this way by a failure in a single AZ in a specific AWS region. But at least the main feature (sending and displaying messages) is still working.

Re: Tell HN: AWS appears to be down again

#196

Post by dijit »

grumple wrote: post #141
Sammi wrote: post #105

Earlier quoted context omitted.

> Does your server at home handle similar traffic to that of US-East-1 since you're comparing uptime? Of course it doesn't. Why are you asking antagonistic questions?

He asked it to demonstrate the point that uptime is trivial for one server with no traffic, and much harder at scale with auto scaling.

Then don't host with so many people?

I don't think people care that AWS has other customers, they want their workload to work, if it doesn't: then that's a today issue.

Re: Tell HN: AWS appears to be down again

#197

Post by tyingq »

jeremyjh wrote: post #158
omh2 wrote: post #153

Earlier quoted context omitted.

AWS doesn't follow their own advice about hosting multi-regional. When us-east-1 is sufficiently borked the management API and IAM services in all regions tend to go down with it. Static infrastructures usually avoid the fallout, but anyone dependent on the API or otherwise dynamically created resources often get caught in the blast regardless of region

I didn't hear any reports of that happening in the most recent outage. The console was inoperable but you could work around using regional console host names.

There are some services that do have hard US-EAST-1 dependencies. Cloudfront, because of certificates. Route53. The control API for IAM (adding/removing roles, etc). And there's also the notion of "global endpoints" like https://sts.amazonaws.com... it's not clear why that exists, because it fails when us-east-1 does. It would be better to only have regional endpoints if the "global" ones are region-specific in reality. The endpoint thing is documented, but it's still confusing to people.

The dependency chains can bite you too. During the us-east-1 outage, a Lambda run by cron-like schedules via EventBridge was itself in an okay state, but the EventBridge events that kick it off were stuck in a queue that was released when the problem was fixed. So if your Lambda wasn't idempotent, and you ran it in another region during the outage, you ended up with problems.

Re: Tell HN: AWS appears to be down again

#198

Post by Hippocrates »

Every time a major cloud provider has an outage, Infra people and execs cry foul and say we need to move to . But does anyone really have an objective measure of how clouds stack up reliability-wise? I doubt it, since outages and their effects are nuanced. The other move is that they want to go multi-cloud... But I’ve been involved in enough multi-cloud initiatives to know how much time and effort those soak up, not to mention the overhead costs of maintaining two sets of infra sub-optimally. I would say that for most businesses, these costs far exceed that occasional six-hour-long outage.

Re: Tell HN: AWS appears to be down again

#199

Post by cassianoleal »

dijit wrote: post #187
cassianoleal wrote: post #183

Earlier quoted context omitted.

Similar with GCP. We had a pretty bad outage once where the status page was showing all green. Google informed us that because the actual issue was further down the stack and didn't trigger any internal SLOs the status didn't get an update. It took them hours to acknowledge and fix it.

Assuming you have a support contract the rep should send out a post-mortem page. This is what happens when we've been affected by outages (even without involving support).

I think they did eventually but it took us quite a bit of troubleshooting, then creating a P1 ticket, then their investigation in order to get to the bottom of it and getting it fixed. And the status page never got an update, which is the subject I was adding to.

Re: Tell HN: AWS appears to be down again

#200

Post by omh2 »

jeremyjh wrote: post #158
omh2 wrote: post #153

Earlier quoted context omitted.

AWS doesn't follow their own advice about hosting multi-regional. When us-east-1 is sufficiently borked the management API and IAM services in all regions tend to go down with it. Static infrastructures usually avoid the fallout, but anyone dependent on the API or otherwise dynamically created resources often get caught in the blast regardless of region

I didn't hear any reports of that happening in the most recent outage. The console was inoperable but you could work around using regional console host names.

If you're referring to Dec 7, it absolutely did. Metrics went down nearly across the board, which also means most auto-scaling setups were non-functional. Cloudfront metrics didn't properly recover until the next day

Logging in with root credentials was not possible in any region, and even logging in with IAM creds in other regions yielded an intermittently buggy console

and as is usual with us-east-1 outages management API calls were a complete crap shoot regardless of region

Post reply on HN