Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

851–860 of 1001 posts

Re: AWS us-east-1 outage

#851

Post by mise_en_place »

ehsankia wrote: post #849
jesboat wrote: post #484

Earlier quoted context omitted.

> This issue is affecting the global console landing page, which is also hosted in US-EAST-1 Even this little tidbit is a bit of a wtf for me. Why do they consider it ok to have anything hosted in a single region? At a different (unnamed) FAANG, we considered it unacceptable to have anything depend on a single region. Even the dinky little volunteer-run thing which ran https://internal.site.example/~someEngineer was…

Forget the number of regions. Monitoring for X shouldn't even be hosted on X at all...

Exactly. And I’m surprised AWS doesn’t have failover. That’s basic SOP for an SRE team.

Re: AWS us-east-1 outage

#852

Post by wolverine876 »

abdabab wrote: post #846
wolverine876 wrote: post #790

Earlier quoted context omitted.

While I support that, how are the people involved evaluated?

If an engineer causes an outage by mistake and then ensures that would never happen again, he has made a positive impact.

I understand that, but eventually they need to evaluate performance, for promotions, demotions, raises, cuts, hiring, firing, etc. How is that done?

Re: AWS us-east-1 outage

#854

Post by throwthere »

PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

Maybe it’s more telling that that engineer no longer works at Amazon.

Re: AWS us-east-1 outage

#855

Post by Rapzid »

bamboozled wrote: post #850

I don't think AWS knows what's going on judging by their updates, yes DynamoDB might be having issues, but so is IAM it seems, we're getting issues terminating resources for example.

They build all their services on top of other services. Hence the 2017 S3 outage that broke the internet.

Re: AWS us-east-1 outage

#856

Post by mananaysiempre »

dylan604 wrote: post #822
codegeek wrote: post #721

Earlier quoted context omitted.

"If you can't connect to the cloud, fail, move on and load the app so that local things are allowed to work." Building fallbacks require work. How much extra effort and overhead is needed to build something like this ? Sometimes the cost vs benefits says that it is ok not to do it. If AWS has an outage like this once a year, maybe we can deal with it (unless you are working with mission critical apps).

Yes, it is a lot of work to test if response code is OK or not, or if a timeout limit has been reached. So much so, I pretty much wrote the test in the first sentence. Phew. 10x coder right here!

Remember the Signal outage? Which is to say, the testing is the problem, not the implementation itself. (Which isn’t to say I think it’s optional—I myself generally don’t run online-only local-by-their-nature things.)

Re: AWS us-east-1 outage

#858

Post by george3d6 »

imiric wrote: post #788
george3d6 wrote: post #765

Earlier quoted context omitted.

This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…

> This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. > Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. That's an overblown argument. Nobody is saying that, but it's clear that businesses that maintain their ow…

There are at least 6 cloud providers I can name that I've used which run their own data centers with capabilities similar to AWSs core products (ec2, route53, s3, cloud watch, rdb)

Ovh, scaleway, online.net, azure, gcp, aws

That's one's I've used in production, I've heard of a dozen more including big names like HP and IBM, I assume they can match aws for the most part.

...

That being said I agree multi tenant is the way to go for reliability. But I was pointing out that in this case even the simple solution of multi region on one provider was not implemented by those affected.

...

As for running your own data center as a small company. I have done it, buying components building servers and all.

Expenses and ISP issues aside, I can't imagine using in house without at least a few outages a year for anywhere near the price of hiring a DevOps person to build a MT solution for you.

If you think you can you've either never tried doing it OR you are being severely underpaid for your job.

Competent teams to build and run reliable in house infrastructure exist, and they can get you SLA similar to multi region AWS or GC (aka 100% over the last 5 years)... But the price tag has 7 to 8 figures in it.

Re: AWS us-east-1 outage

#859

Post by macintux »

alde wrote: post #685

S3 bucket creation is failing across all regions for us, so this isn't an us-east-1 only issue.

Given the global uniqueness requirement, it’s quite possible that S3 is always going to fail for bucket creation if a region is (mostly) offline.

Re: AWS us-east-1 outage

#860

Post by edoceo »

ithkuil wrote: post #713
jesboat wrote: post #484

Earlier quoted context omitted.

> This issue is affecting the global console landing page, which is also hosted in US-EAST-1 Even this little tidbit is a bit of a wtf for me. Why do they consider it ok to have anything hosted in a single region? At a different (unnamed) FAANG, we considered it unacceptable to have anything depend on a single region. Even the dinky little volunteer-run thing which ran https://internal.site.example/~someEngineer was…

One region? I forgot how to count that low

It's like three regions - when two of them explode.

Two is one & one is none.

Post reply on HN