Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

921–930 of 1001 posts

Re: AWS us-east-1 outage

#921

Post by imwillofficial »

PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

This is weird, on my team it’s taken as a learning opportunity. I caused a pretty big outage and we just did a COE.

Re: AWS us-east-1 outage

#922

Post by wahern »

JPKab wrote: post #862
ipmb wrote: post #245

Looks like they've acknowledged it on the status page now. https://status.aws.amazon.com/ > 8:22 AM PST We are investigating increased error rates for the AWS Management Console. > 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US…

As a user of Sagemaker in us-east-1, I deeply fucking resent AWS claiming the service is normal. I have extremely sensitive data, so Sagemaker notebooks and certain studio tools make sense for me. Or DID. After this I'm going back to my previous formula of EC2 and hosting my own GPU boxes. Sagemaker is not working, I can't get to my work (notebook instance is frozen upon launch, with zero way to stop it or restart it…

You don't use AWS because it has better uptime. If you've been around the block enough times, this story has always rung hollow.

Rather, you use AWS because when it is down, it's down for everybody else as well. (Or at least they can nod their head in sympathy for the transient flakiness everybody experiences.) Then it comes back up and everybody forgets about the outage like it was just background noise. This is what's meant by "nobody ever got fired for buying (IBM|Microsoft)". The point is that when those products failed, you wouldn't get blamed for making that choice; in their time they were the one choice everybody excused even when it was an objectively poor choice.

As for me, I prefer hosting all my own stuff. My e-mail uptime is better than GMail, for example. However, when it is down or mail does bounce, I can't pass the buck.

Re: AWS us-east-1 outage

#923

Post by tyingq »

binaryblitz wrote: post #651
takeda wrote: post #635

Earlier quoted context omitted.

IMDB belongs to Amazon, so likely on AWS too. This also confirms it: https://downdetector.com/status/imdb/

TIL Amazon owns IMDB

Explains why nobody else has a feature similar to the "X-ray" thing in Prime video.

Re: AWS us-east-1 outage

#924

Post by orhmeh09 »

rossdavidh wrote: post #395
john37386 wrote: post #387

It seems a bit long to fix! They probably paint themselves in a corner just like facebook few weeks ago. This make me think; Could it be that one day the internet will have a total global outage and it will take few days to recover?

If we have a total global outage, Stack Overflow will be unavailable, and the internet will never be fixed. :) Mostly joking, I hope...

Large portions of stackoverflow are archived and available for download with the Kiwix app (typically used for offline Wikipedia). Same with the arch wiki. Might keep it around for a rainy day.

Re: AWS us-east-1 outage

#926

Post by Ysraes »

AtlasBarfed wrote: post #554
javajosh wrote: post #302

Earlier quoted context omitted.

I firmly believe in the dictum "if you ship it you own it". That means you own all outages. It's not just an operator flubbing a command, or a bit of code that passed review when it shouldn't. It's all your dependencies that make your service work. You own ALL of them. People spend all this time threat modelling their stuff against malefactors, and yet so often people don't spend any time thinking about the threat mo…

That's a great philosophy. Ok, let's take an organization, let's call them, say Ammizzun. Totally not Amazon. Let's say you have a very aggressive hire/fire policy which worked really well in rapid scaling and growth of your company. Now you have a million odd customers highly dependent on systems that were built by people that are now one? two? three? four? hire/fire generations up-or-out or cashed-out cycles ago. S…

Having individuals own systems seems like a terrible practice. You're essentially creating a single point of failure if only one person understands how the system works.

Re: AWS us-east-1 outage

#927

Post by tgtweak »

unethical_ban wrote: post #640
tgtweak wrote: post #549

Earlier quoted context omitted.

There are so many things that can go wrong with a single provider, regardless of how many availability zones you are leveraging, that you cannot depend on 1 cloud provider for your uptime if you require that level of up. Example: Payment/Administrative issues, rogue employee with access, deprecated service, inter-region routing issues, root certificate compromises... the list goes on and it is certainly not limited t…

All cloud tech is proprietary. There is no such thing as trivially setting up a secure, fully automated cloud stack, much less anything like a streamlined cloud agnostic toolset. Deprecated services are not the discussion here. We're talking tactical availability, not strategic tools etc. Rogue employees with access? You mean at the cloud provider or at your company? Still doesn't make sense. Cloud IAM is very diffic…

I advise many fintechs with engineering orgs from 5 to 5000, 2 in top 100 - none are blindly single-cloud and none have 25 people dedicated to each of their public clouds. The largest is not on any public clouds due to regulation/compliance and have several colocation facilities for their mission critical - they have less than 25 dedicated in the entire netsec org. This is a company that maintians strict PCI-DSS1 on thousands of servers and thousands of services. If you're employing 25 people per cloud for netsec in a multi cloud environment you have some seriously deficient DevOps practices or your org is 5-figure deep and has been ignoring devops best practices while on cloud for a half decade. Hahsicorps entire business revolves around cloud agnostic toolkits. All clouds speak kubernetes at this point and unless you have un-cloudable components in your stack (like root cert key signing systems on a proprietary appliance) you really should never find yourself in such a scenario where you have that many people overseeing infra security on a public cloud. It's been proven time and time again that too many people managing security is inversely secure.

Re: AWS us-east-1 outage

#928

Post by unethical_ban »

tgtweak wrote: post #927
unethical_ban wrote: post #640

Earlier quoted context omitted.

All cloud tech is proprietary. There is no such thing as trivially setting up a secure, fully automated cloud stack, much less anything like a streamlined cloud agnostic toolset. Deprecated services are not the discussion here. We're talking tactical availability, not strategic tools etc. Rogue employees with access? You mean at the cloud provider or at your company? Still doesn't make sense. Cloud IAM is very diffic…

I advise many fintechs with engineering orgs from 5 to 5000, 2 in top 100 - none are blindly single-cloud and none have 25 people dedicated to each of their public clouds. The largest is not on any public clouds due to regulation/compliance and have several colocation facilities for their mission critical - they have less than 25 dedicated in the entire netsec org. This is a company that maintians strict PCI-DSS1 on…

I meant at least 25 people in the DevSecOps role per cloud. Security experts, network/ops/systems experts, and automation (gitlab and container underlay) support.

K8s is one of a hundred technologies to learn and use, and just because each cloud is supported by terraform, you can't swap a GCP terraform writer over to Azure in a day.

And no bank is without their uncloudable components.

Re: AWS us-east-1 outage

#930

Post by throwthere »

tgtweak wrote: post #737
anovikov wrote: post #168

Haha my developer called me in panic telling that he crashed Amazon - was doing some load tests with Lambda

Postmortem: unbounded auto-scaling of lambda combined with oversight on internal rate limits caused unforseen internal ddos.

Just wait for the medium article “How I ran up a $400 million AWS bill.”
Post reply on HN