Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

491–500 of 1001 posts

Re: AWS us-east-1 outage

#491

Post by unionpivo »

bandyaboot wrote: post #415
NikolaeVarius wrote: post #370

Earlier quoted context omitted.

> Should I be penalized if an upstream dependency, owned by another team, fails? Yes > Did I lack due diligence in choosing to accept the risk that the other team couldn't deliver? Yes

Where does this mindset end? Do I lack due diligence by choosing to accept that the cpu microcode on the system I’m deploying to works correctly?

If it's brand new RiscV CPU that was just relesed 5 min ago, and nobody really tested then yes.

If its standard CPU that everybody else uses, and its not known to be bad then no.

Same for software. Is it ok to have dependency on AWS services ? Their history shows yes. Dependency on brand new SaaS product ? Nothing mission critical.

Or npm/crates/pip packages. Packages that have been around and seedily maintained for few years, have active users, are worth checking out. Some random project from single developer ? Consider vendoring (and owning if necessary ) it.

Re: AWS us-east-1 outage

#492

Post by gunapologist99 »

kumarakn wrote: post #336
kortex wrote: post #217

Earlier quoted context omitted.

That sounds like the exact opposite of human-factors engineering. No one likes taking blame. But when things go sideways, people are extra spicy and defensive, which makes them clam up and often withhold useful information, which can extend the outage. No-blame analysis is a much better pattern. Everyone wins. It's about building the system that builds the system. Stuff broke; fix the stuff that broke, then fix the t…

I worked at Walmart Technology. I bravely wrote post mortem documents owning the fault of my team (100+ people), owning both technically and also culturally as their leader. I put together a plan to fix it and executed it. Thought that was the right thing to do. This happend two times in my 10 year career there. Both times I was called out as a failure in my performance eval. Second time, I resigned and told them to…

That's shockingly stupid. I also worked for a major Walmart IT services vendor in another life, and we always had to be careful about how we handled them, because they didn't always show a lot of respect for vendors.

On another note, thanks for building some awesome stuff -- walmart.com is awesome. I have both Prime and whatever-they're-currently-calling Walmart's version and I love that Walmart doesn't appear to mix SKU's together in the same bin which seems to cause counterfeiting fraud at Amazon.

Re: AWS us-east-1 outage

#493

Post by errcorrectcode »

300bps wrote: post #402
PragmaticPulp wrote: post #117

I worked at a company that hired an ex-Amazon engineer to work on some cloud projects. Whenever his projects went down, he fought tooth and nail against any suggestion to update the status page. When forced to update the status page, he'd follow up with an extremely long "post-mortem" document that was really just a long winded explanation about why the outage was someone else's fault. He later explained that in his…

being at fault for an outage was one of the worst things that could happen to you Imagine how stressful life would be thinking that you had to be perfect all the time.

That's been most of my life. Welcome to perfectionism.

Re: AWS us-east-1 outage

#494

Post by jtheory »

paulryanrogers wrote: post #291
jedberg wrote: post #137

Earlier quoted context omitted.

It's not really dishonest though because there is nuance. Most everything in EC2 is still working it seems, just the console is down. So is it really down? It should probably be yellow but not red.

SNS seems to be at least partially down as well

My company relies on DynamoDB, so we're totally down.

edit: partly down; it's sporadically failing

Re: AWS us-east-1 outage

#495

Post by Guest19023892 »

bijoo wrote: post #376
Guest19023892 wrote: post #251

Earlier quoted context omitted.

One of my sites went offline an hour ago because the web server stopped responding. I can't SSH into it or get any type of response. The database server in the same region and zone is continuing to run fine though.

Interesting, is the site on a particular type of EC2 instance, e.g. bare metal? I see c4.xlarge is doing fine in us-east-1.

It's just a t3a.nano instance since it's a project under development. However, I have a high number of t3a.nano instances in the same region operating as expected. This particular server has been running for years, so although it could be a coincidence it just went offline within minutes of the outage starting, it seems unlikely. Hopefully no hardware failures or corruption, and it'll just need a reboot once I can get access to AWS again.

Re: AWS us-east-1 outage

#496

Post by thallium205 »

jabiko wrote: post #372
ipmb wrote: post #245

Looks like they've acknowledged it on the status page now. https://status.aws.amazon.com/ > 8:22 AM PST We are investigating increased error rates for the AWS Management Console. > 8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US…

Yeah, but I still have a different understanding what "Increased Error Rates" means. IMHO it should mean that the rate of errors is increased but the service is still able to serve a substantial amount of traffic. If the rate of errors is bigger than, let's say, 90% that's not an increased error rate, that's an outage.

They say that to try and avoid SLA commitments.

Re: AWS us-east-1 outage

#497

Post by javajosh »

NikolaeVarius wrote: post #448
bandyaboot wrote: post #415

Earlier quoted context omitted.

Where does this mindset end? Do I lack due diligence by choosing to accept that the cpu microcode on the system I’m deploying to works correctly?

Yes? If you are worried about CPU microcode failing, then you do a NASA and have multiple CPU arch's doing calculations in a voting block. These are not unsolved problems.

JPL goes further and buys multiple copies of all hardware and software media used for ground systems, and keeps them in storage "just in case". It's a relatively cheap insurance policy against the decay of progress.

Re: AWS us-east-1 outage

#498

Post by amir734jj »

My job (although 50% of time) at Azure is unit testing/monitoring services under different scenarios and flows to detect small failures that will be overlooked in public status page. Our tests run multiple times daily and we have people constantly monitoring logs. It concerns me when I see all AWS services are 100% green when I know there is an outage.

Re: AWS us-east-1 outage

#499

Post by jaywalk »

rossdavidh wrote: post #395
john37386 wrote: post #387

It seems a bit long to fix! They probably paint themselves in a corner just like facebook few weeks ago. This make me think; Could it be that one day the internet will have a total global outage and it will take few days to recover?

If we have a total global outage, Stack Overflow will be unavailable, and the internet will never be fixed. :) Mostly joking, I hope...

Some brave soul at Stack Overflow will have to physically go into the datacenter, roll up a cart with a keyboard, monitor and printer and start printing off a bunch of Networking answers.

Re: AWS us-east-1 outage

#500

Post by itisit »

stephenr wrote: post #488
itisit wrote: post #458

Earlier quoted context omitted.

They just added a banner. My guess is they don't know enough yet to update the respective service statuses.

I have basically zero faith in Amazon at this point. We first noticed failures because a tester happened to be testing in an env that uses the Amazon Pay sandbox. I checked the prod site, and it wouldn't even ask me to login. When I tried to login to SellerCentral to file a ticket - it told me my password (from a pw manager) was wrong. When I tried to reset, the OTP was ridiculously slow. Clicking "resend OTP" gives…

Sorry to hear. If multi-cloud is the answer, I wouldn't be surprised to see folks go back to owning and operating their own gear.
Post reply on HN