Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS us-east-1 outage

status.aws.amazon.com

Post reply on HN
Print view View original

991–1000 of 1001 posts

Re: AWS us-east-1 outage

#991

Post by jesboat »

sangnoir wrote: post #937
jesboat wrote: post #484

Earlier quoted context omitted.

> This issue is affecting the global console landing page, which is also hosted in US-EAST-1 Even this little tidbit is a bit of a wtf for me. Why do they consider it ok to have anything hosted in a single region? At a different (unnamed) FAANG, we considered it unacceptable to have anything depend on a single region. Even the dinky little volunteer-run thing which ran https://internal.site.example/~someEngineer was…

> At a different (unnamed) FAANG I'm guessing Google, on the basis of the recently published (to the public) "I just want to serve 5TB"[1] video. If it isn't Google, then the broccoli man video is still a cogent reminder that unyielding multi-region rigor comes with costs. 1. https://www.youtube.com/watch?v=3t6L-FlfeaI

It's salient that the video is from 2010. Where I was (not Google), the push to make everything multi-region only really started in, maybe, 2011 or 2012. And, for a long time, making services multi-region actually was a huge pain. (Exception: there was a way to have lambda-like code with access to a global eventually-consistent DB.)

The point is that we made it easier. By the time I left, things were basically just multi-region by default. (To be sure, there were still sharp edges. Services which needed to store data (like, databases) were a nightmare to manage. Services which needed to be in the same region specific instances of other services, e.g. something which wanted to be running in the same region as wherever the master shard of its database was running, were another nasty case.)

The point was that every services was expected to be multi-region, which was enforced by regular fire drills, and if you didn't have a pretty darn good story about why regular announced downtime was fine, people would be asking serious questions.

And anything external going down for more than a minute or two (e.g. for a failover) would be inexcusable. Especially for something like a bloody login page.

Re: AWS us-east-1 outage

#992

Post by jesboat »

sheenobu wrote: post #720
jesboat wrote: post #484

Earlier quoted context omitted.

> This issue is affecting the global console landing page, which is also hosted in US-EAST-1 Even this little tidbit is a bit of a wtf for me. Why do they consider it ok to have anything hosted in a single region? At a different (unnamed) FAANG, we considered it unacceptable to have anything depend on a single region. Even the dinky little volunteer-run thing which ran https://internal.site.example/~someEngineer was…

I think I know specifically what you are talking about. The actual files an engineer could upload to populate their folder was not multi-region for a long time. The servers were, because they were stateless and that was easy to multi-region, but the actual data wasn't until we replaced the storage service.

I think the storage was replicated by 2013? Definitely by 2014. It didn't have automated failover, but failover could be done, and was done during the relevant drills for some time.

I think it only stopped when the storage services got to the "deprecated, and we're not bothering to do a failover because dependent teams who care should just use something else, because this one is being shut down any year now". (I don't agree with that decision, obviously ;) but I do have sympathy for the team stuck running a condemned service. Sigh.)

After stuff was migrated to the new storage service (probably somewhere in the 2017-2019 range but I have no idea when), I have no idea how DR/failover worked.

Re: AWS us-east-1 outage

#993

Post by pokot0 »

outworlder wrote: post #889
pokot0 wrote: post #607

Earlier quoted context omitted.

Hiding behind a throw away account does not help your point.

The person is unlikely to have been authorized as a spokesman for AWS. In many workplaces, doing that is grounds for disciplinary action. Hence, throwaway.

Well, I understand that, but if you look at his account history it is only pro-Amazon comments. It feels like propaganda more than information, and all I am saying is that the throwaway does not add credibility or a feeling that his opinion are genuine.

Re: AWS us-east-1 outage

#995

Post by CRConrad »

naikrovek wrote: post #791
george3d6 wrote: post #765

Earlier quoted context omitted.

This seems like an insane stance to have, it's like saying businesses should ship their own stock, using their own drivers, and their in-house made cars and planes and in-house trained pilots. Heck, why stop at having servers on-site? Cast your own silicon waffers, after all you don't want spectrum exploits. Because you are worst at it. If a specialist is this bad, and the market is fully open, then it's because the…

> AWS (and all other IAAS providers) will beg you to use multiple region will they? because AWS still puts new stuff in us-east-1 before anywhere else, and there is often a LONG delay before those things go to other regions. there are many other examples of why people use us-east-1 so often, but it all boils down to this: AWS encourage everyone to use us-east-1 and discourage the use of other regions for the same rea…

> AWS still puts new stuff in us-east-1 before anywhere else, and there is often a LONG delay before those things go to other regions.

Come to think of it (far down the second page of comments): Why east?

Amazon is still mainly in Seattle, right? And Silicon Valley is in California. So one would have thought the high-tech hub both of Amazon and of the USA in general is still in the west, not east. So why us-east-1 before anywhere else, and not us-west-1?

Re: AWS us-east-1 outage

#996

Post by sheenobu »

jesboat wrote: post #992
sheenobu wrote: post #720

Earlier quoted context omitted.

I think I know specifically what you are talking about. The actual files an engineer could upload to populate their folder was not multi-region for a long time. The servers were, because they were stateless and that was easy to multi-region, but the actual data wasn't until we replaced the storage service.

I think the storage was replicated by 2013? Definitely by 2014. It didn't have automated failover, but failover could be done, and was done during the relevant drills for some time. I think it only stopped when the storage services got to the "deprecated, and we're not bothering to do a failover because dependent teams who care should just use something else, because this one is being shut down any year now". (I don'…

Thank you for the sympathy. If we are talking about the same product then it was most likely backed by 3 different storage services over its lifespan, 2013/2014 was a third party product that had some replication/fail-over baked in, 2016-2019 on my team with no failover plans due to "deprecated, dont bother putting anything important here", then 2019 onward with "fully replicated and automatic failover capable and also less cost-per-GB to replicate but less flexible for the existing use cases".

Re: AWS us-east-1 outage

#998

Post by Breza »

madmulita wrote: post #972
throwthere wrote: post #930

Earlier quoted context omitted.

Just wait for the medium article “How I ran up a $400 million AWS bill.”

"On how I learned 'recursion'"

I asked my friend who's a senior dev if he ever uses recursion at work. He said whenever he sees recursion in a code review, he tells the junior dev to knock it off.

Re: AWS us-east-1 outage

#999

Post by Breza »

lsaferite wrote: post #580
minig33 wrote: post #409

Earlier quoted context omitted.

I can’t turn on my lights… the future is weird

And that is why my lighting automation has a baseline req that it works 100% without the internet and preferably without a central controller.

This is an absolute requirement for all of my smart home devices. Not only in case of an outage but also in case the manufacturer decides to stop supporting my device in the future. My Roomba, litter box, washer/dryer, outlets, lights, and all the rest will keep working even if their internet functionality fails. I would like all of those devices to keep working for at least a decade, and I'd be surprised of all the manufacturers keep supporting that old of tech.
Post reply on HN