Potentially-ignoramus comment here, apologies in advance, but amazon.com itself appears to be fine right now. Perhaps slower to load pages, by about half a second. Are they not eating (much of) their own dog food?
AWS multiple services outage in us-east-1
431–440 of 1001 posts
Re: AWS multiple services outage in us-east-1
#432Potentially-ignoramus comment here, apologies in advance, but amazon.com itself appears to be fine right now. Perhaps slower to load pages, by about half a second. Are they not eating (much of) their own dog food?
those sentences aren't logically connected - the outage this post is about is mostly confined to `us-east-1`, anyone who wanted to build a reliable system on top of AWS would do so across multiple regions, including Amazon itself.
`us-east-1` is unfortunately special in some ways but not in ways that should affect well-designed serving systems in other regions.
Re: AWS multiple services outage in us-east-1
#433Potentially-ignoramus comment here, apologies in advance, but amazon.com itself appears to be fine right now. Perhaps slower to load pages, by about half a second. Are they not eating (much of) their own dog food?
They are 100% fully on AWS and nothing else. (I’m Ex-Amazon)
It seems like the outage is only effecting one region so AWS is likely falling back to others. I’m sure parts of the site are down but the main sites are resilient
Re: AWS multiple services outage in us-east-1
#434Seems to have taken down my router "smart wifi" login page, and there's no backup router-only login option! Brilliant work, linksys....
WiFi login portal (Icomera) on the train I'm on doesn't work either.
Re: AWS multiple services outage in us-east-1
#435Just a couple of days ago in this HN thread [0] there were quite some users claiming Hetzner is not an options as their uptime isn't as good as AWS, hence the higher AWS pricing is worth the investment. Oh, the irony. [0]: https://news.ycombinator.com/item?id=45614922
It still can be true that the uptime is better, or am I overlooking something?
Re: AWS multiple services outage in us-east-1
#436Looks like they’re nearly done fixing it. > Oct 20 3:35 AM PDT > The underlying DNS issue has been fully mitigated, and most AWS Service operations are succeeding normally now. Some requests may be throttled while we work toward full resolution. Additionally, some services are continuing to work through a backlog of events such as Cloudtrail and Lambda. While most operations are recovered, requests to launch new EC2…
It's a bit funny that they say "most service operations are succeeding normally now" when, in fact, you cannot yet launch or terminate new EC2 instance, which is basically the defining feature of the cloud...
Re: AWS multiple services outage in us-east-1
#437As this incident unfolds, what’s the best way to estimate how many additional hours it’s likely to last? My intuition is that the expected remaining duration increases the longer the outage persists, but that would ultimately depend on the historical distribution of similar incidents. Is that kind of data available anywhere?
Generally expect issues for the rest of the day, AWS will recover slowly, then anyone that relies on AWS will recovery slowly. All the background jobs which are stuck will need processing.
Re: AWS multiple services outage in us-east-1
#438Potentially-ignoramus comment here, apologies in advance, but amazon.com itself appears to be fine right now. Perhaps slower to load pages, by about half a second. Are they not eating (much of) their own dog food?
I was getting 500 errors a few hours ago on amazon.com
Re: AWS multiple services outage in us-east-1
#439As Amazon moves from day-1 company as it claimed once, to be the sales company like Oracle focusing on raking money, expect more outages to come, and longer to be resolved. Amazon is burning and driving away the technical talent and knowledge knowing the vendor lock-in will keep bringing the sweet money. You will see more sales people hoovering around your c-suites and executives, while you will face even worse techn…
Do you have data suggesting AWS outages are more frequent and/or take longer to resolve?
This is a prediction, not a historical pattern to be observed now. Only future data can verify if this prediction was correct or not.
Re: AWS multiple services outage in us-east-1
#440Every week or so we interview a company and ask them if they have a fall-back plan in case AWS goes down or their cloud account disappears. They always have this deer-in-the-headlights look. 'That can't happen, right?' Now imagine for a bit that it will never come back up. See where that leads you. The internet got its main strengths from the fact that it was completely decentralized. We've been systematically erodin…
First, planning for AWS outage is pointless. Unless you provide service of national security or something, your customers are going to understand that when there's global internet outage your service doesn't work either. The cost of maintaining a working failover across multiple cloud providers is just too high compared to potential benefits. It's astonishing that so few eningeers understand the fact that maintaining a technically beautiful solution costs time and money, which might not make a justified business case.
Second, preparing for the disappearance of AWS is even more silly. The chance that it will happen are orders of magnitude smaller than the chance that the cost of preparing for such an event will kill your business.
Let me ask you: how do you prepare your website for the complete collapse of western society? Will you be able to adapt your business model to post-apocalyptic world where it's only cockroaches?