Earlier quoted context omitted.
Netflix has a chaos gorilla that does something like that: http://techblog.netflix.com/2011/07/netflix-simian-army.html And even more now: http://techblog.netflix.com/2013/12/active-active-for-multi-...
Chaos Engineering was more or less created for a service mesh of microservices. The companies we talk here about are most likely running a monolith app on Centos6..
Amazon packages pile up after AWS outage spawns delivery havoc
191–200 of 219 posts
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#192Earlier quoted context omitted.
HN is a very pro-Amazon place. Regardless of what Amazon does, at least one of the top 3 comments is always justifying Amazon's actions.
That’s not the impression I got from the last 24 hours, nor the last few years. 1. Scale is hard and downtime is hard, HNers either recognize the struggle or appreciate their lack of experience. When AWS fails many armchair architects come out to suggest solutions but many more techies just sympathize with the Amazonians. 2. Technically, Amazon has built something impressive. It might not be what you want, or what ot…
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#193Earlier quoted context omitted.
Us-East-1 took out commuter rail travel in Chicago for like a good 30 minutes, so not just companies
Why is a commuter rail travel system dependent on AWS? O___O
Why shouldn’t Government IT be using the same tools regular companies use for IT?
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#194As bad as this article makes AWS sound, it's actually the reason you should go with AWS over say Azure or GCP; when AWS goes down, its owners actually feel the pain with you, Microsoft and Google run their own stuff elsewhere...
This is simply not true. Microsoft dog foods tons of its own software and all internal cloud services I know of run on azure. Also, having consumed both cloud service offerings, customer support within Azure was far more responsive. Source: used to work as a SWE on a flagship Azure service
so it's even more embarrassing how terrible the software is.
trolling aside - if this is true, the state of e.g: MS Teams is a travesty. Implementation of replies to messages implemented in 2021!? So many bugs, etc. it seriously damages productivity. And don't get me started on sharepoint.
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#195Earlier quoted context omitted.
"postgres replication" would probably be the least of their worries. It's not about "magic clouds". These are services that are handling millions of requests per second and there's a lot going on where they have to maintain consistency and fail predictably. Having some services go down in one region but being back up in another still serving requests and committing transactions is unpredictable and could create a lot…
OK, but this is programmer to manager explanation. I get how computers work. I know simple things can get very complex at scale. I just thought that the huge extra you pay for cloud services is mostly for battle tested solutions for these scale problems. As far as I understand it, if you design is sane, the database/storage handles fallback and recovery. Or maybe in other words - you need to make your service handle…
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#196I've seen mention of a major cable tv operator/last-mile residential ISP that had their entire field-tech dispatch system go down (based on a web gui https app that people can use on their phones) because it's hosted in AWS.
My friend lives in very cold Alaska. She has an app on her car that lets her remotely warm her car, which is a requirement before driving because it's so cold. The AWS outage meant her phone couldn't talk to her car. She was 20 mins late to work because of that. This is the future. AWS has an outage and your car won't work.
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#197Earlier quoted context omitted.
My friend lives in very cold Alaska. She has an app on her car that lets her remotely warm her car, which is a requirement before driving because it's so cold. The AWS outage meant her phone couldn't talk to her car. She was 20 mins late to work because of that. This is the future. AWS has an outage and your car won't work.
5 years ago you couldn't even start your car remotely, and many people lived in Alaska, this is such a first world problem lmao.
Of course you could. There have been remote car starters for 20+ years. They don't rely on internet nonsense to work either.
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#198Re: Amazon packages pile up after AWS outage spawns delivery havoc
#199Earlier quoted context omitted.
Why is a commuter rail travel system dependent on AWS? O___O
From what I’ve seen about Government IT it’s probably a good thing it’s on AWS. Why shouldn’t Government IT be using the same tools regular companies use for IT?
Re: Amazon packages pile up after AWS outage spawns delivery havoc
#200Earlier quoted context omitted.
"postgres replication" would probably be the least of their worries. It's not about "magic clouds". These are services that are handling millions of requests per second and there's a lot going on where they have to maintain consistency and fail predictably. Having some services go down in one region but being back up in another still serving requests and committing transactions is unpredictable and could create a lot…
OK, but this is programmer to manager explanation. I get how computers work. I know simple things can get very complex at scale. I just thought that the huge extra you pay for cloud services is mostly for battle tested solutions for these scale problems. As far as I understand it, if you design is sane, the database/storage handles fallback and recovery. Or maybe in other words - you need to make your service handle…
Even if you create a system that is eventually consistent when availability is restored (a difficult problem all by itself, and probably needs a lot of application layer logic), it may not be worth the trouble. Warehouse workers interact with the "perfect" state of the real world, and if the computers don't have access to that, they aren't very useful.