Live data from Hacker News

Amazon packages pile up after AWS outage spawns delivery havoc

detroitnews.com

191–200 of 219 posts

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#191
post #154

Earlier quoted context omitted.

Netflix has a chaos gorilla that does something like that: http://techblog.netflix.com/2011/07/netflix-simian-army.html And even more now: http://techblog.netflix.com/2013/12/active-active-for-multi-...

Chaos Engineering was more or less created for a service mesh of microservices. The companies we talk here about are most likely running a monolith app on Centos6..

* raises hand :)

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#192

Earlier quoted context omitted.

HN is a very pro-Amazon place. Regardless of what Amazon does, at least one of the top 3 comments is always justifying Amazon's actions.

That’s not the impression I got from the last 24 hours, nor the last few years. 1. Scale is hard and downtime is hard, HNers either recognize the struggle or appreciate their lack of experience. When AWS fails many armchair architects come out to suggest solutions but many more techies just sympathize with the Amazonians. 2. Technically, Amazon has built something impressive. It might not be what you want, or what ot…

[flagged]

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#193

Earlier quoted context omitted.

Us-East-1 took out commuter rail travel in Chicago for like a good 30 minutes, so not just companies

Why is a commuter rail travel system dependent on AWS? O___O

From what I’ve seen about Government IT it’s probably a good thing it’s on AWS.

Why shouldn’t Government IT be using the same tools regular companies use for IT?

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#194

As bad as this article makes AWS sound, it's actually the reason you should go with AWS over say Azure or GCP; when AWS goes down, its owners actually feel the pain with you, Microsoft and Google run their own stuff elsewhere...

This is simply not true. Microsoft dog foods tons of its own software and all internal cloud services I know of run on azure. Also, having consumed both cloud service offerings, customer support within Azure was far more responsive. Source: used to work as a SWE on a flagship Azure service

> Microsoft dog foods tons of its own software and all internal cloud services I know of run on azure

so it's even more embarrassing how terrible the software is.

trolling aside - if this is true, the state of e.g: MS Teams is a travesty. Implementation of replies to messages implemented in 2021!? So many bugs, etc. it seriously damages productivity. And don't get me started on sharepoint.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#195
post #164

Earlier quoted context omitted.

"postgres replication" would probably be the least of their worries. It's not about "magic clouds". These are services that are handling millions of requests per second and there's a lot going on where they have to maintain consistency and fail predictably. Having some services go down in one region but being back up in another still serving requests and committing transactions is unpredictable and could create a lot…

OK, but this is programmer to manager explanation. I get how computers work. I know simple things can get very complex at scale. I just thought that the huge extra you pay for cloud services is mostly for battle tested solutions for these scale problems. As far as I understand it, if you design is sane, the database/storage handles fallback and recovery. Or maybe in other words - you need to make your service handle…

The proper fallback is costly both hardware-wise and in development efforts. It can be cheaper just skip it and tolerate occasional service unavailability.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#196

I've seen mention of a major cable tv operator/last-mile residential ISP that had their entire field-tech dispatch system go down (based on a web gui https app that people can use on their phones) because it's hosted in AWS.

My friend lives in very cold Alaska. She has an app on her car that lets her remotely warm her car, which is a requirement before driving because it's so cold. The AWS outage meant her phone couldn't talk to her car. She was 20 mins late to work because of that. This is the future. AWS has an outage and your car won't work.

5 years ago you couldn't even start your car remotely, and many people lived in Alaska, this is such a first world problem lmao.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#197

Earlier quoted context omitted.

My friend lives in very cold Alaska. She has an app on her car that lets her remotely warm her car, which is a requirement before driving because it's so cold. The AWS outage meant her phone couldn't talk to her car. She was 20 mins late to work because of that. This is the future. AWS has an outage and your car won't work.

5 years ago you couldn't even start your car remotely, and many people lived in Alaska, this is such a first world problem lmao.

> 5 years ago you couldn't even start your car remotely

Of course you could. There have been remote car starters for 20+ years. They don't rely on internet nonsense to work either.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#198
Shit happens. My last employer is now mostly on AWS, and mostly in that region, so things broke all over the place yesterday (oddly enough the only service that is multi region was auth). In the past when we were mostly in our own datacenters we suffered as well, so that's not better either. One time a bad DNS update killed every single service worldwide and it took most of a day to properly restart thousands of services. Build complex systems with thousands or millions of things and it's bound to blow up sometime.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#199
post #193

Earlier quoted context omitted.

Why is a commuter rail travel system dependent on AWS? O___O

From what I’ve seen about Government IT it’s probably a good thing it’s on AWS. Why shouldn’t Government IT be using the same tools regular companies use for IT?

Government IT using AWS is fine. Using it for safety critical things like railways is a lot more of an issue. What if there's any kind of latency etc?

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#200
post #164

Earlier quoted context omitted.

"postgres replication" would probably be the least of their worries. It's not about "magic clouds". These are services that are handling millions of requests per second and there's a lot going on where they have to maintain consistency and fail predictably. Having some services go down in one region but being back up in another still serving requests and committing transactions is unpredictable and could create a lot…

OK, but this is programmer to manager explanation. I get how computers work. I know simple things can get very complex at scale. I just thought that the huge extra you pay for cloud services is mostly for battle tested solutions for these scale problems. As far as I understand it, if you design is sane, the database/storage handles fallback and recovery. Or maybe in other words - you need to make your service handle…

State ("the database") is hard in distributed systems. Was the package picked up or is it still sitting on a shelf waiting? If your distributed system is partitioned, different queries may give different answers and your warehouse workers are going to be running around looking for boxes that aren't there.

Even if you create a system that is eventually consistent when availability is restored (a difficult problem all by itself, and probably needs a lot of application layer logic), it may not be worth the trouble. Warehouse workers interact with the "perfect" state of the real world, and if the computers don't have access to that, they aren't very useful.

Post reply on HN