Live data from Hacker News

Amazon packages pile up after AWS outage spawns delivery havoc

detroitnews.com

151–160 of 219 posts

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#151
post #99

Earlier quoted context omitted.

Did AWS hide her keys, too? This story makes no sense even in block heater Alaska.

Apparently based on another comment the app fails silently. Which is wonderful design.

It's amazing to me that people think they're adding something to the conversation by posting the most banal response.

"Yes, the dumb broad didn't think to walk outside and turn the car on! I shall right this wrong with my clever internet post! Behold my intelligence!"

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#152

Earlier quoted context omitted.

I'm just surprised Amazon was not using something sectioned off from the rest of AWS.

Possibly stems from one of the tenets of the apocryphal Bezos decree that started AWS[1] > All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions. 1: https://nordicapis.com/the-bezos-api-mandate-amazons-manifes...

That attitude, and the ability to enforce it at the scale Amazon is at, are probably the biggest strength of Bezos. It does cover all aspects so, including work culture...

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#153
post #119
post #59

Earlier quoted context omitted.

> Microsoft and Google run their own stuff elsewhere... That is simply not true. https://www.zdnet.com/article/microsoft-moves-closer-to-runn...

true for google though, with their borg-thingy being separate from gcp last time i asked them

Gcp runs on top of borg. (Gcp failure does not mean borg failure though)

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#154

Earlier quoted context omitted.

I'm almost tempted to think that having an explicit policy of forcibly shutting down each region once per month for a few hours (at times that are not publicly announced in advance) would be a worthwhile value-add. A service that is unable to handle such a failure does not qualify as being ready for deployment. And I'm not just saying that to be a self-righteous pedant, I'm saying it because this kind of failure is s…

Netflix has a chaos gorilla that does something like that: http://techblog.netflix.com/2011/07/netflix-simian-army.html And even more now: http://techblog.netflix.com/2013/12/active-active-for-multi-...

Chaos Engineering was more or less created for a service mesh of microservices.

The companies we talk here about are most likely running a monolith app on Centos6..

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#155

Earlier quoted context omitted.

Google does not run the bulk of their services on GCP. Unlike AWS, GCP was not a productization of their existing infrastructure, but rather a separate cloud product developed fairly independently. I'm sure that's changing with time. YouTube also has its own infrastructure independent of GCP and the rest of Google

GCP is not separate from Google's core infrastructure; rather, it's built on top of it. That means that while you can certainly have GCP specific outages, this kind of core infra "everything is down" situation is almost guaranteed to hit everything, GCP and not included. A lot of GCP sub-products are productionizations of existing Google tech; e.g. BigQuery is a public version of Dremel, an internal database/query en…

There aren’t 100k Googlers developing on top off GCP services to get their job done on a daily basis. That’s the big difference between the two clouds level of dog fooding.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#156

As bad as this article makes AWS sound, it's actually the reason you should go with AWS over say Azure or GCP; when AWS goes down, its owners actually feel the pain with you, Microsoft and Google run their own stuff elsewhere...

> Microsoft and Google run their own stuff elsewhere...

Where on Earth did you learn that?

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#157

Earlier quoted context omitted.

I'm almost tempted to think that having an explicit policy of forcibly shutting down each region once per month for a few hours (at times that are not publicly announced in advance) would be a worthwhile value-add. A service that is unable to handle such a failure does not qualify as being ready for deployment. And I'm not just saying that to be a self-righteous pedant, I'm saying it because this kind of failure is s…

Netflix has a chaos gorilla that does something like that: http://techblog.netflix.com/2011/07/netflix-simian-army.html And even more now: http://techblog.netflix.com/2013/12/active-active-for-multi-...

Therefor I didn’t expect Netflix to go down. Would be interesting to learn in a postmortem why Netflix had issues despite Chaos Gorilla.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#158
post #86

Earlier quoted context omitted.

Wouldn't surprise me. I mean, DSPs wouldn't be able to scan packages to pick them up without being able to access the flex app, so unless there's a procedure in place to allow packages to be picked up and manually marked as having been picked up... they wouldn't be able to do anything. Moreover, even if they could pick anything up, they wouldn't have any way to navigate, drop packages off, snap photos, and otherwise…

Failing over to another AWS region is actually pretty difficult for stateful services. Especially if you can't even access data in the primary region at all. Most teams probably don't have the bandwidth to solve this problem given the amount of outages you see in a year (1 or 2). Also, this would be a problem many teams would be solving, so most teams probably just wait and see what leaders have to say about it and w…

Can you elaborate? E.g. postgres replication is pretty straightforward and not a new technology. I'm outside AWS ecosystem and with just dedicated boxes having some DC burn down is manageable. How do magic clouds make that hard?

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#159
post #158
post #86

Earlier quoted context omitted.

Failing over to another AWS region is actually pretty difficult for stateful services. Especially if you can't even access data in the primary region at all. Most teams probably don't have the bandwidth to solve this problem given the amount of outages you see in a year (1 or 2). Also, this would be a problem many teams would be solving, so most teams probably just wait and see what leaders have to say about it and w…

Can you elaborate? E.g. postgres replication is pretty straightforward and not a new technology. I'm outside AWS ecosystem and with just dedicated boxes having some DC burn down is manageable. How do magic clouds make that hard?

"postgres replication" would probably be the least of their worries. It's not about "magic clouds". These are services that are handling millions of requests per second and there's a lot going on where they have to maintain consistency and fail predictably. Having some services go down in one region but being back up in another still serving requests and committing transactions is unpredictable and could create a lot of inconsistencies that would be very difficult to resolve later especially when you have customer facing services like this where someone's package could be lost resulting in bad reviews and other things you don't want to deal with. People here mostly have never even imagined the level of workloads they're handling and are throwing around "easy" solutions like replication or multi-region availability. For something of this scale, it's just not that simple. It would also be incredibly expensive to do this when you could simply shut operations down for a brief period of time. Not like something of this scale has happened that often

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#160

Earlier quoted context omitted.

Wouldn't surprise me. I mean, DSPs wouldn't be able to scan packages to pick them up without being able to access the flex app, so unless there's a procedure in place to allow packages to be picked up and manually marked as having been picked up... they wouldn't be able to do anything. Moreover, even if they could pick anything up, they wouldn't have any way to navigate, drop packages off, snap photos, and otherwise…

It happened once. My expectation of Amazon is that it would be fixed before it became a trend.

Nov 25 2020

https://www.datacenterdynamics.com/en/news/aws-us-east-1-reg...

Post reply on HN