Live data from Hacker News

Amazon packages pile up after AWS outage spawns delivery havoc

detroitnews.com

181–190 of 219 posts

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#181
post #99

Earlier quoted context omitted.

Apparently based on another comment the app fails silently. Which is wonderful design.

It's amazing to me that people think they're adding something to the conversation by posting the most banal response. "Yes, the dumb broad didn't think to walk outside and turn the car on! I shall right this wrong with my clever internet post! Behold my intelligence!"

To be fair if someone was using the “app didn’t work” excuse multiple times, suspicion could be raised.

But for a one off occurrence? Why would you assume a car company knows what they’re doing over the person telling the story? It’s silly.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#183

Earlier quoted context omitted.

Us-East-1 took out commuter rail travel in Chicago for like a good 30 minutes, so not just companies

Why is a commuter rail travel system dependent on AWS? O___O

They decided to run their infra in AWS instead of another cloud provider or on-prem?

why is McDonalds dependent on AWS for their app to work? (/s)

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#184
post #86

Earlier quoted context omitted.

Wouldn't surprise me. I mean, DSPs wouldn't be able to scan packages to pick them up without being able to access the flex app, so unless there's a procedure in place to allow packages to be picked up and manually marked as having been picked up... they wouldn't be able to do anything. Moreover, even if they could pick anything up, they wouldn't have any way to navigate, drop packages off, snap photos, and otherwise…

Failing over to another AWS region is actually pretty difficult for stateful services. Especially if you can't even access data in the primary region at all. Most teams probably don't have the bandwidth to solve this problem given the amount of outages you see in a year (1 or 2). Also, this would be a problem many teams would be solving, so most teams probably just wait and see what leaders have to say about it and w…

"Failing over to another AWS region is actually pretty difficult for stateful services."

I don't care, and neither should you.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#185
post #158

Earlier quoted context omitted.

Can you elaborate? E.g. postgres replication is pretty straightforward and not a new technology. I'm outside AWS ecosystem and with just dedicated boxes having some DC burn down is manageable. How do magic clouds make that hard?

"postgres replication" would probably be the least of their worries. It's not about "magic clouds". These are services that are handling millions of requests per second and there's a lot going on where they have to maintain consistency and fail predictably. Having some services go down in one region but being back up in another still serving requests and committing transactions is unpredictable and could create a lot…

Amazon has essentially unlimited resources and manpower to tackle this problem. These excuses you give are pathetic.

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#186
post #148

Earlier quoted context omitted.

Databases, stateful part of services, have matured to have multi-region support. This isn't new either. Nobody's saying that its easy to have multi-region redundancy for stateful services. Its just something you need to have to prevent nasty single region outages affect your service. This is an excellent example where it would have been better to have degraded performance (higher latency) instead of complete unavaila…

Tell that to management of a medium company showing them the bill for something that has 0.01% chance of happening according to AWS.. Not every workload is of the micro size. MR up our databases would cost around 15mil per year for a company that makes 50mil..

Why are you simping for management?

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#187

Earlier quoted context omitted.

GCP is not separate from Google's core infrastructure; rather, it's built on top of it. That means that while you can certainly have GCP specific outages, this kind of core infra "everything is down" situation is almost guaranteed to hit everything, GCP and not included. A lot of GCP sub-products are productionizations of existing Google tech; e.g. BigQuery is a public version of Dremel, an internal database/query en…

There aren’t 100k Googlers developing on top off GCP services to get their job done on a daily basis. That’s the big difference between the two clouds level of dog fooding.

> There aren’t 100k Googlers developing on top off GCP services to get their job done on a daily basis.

Doing something on top of GCP rather than the normal way at Google was a huge pain. Borg tutorials and documentation were just far superior, I could get a thing running on borg in an hour from not knowing anything about borg, I spent a week trying to get something running internally on GCP but still couldn't get it right (our team wanted to see if we could run things on GCP so I was tasked with testing it, I couldn't find anyone who knew how to do it so we just gave up after I didn't make any real progress). That was the worst documented thing I've ever worked with. And even worse the internal GCP pages were probably running in california and probably weren't tested from Europe, so the page took like 2 seconds between mouse click and it responded to anything.

That was years ago though and I no longer work there, but at least back then the work to make using GCP internally seamless wasn't done. Maybe it is simpler if you run everything in it and don't need it play well with borg, but there is a reason why it isn't popular internally. And likely you wont find many engineers who left Google who recommend you will use it, since they probably didn't test it and if they did it probably was a bad experience (unless they worked on GCP).

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#188

Earlier quoted context omitted.

Why is a commuter rail travel system dependent on AWS? O___O

They decided to run their infra in AWS instead of another cloud provider or on-prem? why is McDonalds dependent on AWS for their app to work? (/s)

Did you just equivocate a public transit service with McDonald's?

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#189
post #89

Earlier quoted context omitted.

That's actually pretty nice though; even though people are missing out on packages at least Amazon won't leave them out in the cold for today.

My inner cynic says Amazon paid Flex drivers because A) they still need drivers available during the christmas run-up and B) screwing drivers would put Amazon in an Ebenezer Scrooge-like PR disaster.

Paying them only to avoid the PR disaster is fine. Kinda what PR disasters are “for”

Re: Amazon packages pile up after AWS outage spawns delivery havoc

#190

Earlier quoted context omitted.

People get slagged on for not having inter-region redundancy. But unless your business model can accommodate that, you’re introducing another failure domain and alot of money for duplicative infrastructure, network fees, etc. For many use cases, it’s acceptable to shrug and blame AWS for a failure. It’s harder when your high availability solution fails independently, which they almost always do more than US-East-1

If your site goes down while everyone else's goes down, it's weirdly forgivable. And being the only one up doesn't win as many market cred points and you'd think.

Exactly. Often people want the perception of reliability.

For many services, it makes more sense to make it reliable than not. For other services, it makes more sense to think about the engineering of the solution in the field.

Example: McDonald’s product images on kiosks are apparently in S3 and not cached locally. Seems like a dumb idea to me, but I wouldn’t try to build a more reliable cloud storage backend to control that risk.

Post reply on HN