Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

aws.amazon.com

Post reply on HN
Print view View original

101–110 of 410 posts

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#101

Post by TheP1000 »

cyounkins wrote: post #73
nemothekid wrote: post #50

Earlier quoted context omitted.

I've been naively setting up our distributed databases in separate AZs for a couple years now, paying, sometimes, thousands of dollars per month in data replication bandwidth egress fees. As far as I can remember I've never never seen an AZ go down, and the only region that has gone down has been us-east-1.

Is that separate AZs within the same region, or AZs across regions? I didn't think there were any bandwidth fees between AZs in the same region.

That is incorrect. Cross az fees are steep.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#102

Post by markranallo »

JCM9 wrote: post #4

Obviously one hopes these things don’t happen, but that’s an impressive and transparent write up that came out quickly.

Its not transparent at all. A massive amount of services were hard down for hours like SNS and were never acknowledged on the status page or in this write-up. This honestly reads like they don't truly understand the scope of things effected.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#103

Post by discodave »

cyounkins wrote: post #42

My favorite sentence: "Our networking clients have well tested request back-off behaviors that are designed to allow our systems to recover from these sorts of congestion events, but, a latent issue prevented these clients from adequately backing off during this event."

I saw pleeeeeenty of untested code at Amazon/AWS. Looking back it was almost like the most important services/code had the least amount of testing. While internal boondoggle projects (I worked on a couple) had complicated test plans and debates about coverage metrics.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#104

Post by dylan604 »

tw04 wrote: post #97
amalter wrote: post #19

Earlier quoted context omitted.

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

Multiple AWS employees have acknowledged it takes VP approval to change the status color of the dashboard. That is absurd and it tells you everything you need to know. The status page isn't about accurate information, it's about plausible deniability and keeping AWS out of the news cycle.

>it's about plausible deniability and keeping AWS out of the news cycle.

How'd that work out for them?

https://duckduckgo.com/?q=AWS+outage+news+coverage&t=h_&ia=w...

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#105

Post by codeduck »

iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

carbon copy of our experience.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#106

Post by ProAm »

amalter wrote: post #19
iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

> You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

Let's not act like this is the first time this has happened. It's bad faith that they do not change when their promise is they hire the best to handle infrastructure so you don't have to. It's clearly not the case. Between this and billing I we can easily lay blame and acknowledge lies.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#107

Post by discodave »

dastbe wrote: post #93
discodave wrote: post #52

Earlier quoted context omitted.

Take DynamoDB as an example. The AWS managed service takes care of replicating everything to multiple AZs for you, that's great! You're very unlikely to lose your data. But, the DynamoDB team is running a mostly-regional service. If they push bad code or fall over it's likely going to be a regional issue. Probably only the storage nodes are truly zonal. If you wanted to deploy something similar, like Cassandra across…

> But, the DynamoDB team is running a mostly-regional service. this is both more and less true than you might think. for most regional endpoints teams leverage load balancers that are scoped zonally, such that ip0 will point at instances in zone a, ip1 will point at instances in zone b, and so on. Similarly, teams who operate "regional" endpoints will generally deploy "zonal" environments, such that in the event of a…

Yeah, teams go to a lot of effort to have zonal environments/fleets/deployments... but there are still many, many regional failure modes. For example, even in a foundational service like EC2 most of their APIs touch regional databases.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#108

Post by ProAm »

systemvoltage wrote: post #54
amalter wrote: post #19

Earlier quoted context omitted.

I mean, not to defend them too strongly, but literally half of this post mortem is addressing the failure of the Service Dashboard. You can take it on bad faith, but they own up to the dashboard being completely useless during the incident.

People of HN has been extremely unprofessional with regards to AWS's downtime. Some kind of a massive zeitgeist against Amazon, like a giant hive mind that spews hate. Why are we doing this folks? What's making you so angry and contemptful? Literally try searching the history of downtimes and it was always professional and respectful. Yesterday, my comment was fricking flagged for asking people to be nice to which pe…

> Why are we doing this folks? What's making you so angry and contemptful?

Because Amazon kills industries. Takes job. They do this because they promise they hire the best people that can do this better than you and for cheaper. And it's rarely true. And then they lie about it when things hit the fan. If you're going to be the best you need to act like the best, and execute like the best. Not build a walled garden that people cant see into, and hard to leave.

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#109

Post by discodave »

soheil wrote: post #89

Having an internal network like this that everything on the main AWS network so heavily depends on is just bad design. One does not create a stable high tech spacecraft and then fuels it with coal.

It's 2006, you work for an 'online book store' that's experimenting with this cloud thing. Are you going to build a whole new network involving multi-million dollar networking appliances?

Re: Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region

#110

Post by tempnow987 »

ALittleLight wrote: post #48
iwallace wrote: post #10

My company uses AWS. We had significant degradation for many of their APIs for over six hours, having a substantive impact on our business. The entire time their outage board was solid green. We were in touch with their support people and knew it was bad but were under NDA not to discuss it with anyone. Of course problems and outages are going to happen, but saying they have five nines (99.999) uptime as measured by…

I worked at Amazon. While my boss was on vacation I took over for him in the "Launch readiness" meeting for our team's component of our project. Basically, you go to this meeting with the big decision makers and business people once a week and tell them what your status is on deliverables. You are supposed to sum up your status as "Green/Yellow/Red" and then write (or update last week's document) to explain your stat…

This is not unique. The reason is simple.

1) If you keep status green for 5 years, while not delivering anything, the reality is the folks at the very top (who can come and go) just look at these colors and don't really get into the project UNLESS you say you are red :)

2) Within 1-2 years there is always going to be some excuse for WHY you are late (people changes, scope tweaks, new things to worry about, covid etc)

3) Finally you are 3 years late, but you are launching. Well, the launch overshadows the lateness. Ie, you were green, then you launched, that's all the VP really sees sometime.

Post reply on HN