Live data from Hacker News

AWS's us-east-1 region is experiencing issues

health.aws.amazon.com

71–80 of 168 posts

Re: AWS's us-east-1 region is experiencing issues

#71
post #38

What’s up with all of the multi-platform outages lately? Seems abnormal looking at historical data. Are there issues affecting the internet backbone or something? Or just a coincidence?

Increase in attrition across the industry. A lot of institutional knowledge in these massive tech corporations is disappearing and we're starting to reach the tipping point.

See my comment here: https://news.ycombinator.com/item?id=30621190

Re: AWS's us-east-1 region is experiencing issues

#72

I can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in these massive tech corporations is disappearing? Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.

Disc: Googler opinions are my own. I've been an SRE on a tier 1 GCP product for over three years this is not the case. In my experience, our systems have only gotten more reliable and easier to understand since I joined. It's not like there are only a few key knowledge holders that single handedly prevent outages from happening. In reality, you don't need to know shit about how a system works internally to prevent ou…

I’ll be honest. My external impression of Amazon and Google could not be more distant in this regard.

Google doesn’t have nearly as hard a time retaining good people as Amazon does.

Re: AWS's us-east-1 region is experiencing issues

#73
post #26

Earlier quoted context omitted.

Just like the tech priests in Warhammer 40k, keeping occult old engineering, thatno one could build anymore, running

If we want to normalize letting long term support people call themselves tech priests I'd very much appreciate it. "What were your duties at your last position?" "Performing the daily ministrations and singing the praise of the machine god."

[deleted]

Re: AWS's us-east-1 region is experiencing issues

#74
post #56

Earlier quoted context omitted.

Not familiar with 40k. Was it a similar idea to nuclear-power-as-religion from Foundation?

Or how about Anathem, with the Ita class doing computer things and nuclear materials cared for by a select group?

That was probably one of Stephenson’s most insightful moments of science fiction.

Scientists only get to talk to the public every 100 years or something, wasn’t it?

IT was never allowed to talk to scientists.

Seemed like a modernist idea, even at the time of publishing.

Re: AWS's us-east-1 region is experiencing issues

#75

I can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in these massive tech corporations is disappearing? Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.

You're right, but that's been true since the beginning of the tech boom (but isn't exclusive to tech) when no one works for a place for several decades. Companies weather this in different ways but attrition has always been around. What's causing people to believe that the latest round of attrition is any different?

> What's causing people to believe that the latest round of attrition is any different?

The Great Recession

The Great Resignation

The Great Dying (due to COVID-19)

Re: AWS's us-east-1 region is experiencing issues

#76
post #55

Earlier quoted context omitted.

Not familiar with 40k. Was it a similar idea to nuclear-power-as-religion from Foundation?

Not far off. The “golden age” of humanity was shattered long ago, with the mortal wounding of the god emperor, and knowledge of most of the greatest technology was lost.Millennia later, a cult has grown up that both worships and maintains technology as having machine spirits, which are somehow linked to the machine god itself. That god may or may not be the same or related to the god emperor of mankind, depending on…

FWIW hardline tech priests view the machine god as separate from the emperor. Hardline imperium officials view the emperor as separate from the machine god. The official party line is that the emperor and the machine god are the same, with the emperor perhaps being an avatar.

It seems like both sides are fine to have them be reconciled, but it's an important narrative gadget that can be used to get humanity to fight itself in-universe.

Also interesting is that humanity's "lost" technological progress seems to eclipse some of the other races in the W40K narrative, with even the Tau (space dwarves with robots) and Eldar (space elves with crystals) freaking out when humanity brings giant robots, because the sheer physical impracticality of a gigantic human shaped robot is noted, with nobody aware of how they continue to work.

Re: AWS's us-east-1 region is experiencing issues

#78
post #72

Earlier quoted context omitted.

Disc: Googler opinions are my own. I've been an SRE on a tier 1 GCP product for over three years this is not the case. In my experience, our systems have only gotten more reliable and easier to understand since I joined. It's not like there are only a few key knowledge holders that single handedly prevent outages from happening. In reality, you don't need to know shit about how a system works internally to prevent ou…

I’ll be honest. My external impression of Amazon and Google could not be more distant in this regard. Google doesn’t have nearly as hard a time retaining good people as Amazon does.

From within AWS, it just feels like we push people too hard. Service teams are too small relative to their goals, sales teams have unrealistic growth targets and also double as support for needy/incompetent customers, and professional services have billable hour requirements which are as high as any major consultancy and with additional pre-sales support expectations.

Strict adherence to the "hiring bar" means we fail to bring in good people who aren't desperate enough to act out the cultish LP dance during their interview. Hiring new grads seems to be the only area where growth is not stalling - but that can obviously only help so much.

My team is hiring for 2-3 people and we are being buried alive without that growth happening sooner - but I can't in good conscience recommend this place to anyone I respect or like.

Re: AWS's us-east-1 region is experiencing issues

#79
post #26

Earlier quoted context omitted.

Just like the tech priests in Warhammer 40k, keeping occult old engineering, thatno one could build anymore, running

If we want to normalize letting long term support people call themselves tech priests I'd very much appreciate it. "What were your duties at your last position?" "Performing the daily ministrations and singing the praise of the machine god."

As an archivist/librarian-programmer, I'm totally here for calling myself a tech priestess.

Re: AWS's us-east-1 region is experiencing issues

#80

What’s up with all of the multi-platform outages lately? Seems abnormal looking at historical data. Are there issues affecting the internet backbone or something? Or just a coincidence?

A handful of large-traffic sites have recently, and relatively suddenly, started blocking traffic from a large region. That's a major change in flow.
Post reply on HN