Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS's us-east-1 region is experiencing issues

health.aws.amazon.com

Post reply on HN
Print view View original

51–60 of 168 posts

Re: AWS's us-east-1 region is experiencing issues

#51

Post by hkt »

fragmede wrote: post #41
halestock wrote: post #17

I can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in these massive tech corporations is disappearing? Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.

You're right, but that's been true since the beginning of the tech boom (but isn't exclusive to tech) when no one works for a place for several decades. Companies weather this in different ways but attrition has always been around. What's causing people to believe that the latest round of attrition is any different?

I'd speculate that perhaps more senior people are moving, and/or a greater overall rate of attrition combined with much more complex technologies and organisations. In other words, it might be harder to become good at jobs now, and fewer people stick with them. Just a hunch but definitely seems to be where the incentives point with loyalty penalties and tech bloat.

Re: AWS's us-east-1 region is experiencing issues

#53

Post by Spooky23 »

fragmede wrote: post #42
frays wrote: post #38

Earlier quoted context omitted.

Increase in attrition across the industry. A lot of institutional knowledge in these massive tech corporations is disappearing and we're starting to reach the tipping point.

But there's always been attrition. What are some of the ways that is now different that is affecting attrition rates and their effects?

Things are bigger anywhere. My colleagues and I thought we’re hot shit managing 5-7k applications and infrastructure. Amazon probably runs 20,000 orgs like mine.

Also, times are good and rates are crazy. Even at VARs, you can make a lot of cash. I have a buddy who went from $150k to $600k. The guy paid off his mortgage and is at a point where he could burn out and work at Home Depot if he needed to.

Re: AWS's us-east-1 region is experiencing issues

#54

Post by hughrr »

zwirbl wrote: post #26
halestock wrote: post #17

I can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in these massive tech corporations is disappearing? Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.

Just like the tech priests in Warhammer 40k, keeping occult old engineering, thatno one could build anymore, running

So today I find out my job title is tech priest. I was happy with necromancer before. Does it come with a pay rise?

Re: AWS's us-east-1 region is experiencing issues

#55

Post by atty »

viraptor wrote: post #33
zwirbl wrote: post #26

Earlier quoted context omitted.

Just like the tech priests in Warhammer 40k, keeping occult old engineering, thatno one could build anymore, running

Not familiar with 40k. Was it a similar idea to nuclear-power-as-religion from Foundation?

Not far off. The “golden age” of humanity was shattered long ago, with the mortal wounding of the god emperor, and knowledge of most of the greatest technology was lost.Millennia later, a cult has grown up that both worships and maintains technology as having machine spirits, which are somehow linked to the machine god itself. That god may or may not be the same or related to the god emperor of mankind, depending on the interpretation.

Honestly the lore of w40k is quite fun to read, if you’re into dystopian and fantasy sci-fi.

Re: AWS's us-east-1 region is experiencing issues

#56

Post by aaronax »

viraptor wrote: post #33
zwirbl wrote: post #26

Earlier quoted context omitted.

Just like the tech priests in Warhammer 40k, keeping occult old engineering, thatno one could build anymore, running

Not familiar with 40k. Was it a similar idea to nuclear-power-as-religion from Foundation?

Or how about Anathem, with the Ita class doing computer things and nuclear materials cared for by a select group?

Re: AWS's us-east-1 region is experiencing issues

#58

Post by hughrr »

nyellin wrote: post #45
halestock wrote: post #17

I can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in these massive tech corporations is disappearing? Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.

That's the problem we're out to solve with robusta.dev. We're slowly but surely converting the world's institutional technical knowledge into re-usable and automated runbooks.

I’m just going to have to spend all day fixing the runbooks as well as the technology ;)

Re: AWS's us-east-1 region is experiencing issues

#59

Post by Traster »

halestock wrote: post #17

I can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in these massive tech corporations is disappearing? Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.

I think this is a transient issue. When you're in growth mode you make a huge series of hacks to just keep things running and then when you leave.... well, it's a problem. But if the business is robust, and lives beyond you, what replaces your work is better documented, better tested, and maintainable.

That's the dream. Obviously there are companies that sink between v1 and v2, but that's life.

Fundamentally I think the cloud business is robust, it's a fundamentally reasonable way of organising things (for enough people), which is why it attracts customers despite being arguably more expensive.

I've been in this situation in much smaller scales, and yes, you'll see massive drop in productivity but that's the cost of going from prototype to product.

Re: AWS's us-east-1 region is experiencing issues

#60

Post by hughrr »

didip wrote: post #31
xilni wrote: post #8

This is why you are strongly urged not to rely on one region or AZ.

Multi AZ is great and should be by default, but multi Region is expensive.

This. We have multi AZ in more than one region and I occasionally dream of Bezos wearing only a top hat and waistcoat laughing manically while diving into a large vat of gold coins.
Post reply on HN