Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

791–800 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#791

Post by trashtester »

matwood wrote: post #752
itsoktocry wrote: post #676

Earlier quoted context omitted.

> removing engineers won't instantly crash the product. It'll happen slowly It's amazing to me how many people following the Twitter saga, some familiar with or actually working in technology, thought that Twitter would crash within days of the engineers being fired. And because it didn't, the job cuts are justified.

I agree those were odd takes. I've likened firing most of the engineers to taking your hands off the wheel in the car. It won't crash immediately, but it doesn't mean the car can go driverless. With that said, there are differences between internal systems and something like Twitter on the public internet. I assume that Twitter is a system under constant attack. What happens when the next log4shell level vulnerabilit…

> It won't crash immediately, but it doesn't mean the car can go driverless.

Unless the company that creates it is owned by Elon ;)

Re: Why Twitter didn’t go down: From a real Twitter SRE

#793

Post by matwood »

kortilla wrote: post #565
mschuster91 wrote: post #517

Earlier quoted context omitted.

> I mean wholly crap, this reminds me of old German industry where people retire in the place they started. German here. I think this actually is a huge part of the success of the famous Mittelstand - all that institutional knowledge these people have is extremely valuable. It's not just basic stuff like "know time tracking, billing and other admin systems and internal processes", but also the stuff that really can s…

> the problem is US-centric capitalism cannot quantify that impact (and it doesn't want to, given that attrition-related problems are long-term issues with years of time to impact) This is false. People invest on many year horizons all of the time when they believe in the company. If there was good evidence that having an average tenure of 5+ years was a great boost to the company, people would clamor to invest in co…

> Long tenure alone is absolutely not an advantage.

It's also not necessarily a disadvantage. The real argument is that people management is important. Everyone from the new person who just joined to the decade long senior engineer need to be held to high standards. Which brings me to my next point.

The big problem with Twitter was management. Dorsey was barely a CEO for many years. Parag didn't seem to want to try anything. I think people routinely underestimate how important good leaders are. "There are no bad teams, only bad leaders" comes to mind. So far Elon has shown himself to be terrible in this regard. With SpaceX and Tesla he communicated a vision, with Twitter he's communicated very little and just instituted chaos.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#794

Post by ransom1538 »

robswc wrote: post #43

It was if I was in two realities the other day. I did experience a few bugs but twitter never went "down" for me, despite everyone claiming that 1) it would and 2) it already did. All 10 of the "top trends" for me were about how twitter was dead/dying and armchair experts on reddit said it was only a matter of hours at this point. Maybe it was close (in reference to external factors not implied in your very insightfu…

As a SRE one of two things happen when you are laid off:

1) The site goes down [you look like you can't do you job]

2) The site stays up [you can do your job] but the lay off looks like a good decision.

You are screwed either way.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#795

Post by jokethrowaway »

joenot443 wrote: post #763
matwood wrote: post #752

Earlier quoted context omitted.

I agree those were odd takes. I've likened firing most of the engineers to taking your hands off the wheel in the car. It won't crash immediately, but it doesn't mean the car can go driverless. With that said, there are differences between internal systems and something like Twitter on the public internet. I assume that Twitter is a system under constant attack. What happens when the next log4shell level vulnerabilit…

If Twitter went another month without an outage, how would you adjust your opinion? How about a year? The car analogy is amusing, but how much does it really hold up? Have we ever seen another major social media company drop this much of its staff in one go? I certainly can’t think of an example. I think we’re in somewhat uncharted waters here. A driverless car won’t last long, we know that for a fact. I think it rem…

I've seen that happens on a smaller scale.

Company soft killed the product, everybody left, they didn't hire anyone to replace. We went from 20 engs to 3. Worst codebase ever made by ex FANGs hotshots who thought they understood something about system architecture. Very "clever" and complicated. Data consistency issues happening everyday, likely due to misuse of messagging queues. Chargebacks being ignored and mailed in physical letters every month. A couple of millions going through the platform every year.

My task was to run a team of mostly juniors maintaining and adding features to that mess.

I had no clue what that codebase was doing. We just left things as they were, fixing fires as they came. Nothing too bad happened. Slowly built a leaner replacement for some components. We simplified things over time and we even rebuilt some of the knowledge of the old platform, which helped with the daily outages.

The issues started happening not as often. Eventually. I moved on from that company, removing again a big chunk of knowledge. Over time I've heard tales of other people coming in and rebuilding that knowledge, over and over.

The platform is still standing.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#796

Post by toss1 »

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

>>If anything, the article might actually persuade me that it was all bloat.

Not for me

This is almost exactly like the new manager coming in, noticing that the floors and surfaces are all clean, all the systems work, the trash is emptied, etc., and so deciding that the entire maintenance staff is unnecessary and firing them.

The place doesn't become a decrepit pigsty the next morning; it slowly degrades.

Same for these systems. They were designed, built, tuned, and maintained over the course of years to go from requiring constant manual intervention to running largely unattended and with a good buffer of ready hardware and automatic failover for failures. That "largely" in "largely unattended" is doing some very heavy lifting.

The system WILL require human intervention to keep running, and more than just a skeleton crew. The only question is whether it will happen before the new crew gets up to speed to handle the inevitable degradation.

This does NOT mean that the SREs were bloat - it means that they were doing an excellent job and could safely take a break. We're now in just the two-week vacation zone - same as if the entire SRE team went on a holiday. We'd expect it to work. Now let's see what happens in two months.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#798

Post by kranke155 »

entropy_ wrote: post #770
kranke155 wrote: post #705

Earlier quoted context omitted.

Software engineers at Twitter got used to working on an money losing company for 10 years and being told they’re great at their job. Then they were fired en masse because someone’s called them out on it. If your company is losing money all this time you are likely to be fired eventually in the real world. Job security in sw world had become so high that no one seemed to expect it. Everyone assumed “sure we’re losing…

This is factually untrue though. Twitter was making money in 2018 and 2019 (to the tune of ~1.2B/year in net profit out of ~3B revenue, which is a fairly high profit margin) they lost quite a bit of money in 2020 and less in the years thereafter. However, even in the years where they had negative net income EBITDA remained positive suggesting the losses were upfront investments that would be expected to be amortized…

I stand corrected. My impression was they always lost money.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#799

Post by acdha »

NaturalPhallacy wrote: post #291
acdha wrote: post #95

Earlier quoted context omitted.

Want to bet that’s not unconnected to sacking the people who tried to block bots?

Sure. Who do you have on the inside who can publicly verify that you're correct?

Their previous CEO said they saw on the order of 500k bot signup attempts daily, and the reports last week specifically mentioned layoffs affecting those teams and things like reporting. Put that together and I’d be careful about comparing numbers before and after this month since they’re almost certainly not measuring the same thing.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#800

Post by ilaksh »

jeffbee wrote: post #81
robswc wrote: post #43

It was if I was in two realities the other day. I did experience a few bugs but twitter never went "down" for me, despite everyone claiming that 1) it would and 2) it already did. All 10 of the "top trends" for me were about how twitter was dead/dying and armchair experts on reddit said it was only a matter of hours at this point. Maybe it was close (in reference to external factors not implied in your very insightfu…

A few parts clearly did go down. 2FA login was just serving error codes all day a few days ago. On Saturday people were posting entire feature films, 2 minutes per tweet, because apparently their copyright content matching wasn’t working.

I would use it as a case in point. You have one group claiming that Twitter is completely on fire and soon to be closed, and the other saying it's never been better and there was zero fallout from rapidly firing thousands of people including a lot of engineers.

Neither of those extreme viewpoints reflected reality accurately. There were significant problems, but they did not take the entire site down or anything. In fact, for millions who did not have 2FA or use it at that point, they did not see any issue. Whereas people that wanted to find an issue could go looking for the movies. So each side was able to find ways to reinforce their alternate realities.

Post reply on HN