Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

231–240 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#231

Post by [deleted] »

AtlasBarfed wrote: post #172
mikeweiss wrote: post #135

Earlier quoted context omitted.

Wait Musk saddled Twitter with debt? Please elaborate...

Yeah, it's private equity 101. I can't believe it happens. A bunch of MBA sharks get loans to buy something with some chop shop plan or other ruthless scheme to carve up or spitshine a company. The loans are then assigned to the company they bought, rather than the core sharks ... uh investors. The "investors" then repackage/selloff/spit shine as necessary to get it resold to some sucker. The investors get THEIR mone…

[deleted]

Re: Why Twitter didn’t go down: From a real Twitter SRE

#233

Post by afavour »

Andrew_nenakhov wrote: post #193
inferiorhuman wrote: post #185

Earlier quoted context omitted.

He rolled in, sold most of the fire extinguishers, and made a big show of trying to make cherries jubilee while shoving his dick in one of the few remaining fire extinguishers. Let's be clear, it isn't just the erratic layoffs it's Musk's incessant meddling that's going to be Twitter's downfall. He literally took down SMS based 2FA because "microservices bad". He fired the payroll and tax departments (HR too?). He's…

> However failure is pretty much inevitable at this point I'm pretty sure there will be no failure at all, and Twitter will work just fine.

There have already been failures. If I recall the system for sending two factor auth codes via SMS was down for the best part of a day.

Not the end of the site by any means but cracks are showing.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#234

Post by nprateem »

MuffinFlavored wrote: post #173
frognumber wrote: post #155

Earlier quoted context omitted.

It's not goofballs. It's generally misaligned incentives. Managing a 10,000 org leads to better job prospects than a 1000 person org, than a 100 person org, than a pizza box team. Organizations tend to bloat. Random, rapid cuts might not be the fix here, but headcount was too high.

> Organizations tend to bloat. Say I'm an engineer. And I write code all day. And I get paid by a company to do it. And it blows up in production. And I tell my boss "code tends to blow up". Or I tell my boss "code tends to take longer than estimated to deliver". I wouldn't be given lots of promotions/bonuses with that outlook. We're talking about a Twitter CEO with $30m/yr+ in total compensation. There's no way his/…

This is exactly what happens. It's also why CEOs buy other firms even though research shows the majority of takeovers fail. Just pick up any management book in the last 20 years.

There's a clear correlation between org size and CEO pay, even when not in the company's interests. Some CEOs and managers also love the power. Politics is also at work.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#235

Post by midoridensha »

bigtones wrote: post #17
uluyol wrote: post #8

I wonder how big Twitter's infra costs are (capex). Based on the article, they need to build capacity > 2x peak loaf to tolerate one of the DCs failing. But if they instead had 4 and still only tolerated 1 failure, they'd only need capacity > 1.33x peak load. This doesn't account for the extra overhead associated with extra DCs, but it seems like there's opportunities for major effeciency wins.

Well Elon is planning on shutting down one of their three data centers, the Sacramento data center - so they must have a lot of extra capacity. https://www.datacenterdynamics.com/en/news/report-elon-musk-...

Go Elon! Shutting down a datacenter will save a lot of energy. I applaud this move. (Though I suppose they could just sell it to some other company.)

Now he just needs to shut down the other 2.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#236

Post by jonas21 »

fenomas wrote: post #139
emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...

Perhaps a closer analogy would be that the new boss rolled in and threw away 80% of the fire extinguishers.

Whether that will spell disaster when there's a fire depends on whether the building had too many fire extinguishers to begin with and whether the boss can buy new, better fire extinguishers to replace some of them before there's a fire.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#237

Post by oska »

raldi wrote: post #24

The most helpful thing to reflect on in these Twitter operational discussions is the difference between homeostasis and evolution. You can get rid of 80% of the work force and the existing homeostasis systems will keep things running smoothly despite known day-to-day chaos. Where you’re really going to run into trouble is inventing responses to novel chaos and gradually changing times.

We had an obituary for Fred Brooks on here just the other day. I'd suggest that his thesis in The Mythical Man-Month conflicts with your comment above (that reduction in staff count for a software project has a good correlation with the ability to maintain it / evolve it / innovate on top of it).

Re: Why Twitter didn’t go down: From a real Twitter SRE

#238

Post by DSMan195276 »

emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

> but it was never going to happen overnight and I'm not sure why news outlets were saying that

I don't think it's that obvious, it's one thing to just leave everything running but Elon was also talking about changes he was making (Ex. turning off a bunch of "microservices" because they don't do anything). If you turn off the wrong thing and don't have anybody left who knows how to properly turn it back on again, then you're in a pretty bad situation. It doesn't seem like that happened, but I don't also think we have enough information to say how close it was to happening.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#239

Post by bboygravity »

jandrese wrote: post #220
libraryofbabel wrote: post #178

Earlier quoted context omitted.

Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to. So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building . But eventually people are going to start turning on stoves again… an…

Except that he at the same time demanded that people invent an entirely new dish by the end of the week, and now they are scrambling to try to figure it out. Already the DMCA auto-takedown bot is apparently broken and people are posting entire movies on Twitter. I would expect other peripheral systems to start breaking down as nobody is maintaining them even as other parts of the system are being changed.

oh noes! Not the DMCA auto-takedown bot!

Said no-one in the entire world except a hand-full of Hollywood studio owners.

That bot shouldn't have existed in the first place, but I know that that falls under "just world fallacy" and is a naive thought.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#240

Post by inferiorhuman »

Andrew_nenakhov wrote: post #193
inferiorhuman wrote: post #185

Earlier quoted context omitted.

He rolled in, sold most of the fire extinguishers, and made a big show of trying to make cherries jubilee while shoving his dick in one of the few remaining fire extinguishers. Let's be clear, it isn't just the erratic layoffs it's Musk's incessant meddling that's going to be Twitter's downfall. He literally took down SMS based 2FA because "microservices bad". He fired the payroll and tax departments (HR too?). He's…

> However failure is pretty much inevitable at this point I'm pretty sure there will be no failure at all, and Twitter will work just fine.

On Hacker News, everyone's a comedian! And, yeah, as the sibling comment pointed out there've already been failures as a result of some musky action. While Twitter isn't likely to fail on its own, E-Lon is actively causing problems. You need people to deal with that, and even if he had motivated, relevant, and competent engineers… how long will they stay motivated without a paycheck?

Let's not forget that whatever code monkeys are left are now personally liable for running afoul of the FTC. Whatever motivation they may have now will run out pretty damn quick once they stop getting paid.

Post reply on HN