Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

171–180 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#171

Post by [deleted] »

sroussey wrote: post #101
porknubbins wrote: post #69

Earlier quoted context omitted.

The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivat…

I had MySQL running on some bare metal for many years without a restart. I was terrified to update the kernel at that point, knowing that system disk had been running continuously for many years, and had no faith it would restart successfully. Finally got two new servers to replace these (with these new SSD things!) and after migration, sure enough, one of the old servers failed to boot.

[deleted]

Re: Why Twitter didn’t go down: From a real Twitter SRE

#172

Post by AtlasBarfed »

mikeweiss wrote: post #135
hn_throwaway_99 wrote: post #123

Earlier quoted context omitted.

I did see a lot of politically-motivated salivating which I thought was woefully premature. That said, I see 2 outcomes now: 1. Generally, with large, complex systems like this, everything works, until it doesn't. All the big boys have major outages periodically. I just can't fathom how Twitter is going to handle the eventual certainty of a major outage when, as the author notes, in some cases there are teams that ha…

Wait Musk saddled Twitter with debt? Please elaborate...

Yeah, it's private equity 101. I can't believe it happens. A bunch of MBA sharks get loans to buy something with some chop shop plan or other ruthless scheme to carve up or spitshine a company.

The loans are then assigned to the company they bought, rather than the core sharks ... uh investors. The "investors" then repackage/selloff/spit shine as necessary to get it resold to some sucker. The investors get THEIR money back with a profit, leave the debt to the company and the sucker who buys it, and look for the next victim/target.

Musk likely went the rapacious route because he made a dumb move, and this is the only way he's getting the money he committed to the deal back.

Oh yeah, and when they buy the company, they are in charge of it. So they can pay themselves whatever they want in executive bonuses / etc.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#173

Post by MuffinFlavored »

frognumber wrote: post #155
MuffinFlavored wrote: post #131

Earlier quoted context omitted.

> Twitter grew 3x on the headcount front There were multiple executives making $10m/yr+ There were board members There were shareholders Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"? These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc. How can us commenters on HackerNews sit from our arm…

It's not goofballs. It's generally misaligned incentives. Managing a 10,000 org leads to better job prospects than a 1000 person org, than a 100 person org, than a pizza box team. Organizations tend to bloat. Random, rapid cuts might not be the fix here, but headcount was too high.

> Organizations tend to bloat.

Say I'm an engineer. And I write code all day. And I get paid by a company to do it. And it blows up in production. And I tell my boss "code tends to blow up".

Or I tell my boss "code tends to take longer than estimated to deliver".

I wouldn't be given lots of promotions/bonuses with that outlook.

We're talking about a Twitter CEO with $30m/yr+ in total compensation.

There's no way his/her view was "shrug, organizations tend to bloat" while racking in extremely competitive + good pay as somebody whose main job is to drive a company towards maximum growth/profitability.

There's no way the CEO was able to convince a majority of people responsible for paying him (the board), to pay award him $30m/yr in stock options, while he was also "bad at his job enough" to let the organization bloat without as much as an afterthought to it.

We, the people on the outside looking in, have to be missing something.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#174

Post by inferiorhuman »

subroutine wrote: post #115
halfmatthalfcat wrote: post #70

Earlier quoted context omitted.

This is the real question? Your question has nothing to do with the blog post and if you take a look around, what Twitter did was literally done across the entire industry, hence all the layoffs recently. There was a hiring glut to take advantage of cheap capital during COVID recovery. The capital has dried up, glut has ended and a lot of people lost their jobs. Why is that so hard to see? None of this is unique in a…

A bunch of people just got axed from Twitter because covid cash dried up? Shit I thought it was because Elon took over and fired anyone who refused to work at the office instead of at home.

Now to be fair, as of last week Musk-o backed off the mandated return-to-office. Turns out that didn't quite go as planned.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#176

Post by randyrand »

dboreham wrote: post #151

...because it's only been a week and it takes longer for things to rot badly?

servers don’t really rot if they don’t rely on 3rd party cloud software.

Linux doesn’t upgrade itself.

the web is very standards based.

Hacker news has pretty much been running for decades on the same software hardware stack

Re: Why Twitter didn’t go down: From a real Twitter SRE

#177

Post by dcow »

cratermoon wrote: post #156
fenomas wrote: post #139

Earlier quoted context omitted.

To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...

He also sold all the smoke detectors.

Credit where it’s due though: he did supply a spare sink.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#178

Post by libraryofbabel »

fenomas wrote: post #139
emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...

Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to.

So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building. But eventually people are going to start turning on stoves again… and then…

Re: Why Twitter didn’t go down: From a real Twitter SRE

#179

Post by wwweston »

taolegal wrote: post #41
philistine wrote: post #33

Earlier quoted context omitted.

And Twitter's problem are nowhere near technological. The site needed to make more money, not reengineer the whole thing while advertisers are fleeing because Trump is back on on a whim!

So should the platform be guided by advertisers? Especially one that’s apparently the de facto public square?

If that's it's business model then sure, that's how it works.

A "de facto public square" would be public in conception, construction, and support from the start, which is one of the ways we know that Twitter is no such thing. Though it would likely also have some rules for how speech is/isn't conducted.

And all things considered, advertiser-friendliness is a sort of low-resolution but approximate passable democratic mechanism for marking boundaries of civilized discourse.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#180

Post by 8n4vidtmkvmk »

emodendroket wrote: post #66
raldi wrote: post #24

The most helpful thing to reflect on in these Twitter operational discussions is the difference between homeostasis and evolution. You can get rid of 80% of the work force and the existing homeostasis systems will keep things running smoothly despite known day-to-day chaos. Where you’re really going to run into trouble is inventing responses to novel chaos and gradually changing times.

I think this is kind of baked in though. Part of the thought process seems to be, at least for non-paying customers, it's not actually necessary to have five nines for Twitter, because people will just put up with it if it's less reliable.

true. if twitter, Facebook, reddit, and hackernews go down for a couple days it wouldn't affect me at all. if GitHub and npm went down I'd me mildly annoyed but could still work.
Post reply on HN