Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

691–700 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#691

Post by usrusr »

AtlasBarfed wrote: post #164

Twitter is a lot of drama, but ultimately this is a common private equity play: 1) take private 2) fire/layoff until things break, patch up / rehire with cheaper labor. Repeat as necessary 3) spit shine/repackage what's left with a theoretically more appealing balance sheet. 4) resell to another sucker...uh, buyer, or take public again. Fundamentally Musk bought 44 billion for 5 billion of annual revenue, and presuma…

4) who'd that sucker be? How much hubris would it take to sell investors on the idea of salvaging a failing company that "the guy who killed the internal combustion engine" and "the guy who made rockets land on their tail" couldn't make profitable? If it comes to selling off a failure, Musk's reputation will be quite a dealbreaker. The days of whatever happens, if all else fails we can still sell to Yahoo are long gone. Perhaps China might be interested, for obvious reasons, but surely not after Twitter has gone down fighting, fading away into irrelevancy in the process.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#692

Post by marstall »

dfghbbjiiugb wrote: post #503
mjr00 wrote: post #451

Earlier quoted context omitted.

It's crazy to think about, but many people who use and build software today, including HN readers/commenters, are young enough to have only been exposed to the SaaS, cloud-first era, where software built with microservices deployed from CI/CD systems multiple times per day is just the way things are done. You're totally right; if you don't make changes to the software, it's unlikely to spontaneously stop working, esp…

Your broad point is obviously correct (most outages are caused by code or config changes) but there are still classes of failures that can happen without any real changes, like various performance degradations (maybe your table grows too large) or occasional catastrophic failures from things like disk space or id overflow or something. There's also the stability of third party systems: forced deprecations, security E…

also: regulatory changes. if you can't function within the law, that's tantamount to a critical bug.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#693

Post by pjc50 »

drcross wrote: post #592
Moissanite wrote: post #580

Earlier quoted context omitted.

The fact that he was able to "buy" Twitter and yet transfer a significant amount of debt to the company rather than being liable himself is just another sign of how different "rich people accounting" is. He will walk away from this with a bit less theoretical money but no material impact to his life, while thousands of people are having their lives up-ended. How long are we going to keep letting shit like this happen…

Debts from where? Any evidence of this at all? It seems like fantasy considering how much money the rest of Musks companies have in the bank.

https://www.thenationalnews.com/business/banking/2022/10/09/...

"Lenders that also include Bank of America, Barclays and Mitsubishi UFJ Financial Group committed to provide $13 billion of debt financing for the deal. Their losses would amount to $500 million or more if the debt were to be sold now, according to Bloomberg calculations."

Re: Why Twitter didn’t go down: From a real Twitter SRE

#694

Post by gjulianm »

jillesvangurp wrote: post #514
ulkesh wrote: post #27

All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…

You underestimate Elon Musk. Many people have done that before and lost that bet. If anything, he repeatedly succeeded in building world class software and hardware teams for Tesla, SpaceX, and a few other companies. The notion that he won't be able to attract world class talent is ludicrous. Yes, he is a bit of a liability and his management style is obnoxious and unconventional. But he does get things right once in…

> If anything, he repeatedly succeeded in building world class software and hardware teams for Tesla, SpaceX, and a few other companies

The point is that Twitter doesn't really needed someone to build a world class software and hardware team. The technical challenges in reliability and speed seemed pretty much solved or on track to be solved already. The problem of Twitter was that they never knew how to properly manage the community and make the company profitable.

Twitter doesn't have a tech problem, it has a community problem.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#695

Post by pjc50 »

EricLeer wrote: post #618
unculture wrote: post #590

Earlier quoted context omitted.

It’s going to be something like that. He bought it for 44 billion. That was mostly loans. Those loans are transferred to the company and the company pays interest on them.

But who is issuing these loans? It should be pretty obvious that you would never see that money back right?

Per link upthread: Bank of America, Barclays and Mitsubishi UFJ Financial Group, apparently. I've also heard Morgan Stanley mentioned.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#696

Post by pjc50 »

_3u10 wrote: post #593
Dave3of5 wrote: post #571

I don't think it will fail on a technical level. As this article says lots of engineering has gone in to make the thing pretty resilient. I would also say that there are still enough engineers who work there who can figure out what's gone wrong and "turn it on and off again" or w/e makes it splutter back to life. In terms of changes to the Platform ditto. It's not difficult to make these changes that a team of 100's…

If you consider that engineers cost about $1000 a day and he got rid of 4,000 of them… oddly it works out to about $4 million per day. I’m not an expert in math but it’s seems pretty possible. Plus they’ll have a lot more impressions to sell now that people are allowed to speak. Ad rates might drop but then someone will write an article about how they are getting great CPC on Twitter and everything will be back to no…

> allowed to speak

Advertisers have already said they don't want their ads screenshot next to slurs.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#697

Post by a1445c8b »

8n4vidtmkvmk wrote: post #180
emodendroket wrote: post #66

Earlier quoted context omitted.

I think this is kind of baked in though. Part of the thought process seems to be, at least for non-paying customers, it's not actually necessary to have five nines for Twitter, because people will just put up with it if it's less reliable.

true. if twitter, Facebook, reddit, and hackernews go down for a couple days it wouldn't affect me at all. if GitHub and npm went down I'd me mildly annoyed but could still work.

As long as you don’t mention Stack Overflow, I agree!

Re: Why Twitter didn’t go down: From a real Twitter SRE

#698

Post by itsoktocry »

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

>The biggest thing that brings down a site is changes.

Well, Elon is talking about a massive amount of changes coming down the pipe, so I guess we'll see how that goes!

Re: Why Twitter didn’t go down: From a real Twitter SRE

#699

Post by scott_w »

coldtea wrote: post #535
oxfordmale wrote: post #443

Earlier quoted context omitted.

Let's do a thought experiment and see what functions aren't needed to keep the light on for 30 days: 1) HR 2) Legal 3) Sales 4) Marketing 5) Payroll 6) Admin staff 7) Most of Engineering, other than the bare minimum of L1/L2/3 support. As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend.

> As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend. On the other hand, a car without a second and third steering wheel, 20 windscreen wipers, and an oven in the back, keeps running just fine, even after the first bend...

That might be fine but do you want to fly in a plane that only has one engine? What about if it has no fallback if the main sensors freeze?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#700

Post by goosedragons »

fnord123 wrote: post #472
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

Twitter had 7,500 employees. most of the roles you mention (security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, scrum master) are not bloat. So the question is what are the other 7000 people doing?

They had over a thousand moderators. So maybe your estimate of how many people is required is a bit off.
Post reply on HN