Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

311–320 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#311

Post by silisili »

dcow wrote: post #110
porknubbins wrote: post #69

Earlier quoted context omitted.

The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivat…

I have a piece of Rust software that has not gone down in its entire lifetime.

The infra usually matters way more than the code. RAM or a disk will typically fail before the Linux kernel, and it's written in the boogeyman language.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#312

Post by whatshisface »

memish wrote: post #284
headsoup wrote: post #42

Earlier quoted context omitted.

Because politics and ideologues are so prevalent and everyone's attaching 'the other' (both sides) for the own. And of course people in 'important' roles who've been laid off are going to say the company is doomed, these are the probably the worst source to go off. They're not going to say "oh yeah I didn't do much at all really, just bossed people around and spent my budget every year."

It's actually scary how many people, even engineers, put their reputation on the line saying Twitter wouldn't survive the weekend. It wasn't just Twitter employees. It's like a mass psychosis of some kind. It comes off as a kind of desperation, as though they need Elon to fail. Why? What's driving that response?

The only way a herd can move is by becoming confident that it is moving.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#313

Post by elgenie »

mikeweiss wrote: post #135
hn_throwaway_99 wrote: post #123

Earlier quoted context omitted.

I did see a lot of politically-motivated salivating which I thought was woefully premature. That said, I see 2 outcomes now: 1. Generally, with large, complex systems like this, everything works, until it doesn't. All the big boys have major outages periodically. I just can't fathom how Twitter is going to handle the eventual certainty of a major outage when, as the author notes, in some cases there are teams that ha…

Wait Musk saddled Twitter with debt? Please elaborate...

It’s quite bad for the business as a going concern and its ability to serve customers and employ employees when it suddenly, and for no obvious reason, takes on tons of debt. Money isn’t free and loans demand payment.

In Twitter’s case, its $13B in new debt on the balance sheet means that, every year, they have to come up with $13B times the interest rate in additional revenue and/or reduced cost merely to be in the same place profit/loss-wise. Elon already massively overpaid what Twitter’s business-as-usual can generate even before accounting for that $13B; as a result, the post-acquisition Twitter has to try a bunch of negative-expected-value moonshots in the unlikely hope that one of them hits against the odds and the others don’t turn out fatal, because doing nothing or iterating sustainably kills the company via debt service.

In a lot of cases the best solution would be for the company to declare bankrupcy, reorganize, and discharge the debt by convincing creditors to take pennies on the dollar and a share of the resulting smarter-run healthier company, and a judge that the plan is reasonable. However, Elon both poisoned that well by firing people, angering advertisers, and bumbling around product-wise, and also staked a bunch of $TSLA that would need to be liquidated to go through with the bankruptcy.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#314

Post by npn »

jdminhbg wrote: post #157
dboreham wrote: post #151

...because it's only been a week and it takes longer for things to rot badly?

I think this post is aimed at non-technical "RIP Twitter" posters from last week who are surprised it didn't fall apart in 24 hours.

There was a long twitter thread from some one proclaimed 10+ years SRE listing a lot of things that could bring the server down. Lot of them are pretty far-fetched and outdated.

Too bad I did not bother to save the link.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#316

Post by edgyquant »

emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

This whole thing has been quite telling for me. I don’t use Twitter and am not a Musk fan, but seeing someone try something to move away from ad driven seemed like a good possibility for me.

Then I saw how bitter and nasty a lot of the online communities I belonged to were. I mention that advertisers were of course mad, the stated goal is to reduce reliance on ad revenue. I was met with people attacking me as an idiot with clickbait articles of “proof” that ad models were irrelevant to twitters problems etc, as if I was doing anything other than quoting musks official reasons for the purchase. Suddenly Twitter was this great beacon of graceful discussion and Musk has ruined it.

People just love to be mad, no one had anything nice to say about Twitter and in a flick of a switch they’re holding completely opposite opinions. We’ve always been at war with Eurasia, it seems

Re: Why Twitter didn’t go down: From a real Twitter SRE

#317

Post by s1artibartfast »

pooper wrote: post #302
eru wrote: post #259

Earlier quoted context omitted.

Generally agreed. However keep in mind that bankruptcy is a purely financial event and doesn't necessarily have to have any impact on operations. Airlines are notoriously for going bankrupt regularly.

Airlines get free government money though. Will twitter? If not, Twitter will need to borrow more money and at higher interest rates compared to market rates if they go through with bankruptcy.

Funny thing about the bankruptcy hypothesis is that the owner of Twitter is the richest man in the world and could literally pay old twitters expenses for decades even if it never made a penny. If we are just talking about the debt payments, he has a 200 year runway.

Pure copium.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#318

Post by edgyquant »

jandrese wrote: post #220
libraryofbabel wrote: post #178

Earlier quoted context omitted.

Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to. So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building . But eventually people are going to start turning on stoves again… an…

Except that he at the same time demanded that people invent an entirely new dish by the end of the week, and now they are scrambling to try to figure it out. Already the DMCA auto-takedown bot is apparently broken and people are posting entire movies on Twitter. I would expect other peripheral systems to start breaking down as nobody is maintaining them even as other parts of the system are being changed.

Are we in support of DMCA now?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#319

Post by PM_me_your_math »

HyperSane wrote: post #192
throwawaylinux wrote: post #109

Earlier quoted context omitted.

I think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.

It would be funny for Musk to share privately with friends. It is a deeply strange and inappropriate thing for the CEO of a billion dollar company to post publicly. Something is deeply wrong with Musk's decision making process.

His process tree has only produced private and reusable rockets, space internet, electric vehicles, and hundreds of billions in value. But he makes crude jokes like 90% of the men on the whole planet. Clearly there's something deeply wrong.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#320

Post by jansan »

swellguy wrote: post #34

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

Did't you see the "Day in my life at the Twitter office video!"?

https://www.tiktok.com/@realpankhilpatel/video/7159187292631...

Normal people don't have vacations like that.

Post reply on HN