Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

611–620 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#611

Post by Lutger »

H8crilA wrote: post #573
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

This is a PR article that tries to push the idea that Twitter is OK (to work at, and maybe also to buy ad space from). Damage control.

I read it as the exact opposite: the reason Twitter is still ok* is not because all these people were just browsing reddit at work. You can't just gut Twitter to run on only a couple of hundred people and still expect the same results in the longterm.

Twitter was not a leftist commie welfare company as Musk and its fans want it to be. It was actually the fine work SRE (amongst others) put into it that makes it still tick along as it does...for now.

* actually some things are already breaking, but it will take some time for the real damage to surface on a technical level

Re: Why Twitter didn’t go down: From a real Twitter SRE

#612

Post by yywwbbn »

carabiner wrote: post #477
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Yeah it's dancing around the question: Was Musk right? All signs so far are pointing to, yes . MBA's will be studying this for years.

The only winners here are former Twitter shareholders no matter what happens next.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#613

Post by jasode »

unculture wrote: post #590
larsnystrom wrote: post #578

Earlier quoted context omitted.

Are you saying Twitter is paying ~$1.5B in interest rates every year?

It’s going to be something like that. He bought it for 44 billion. That was mostly loans. Those loans are transferred to the company and the company pays interest on them.

>He bought it for 44 billion. That was mostly loans.

That's not true if media reports about the purchase was accurate. The debt financing was only $13 billion: https://www.google.com/search?q=musk+buys+twitter+%22%2413+b...

The more complete funding structure[1] looked like this:

+ $24 billion : Elon Musk personal wealth (e.g. selling some of his TESLA stock to raise money)

+ $7 billion : other equity investors (Saudi Arabia prince, Larry Ellison, etc.)

+ $13 billion: bank loans

The Twitter puchase was mostly Elon's personal money (~54%) and not debt.

[1] https://archive.ph/L6Mhv

Re: Why Twitter didn’t go down: From a real Twitter SRE

#614

Post by gjulianm »

lelanthran wrote: post #600
gjulianm wrote: post #583

Earlier quoted context omitted.

Well, I think the issue is precisely considering Twitter a "simple technology project", and it's the same mistake that Musk does. Twitter isn't a "software and servers business" as he said. Twitter is a social community, and while in some regards it might be easier, it's also far more difficult in others. Just compare how many business and institutions can reliably launch rockets or create cars, and how many can reli…

> Well, I think the issue is precisely considering Twitter a "simple technology project" But I didn't call it "simple", I called it "simpler", and it is.

Point still stands if you call it a "simpler technology project". It's not about the tech, it's about the community, and communities are far harder to predict and manage than a rocket or a car.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#615

Post by adql »

pcthrowaway wrote: post #504
tempestn wrote: post #452

Earlier quoted context omitted.

I think you're misinterpreting the comment you're replying to. They would agree with you that the tiny SRE team described in the article sounds very effective, and likely have a lot to do with why the site is still up and running currently. Work like that should continue. But if 1-3 people can have that degree of impact, what are the other 8000 doing? (Again, this is just me attempting to interpret the point made by…

The SRE team mentioned in the article is SRE for one component of a complex architecture. There are probably many such components; I'd imagine SRE alone would be 200+ people How many of the remaining staff have the knowledge required to keep all of those components running smoothly?

Once you get the automation going the number itself doesn't matter that much.

You might have 200 different apps (hell, we have close to that, only 3 people in ops) but competent team will make sure they deploy in same way and are monitored in same way.

And once you go from "a server" to multiple servers, whether the end number ends up being 20 or 200 isn't that important till you start hitting say switching capacity, and if you're in cloud that's usually not your concern anyway.

Our biggest site (about dozen million users, a bunch of services and caching underneath, few gbits of traffic) took zero actual maintenance for 2022, "it just works", any job was implementing new stuff. It took some time to get to that state but once you do aside from hardware failures it "runs itself"

Re: Why Twitter didn’t go down: From a real Twitter SRE

#616

Post by guiriduro »

_3u10 wrote: post #606
guiriduro wrote: post #599

Earlier quoted context omitted.

Yeah I love it. I would like to "buy" a multi-million dollar mansion in the Bahamas with big bank money, but have the bank only encumber the property itself, and I get to live in it and pay service staff for my luxury, maybe the odd AirBnB letting to maintain the occasional pretense of repayment, up until the time 'it' can't sustain 'itself' and I walk away debt-free. But apparently I'm not a rich person so that kind…

It’s really easy to get a mortgage for 20% of a property and assign it to a corporation. In fact most banks will go as high as 80%, especially if the house can rent for the mortgage payment. And they will definitely do that deal if you’re willing to pay 10% interest.

True, but the difference is all about what happens when you can't pay the 10% interest or get into significant negative equity. If you can just walk away, then you have "rich person accounting."

Re: Why Twitter didn’t go down: From a real Twitter SRE

#617

Post by Dave3of5 »

jiggawatts wrote: post #518
underwater wrote: post #497

Earlier quoted context omitted.

Look at his claim that Twitter is slow in India because of poorly batched backend calls [1]. He's simply regurgitating what he thinks he heard from others without really understanding how anything works. He knows less about software architecture than an intern picked at random. 1. https://mobile.twitter.com/elonmusk/status/15921762028730859...

Stop repeating this, because Elon had the gist of it correct, if not the specifics. a) Twitter is atrociously slow for something that displays a mere kilobyte of text at a time. b) Much of that is because of hundreds of client-server JavaScript requests on first load. Not technically RPCs, but just as slow and looking nearly identical on the network. c) Most of the rest is because of a thousand microservices chatting…

> Much of that is because of hundreds of client-server JavaScript requests on first load. Not technically RPCs, but just as slow and looking nearly identical on the network.

This is not true at all. I have a look and it's not that there are many REST requests it's that the requests send back a load of data. For Example the timeline sends back a 1MB JSON response. It's heavily gzipped so comes in at 100KB over the network but it takes quite a while for the whole request.

In places like india the internet is slow and many people are using VPNs. I emulated this and I can see that over VPN on a slow internet connection it's like 10 seconds to load the home page.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#618

Post by EricLeer »

unculture wrote: post #590
larsnystrom wrote: post #578

Earlier quoted context omitted.

Are you saying Twitter is paying ~$1.5B in interest rates every year?

It’s going to be something like that. He bought it for 44 billion. That was mostly loans. Those loans are transferred to the company and the company pays interest on them.

But who is issuing these loans? It should be pretty obvious that you would never see that money back right?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#619

Post by Nokinside »

More than 50% of large distributed system downtime is caused by configuration errors, failed software updates, or network attacks.

With configuration error, or failed software update everything can go down at once, or there are cascades of restarts that kill the system.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#620

Post by nikau »

acknorabotr wrote: post #278
ulkesh wrote: post #27

All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…

It's crazy to think a guy who builds reusable rockets thinks he can run a complex technology operation like Twitter.

s/builds/obtains funding to get other people to build
Post reply on HN