Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

651–660 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#651

Post by throwaway290 »

fastball wrote: post #470
summerlight wrote: post #348

Earlier quoted context omitted.

Twitter used to experience significant downtime compared to all other major platforms and one of the reason was its lack of redundancies across everything. Headcount is one such thing and it takes manpower to automate infrastructures as discussed in the post. Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usuall…

The headcount at WhatsApp in 2013 was somewhere between 50-100, at which time they were servicing approx 400m MAU, which is more than users than Twitter has been able to boast for most of their existence. Coincidentally, in 2013 SpaceX was just starting to provide commerical launch capacity, at which point I think they too had Not surprised Elon Musk thinks he can run twitter with a skeleton crew.

1) And what was their uptime in 2013? How did uptime change as the service grew in popularity?

2) WhatsApp does not support the type of public broadcasts done at Twitter, and due to its e2ee doesn't require much human moderation.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#652

Post by psychoslave »

jasode wrote: post #613
unculture wrote: post #590

Earlier quoted context omitted.

It’s going to be something like that. He bought it for 44 billion. That was mostly loans. Those loans are transferred to the company and the company pays interest on them.

>He bought it for 44 billion. That was mostly loans. That's not true if media reports about the purchase was accurate. The debt financing was only $13 billion: https://www.google.com/search?q=musk+buys+twitter+%22%2413+b... The more complete funding structure[1] looked like this: + $24 billion : Elon Musk personal wealth (e.g. selling some of his TESLA stock to raise money) + $7 billion : other equity investors (Saud…

https://thepolicytimes.com/top-10-richest-people-worth-more-...

So, a handle of individuals can dwarf most countries GPD.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#653

Post by nebula8804 »

radu_floricica wrote: post #420
throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

I have an old project I gave up on - haven't touched it, done any code changes or maintenance in... almost a decade? At least a stubborn client is still using it, successfully. And it's not an old guy in a living room, but an honest small sized company that has this software as the core of its operations. So yeah, I totally agree with you. No code changes = long life.

You should be proud. I hope that one day some software I write can serve people for that long.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#654

Post by lukaslalinsky »

randmeerkat wrote: post #336
ezoe wrote: post #246

Earlier quoted context omitted.

Redundancy. Now the systems are stable but human workers either be sick, leave, or die eventually. Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them. The article wrote about achieving stability by the distributed system so an unexpected death of one rack doesn't affect the service availability. The same can be done for…

> Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them. People would absolutely be more engaged and more excited about their work if they were paid more. The only reason people work is literally for money…

This really only works in situations where people are typically underpaid and need to get up to some living standard.

In most software engineering jobs, people have enough money to live an enjoyable life. At that point, having more money is not really such a strong motivator.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#655

Post by TheOtherHobbes »

ivanhoe wrote: post #581
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

> If the current state of affairs at Twitter keeps up, it'll probably be a slow descent into chaos. It's not like Twitter was bug free before. How many times it annoyingly refreshed the timeline while I was reading something, or when it shows notification that it failed to send the DM, and when you retry it says "you've already wrote this", or you open the reply dialog, but it freezes, has no send button at all, so y…

They basically gave an established global corporation with around 200mn active users a startup tech, marketing, and support stack.

Anyone thinking "huge success" is unrealistically optimistic, IMO.

It's going to be another MySpace/AOL/Bebo, added to the list of dumbest purchases ever.

And that's still going to be true if the point was to destroy the original community and replace it with a different political orientation.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#656

Post by reacharavindh »

Has anybody suggested the idea of rewriting twitter in Rust yet to Elon? Memory safe, needing fewer servers(hence less $), and the story sells itself to someone like Elon who jumps on such impulses...

^ is a snark. I'm not suggesting that they should..

Re: Why Twitter didn’t go down: From a real Twitter SRE

#658

Post by stefan_ »

EricLeer wrote: post #618
unculture wrote: post #590

Earlier quoted context omitted.

It’s going to be something like that. He bought it for 44 billion. That was mostly loans. Those loans are transferred to the company and the company pays interest on them.

But who is issuing these loans? It should be pretty obvious that you would never see that money back right?

The banks committed to the loans at the point of the takeover proposal (end of April). The macroeconomic lookout was better. Since then, just about every tech stock is down >20% and Musk has helpfully made unfounded accusations of bots and spam and whatnot in a pointless attempt to delay the acquisition, meaning the banks now have to try and unload these loans in a recession. I think last I heard they are giving a 40% discount or just keeping them on their books, hoping for a recovery as Musk trashes the company value further with every week that passes.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#659

Post by throwuwu »

EricLeer wrote: post #618
unculture wrote: post #590

Earlier quoted context omitted.

It’s going to be something like that. He bought it for 44 billion. That was mostly loans. Those loans are transferred to the company and the company pays interest on them.

But who is issuing these loans? It should be pretty obvious that you would never see that money back right?

Only if the people issuing the loans are basing their decisions off of media coverage.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#660

Post by Yoric »

chrsig wrote: post #634
andreyk wrote: post #577

Earlier quoted context omitted.

Yep, agree. Twitter's revenue was primarily from ads, and now I'd bet their ads revenue has dropped a huge amount. Given how Musk is behaving now (reinstating Trump etc.), and losing many pivotal sales people, and the firing of a ton of people in charge of dealing with hate speech and such, it seems unlikely advertisers will return now.

advertisers wont care in a couple months. the situation has lots of public attention now, which is what the advertisers are actually afraid of. They don't want their brand associated with the craziness. They don't actually care one way or the other about any choice musk makes. They're not going to just walk away from a 300+ million person audience permanently just due to principles

No but if there is no content moderation and Twitter becomes even more of an open cesspit than it already was, announcers are going to stay away because most of them do not want to be associated with toxicity. Perhaps politically active organizations will want to be there (e.g. the NRA or the ACLU), but nothing family-friendly (e.g. Disney) and most likely nothing Christmas-y during Christmas season.
Post reply on HN