Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

541–550 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#541

Post by entropi »

rjzzleep wrote: post #429
poulsbohemian wrote: post #87

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…

The reality is most huge companies are majority bloat. The hiring numbers are also in part crap that goes into Series X Raise pitch decks. Oftentimes a lot of the new bloat pisses off competent people, because their work doesn't actually get less, it becomes more. Not only do they have to now nanny people that are often not actually competent in their job, they just happened to go through the coding interview with wh…

> The reality is most huge companies are majority bloat.

This is true, and in my opinion, true for a reason. And that reason is not "most huge companies are dumb", as opposed to what Musk's cult seem to believe. The reality is, measuring what exactly is "bloat" and precisely cutting that bloat is extremely difficult and firing more than half of your workforce is probably like using a warhammer to do brain surgery.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#542

Post by skc »

Twitter pretty much passed it's first real test with flying colors since Elon took over.

The world cup is on and the site didn't collapse.

That is a huge win and while I don't believe Twitter will ever be profitable, I think Elon will be feeling rather smug right now with his skeleton staff.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#543

Post by caporaltito »

kakwa_ wrote: post #524
holoduke wrote: post #506

Earlier quoted context omitted.

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

That's likely optimistic. Maybe the user facing site, but that's just the tip of the iceberg. There are plenty of internal/backend/restricted systems to support and/or monetize this part. And that's not counting the huge number of support people & moderators needed.

For the ad platform only, this would require already a hundred of people. Building an SSP is not an easy task. You are right about being optimistic.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#544

Post by bandrami »

diydsp wrote: post #511
riffraff wrote: post #491

Earlier quoted context omitted.

> All signs so far are pointing to, yes which signs? They launched a new feature (blue checks for $8) and had to turn it off immediately because it was bleeding money and ruining the platform and they have less ad revenue booked for next year than they had at the same time last time. I don't think one should judge the new twitter course yet, but "well, the site is still up" is a very bad measure of success.

Didnt TFA break as well? And im seeing mass twitter account dumping reported via reddit.

It broke completely for 2 days and still doesn't fully work

Re: Why Twitter didn’t go down: From a real Twitter SRE

#545

Post by djtango »

donkeyd wrote: post #383
summerlight wrote: post #348

Earlier quoted context omitted.

Twitter used to experience significant downtime compared to all other major platforms and one of the reason was its lack of redundancies across everything. Headcount is one such thing and it takes manpower to automate infrastructures as discussed in the post. Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usuall…

> Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usually have higher expectations for established companies. This exactly. During the recent Whatsapp outage, many threads popped up on HN about how big of an issue this is in Europe, since Whatsapp is the main messaging platform in Europe. Thankfully, these outage…

Whatsapp is even more essential in most of APAC and South America

Re: Why Twitter didn’t go down: From a real Twitter SRE

#546

Post by djtango »

bojan wrote: post #490
donkeyd wrote: post #383

Earlier quoted context omitted.

> Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usually have higher expectations for established companies. This exactly. During the recent Whatsapp outage, many threads popped up on HN about how big of an issue this is in Europe, since Whatsapp is the main messaging platform in Europe. Thankfully, these outage…

I don't remember WhatsApp being less stable before it was bought by Meta. And it was just as essential back then too.

Yes it felt more reliable when it was on their own infra before they migrated to FB internal infra.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#547

Post by coldtea »

tempestn wrote: post #487
carabiner wrote: post #477

Earlier quoted context omitted.

Yeah it's dancing around the question: Was Musk right? All signs so far are pointing to, yes . MBA's will be studying this for years.

Musk might have been right about some things. There probably was some degree of bloat. But to say he's badly mishandled this whole saga is a gross understatement. It is very difficult to utterly kill a site like Twitter; the fact that we're even considering that as a realistic possibility shows just how badly. I think Musk is used to Tesla and SpaceX, which are both companies that a lot of people are (or at least wer…

>the fact that we're even considering that as a realistic possibility shows just how badly.

Do we have much to go with to "really consider it", or is it just sensenationalist headlines, as Musk went ahead the accepted orthodoxy on Twitter and content moderation and so "he must pay"?

That would be the same news outlets that built him up as the real life Tony Stark and propped up his quite ordinary companies, one of which is building electric cars like everybody else, just doing it in style as well-off upper-middle class people's toys, and another is doing space tech that was already a thing we were pursuing 40 years ago with a little modern engineering thrown in...

Re: Why Twitter didn’t go down: From a real Twitter SRE

#548

Post by rajamaka »

ab8 wrote: post #296
swellguy wrote: post #34

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

Tesla has 99k employees

Tesla's product can't be replicated with a 2000 line CRUD app

Re: Why Twitter didn’t go down: From a real Twitter SRE

#550

Post by spaceman_2020 »

ulkesh wrote: post #27

All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…

Man, it's not about tech. This is simply about leveraging the "Musk halo effect" on a highly visible, failing business.

You can screenshot this: Musk will cut down costs, make Twitter profitable, and take it public again when markets are better placed.

The markets will give the newly listed Twitter the same "Musk-boost" as Tesla and ramp up the valuation to $100B.

Post reply on HN