Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

571–580 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#571

Post by Dave3of5 »

I don't think it will fail on a technical level. As this article says lots of engineering has gone in to make the thing pretty resilient. I would also say that there are still enough engineers who work there who can figure out what's gone wrong and "turn it on and off again" or w/e makes it splutter back to life.

In terms of changes to the Platform ditto. It's not difficult to make these changes that a team of 100's of devs who are not 100% aware of what's there already can't figure it out. I've taken over systems that I knew very little of and were pretty big (not as big as twitter) and I managed for years to make changes without drastically breaking stuff. In any event if they do break stuff they will be able to fix what they have broken.

No, the real failure here is the massive debt burden and the fact that there is no way that twitter can ever service that amount of debt. Note that before EM took over it was ticking along with a relatively small loss. If they had cut headcount by maybe 10% they would have been break even easily. There is no way that's possible with $4million of interest per day. They have to radically change the way they monetise the platform to get to that level. I don't think they will ever get there and Musk will sell off at a bargain basement price at some point in the future to pay back the debt.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#572

Post by [deleted] »

oska wrote: post #237
raldi wrote: post #24

The most helpful thing to reflect on in these Twitter operational discussions is the difference between homeostasis and evolution. You can get rid of 80% of the work force and the existing homeostasis systems will keep things running smoothly despite known day-to-day chaos. Where you’re really going to run into trouble is inventing responses to novel chaos and gradually changing times.

We had an obituary for Fred Brooks on here just the other day. I'd suggest that his thesis in The Mythical Man-Month conflicts with your comment above (that reduction in staff count for a software project has a good correlation with the ability to maintain it / evolve it / innovate on top of it).

[deleted]

Re: Why Twitter didn’t go down: From a real Twitter SRE

#573

Post by H8crilA »

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

This is a PR article that tries to push the idea that Twitter is OK (to work at, and maybe also to buy ad space from). Damage control.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#574

Post by reverius42 »

Operative0198 wrote: post #206
hairofadog wrote: post #105

Regardless of what’s up with the tech stack, I find it difficult to want to be a part of whatever this is: https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-buc... ) If this is the new town square, you can forward my mail to a cabin in the woods.

You do know that you can block Elon's account.

If you have to block the account of the CEO of the service you are using, perhaps you should just stop using that service?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#575

Post by Thorentis »

wraptile wrote: post #508
minhazm wrote: post #373

Earlier quoted context omitted.

Many people who have worked with Musk have shared similar sentiments in interviews. But it seems that people just refuse to believe any of it. People think that there's no way it's possible for someone to be that deeply technical and be a CEO of multiple companies at the same time. I've talked to people about it and they straight up refuse to believe it saying that it's impossible and that any evidence of him being t…

Something doesn't compute in this scenario though. Either his tricking everyone around him or is unfortunate enough to slip up publically. Not knowing what GraphQL is and talking about RPCs in HTTP is a very revealing slip up. My guess would be that he has some knowledge but also is very good at faking it which is not necessarily a bad thing - those are good traits for a CEO. Though people should be aware of this fac…

What's wrong with his RPCs tweet? If Twitter is using microservices that make RPCs to fetch data and render content then it makes sense.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#576

Post by gjulianm »

powerapple wrote: post #391
summerlight wrote: post #348

Earlier quoted context omitted.

Twitter used to experience significant downtime compared to all other major platforms and one of the reason was its lack of redundancies across everything. Headcount is one such thing and it takes manpower to automate infrastructures as discussed in the post. Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usuall…

I think the opposite. Many softwares at its best when the team was small. Software companies have to hire many people because it needs to report growth to investors, headcount is one of the measurement of growth. It is not necessarily good for the product, actually many times, it hurts the product, but overall it is good for the company, the company will enter new areas, can explore new things. What Twitter is doing…

> What Twitter is doing is to scale down first

This is not a trivial task. With such a heavy reduction and how entire teams have been completely decimated, there will be a lot of lost knowledge. I'm sure there will even be cases where the people who stay don't even know what knowledge was lost.

All that puts Twitter in a very risky position, specially in a product of such complexity that does a lot of things in-house. It shouldn't be underestimated.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#577

Post by andreyk »

Dave3of5 wrote: post #571

I don't think it will fail on a technical level. As this article says lots of engineering has gone in to make the thing pretty resilient. I would also say that there are still enough engineers who work there who can figure out what's gone wrong and "turn it on and off again" or w/e makes it splutter back to life. In terms of changes to the Platform ditto. It's not difficult to make these changes that a team of 100's…

Yep, agree. Twitter's revenue was primarily from ads, and now I'd bet their ads revenue has dropped a huge amount. Given how Musk is behaving now (reinstating Trump etc.), and losing many pivotal sales people, and the firing of a ton of people in charge of dealing with hate speech and such, it seems unlikely advertisers will return now.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#578

Post by larsnystrom »

Dave3of5 wrote: post #571

I don't think it will fail on a technical level. As this article says lots of engineering has gone in to make the thing pretty resilient. I would also say that there are still enough engineers who work there who can figure out what's gone wrong and "turn it on and off again" or w/e makes it splutter back to life. In terms of changes to the Platform ditto. It's not difficult to make these changes that a team of 100's…

Are you saying Twitter is paying ~$1.5B in interest rates every year?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#579

Post by mopsi »

holoduke wrote: post #506
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

A script that takes text from input field and displays it along with previous submissions in chronological order is trivial; the difficulty comes from serving millions of people.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#580

Post by Moissanite »

Dave3of5 wrote: post #571

I don't think it will fail on a technical level. As this article says lots of engineering has gone in to make the thing pretty resilient. I would also say that there are still enough engineers who work there who can figure out what's gone wrong and "turn it on and off again" or w/e makes it splutter back to life. In terms of changes to the Platform ditto. It's not difficult to make these changes that a team of 100's…

The fact that he was able to "buy" Twitter and yet transfer a significant amount of debt to the company rather than being liable himself is just another sign of how different "rich people accounting" is. He will walk away from this with a bit less theoretical money but no material impact to his life, while thousands of people are having their lives up-ended. How long are we going to keep letting shit like this happen?
Post reply on HN