Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

481–490 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#481
post #418

Earlier quoted context omitted.

> He proved over and over again. [Narrator voice] He did not.

Did SpaceX failed for bankruptcy or something? Did Tesla not change the whole industry?

Funny enough it would be hard to find anyone with more CEO experience than Musk. This doesn't mean he does all the management, often the work of these guys is to find the right people to do the management for them.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#482

Earlier quoted context omitted.

I think Twitter was stuck in a not-especially profitable niche. They shift into fast-mode to get out of it and find a better spot, then they can shift back into stable mode once they occupy a better equilibrium. That said, there are lots of bugs in Twitter now, today, when they presumably had the benefit of being in stable mode for a long time. For example, Twitter regularly refreshes and loads new tweets while I'm r…

That's not a bug, it's a 'feature', tons of sites do it for whatever asinine reason.

Like ... displaying new / refreshing ads ?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#483

Earlier quoted context omitted.

I think Twitter was stuck in a not-especially profitable niche. They shift into fast-mode to get out of it and find a better spot, then they can shift back into stable mode once they occupy a better equilibrium. That said, there are lots of bugs in Twitter now, today, when they presumably had the benefit of being in stable mode for a long time. For example, Twitter regularly refreshes and loads new tweets while I'm r…

That's not a bug, it's a 'feature', tons of sites do it for whatever asinine reason.

Keeping vertical scroll position stable while asynchronously loading lots of randomly-sized content is harder than it seems.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#485

Regardless of what’s up with the tech stack, I find it difficult to want to be a part of whatever this is: https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-buc... ) If this is the new town square, you can forward my mail to a cabin in the woods.

You do know that you can block Elon's account.

but hairofadog needs a reason to be hysterical.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#486

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> I was partly expecting the rest of the article to explain to my why exactly it wasn't just bloat Same here. I guess his header was on point in why Twitter is still up; but I was also interested in hearing about why Twitter actually needs all those people. If it can be run with 50-80% of the staff gone, that does sound like some bloat at least.

Slack space leads to innovations, like developing infrastructure automation and improving capacity planning. SRE as a practice needs slack space for operations teams to work on improvements and fixes in addition to BAU fault fixing, deployments and patching.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#487

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Yeah it's dancing around the question: Was Musk right? All signs so far are pointing to, yes . MBA's will be studying this for years.

Musk might have been right about some things. There probably was some degree of bloat. But to say he's badly mishandled this whole saga is a gross understatement. It is very difficult to utterly kill a site like Twitter; the fact that we're even considering that as a realistic possibility shows just how badly.

I think Musk is used to Tesla and SpaceX, which are both companies that a lot of people are (or at least were) excited to work for because they believe in the mission and what's being created. Plus there aren't many alternatives if you want to do that work. Twitter really isn't like that for most people; a Twitter developer has many other options to do similar work. Add to that the fact that he's both cranked up the intensity of the abuse and that it's more visible to everyone, and you can't expect a lot of good people to stick around. And despite the fact that it might coast for quite a while on the back of excellent work in the past, eventually you do need good people to keep a business going. (This is leaving aside the direct impacts of his actions on users and advertisers!)

Re: Why Twitter didn’t go down: From a real Twitter SRE

#488
post #449

Earlier quoted context omitted.

It’s easy to think it’s bloat at a steady state. When something important goes down and nobody knows how to fix it, it looks different.

That being said it's not like twitter is a massively complex product with lots of different features. I can imagine you could keep it running with a skeletton team. Liasing with ads buyers excepted.

I'm pretty sure one of the most complicated things is preventing automation of content - bots. which would be an arms race type condition. Bots you want to prevent - people not using twitter's api to do their bot stuff. Why would people not use twitter's api?

1. who would trust twitter not to change API and make code worthless

2. people who want to do stuff Twitter doesn't want you to do in an automated fashion.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#489

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

And vulnerabilities.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#490

Earlier quoted context omitted.

Twitter used to experience significant downtime compared to all other major platforms and one of the reason was its lack of redundancies across everything. Headcount is one such thing and it takes manpower to automate infrastructures as discussed in the post. Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usuall…

> Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usually have higher expectations for established companies. This exactly. During the recent Whatsapp outage, many threads popped up on HN about how big of an issue this is in Europe, since Whatsapp is the main messaging platform in Europe. Thankfully, these outage…

I don't remember WhatsApp being less stable before it was bought by Meta. And it was just as essential back then too.
Post reply on HN