Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

411–420 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#411

Post by yalogin »

trollied wrote: post #395
yalogin wrote: post #387

The reason why Twitter hasn’t crashed is it’s well written and well orchestrated. Once a bug comes in and crashes something that is when the chaos starts. It’s almost a guarantee that the current crop do not know how to fix the bug. It will be interesting to see how they handle that.

> It’s almost a guarantee that the current crop do not know how to fix the bug Seeing lots of comments like this. Why do you feel it’s ok to say that, when you don’t actually work there or have intimate knowledge of who is currently working there? The pure speculation and nonsense surrounding Twitter at the moment is plain awful.

It’s just statistical probability. The number of employees left is small and the amount of code is huge. It’s not a huge leap to say that a bug is unlikely to be in the area of expertise of the folks that are left. I would say it’s the probability is greater than 50%. You should notice the probability is not proportional to the employees still remaining. That is because I am guessing some good folks are still there.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#412

Post by Karrot_Kream »

npn wrote: post #314
jdminhbg wrote: post #157

Earlier quoted context omitted.

I think this post is aimed at non-technical "RIP Twitter" posters from last week who are surprised it didn't fall apart in 24 hours.

There was a long twitter thread from some one proclaimed 10+ years SRE listing a lot of things that could bring the server down. Lot of them are pretty far-fetched and outdated. Too bad I did not bother to save the link.

You're thinking of https://twitter.com/MosquitoCapital/status/15935411779656785.... Remember, on Twitter your likes and follower counts are completely independent of your actual skill area and level. Most of that list reminded me of 2009-era site operations, nothing like what operating a high-traffic site is like in 2023.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#414

Post by [deleted] »

JohnBooty wrote: post #332
memish wrote: post #284

Earlier quoted context omitted.

It's actually scary how many people, even engineers, put their reputation on the line saying Twitter wouldn't survive the weekend. It wasn't just Twitter employees. It's like a mass psychosis of some kind. It comes off as a kind of desperation, as though they need Elon to fail. Why? What's driving that response?

There are a lot of places where the systems would start to fall over if 70-80% of the team departed. Especially since a lot of folks left on bad terms and/or were suddenly terminated. It was the opposite of a smooth handover. So it wasn't unreasonable to think that Twitter would begin experiencing problems. I didn't think that Twitter was going to literally have an unrecoverable system crash and permanently shut its…

[deleted]

Re: Why Twitter didn’t go down: From a real Twitter SRE

#415

Post by radu_floricica »

poulsbohemian wrote: post #87

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…

Yeah, but that's not what's happening here, is it? Musk is pretty obviously looking for a corporate culture shift, not turning the company into a cash cow - and for this to work, entropy needs only to work slower than half a year. Which this article argues pretty convincingly is going to happen.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#416

Post by smaudet »

TranquilMarmot wrote: post #380
throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

> As long as nothing changes, and you don't run out of disk space (from logs for example), things stay working pretty much just fine. > ... > There are other things that can bring a site down, like security issues, or bugs triggered by unusual states, too much traffic, etc. But generally speaking those things are rare and don't bring down an entire site. Aren't these changes inevitable, though? There is no such thing…

Software never goes stale, it's the environment around it which stales.

Something from the 70s works perfectly fine, except it can't run on anything bare any longer, and the hard drives etc. have all long since failed or their PSU capacitors have blown....so Twitter will absolutely rot, how fast depends on several factors.

I personally suspect the infrastructure used to build Twitter will rot faster than Twitter itself, and of course the largest most dramatic source of rot is the power required to run it - several large communities have abandoned it already, making it less much less relevant, meaning the funding for it will also dry up, meaning more wasted cpu cycles and the like.

Thats of course assuming its left in some sort of limbo, it doesnt sound like thats the case with the current management, its only a matter of time before it topples over from shitty low-rate contractor code. Honestly, the app worked like so much hot garbage already, I could see it falling over itself and imploding with a couple poorly placed loops...

Re: Why Twitter didn’t go down: From a real Twitter SRE

#417

Post by kibibyte »

powerapple wrote: post #391
summerlight wrote: post #348

Earlier quoted context omitted.

Twitter used to experience significant downtime compared to all other major platforms and one of the reason was its lack of redundancies across everything. Headcount is one such thing and it takes manpower to automate infrastructures as discussed in the post. Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usuall…

I think the opposite. Many softwares at its best when the team was small. Software companies have to hire many people because it needs to report growth to investors, headcount is one of the measurement of growth. It is not necessarily good for the product, actually many times, it hurts the product, but overall it is good for the company, the company will enter new areas, can explore new things. What Twitter is doing…

Scaling back up is really hard though. We had a de facto freeze on hiring (not exactly hiring freeze; more of a headcount cap) just shy of a decade ago to focus on our product. During that time, some of our best recruiters left because they basically had nothing to do anymore.

The freeze worked: we got rid of some products that weren't getting traction and were able to improve the products that did have traction. But the cost of the freeze lingered for at least a year; it reset the hiring pipeline, we couldn't grow fast when we needed to because the limited number of recruiters we had were already overworked, and the limited number of engineers had to balance interviewing needs with their real work. This all happened when my employer was Twitter is simply at a different scale. 7500 -> 2500 employees is a 66% reduction. Going 2500 -> 7500 is a 200% increase. Recruiting is likely totally gutted, and the current 2500 employees have to support systems previously maintained by a 7500-person company. If they decide they need to grow, it'll have to restart at a snail's pace, and they'll have to make sacrifices on feature development or stability along the way.

Edit: for what it's worth, the fastest way to regrow back 200% is to rehire the people laid off. But, given that I happened to interview earlier today an ex-Twitter candidate who didn't make it through the Elon snap, that route is rapidly closing up.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#418

Post by zichy »

mirzap wrote: post #402
poulsbohemian wrote: post #87

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…

Quoted post unavailable.

> He proved over and over again.

[Narrator voice] He did not.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#419

Post by 28304283409234 »

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Argh. "It works now, so it will work until forever."

It takes _effort_ to make it work this smoothly now, _and in the future_.

SRE is about _preventing_ issues. Not mopping up after them.

To me, the article read like every succesfull sysadmin story: there's no fires, so sysadmin must be bloat.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#420

Post by radu_floricica »

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

I have an old project I gave up on - haven't touched it, done any code changes or maintenance in... almost a decade? At least a stubborn client is still using it, successfully. And it's not an old guy in a living room, but an honest small sized company that has this software as the core of its operations.

So yeah, I totally agree with you. No code changes = long life.

Post reply on HN