Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

741–750 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#741

Post by ryanfreeborn »

28304283409234 wrote: post #419
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Argh. "It works now, so it will work until forever." It takes _effort_ to make it work this smoothly now, _and in the future_. SRE is about _preventing_ issues. Not mopping up after them. To me, the article read like every succesfull sysadmin story: there's no fires, so sysadmin must be bloat.

No, I think the article makes it very clear what the value and function of SRE is. The point of the comment you're responding to is that the author was the only one doing this—not a team of ten, not even a team of two. This is Twitter's whole cache system! Probably the most important part of their hardware stack, in terms of "is the site performing well for users". There are other SRE needs at Twitter, but not that that many. What were the other 9k people at the company? It begs the question.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#742

Post by hgsgm »

john_the_writer wrote: post #712
b3lvedere wrote: post #559

Earlier quoted context omitted.

Whenever clients complain about those costs and efforts, i tell them it's just like their car. Your car is working perfectly fine so why should you pay for maintenance?

software doesn't break down from heat. An app I write today will run until the hardware dies. I have a palm_os app I wrote in 1998 that still runs perfectly.

No post body was provided.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#743

Post by nigamanth »

Twitter won't go down that easily. Besides, Elon Musk hasn't bought it for no reason. I'm sure that he has his plans with Twitter to implement new features and whatnot to restore it.

Till now, he's fired a lot of engineers probably because:

a) he doesn't need so many. b) he thinks he can do the job himself.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#744

Post by throw0101c »

donkeyd wrote: post #372
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

> In a start-up many of these roles can be ignored, becaus growth > stability. In a large organization, part of the bloat helps insure a certain amount of stability that's necessary to keep an organization alive.

It also (a) increases the bus factor, [1] and (b) allows people to take vacations and time off without having to watch their phones like hawk.

[1] https://en.wikipedia.org/wiki/Bus_factor

Re: Why Twitter didn’t go down: From a real Twitter SRE

#745

Post by CPLX »

fnord123 wrote: post #472
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

Twitter had 7,500 employees. most of the roles you mention (security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, scrum master) are not bloat. So the question is what are the other 7000 people doing?

If you spend a million dollars with a company do you expect them to call you back if you reach out to them?

How many millions are in a billion?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#746

Post by ryanfreeborn »

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

I have been under the good faith assumption that most (though definitely not all) of the employees that have departed Twitter were probably necessary and valuable to the company. I left the article with the same impression as you. This single person did this very important job, seemingly well, and didn't appear to be drowning in the work. What were the other 8-9k doing?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#747

Post by grishka »

gizmo wrote: post #714
pas wrote: post #641

Earlier quoted context omitted.

That's tiny bit of an exaggeration, maaaybe. Maybe completely prophetic, though! That said, Instagram was run by just a dozen people back then, while it had hundreds of millions of users, right? So it's not a new data point. Storing, retrieving, indexing, managing 280 char blobs (with links, threads, embeds) is not exactly the most hardcore of a problem domains. Microblogs are the typical tutorial topic, and twitter'…

Twitter is basically a real-time database where everything is interconnected. It's one of the harder things to scale because it doesn't allow for easy segmentation.

A lot of things in a typical social media service can be segmented trivially. The hardest part is the feed because it requires querying many database servers at once, but then it can be cached and served quicker.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#748

Post by anon400232 »

hef19898 wrote: post #726
Aeolun wrote: post #719

Earlier quoted context omitted.

> thought that Twitter would crash within days of the engineers being fired Because they work for companies where the product would fail within days of them being fired themselves.

That's what people keep telling themselves. Truth is, everybody is replaceable. And usually, loss of institutional knowledge takes a while to show real effects.

Saying everyone is replaceable is a first-degree approximation. The next level of detail would talk about, at least: (a) how much knowledge gets lost; (b) how many person hours -and- "wall clock time" it takes to rebuild that knowledge; (c) the amortized cost (or benefit) of losing that person

Re: Why Twitter didn’t go down: From a real Twitter SRE

#749

Post by CPLX »

itsoktocry wrote: post #676
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

> removing engineers won't instantly crash the product. It'll happen slowly It's amazing to me how many people following the Twitter saga, some familiar with or actually working in technology, thought that Twitter would crash within days of the engineers being fired. And because it didn't, the job cuts are justified.

There are fairly widespread reports of technical issues. I’ve seen a few about 2FA failing, strange undefined behavior in the ad serving platform, and others.

Of course it could go either way but the jury is currently out. It’s entirely possible that severe company-impairing technical breakdowns are already in progress and unrecoverable.

Or maybe not.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#750

Post by ojosilva »

rvba wrote: post #529

> When I joined the team the first project I had was to swap old machines that were being retired for new machines. There were no tools or automation to do this, I was given a spreadsheet with server names. Writing a program to store a list of servers to be swapped instead of keeping them in a spreadsheet sounds a bit like buying a brewery when you want to drink 1 beer. Program used by a team of one sounds like over-…

This spreadsheet was, from the OPs narrative, the master-list used by the whole operations team. An operations team that is now reduced to "ashes" - therefore not a bad idea at all to have your server inventory centralized.
Post reply on HN