Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

471–480 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#471

Post by bartvk »

rjzzleep wrote: post #429
poulsbohemian wrote: post #87

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…

The reality is most huge companies are majority bloat. The hiring numbers are also in part crap that goes into Series X Raise pitch decks. Oftentimes a lot of the new bloat pisses off competent people, because their work doesn't actually get less, it becomes more. Not only do they have to now nanny people that are often not actually competent in their job, they just happened to go through the coding interview with wh…

> this reminds me of old German industry where people retire in the place they started

What is wrong with that?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#472

Post by fnord123 »

donkeyd wrote: post #372
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

Twitter had 7,500 employees. most of the roles you mention (security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, scrum master) are not bloat. So the question is what are the other 7000 people doing?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#474

Post by bryanrasmussen »

donkeyd wrote: post #372
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

>Bugs will creep in,

https://twitter.com/IlluminatiGanga/status/15946097904324444...

new members joining in 1970. hmmm.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#475

Post by __bjoernd »

fastball wrote: post #453
gitgud wrote: post #433

Earlier quoted context omitted.

Quoted post unavailable.

Literally his first two companies, Zip2 (CTO) and x.com (CEO), were pure software companies? Another company he founded, SpaceX, is the first space launch provider to ever achieve propulsive landing and re-use of an orbital-class rocket, a feat which includes some of the hardest software engineering you can do. Another company he runs, Tesla, is famous for being one of the few car companies that makes software for th…

Knowing how to run a software company and knowing how to build software are two things. The first thing usually implies hiring the right people for the second thing. I’m sure Musk knows at least a bit of that, but everything we see with Twitter is that he’s axing those people at a rate that you can’t keep up hiring at. So there is some obvious disconnect here and I don’t think “Ah, he knows what he is doing “ is a sufficient explanation.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#476

Post by akira2501 »

minhazm wrote: post #373
memish wrote: post #326

Earlier quoted context omitted.

John Carmack, "Elon is definitely an engineer. He is deeply involved with technical decisions at spacex and Tesla. He doesn’t write code or do CAD today, but he is perfectly capable of doing so." Kevin Watson, who developed the avionics for Falcon 9 and Dragon and previously managed the Advanced Computer Systems and Technologies Group within the Autonomous Systems Division at NASA's Jet Propulsion laboratory: "Elon i…

Many people who have worked with Musk have shared similar sentiments in interviews. But it seems that people just refuse to believe any of it. People think that there's no way it's possible for someone to be that deeply technical and be a CEO of multiple companies at the same time. I've talked to people about it and they straight up refuse to believe it saying that it's impossible and that any evidence of him being t…

With a handful of tricks or a patsy in your pocket, it's easy enough to pull things like this off. These are all self-reported encounters, which lends some doubt to them; as I've never seen any public performance of his that suggests he has this exceptional intelligence or that he isn't subject to the same amount of irrational thinking that most humans are. You may be able to do some type of rocket equation in your head, but if you constantly promise things that aren't ultimately delivered.. people have good reason to question this narrative.

He clearly does know how to make incredible sums of money. Why that's not enough and people need to find excuses to exaggerate or demean his intelligence is beyond me.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#477

Post by carabiner »

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Yeah it's dancing around the question: Was Musk right? All signs so far are pointing to, yes. MBA's will be studying this for years.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#478

Post by torben-friis »

usgroup wrote: post #467
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

I expect it all to work better with Musk in charge. He knows how to make scalable software and he knows about performant teams. That he’s not going to realise these totally obvious first order consequences people are raising seems unlikely.

I honestly can’t tell if you’re being sarcastic.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#479

Post by __bjoernd »

drdrey wrote: post #162
throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

Another thing we noticed at Netflix was that after services didn’t get pushed for a while (weeks), performance started degrading because of things like undiscovered memory leaks, threads leaks, disks filling up. You wouldn’t notice during normal operations because of regular autoscaling and code pushes, but code freezes tended to reveal these issues.

I once debugged a kernel memory leak in an internal module that manifested after around 6 years of (physical) server uptime. There are surprises lurking very far down the road.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#480

Post by FranzFerdiNaN »

usgroup wrote: post #467
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

I expect it all to work better with Musk in charge. He knows how to make scalable software and he knows about performant teams. That he’s not going to realise these totally obvious first order consequences people are raising seems unlikely.

Everything that is happening with Twitter proves Musk is yet another wealthy idiot who doenst know shit about shit, except how to blow his own horn. Musk is simlpy lucky that Twitter used to have such excellent engineers like this SRE so that the site isnt yet on fire.
Post reply on HN