Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

561–570 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#561

Post by onethought »

coldtea wrote: post #547
tempestn wrote: post #487

Earlier quoted context omitted.

Musk might have been right about some things. There probably was some degree of bloat. But to say he's badly mishandled this whole saga is a gross understatement. It is very difficult to utterly kill a site like Twitter; the fact that we're even considering that as a realistic possibility shows just how badly. I think Musk is used to Tesla and SpaceX, which are both companies that a lot of people are (or at least wer…

> the fact that we're even considering that as a realistic possibility shows just how badly. Do we have much to go with to "really consider it", or is it just sensenationalist headlines, as Musk went ahead the accepted orthodoxy on Twitter and content moderation and so "he must pay"? That would be the same news outlets that built him up as the real life Tony Stark and propped up his quite ordinary companies, one of w…

Are you purposefully misrepresenting this? Or am I missing sarcasm?

Why aren’t the other space programs/companies landing rockets?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#562

Post by raldi »

kweingar wrote: post #211
emodendroket wrote: post #66

Earlier quoted context omitted.

I think this is kind of baked in though. Part of the thought process seems to be, at least for non-paying customers, it's not actually necessary to have five nines for Twitter, because people will just put up with it if it's less reliable.

I don’t have personal experience in this, so obviously I can’t speak with any authority. But I have heard from colleagues that tons of little factors can dramatically affect user engagement. For example, even a couple dozen milliseconds of longer load times can push a noticeable number of users away from your app.

This is true, an insightful add-on point, and one Larry Page’s favorite pearls of wisdom.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#563

Post by PeterStuer »

In every software company of over 50 employees I have seen up close it would be fairly to identify 50% of people that had minimal to negative contribution.

That said, it takes some time to get to know the real workings of a company, knowledge required to select the right 50%.

That said, afaict the current mo is keep the lights on for the current offering while a new team builds a new product for that enormous user base that offers something much more profitable.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#564

Post by cdf »

holoduke wrote: post #506
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

Getting it to about 90% working in a single country or just the general western sphere, probably 100 will suffice.

Making it globally available and legally compliant, that's where the next few thousand folks come in.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#565

Post by kortilla »

mschuster91 wrote: post #517
rjzzleep wrote: post #429

Earlier quoted context omitted.

The reality is most huge companies are majority bloat. The hiring numbers are also in part crap that goes into Series X Raise pitch decks. Oftentimes a lot of the new bloat pisses off competent people, because their work doesn't actually get less, it becomes more. Not only do they have to now nanny people that are often not actually competent in their job, they just happened to go through the coding interview with wh…

> I mean wholly crap, this reminds me of old German industry where people retire in the place they started. German here. I think this actually is a huge part of the success of the famous Mittelstand - all that institutional knowledge these people have is extremely valuable. It's not just basic stuff like "know time tracking, billing and other admin systems and internal processes", but also the stuff that really can s…

> the problem is US-centric capitalism cannot quantify that impact (and it doesn't want to, given that attrition-related problems are long-term issues with years of time to impact)

This is false. People invest on many year horizons all of the time when they believe in the company. If there was good evidence that having an average tenure of 5+ years was a great boost to the company, people would clamor to invest in companies like that.

This was basically Google’s entire pitch when they were an early public company. Happy employees == great things. I remember when 60 minutes or dateline did some special about google before 2010 and people were floored by how good the employees got it. However, hiring standards relaxed and now google is slowly rotting from the inside to realize it’s the new IBM.

Long tenure alone is absolutely not an advantage. It’s very easy to have a bunch of dead weight that just looks like they know what’s going on.

So back to your point about experience mattering. That should show up in customer satisfaction, project turnaround speed, success rates, etc. Otherwise it’s pretty meaningless. And guess who already measures those KPIs…

Re: Why Twitter didn’t go down: From a real Twitter SRE

#566

Post by saiya-jin »

mirzap wrote: post #402
poulsbohemian wrote: post #87

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…

Quoted post unavailable.

Yeah he is so smart, he locked himself out in firing spree during some manic phase and then had to beg people he just fired to let him in. If anything, this twitter saga showed serious cracks (that were there before if you looked closely) in his persona. Vengeful, petty childish behavior was there before (ie Musk accusing main Thai cave rescuer for being a pedophile, because he refused his submarine concept? Plenty of similar stories)

Its a matter of approach - if you focus on all his success, you can continue worshiping him as some capitalist messiah. If you focus on bad stuff, he is a fucked up mess to be polite, a bad parent and husband, an even worse human being in general, utterly horrible boss, person with deep mental issues that aren't clearly tackled in appropriate ways.

The truth, somewhere in the middle of those 2 extremes, still ain't that nice IMHO. He burned basically all good PR capital be built over the years, basically nobody sees him now as good person, just another ultra rich a-hole like Bezos that needs to be kept in check. As he will find in following years, people's opinions trickle down to cash flow. If I would be buying an electric car right now (which doesn't make much sense for another decade to me), Tesla would be off the table for example.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#567

Post by birdyrooster »

donkeyd wrote: post #372
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

They will stagnate and have more failed launches than Space X.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#568

Post by kortilla »

henrikschroder wrote: post #406
mirzap wrote: post #402

Earlier quoted context omitted.

Quoted post unavailable.

Your reply is indistinguishable from a religious person exclaiming that "God is omniscient and moves in mysterious ways!" Stop worshipping billionaires.

> Stop worshipping billionaires.

This is a vapid over-generalization designed to dismiss any statements by attaching them to the wealth, not the person. Don’t do this.

Nobody is talking about the Waltons, Buffet, Larry Ellison, etc. This is a discussion about Musk specifically and his history of operating companies.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#569

Post by mihaaly »

donkeyd wrote: post #372
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

> The article touches a really small [...] subject.

That's how it couldn't prove the claim.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#570

Post by andy_ppp »

There seems to be a load of people in this thread making statements on both sides that a) the article doesn’t prove there were too many people at twitter b) the article implies there were too many people at Twitter c) Twitter is a simple app why so complicated etc.

I would say this; Twitter at this scale is extremely complicated. We don’t have enough information to know what all these roles were doing and how many were truly superfluous. The article gives an interesting insight into something extremely complicated- it might be a slice repeated in a lot of areas. I’m of the opinion that you can do a lot more with less but there’s no need to send emails to everyone in the organisation being an arsehole and essentially accusing them of being lazy, then rehiring people who never worked there just for the meme.

Anyway Musk will blow up some rockets, put enormous pressure on people to fix them, and come up smelling of roses again even if he treats a load of people like crap along the way. His fans will say he’s a genius and his enemies will say he’s a dickhead. And so it goes.

Post reply on HN