Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

551–560 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#551

Post by objclxt »

holoduke wrote: post #506
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

> I am convinced you can run the entire tech stack with a team of a 100 people.

This is because you don't see the complexity. What you see as a Twitter user is a fraction of what's actually there.

You have to build a platform for ads. Not just serving ads, but allowing advertisers to prepare their collateral, preview them, get their results, and be billed. So that's an entire content and invoicing platform separate from your main feed.

And since your platform is all user generated content, you've got to build a moderation pipeline. A place for users to make reports, but also an interface for your content moderators to view content and make decisions. Oh, and while you're there you'd better build a portal for law enforcement to make data requests, along with your DMCA takedowns. Oh yeah, DMCA - that's another whole thing you've got to worry about.

Then the EU comes along and needs you to build something to support your GDPR obligations. Then India wants something similar, but only for its citizens. Your users also want verification, so better build that platform for securely verifying accounts and awarding checkmarks.

It snowballs. Was Twitter's engineering group bloated? Probably. Most large companies are. Could you run the whole Twitter tech stack as it exists today with a hundred people? Absolutely not.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#552

Post by DSMan195276 »

jiggawatts wrote: post #518
underwater wrote: post #497

Earlier quoted context omitted.

Look at his claim that Twitter is slow in India because of poorly batched backend calls [1]. He's simply regurgitating what he thinks he heard from others without really understanding how anything works. He knows less about software architecture than an intern picked at random. 1. https://mobile.twitter.com/elonmusk/status/15921762028730859...

Stop repeating this, because Elon had the gist of it correct, if not the specifics. a) Twitter is atrociously slow for something that displays a mere kilobyte of text at a time. b) Much of that is because of hundreds of client-server JavaScript requests on first load. Not technically RPCs, but just as slow and looking nearly identical on the network. c) Most of the rest is because of a thousand microservices chatting…

> Much of that is because of hundreds of client-server JavaScript requests on first load.

Isn't this not true though? Based on the responses to his Tweet only one GraphQL request is made which batches the whole thing.

> Most of the rest is because of a thousand microservices chatting away at each other. Not technically client-to-server, but still taking significant time.

But what does this have to do with India? If the slowness is due to calls between microservices hosted on (from my understanding) their own hardware, why would it matter if the initial GraphQL request comes in from India, since all the rest of the backend requests would be the same?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#553

Post by wikfwikf »

donkeyd wrote: post #372
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

data architecture is bloat?

Have you implemented a system which stores hundreds of billions of pieces of media content and makes different slices of them immediately available to hundreds of millions of users?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#554

Post by spaceman_2020 »

edmundsauto wrote: post #277
WalterBright wrote: post #165

Earlier quoted context omitted.

America is such a great country that a random person can just fecklessly blunder into creating a revolutionary electric car company and cluelessly blunder into creating a rocket company that is the envy of the world.

Automotive and aerospace are not that similar to social media. People buying into the vision of "get the planet off fossil fuels for transport" and "get this species to Mars" are probably willing to make sacrifices that people working on social media are not. It's the Halo Effect fallacy to think competence in one field automatically translates to another. Especially when the founder in question has displayed increas…

> Automotive and aerospace are not that similar to social media.

Yes. Social media is easier.

> It's the Halo Effect fallacy to think competence in one field automatically translates to another.

This is precisely about leveraging the Halo Effect fallacy. Elon Musk might not know social media, but the markets don't know that, nor do they care. The average retail trader sees "Elon Musk's company" and buys and holds, regardless of absurd PEs.

Musk knows the power his brand has. He's simply going to use that to pump up Twitter's valuation, all through the virtue of his "halo"

Re: Why Twitter didn’t go down: From a real Twitter SRE

#555

Post by kortilla »

bartvk wrote: post #471
rjzzleep wrote: post #429

Earlier quoted context omitted.

The reality is most huge companies are majority bloat. The hiring numbers are also in part crap that goes into Series X Raise pitch decks. Oftentimes a lot of the new bloat pisses off competent people, because their work doesn't actually get less, it becomes more. Not only do they have to now nanny people that are often not actually competent in their job, they just happened to go through the coding interview with wh…

> this reminds me of old German industry where people retire in the place they started What is wrong with that?

Depends on if you want innovation or stability. For innovation, people building fiefdoms over decades of political maneuvering is terribly destructive to change.

Very few industries require constant learning for the business to compete so a highly tenured employee likely hasn’t learned anything new beyond minute process changes for 10+ years. Once people are ossified into a role like that, they will do anything and everything to shut down anything that has a whiff of threatening their current role.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#556

Post by busymom0 »

swellguy wrote: post #34

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

Not just that. From my personal experience, twitter had to be one of the slowest websites I use. Even in my 2020 mac, it often shows the memory warning in safari. Things take a while to load. And the UX is terrible with having to constantly click to read child comments, having to click on “show hidden replies” etc. I honestly have no idea how a company with thousands of employees and a billion in loss was able to operate such a terribly performing website.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#557

Post by hef19898 »

jiggawatts wrote: post #518
underwater wrote: post #497

Earlier quoted context omitted.

Look at his claim that Twitter is slow in India because of poorly batched backend calls [1]. He's simply regurgitating what he thinks he heard from others without really understanding how anything works. He knows less about software architecture than an intern picked at random. 1. https://mobile.twitter.com/elonmusk/status/15921762028730859...

Stop repeating this, because Elon had the gist of it correct, if not the specifics. a) Twitter is atrociously slow for something that displays a mere kilobyte of text at a time. b) Much of that is because of hundreds of client-server JavaScript requests on first load. Not technically RPCs, but just as slow and looking nearly identical on the network. c) Most of the rest is because of a thousand microservices chatting…

A good engineer (!) would have been able to understand the difference, just saying.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#558

Post by dcow »

EdwardDiego wrote: post #270
dcow wrote: post #110

Earlier quoted context omitted.

I have a piece of Rust software that has not gone down in its entire lifetime.

Good for you! Now, if your Rust code was a distributed system that handles spiky loads from ~330m users, and processes petabytes of data, then I'd consider your comparison relevant to Twitter. But I'm going to assume it's not relevant. P.S., I've written Java services that never went down, because they had a well defined domain and all potential errors were handled. But, I'm not about to compare that to all of friggi…

I wasn’t attempting to compare anything to Twitter…

Re: Why Twitter didn’t go down: From a real Twitter SRE

#559

Post by b3lvedere »

28304283409234 wrote: post #419
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Argh. "It works now, so it will work until forever." It takes _effort_ to make it work this smoothly now, _and in the future_. SRE is about _preventing_ issues. Not mopping up after them. To me, the article read like every succesfull sysadmin story: there's no fires, so sysadmin must be bloat.

Whenever clients complain about those costs and efforts, i tell them it's just like their car.

Your car is working perfectly fine so why should you pay for maintenance?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#560

Post by ravloony »

weinzierl wrote: post #464
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

No matter if it was or not and for better or worse: If Twitter survives this without any major harm it will have profound consequences for the whole software industry.

This is exactly why this thread is full of slightly insecure comments making vague predictions. I'd suggest to most of them to get off HN and back to work, now is the time to make yourself useful!
Post reply on HN