Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

731–740 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#731

Post by acdha »

user_ wrote: post #147
acdha wrote: post #95

Earlier quoted context omitted.

Want to bet that’s not unconnected to sacking the people who tried to block bots?

I'm sure it's connected, but not because of the bots. Millions of political moderates are stopping by to see what happens when a large-scale public forum begins to tolerate freedom of speech.

Exactly what kind of views do you think moderates were worried about not being tolerated? Going by the definition of the term it seems unlikely that anyone, much less millions, was previously deterred by policy.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#733

Post by sgu999 »

jiggawatts wrote: post #723
karpathy wrote: post #539

Earlier quoted context omitted.

Elon also understands deep neural nets a lot more than I think people imagine. He starts with good intuitions and mental models, but also actively asks for technical deep dives, and has very good retention. E.g. I recall teaching him about our use of focal loss in contrast to binary cross-entropy for the object detection neural net (I said it had given us a 5% bump and he asked to know more) and he understood how it…

I think what upsets a lot of the Silicon Valley types here on HN is that people just like them are being called on their bullshit and fired en-masse for it. That has got to be uncomfortable. You know the old saying: "It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It." That's what's happening here. That engineer that was humiliated publicly for defending a slow-…

> That engineer that was humiliated publicly for defending a slow-as-molasses JavaScript-heavy microservices Rube Goldberg machine?

Citation needed ;D

Re: Why Twitter didn’t go down: From a real Twitter SRE

#735

Post by HeavyStorm »

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Indeed!

Re: Why Twitter didn’t go down: From a real Twitter SRE

#736

Post by jiggawatts »

Marazan wrote: post #605

Once thing I've seen that I think is overlooked in the whole "is Twitter's tech stack inefficient" chat is the nature of the service. A lot of the "Twitter could be run off my laptop" style comments seem to come from people who run, effectively, Read-Only services. They might serve data at thousands of queries per second but the data itself is _slow_. It is video files or music streams or other data that updates infr…

I've heard this elsewhere, and when Twitter was originally founded in 2006 it wouldn't have been a particularly hard problem.

Twitter processes roughly 10K tweets per second. Even if you bloat out the text quite a lot to account for encoding overheads, metadata, etc, etc... and assume that each one is 10KB, then this is just 1 Gbps. A single NIC on an old server.

Okay, I get it, Twitter needs a lot of data too. Lists of users, etc...

Twitter has 450 million monthly active users, which sounds like a lot, but even if there's 1MB of profile data per user such as who they are following, that's just 400 TB.

That's... not that much these days. A large-but-not-enormous database cluster.

Sure, there's "historical" data, but that can be compressed and put on cheap cold storage, like S3 or whatever.

Give me a few million annually as an opex budget, a small team of decent developers, and I can guarantee you I could whip up a cloud-hosted service that can process tweets at Twitter's scale.

Obviously, what I can't replicate is the much larger set of tools and systems behind the scenes that are used for moderation, analytics, ad sales, etc...

Re: Why Twitter didn’t go down: From a real Twitter SRE

#737

Post by geysersam »

carabiner wrote: post #477
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Yeah it's dancing around the question: Was Musk right? All signs so far are pointing to, yes . MBA's will be studying this for years.

Come on. There's no way to draw any conclusions only a couple of months after the change.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#738

Post by mschuster91 »

kortilla wrote: post #565
mschuster91 wrote: post #517

Earlier quoted context omitted.

> I mean wholly crap, this reminds me of old German industry where people retire in the place they started. German here. I think this actually is a huge part of the success of the famous Mittelstand - all that institutional knowledge these people have is extremely valuable. It's not just basic stuff like "know time tracking, billing and other admin systems and internal processes", but also the stuff that really can s…

> the problem is US-centric capitalism cannot quantify that impact (and it doesn't want to, given that attrition-related problems are long-term issues with years of time to impact) This is false. People invest on many year horizons all of the time when they believe in the company. If there was good evidence that having an average tenure of 5+ years was a great boost to the company, people would clamor to invest in co…

> This is false. People invest on many year horizons all of the time when they believe in the company.

The perfect example of why that is not the case at least in Silicon Valley is Google and their messengers and social networks... just how many of them exist(ed)? Four? Five? Each of them was someone's promotion project, left to languish, suffer and eventually die afterwards. Honestly it's a wonder Gmail and Maps are still around, their UI hasn't been updated much in ages despite obvious potential for improvements... I guess the only reason why they are alive is that they generate absurd amounts of data.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#739

Post by ColonelPhantom »

8n4vidtmkvmk wrote: post #187
jeffbee wrote: post #81

Earlier quoted context omitted.

A few parts clearly did go down. 2FA login was just serving error codes all day a few days ago. On Saturday people were posting entire feature films, 2 minutes per tweet, because apparently their copyright content matching wasn’t working.

that's surprising actually. I'd expect my sites to run for about a year if i suddenly died. maybe longer. certs will renew themselves, servers will restart themselves. everything is on autopay. the thing to kill it would probably be api breaks by 3rd parties. maybe disk space would kill me eventually too.. i don't have auto resizing for my sql database but its got a long runway yet. or god forbid my site grows on its…

Yes, but when you suddenly die, you don't think "hey, let's bring down this website" like Elon has: https://twitter.com/elonmusk/status/1592177471654604800

Re: Why Twitter didn’t go down: From a real Twitter SRE

#740

Post by anon400232 »

cal85 wrote: post #512
fnord123 wrote: post #472

Earlier quoted context omitted.

Twitter had 7,500 employees. most of the roles you mention (security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, scrum master) are not bloat. So the question is what are the other 7000 people doing?

Sitting on beanbags in group therapy sessions talking about imposter syndrome.

One problem with a humorous (yes, I am not all serious all the time) comment like this is while there could be some degree truth to it -- provide a source if so, please -- it smacks of exaggeration.
Post reply on HN