Why Twitter didn’t go down: From a real Twitter SRE

Post by saagarjha » Tue, Nov 22, 2022, 11:42 AM UTC

karpathy wrote: post #539

memish wrote: post #326

Earlier quoted context omitted.

John Carmack, "Elon is definitely an engineer. He is deeply involved with technical decisions at spacex and Tesla. He doesn’t write code or do CAD today, but he is perfectly capable of doing so." Kevin Watson, who developed the avionics for Falcon 9 and Dragon and previously managed the Advanced Computer Systems and Technologies Group within the Autonomous Systems Division at NASA's Jet Propulsion laboratory: "Elon i…

Elon also understands deep neural nets a lot more than I think people imagine. He starts with good intuitions and mental models, but also actively asks for technical deep dives, and has very good retention. E.g. I recall teaching him about our use of focal loss in contrast to binary cross-entropy for the object detection neural net (I said it had given us a 5% bump and he asked to know more) and he understood how it…

Post by saagarjha » Tue, Nov 22, 2022, 11:44 AM UTC

spaceman_2020 wrote: post #554

edmundsauto wrote: post #277

Earlier quoted context omitted.

Automotive and aerospace are not that similar to social media. People buying into the vision of "get the planet off fossil fuels for transport" and "get this species to Mars" are probably willing to make sacrifices that people working on social media are not. It's the Halo Effect fallacy to think competence in one field automatically translates to another. Especially when the founder in question has displayed increas…

> Automotive and aerospace are not that similar to social media. Yes. Social media is easier. > It's the Halo Effect fallacy to think competence in one field automatically translates to another. This is precisely about leveraging the Halo Effect fallacy. Elon Musk might not know social media, but the markets don't know that, nor do they care. The average retail trader sees "Elon Musk's company" and buys and holds, re…

Post by saagarjha » Tue, Nov 22, 2022, 11:46 AM UTC

kweingar wrote: post #211

emodendroket wrote: post #66

Earlier quoted context omitted.

I think this is kind of baked in though. Part of the thought process seems to be, at least for non-paying customers, it's not actually necessary to have five nines for Twitter, because people will just put up with it if it's less reliable.

I don’t have personal experience in this, so obviously I can’t speak with any authority. But I have heard from colleagues that tons of little factors can dramatically affect user engagement. For example, even a couple dozen milliseconds of longer load times can push a noticeable number of users away from your app.

Post by kenjackson » Tue, Nov 22, 2022, 11:49 AM UTC

smaudet wrote: post #416

TranquilMarmot wrote: post #380

Earlier quoted context omitted.

> As long as nothing changes, and you don't run out of disk space (from logs for example), things stay working pretty much just fine. > ... > There are other things that can bring a site down, like security issues, or bugs triggered by unusual states, too much traffic, etc. But generally speaking those things are rare and don't bring down an entire site. Aren't these changes inevitable, though? There is no such thing…

Software never goes stale, it's the environment around it which stales. Something from the 70s works perfectly fine, except it can't run on anything bare any longer, and the hard drives etc. have all long since failed or their PSU capacitors have blown....so Twitter will absolutely rot, how fast depends on several factors. I personally suspect the infrastructure used to build Twitter will rot faster than Twitter itse…

Post by Aeolun » Tue, Nov 22, 2022, 11:51 AM UTC

itsoktocry wrote: post #676

donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

> removing engineers won't instantly crash the product. It'll happen slowly It's amazing to me how many people following the Twitter saga, some familiar with or actually working in technology, thought that Twitter would crash within days of the engineers being fired. And because it didn't, the job cuts are justified.