Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

921–930 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#922
post #326

Earlier quoted context omitted.

John Carmack, "Elon is definitely an engineer. He is deeply involved with technical decisions at spacex and Tesla. He doesn’t write code or do CAD today, but he is perfectly capable of doing so." Kevin Watson, who developed the avionics for Falcon 9 and Dragon and previously managed the Advanced Computer Systems and Technologies Group within the Autonomous Systems Division at NASA's Jet Propulsion laboratory: "Elon i…

Elon also understands deep neural nets a lot more than I think people imagine. He starts with good intuitions and mental models, but also actively asks for technical deep dives, and has very good retention. E.g. I recall teaching him about our use of focal loss in contrast to binary cross-entropy for the object detection neural net (I said it had given us a 5% bump and he asked to know more) and he understood how it…

Has he ever make a public statement, at least a paragraph in length, explaining something technical?

It's hard to blame people based on his decades of public behavior and lying about his education, falsely claiming to have a physics degree and to have been admitted to grad school.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#923

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

There's an incredible amount of bloat in big tech.

It's become an adult daycare, https://twitter.com/DavidSacks/status/1561096423243800576

Twitter's layoffs followed by that 1AM photo of hackers at work is terrifying to lifestyle employees.

It's the Return of the Nerds.

Google, Meta, Netflix, Microsoft are all watching.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#924
post #923

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

There's an incredible amount of bloat in big tech. It's become an adult daycare, https://twitter.com/DavidSacks/status/1561096423243800576 Twitter's layoffs followed by that 1AM photo of hackers at work is terrifying to lifestyle employees. It's the Return of the Nerds. Google, Meta, Netflix, Microsoft are all watching.

what does working until 1AM have anything to do with being a nerd or a hacker?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#925

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

Googling "how many people work at stack overflow" gives a number less than 400. If true I think you could put together and run Twitter with less than 400 people from an engineering stand point. However Twitter probably has a much higher soft skills head count for moderation and what not. Though it wouldn't surprise me if a lot of that was contractors not direct head count.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#926
post #86
post #27

All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…

Judging by some of the old patents he's filed [1], I'd guess he has at least a decent understanding of the tech involved. Probably less so, when it comes to the details of more modern distributed systems, but I also wouldn't be surprised if he's spent some effort towards all that as well - he's been working in/around pretty cutting edge tech for quite a while. Could he sit down and code it himself? probably not, but…

The CEO get to put his name on the company's patents, yes.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#927

Earlier quoted context omitted.

No matter if it was or not and for better or worse: If Twitter survives this without any major harm it will have profound consequences for the whole software industry.

And it's almost inevitable. That's why the overwhelming "Elon is so dumb" narrative seems so odd to me. "Major harm," to me, would be either bankruptcy or a competitor overtaking significant chunks of Twitter's users. Even if Elon literally has to re-hire half the roles he fired for, or Twitter is down for a few days or a week, or they struggle to get advertisers for a little while, that's nothing for the long-term.…

> "Major harm," to me, would be either bankruptcy or a competitor overtaking significant chunks of Twitter's users.

With the takeover, Twitter has been saddled with massive new debt and would have to become significantly more profitable to service it. Instead, they are losing ad revenue, and any savings from reduced head count will not manifest four at least 4 months because of the severance payments for the laid off workers.

Musk could of course keep the company going as a hobby for a few more years, but he strikes me as more likely to cut his losses early — preferably by finding somebody to take the problem off his hand, but, if necessary, by bankruptcy.

> Even if Elon literally has to re-hire half the roles he fired for

I have this nagging suspicion that the kind of people who line up for an opportunity to work without equity, lousy job security, and send weekly code printouts to the CEO are not necessarily the highest quality hires in the field.

> or Twitter is down for a few days or a week

The Twitter user base has quite a bit of inertia keeping them from moving to competing platforms — lots of people I follow have made backup plans on Mastodon, but not many have moved their focus there. But Twitter becoming literally impossible to use would probably be the one thing precipitating such a process.

> they struggle to get advertisers for a little while

And after that? Are the major brands going to decide that they like advertising on a Gab-lite Twitter after all? Or is Twitter going to strike it rich with ads for dietary supplements, gold coins, non-perishable meals, and tactical pants?

> In six months, the chances that it'll look like these firings were a bad idea are minimal.

The targeted firings were bad enough (To paraphrase someone else "Imagine ranking engineers by # of lines of code added, and then keeping the ones with the HIGHEST number"), but the voluntary departures are likely to be even more consequential. After the way the workforce was treated, what are the odds that the people left are the "best", rather than the ones with a lack of alternatives (e.g. for visa reasons), or Musk fans who may or may not be qualified?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#928

Now that Elon has cleaned house, when is Twitter hiring again? It might be a good opportunity to get in on the new ground floor.

Who sees the foundation of a high-rise starting to crack and thinks, "perfect time to get in on the ground floor"? Ground floor is exactly where you don't want to be when the structure collapses under its own weight. Rather than bring in additional specialists to reinforce it, Musk has partially evacuated the building at least.

It’s Elon, don’t think he going to take a 44 billion hit on this, he is going to make his (and his co-financiers) money back.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#929

Earlier quoted context omitted.

I'm still on the fence. As an engineering manager, I tend to attach faces to those "jobs". So seeing cuts, I imagine a ton of people that had to go home and tell their friends/family/etc that they no longer had a job. On the other hand. As an engineer, we tend to attach way too much self importance to our roles. Like if we're not there entering the "numbers" 4 6 15 16 24 32 every 108 minutes, the entire business is g…

> Like if we're not there entering the "numbers" 4 6 15 16 24 32 every 108 minutes, the entire business is going to crumbl Never I have encountered an engineer that thought that.

>> Like if we're not there entering the "numbers" 4 6 15 16 24 32 every 108 minutes, the entire business is going to crumbl

> Never I have encountered an engineer that thought that.

there are people all up and down this page saying effectively just that. well i guess i'm assuming most of these people are engineers in the software sense of the word.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#930

Earlier quoted context omitted.

I agree those were odd takes. I've likened firing most of the engineers to taking your hands off the wheel in the car. It won't crash immediately, but it doesn't mean the car can go driverless. With that said, there are differences between internal systems and something like Twitter on the public internet. I assume that Twitter is a system under constant attack. What happens when the next log4shell level vulnerabilit…

If Twitter went another month without an outage, how would you adjust your opinion? How about a year? The car analogy is amusing, but how much does it really hold up? Have we ever seen another major social media company drop this much of its staff in one go? I certainly can’t think of an example. I think we’re in somewhat uncharted waters here. A driverless car won’t last long, we know that for a fact. I think it rem…

> If Twitter went another month without an outage, how would you adjust your opinion? How about a year?

I think it's hard to just use time as a measure. If there are no security issues and they add no features, then things should run fine. Which, ironically points to how solid the team was that was fired.

Of course if Twitter not only limps along, but thrives in this new setup then I'll definitely change my opinion. Being in the US, this might end up the case while Twitter for the rest of the world falls apart.

Keep in mind, that I think Twitter was bloated and needed a big shakeup. Randomly dumping people and those who tried to correct me is not the heuristic I would have used.

With that said, Twitter still has 2 huge problems. No vision and saddled with an enormous amount of debt. Right now, Musk is taking the PE approach to cut and milk what's there. The problem is, there isn't much to milk.

Post reply on HN