Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

531–540 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#531

Post by wmil »

fnord123 wrote: post #472
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

Twitter had 7,500 employees. most of the roles you mention (security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, scrum master) are not bloat. So the question is what are the other 7000 people doing?

Twitter had a lot of people in marketing.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#533

Post by jiggawatts »

jdminhbg wrote: post #152
MuffinFlavored wrote: post #131

Earlier quoted context omitted.

> Twitter grew 3x on the headcount front There were multiple executives making $10m/yr+ There were board members There were shareholders Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"? These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc. How can us commenters on HackerNews sit from our arm…

> How can us commenters on HackerNews sit from our armchair and say "ah, goofballs should've just not let headcount get so high!" The cliche HN comment on sites like Twitter (and many, many others, any time headcount comes up) has always been "why do they need so many people?" I've mostly dismissed it the same way I dismiss "I could build Uber in a weekend," but with every other tech giant laying people off, maybe I…

A friend of mine used to work for a private equity firm that had about a hundred C++ and Java developers working on a stock trading and prediction system. He quit and rewrote the essence of that system in F# single-handedly.

If you watch interviews with famous developers like John Carmack, they'll mention that working alone scales to about the equivalent of 5x developers. That is, adding 1-3 extra people might slow you down because of the overheads of communication and coordination. It's only around 5+ in a team that there is a definite advantage.

But what are the chances of putting together a team of 5 rockstar developers that all agree on language, style, and vision? Basically zero. So you have to settle for mediocrity. Popular languages, simple approaches, established design patterns.

If you're an experienced "rockstar" developer coding by yourself and use a fancy language like F#, you can outperform a team of 10+ people. If you're replicating a system you've seen already, 20-50 might not be out of the question, especially if you're smart enough to avoid "tarpits" and instead rely on good quality libraries and CotS components like databases, PaaS, and the like.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#534

Post by katekarin »

Varriount wrote: post #386
lcw wrote: post #366

Earlier quoted context omitted.

Agreed, one of the craziest bugs I had to deal with was we had a distributed system using lots of infrastructure. Said distributed system started having trouble communicating with random nodes and sub-systems. I spent 3 hard days finding a Linux kernel bug where the ARP cache was not removing least recently accessed network addresses. Normally, this wouldn't be a big deal for a typical network because few networks wo…

Goodness, what kind of process/tools did you use to track that problem down?

My team had a similar issue with the ARP cache on AWS when we used Amazon Linux as an OS for cluster nodes, and Debian for the database host. When new tasks were starting some had random timeouts when connecting to the database.

It turned out that the Debian database host had bad ARP entries (an IP address was pointing to a non-existing MAC Address) caused by frequent reuse of the same IP addresses.

Debian has a default ARP cache size that's larger than Amazon Linux (I think it's entirely disabled on AL?).

As for the tooling we used to track it down, it was tcpdump. We saw SYN's getting sent, but not ACK's back. Few more tcpdump flags (-e shows the hardware addresses) and we discovered mismatched MAC addresses.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#535

Post by coldtea »

oxfordmale wrote: post #443
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Let's do a thought experiment and see what functions aren't needed to keep the light on for 30 days: 1) HR 2) Legal 3) Sales 4) Marketing 5) Payroll 6) Admin staff 7) Most of Engineering, other than the bare minimum of L1/L2/3 support. As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend.

>As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend.

On the other hand, a car without a second and third steering wheel, 20 windscreen wipers, and an oven in the back, keeps running just fine, even after the first bend...

Re: Why Twitter didn’t go down: From a real Twitter SRE

#536

Post by spaceman_2020 »

mirzap wrote: post #407
swellguy wrote: post #34

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

Yeah, and they were even profitable with ~3k employees. Then the hiring spree started and they went negative. Even if there wasn't Musk they would have to let go at least 30% of the people.

The stock was rightly crashing when the company was public. A social media site that manges to lose money during the pandemic is truly mismanaged.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#537

Post by 867-5309 »

yunohn wrote: post #444
rjzzleep wrote: post #429

Earlier quoted context omitted.

The reality is most huge companies are majority bloat. The hiring numbers are also in part crap that goes into Series X Raise pitch decks. Oftentimes a lot of the new bloat pisses off competent people, because their work doesn't actually get less, it becomes more. Not only do they have to now nanny people that are often not actually competent in their job, they just happened to go through the coding interview with wh…

> I mean wholly crap, this reminds me of old German industry where people retire in the place they started. This is honestly uncalled for. Job hopping every 2-3yrs should not be an expected task.

is that down to the employer or employee?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#539

Post by karpathy »

memish wrote: post #326
ulkesh wrote: post #27

All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…

John Carmack, "Elon is definitely an engineer. He is deeply involved with technical decisions at spacex and Tesla. He doesn’t write code or do CAD today, but he is perfectly capable of doing so." Kevin Watson, who developed the avionics for Falcon 9 and Dragon and previously managed the Advanced Computer Systems and Technologies Group within the Autonomous Systems Division at NASA's Jet Propulsion laboratory: "Elon i…

Elon also understands deep neural nets a lot more than I think people imagine. He starts with good intuitions and mental models, but also actively asks for technical deep dives, and has very good retention. E.g. I recall teaching him about our use of focal loss in contrast to binary cross-entropy for the object detection neural net (I said it had given us a 5% bump and he asked to know more) and he understood how it works about as quickly as you'd expect a PhD student to. The fact that he can do this across many technical disciplines is impressive and borderline superhuman. I don't think people understand or would believe how low-level and technical typical meetings with him are. Just saying because I get triggered reading way off innacurate takes on this topic (original comment).

Re: Why Twitter didn’t go down: From a real Twitter SRE

#540

Post by RivieraKid »

kakwa_ wrote: post #524
holoduke wrote: post #506

Earlier quoted context omitted.

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

That's likely optimistic. Maybe the user facing site, but that's just the tip of the iceberg. There are plenty of internal/backend/restricted systems to support and/or monetize this part. And that's not counting the huge number of support people & moderators needed.

Frontend, Android app and iOS app should be doable by 1 person each. Then one graphic designer and a project manager for all of the 3 frontends.
Post reply on HN