Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

721–730 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#721

Earlier quoted context omitted.

Your reply is indistinguishable from a religious person exclaiming that "God is omniscient and moves in mysterious ways!" Stop worshipping billionaires.

You could say that if he was new to the party. But he did change the world with his companies already, didn't he?

Yes, and that's why he's getting worshipped, and that's what's making him go off the rails here.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#722
post #512

Earlier quoted context omitted.

Twitter had 7,500 employees. most of the roles you mention (security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, scrum master) are not bloat. So the question is what are the other 7000 people doing?

Sitting on beanbags in group therapy sessions talking about imposter syndrome.

what's sad is that the people who were there doing real work are being lumped in with the people who were there for adult daycare.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#723
post #326

Earlier quoted context omitted.

John Carmack, "Elon is definitely an engineer. He is deeply involved with technical decisions at spacex and Tesla. He doesn’t write code or do CAD today, but he is perfectly capable of doing so." Kevin Watson, who developed the avionics for Falcon 9 and Dragon and previously managed the Advanced Computer Systems and Technologies Group within the Autonomous Systems Division at NASA's Jet Propulsion laboratory: "Elon i…

Elon also understands deep neural nets a lot more than I think people imagine. He starts with good intuitions and mental models, but also actively asks for technical deep dives, and has very good retention. E.g. I recall teaching him about our use of focal loss in contrast to binary cross-entropy for the object detection neural net (I said it had given us a 5% bump and he asked to know more) and he understood how it…

I think what upsets a lot of the Silicon Valley types here on HN is that people just like them are being called on their bullshit and fired en-masse for it. That has got to be uncomfortable.

You know the old saying: "It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It."

That's what's happening here.

That engineer that was humiliated publicly for defending a slow-as-molasses JavaScript-heavy microservices Rube Goldberg machine? Hacker News is filled with people just like him that have built near identical software in other orgs.

Understandably they're upset and are looking for any excuse to dismiss Elon's criticism of not just Twitter, but their entire industry.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#724

Earlier quoted context omitted.

> As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend. On the other hand, a car without a second and third steering wheel, 20 windscreen wipers, and an oven in the back, keeps running just fine, even after the first bend...

That might be fine but do you want to fly in a plane that only has one engine? What about if it has no fallback if the main sensors freeze?

>That might be fine but do you want to fly in a plane that only has one engine?

I've done that lots of times. As do almost all people flying Cessnas, Moonys, Pipers and the like...

Re: Why Twitter didn’t go down: From a real Twitter SRE

#725

Earlier quoted context omitted.

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

> I am convinced you can run the entire tech stack with a team of a 100 people Please tell us in detail about the Twitter stack. Because I always find it fascinating how people think they can estimate the effort to maintain it whilst having next to no understanding what so ever of the tech stack.

I couldn’t give you details, but I do know Instagram had 13 employees when Facebook bought them. Is Twitter really two-orders of magnitude more difficult to run?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#726
post #719

Earlier quoted context omitted.

> removing engineers won't instantly crash the product. It'll happen slowly It's amazing to me how many people following the Twitter saga, some familiar with or actually working in technology, thought that Twitter would crash within days of the engineers being fired. And because it didn't, the job cuts are justified.

> thought that Twitter would crash within days of the engineers being fired Because they work for companies where the product would fail within days of them being fired themselves.

That's what people keep telling themselves. Truth is, everybody is replaceable. And usually, loss of institutional knowledge takes a while to show real effects.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#727
post #162

Earlier quoted context omitted.

Another thing we noticed at Netflix was that after services didn’t get pushed for a while (weeks), performance started degrading because of things like undiscovered memory leaks, threads leaks, disks filling up. You wouldn’t notice during normal operations because of regular autoscaling and code pushes, but code freezes tended to reveal these issues.

We used to have a horribly written node process that was running in a Mesos cluster (using Marathon). It had a memory leak and would start to fill up memory after about a week of running, depending on what customers were doing and if they were hitting it enough. The solution, rather than investing time in fixing the memory leak, was to add a cron job that would kill/reset the process every three days. This was easier…

This is a feature in many HTTPDs, WSGI/FastCGI apps, possibly even in K8s. After X requests/time, restart worker process. Old tricks are the best tricks ;)

Re: Why Twitter didn’t go down: From a real Twitter SRE

#728

Now that Elon has cleaned house, when is Twitter hiring again? It might be a good opportunity to get in on the new ground floor.

Quoted post unavailable.

Maybe it’s worth the money for guys who are okay with that, otherwise who would work for him?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#729
post #68

Earlier quoted context omitted.

So should the platform be guided by advertisers? Especially one that’s apparently the de facto public square?

What choice do they have? The company was hovering around profitable (2021 would’ve been without a lawsuit settlement) but that was before they were saddled with a ton of new debt. It’s possible that they could find new revenue models - that pay for checkmark scheme isn’t it but maybe a smart businessman could come up with a better variant - but that takes time and has to be done carefully since they have to keep the…

Debt and equity are just different ways for investors to invest in a company. If company isn’t generating any return on investment, keeping the lights on is not sustainable.
Post reply on HN