Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

771–780 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#771

Post by housecarpenter »

ggm wrote: post #230

The article is good, and informative but a little odd in one respect. It uses air-quotes to introduce concepts like "rack" and "shelf" in a DC, but does O(logn) notation mid-flow. If you don't know what a rack is, how are you meant to know what the Order of scaling function means? Thats a highly computer sciences specific notation, and if you grok O(n) you know what a rack, a host, a DC is.

I get your point that both of these are specialized notations/terminologies, but it's entirely possible for somebody to understand algorithmic complexity without knowing what "a rack, a host, a DC" is, there's no inherent connection between the two. As somebody whose interests tend to be more theoretical than practical, myself prior to reading the article would be an example.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#773

Post by perihelions »

saagarjha wrote: post #701
bombolo wrote: post #677

Earlier quoted context omitted.

USA at some point had an anti-missile system that needed periodical reboots because it was originally designed for short deployments, so the floating point variable for the clock would start to lose precision after a while.

Which, of course, led to people dying when the drift was too great.

Context for those unfamiliar:

https://www-users.cse.umn.edu/~arnold/disasters/patriot.html

https://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dha...

https://hn.algolia.com/?query=patriot%20missile (HN threads)

Re: Why Twitter didn’t go down: From a real Twitter SRE

#774

Post by matwood »

carabiner wrote: post #477
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Yeah it's dancing around the question: Was Musk right? All signs so far are pointing to, yes . MBA's will be studying this for years.

Way too soon to declare that Musk was right. I don't even think signs are pointing there. Twitter is bleeding some of its most valuable users, the content creators, to things like Mastodon. There do appear to be cracks happening at the edges. Bots and hate speech do appear to have increased.

Thing is, I think Twitter was bloated and it needed a kick in the rear. Pre-acquisition I heard the same from many I follow. How Musk has gone about it has been the problem. Ignoring his perpetual hates, he had a decent amount of goodwill the day the deal closed. Then, he squandered it with all his antics. A transparent content moderation board turns out to be a game-able Twitter poll. Blue check for all was completely missing any point. No one wants a blue check for money w/o the associated verification. Verification for all would have been awesome.

Ads quality has dropped from what I've seen. It looks like people are pulling out, albeit slowly. MBAs will be studying this, but how things are going means we may look back and see this as Twitters Yahoo/AOL moment when it sells for a few billion in a couple years.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#775

Post by ce4 »

j3th9n wrote: post #436
aeyes wrote: post #404

Earlier quoted context omitted.

You don't even need that, the kernel OOM killer would take care of this eventually. Unless its something like Java where the garbage collector would begin to burn CPU.

The OOM killer doesn't restart (randomly, unless configured) killed processes, it just kills.

Unless the OOM-killer kills the wrong process. Ages ago we had a userspace filesystem (gpfs) that was of course one of the oldest processes around and it consumed lots of RAM. When the oom killer started looking for a target, of course one of the mmfsd processes was selected and it resulted in instantaneous machine lockup (any access to that filesystem would be blocked forever in the system call which depended on the userspace daemon to return, alas never returning). Was funny to debug

Re: Why Twitter didn’t go down: From a real Twitter SRE

#776

Post by qwytw »

mirekrusin wrote: post #427
henrikschroder wrote: post #406

Earlier quoted context omitted.

Your reply is indistinguishable from a religious person exclaiming that "God is omniscient and moves in mysterious ways!" Stop worshipping billionaires.

You could say that if he was new to the party. But he did change the world with his companies already, didn't he?

Any optimization he might do is unlikely to offset the cost of servicing the debt he offloaded to Twitter's balance sheet after the acquisition.

So at 5% interest the yearly cost is 650 million, even at some obscene and unrealistic payroll per employee (including not only developers), like let's say 400k he'd have to fire over 1600 people just to balance this out. There is no way to spin this positively. Also due to no more RSUs cash comp for remaining employee would need to increase.

Add to that the the loss in advertisement revenue do to obvious reasons.

Taking these facts into account could you please explain how is Musk NOT a liability to Twitter? He basically spent 31 billion (plus 13 billion he offload to Twitter) just to buy the trademark and the user base (and some proportion of advertisers...). From a financial standpoint that's an objectively terrible deal. Musk might have other goals and that's great but in no way this is comparable to Tesla or SpaceX.

>But he did change the world with his companies already, didn't he?

History is full of examples of extraordinary successful people who get blinded by their own success and start thinking that they are infallible. This ussually results which in pretty unhinged behavior when they get older. Musk does not seem like the most grounded person ever so he's likely at a much higher risk of this than almost every tech billionaire I can think of.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#777

Post by pmoriarty »

john_the_writer wrote: post #712
b3lvedere wrote: post #559

Earlier quoted context omitted.

Whenever clients complain about those costs and efforts, i tell them it's just like their car. Your car is working perfectly fine so why should you pay for maintenance?

software doesn't break down from heat. An app I write today will run until the hardware dies. I have a palm_os app I wrote in 1998 that still runs perfectly.

"software doesn't break down from heat. An app I write today will run until the hardware dies. I have a palm_os app I wrote in 1998 that still runs perfectly."

In an organization of any appreciable size, things change all the time.. and I'm not just talking about code (for which you could have a code freeze in an emergency situation like this), but the external systems you're connected to could change for reasons completely out of your control. Content changes can break stuff because of bugs in your code. Legacy systems could require all sorts of ongoing tweaking and maintenance. And, yes, heat can break your software if the server it's running on overheats.

Twitter is not a palm_os app.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#779

Post by badpun »

randmeerkat wrote: post #336
ezoe wrote: post #246

Earlier quoted context omitted.

Redundancy. Now the systems are stable but human workers either be sick, leave, or die eventually. Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them. The article wrote about achieving stability by the distributed system so an unexpected death of one rack doesn't affect the service availability. The same can be done for…

> Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them. People would absolutely be more engaged and more excited about their work if they were paid more. The only reason people work is literally for money…

I've found that to be absolutely not the case, at least for me. If the job is super disorganized, boring, stresful for stupid reasons or just plain not interesting for me, no amount of money will make me engaged or excited. I've been in such jobs which paid a lot of money (for me) and the result was golden handcuffs - I didn't want to lose the money, so I was basically pretending to be way more engaged than I really was. I suspect majority of FAANG employees are like that - the money's just way too good to not put up a charade and keep milking the cow.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#780

Post by taolegal »

wwweston wrote: post #179
taolegal wrote: post #41

Earlier quoted context omitted.

So should the platform be guided by advertisers? Especially one that’s apparently the de facto public square?

If that's it's business model then sure, that's how it works. A "de facto public square" would be public in conception, construction, and support from the start, which is one of the ways we know that Twitter is no such thing. Though it would likely also have some rules for how speech is/isn't conducted. And all things considered, advertiser-friendliness is a sort of low-resolution but approximate passable democratic…

> advertiser-friendliness is a sort of low-resolution but approximate passable democratic mechanism for marking boundaries of civilized discourse

This reminds me that progressives have historically always supported corporations as complex hierarchies, scientific enterprises, run (ideally) by “experts.”

Post reply on HN