Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

951–960 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#951

Post by [deleted] »

WalterBright wrote: post #165
ulkesh wrote: post #27

All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…

America is such a great country that a random person can just fecklessly blunder into creating a revolutionary electric car company and cluelessly blunder into creating a rocket company that is the envy of the world.

[deleted]

Re: Why Twitter didn’t go down: From a real Twitter SRE

#952

Post by bumby »

WalterBright wrote: post #936
trollerator23 wrote: post #867

Earlier quoted context omitted.

I _think_ you are joking. No?

I'm ridiculing the popular notion that Musk is the poster boy for "you didn't build that".

98% of SpaceX contracts are govt. That business has no viability without taxpayer money.

Tesla, to this day but especially early, had the govt subsidize their products to help make them more competitive.

I don’t think those are even a bad thing, but it isn’t a supportive argument that he’s a great free market capitalist.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#953

Post by dpedu »

hairofadog wrote: post #105

Regardless of what’s up with the tech stack, I find it difficult to want to be a part of whatever this is: https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-buc... ) If this is the new town square, you can forward my mail to a cabin in the woods.

What's objectionable about this?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#954

Post by dpedu »

jeffbee wrote: post #81
robswc wrote: post #43

It was if I was in two realities the other day. I did experience a few bugs but twitter never went "down" for me, despite everyone claiming that 1) it would and 2) it already did. All 10 of the "top trends" for me were about how twitter was dead/dying and armchair experts on reddit said it was only a matter of hours at this point. Maybe it was close (in reference to external factors not implied in your very insightfu…

A few parts clearly did go down. 2FA login was just serving error codes all day a few days ago. On Saturday people were posting entire feature films, 2 minutes per tweet, because apparently their copyright content matching wasn’t working.

The extent of the 2FA issue I saw - which, by the way, was w.r.t. sms-based 2fa login, not other 2fa methods - was a single screenshot making this claim.

Was there more I missed?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#955

Post by moduspol »

microtherion wrote: post #927
moduspol wrote: post #838

Earlier quoted context omitted.

And it's almost inevitable. That's why the overwhelming "Elon is so dumb" narrative seems so odd to me. "Major harm," to me, would be either bankruptcy or a competitor overtaking significant chunks of Twitter's users. Even if Elon literally has to re-hire half the roles he fired for, or Twitter is down for a few days or a week, or they struggle to get advertisers for a little while, that's nothing for the long-term.…

> "Major harm," to me, would be either bankruptcy or a competitor overtaking significant chunks of Twitter's users. With the takeover, Twitter has been saddled with massive new debt and would have to become significantly more profitable to service it. Instead, they are losing ad revenue, and any savings from reduced head count will not manifest four at least 4 months because of the severance payments for the laid off…

No post body was provided.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#956

Post by bumby »

donkeyd wrote: post #372
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

Your post made me think of this discussion about how mature organizations naturally have to pivot towards maintenance:

https://freakonomics.com/podcast/in-praise-of-maintenance/

Re: Why Twitter didn’t go down: From a real Twitter SRE

#957

Post by sundarurfriend »

no_op wrote: post #648
thefz wrote: post #515

Earlier quoted context omitted.

> "Anyone who actually writes software, please report to the 10th floor at 2 pm today. Before doing so, please email a bullet point summary of what your code commands have achieved in the past ~6 months, along with up to 10 screenshots of the most salient lines of code" Actual quote. Anyone using the term "code commands" comes out a little detached from programming reality, let alone the rest of this request, it is o…

"Code commands" is very plausibly an autocomplete flub of what was supposed to be "code commits." When I type "code comm" my iPhone offers up "commands" as the completion. I've seen a lot of mockery of this request, but I suspect people aren't considering the wide variance in employee quality that can exist within a mismanaged organization. What Musk was asking for here wouldn't be a good way to evaluate skilled, con…

> What Musk was asking for here wouldn't be a good way to evaluate skilled, conscientious developers, but it would be a pretty effective way to rapidly identify people who are basically incompetent or just aren't really doing anything.

So it's basically a FizzBuzz test, but for existing employees?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#958

Post by drdrey »

polio wrote: post #250
drdrey wrote: post #162

Earlier quoted context omitted.

Another thing we noticed at Netflix was that after services didn’t get pushed for a while (weeks), performance started degrading because of things like undiscovered memory leaks, threads leaks, disks filling up. You wouldn’t notice during normal operations because of regular autoscaling and code pushes, but code freezes tended to reveal these issues.

If resource leaks became a serious issue I imagine they could buy time by restarting. I'm curious what the causes were for code freezes. At Meta they would freeze around Thanksgiving and NYE because of unusually high traffic.

Same, code freezes were typically around holidays (when you know traffic will be elevated, engineers will be less available and you want increased stability)

Re: Why Twitter didn’t go down: From a real Twitter SRE

#959

Post by drdrey »

r3trohack3r wrote: post #890
drdrey wrote: post #162

Earlier quoted context omitted.

Another thing we noticed at Netflix was that after services didn’t get pushed for a while (weeks), performance started degrading because of things like undiscovered memory leaks, threads leaks, disks filling up. You wouldn’t notice during normal operations because of regular autoscaling and code pushes, but code freezes tended to reveal these issues.

We joked about adding this to the NodeQuark platform: // Fix Slow Memory Leaks setTimeout(() => process.exit(1), 1000 * 60 * 60 * 24)

might want to add some random jitter in there :) Imagine your entire NodeQuark cluster decided to restart at the same time

Re: Why Twitter didn’t go down: From a real Twitter SRE

#960

Post by greycol »

polynomial wrote: post #874
donkeyd wrote: post #830

Earlier quoted context omitted.

Good point. I know this as the Mack Truck Theory. For the project I'm working on right now, there's a couple of incredibly valuable people that would cause a pretty significant issue if they disappeared.

Sensitive workplaces (which seems like most these days) have taken to calling this the Lottery Factor (as in team members quit bc they hit the lottery) to spare more delicate types the pain of imagining their peers run over in traffic accidents.

There's advantages to both terms. Bus factor maintains the (possible) fiction that people care about their coworkers/bosses enough that they'd transition out if they had the choice. Bus factor highlights that there's no way to get that information even if the person is nice enough to help for effectively free.
Post reply on HN