Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

181–190 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#181

Post by kweingar »

nemo44x wrote: post #90
headsoup wrote: post #42

Earlier quoted context omitted.

Because politics and ideologues are so prevalent and everyone's attaching 'the other' (both sides) for the own. And of course people in 'important' roles who've been laid off are going to say the company is doomed, these are the probably the worst source to go off. They're not going to say "oh yeah I didn't do much at all really, just bossed people around and spent my budget every year."

Quoted post unavailable.

I will admit to this. I was so angry at how Musk treated his employees, and I took them at face value when they predicted outages. I still think that we will see outages in the coming weeks and months, but probably not in spectacular fashion.

A lot of Musk’s fans get emotional about this too. I remember people gloating when they rolled out the new Twitter Blue, saying that Elon got more done in a couple weeks than Twitter had gotten done in years beforehand. And then they shut it off, and now they’re delaying the relaunch. People were praising Musk’s “moderation council” concept, and then he promptly abandoned it to make these decisions by fiat (and in one case, a poll).

Re: Why Twitter didn’t go down: From a real Twitter SRE

#182

Post by subroutine »

inferiorhuman wrote: post #174
subroutine wrote: post #115

Earlier quoted context omitted.

A bunch of people just got axed from Twitter because covid cash dried up? Shit I thought it was because Elon took over and fired anyone who refused to work at the office instead of at home.

Now to be fair, as of last week Musk-o backed off the mandated return-to-office. Turns out that didn't quite go as planned.

What a wild card. Has Musk said why he purchased twitter in the first place? IRL he seems so introverted. It's just odd to me he'd have any interest in a social media company.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#183

Post by polio »

dcow wrote: post #110
porknubbins wrote: post #69

Earlier quoted context omitted.

The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivat…

I have a piece of Rust software that has not gone down in its entire lifetime.

By definition, no software goes down until its lifetime ends.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#184

Post by bagels »

brookst wrote: post #142
porknubbins wrote: post #69

Earlier quoted context omitted.

The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivat…

> all software goes down and requires specialist intervention eventually Well, that’s it, isn’t it? How many software systems need to keep running for Twitter to remain more or less functional? If there are 10 critical systems that are running at four 9’s, you’d expect 3.6 hours of downtime a year, or about 90 days of uptime at a stretch if I have my math right. If there are 100 critical systems running at 3 9’s, you…

Some of those 9s come with an assumption that someone is around who knows what buttons to push when it really goes awry.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#185

Post by inferiorhuman »

fenomas wrote: post #139
emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...

He rolled in, sold most of the fire extinguishers, and made a big show of trying to make cherries jubilee while shoving his dick in one of the few remaining fire extinguishers. Let's be clear, it isn't just the erratic layoffs it's Musk's incessant meddling that's going to be Twitter's downfall. He literally took down SMS based 2FA because "microservices bad". He fired the payroll and tax departments (HR too?). He's scared off Twitter's main source of income while saddling it with significant debt.

As an SRE I would have been shocked if Twitter failed catastrophically (well moreso than broadly disabling authentication) in short order. However failure is pretty much inevitable at this point given the damage that E-Lon is actively doing.

Whatever. Twitter and Musk deserve each other.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#186

Post by emodendroket »

viraptor wrote: post #106
emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

> as though tons of tech sites (including big names like Amazon) don't just devolve this work to their on-call rotations Not everyone has the same knowledge and skills. Not everything is documented. Not everything in the documentation is current and correct. (especially after recent changes) It's not even that they won't stay up, but depending on who left the company, the oncall response may look radically different…

Yeah, but I think a judgment is being made that having the service be less reliable is tolerable.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#187

Post by 8n4vidtmkvmk »

jeffbee wrote: post #81
robswc wrote: post #43

It was if I was in two realities the other day. I did experience a few bugs but twitter never went "down" for me, despite everyone claiming that 1) it would and 2) it already did. All 10 of the "top trends" for me were about how twitter was dead/dying and armchair experts on reddit said it was only a matter of hours at this point. Maybe it was close (in reference to external factors not implied in your very insightfu…

A few parts clearly did go down. 2FA login was just serving error codes all day a few days ago. On Saturday people were posting entire feature films, 2 minutes per tweet, because apparently their copyright content matching wasn’t working.

that's surprising actually. I'd expect my sites to run for about a year if i suddenly died. maybe longer. certs will renew themselves, servers will restart themselves. everything is on autopay. the thing to kill it would probably be api breaks by 3rd parties. maybe disk space would kill me eventually too.. i don't have auto resizing for my sql database but its got a long runway yet. or god forbid my site grows on its own and the qps kills it.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#188

Post by throwawaylinux »

hairofadog wrote: post #149
throwawaylinux wrote: post #109

Earlier quoted context omitted.

I think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.

It doesn’t bother me because it’s lascivious, it bothers me because it’s the comedy equivalent of fart noises. It was one thing when he was just a user of Twitter, but now he’s in charge, and it seems like a tone is being set that’s not really the vibe I’m into. Whatever floats your boat, I guess.

I would say fart noises are the comedy equivalent of fart noises, but in any case, I sure do love people getting upset by the comedy equivalent of fart noises. Speaking of which, I would actually pay Musk $8 if it meant he would post a video of himself doing the old underarm fart noise thing and I could see all the fantastic "well I never" reactions from everybody who got upset by it.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#189

Post by dehrmann »

user_ wrote: post #79
lolsoftware wrote: post #11

I'm curious how/if the ostensible decline in active users is affecting things, and to what degree.

> Twitter added 1.6M daily active users this past week, another all-time high https://twitter.com/elonmusk/status/1594865247323660290

Louis Rossmann said something like "there's a difference between people showing up to laugh at you vs. laugh with you."

Re: Why Twitter didn’t go down: From a real Twitter SRE

#190

Post by Robotbeat »

xedrac wrote: post #98
swellguy wrote: post #34

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

I think a likely answer is "moderating". That would explain why Elon was ok letting so many go so quickly.

From what I understand, contractors were used for moderating.
Post reply on HN