Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

351–360 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#351

Post by namdnay »

jdminhbg wrote: post #152
MuffinFlavored wrote: post #131

Earlier quoted context omitted.

> Twitter grew 3x on the headcount front There were multiple executives making $10m/yr+ There were board members There were shareholders Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"? These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc. How can us commenters on HackerNews sit from our arm…

> How can us commenters on HackerNews sit from our armchair and say "ah, goofballs should've just not let headcount get so high!" The cliche HN comment on sites like Twitter (and many, many others, any time headcount comes up) has always been "why do they need so many people?" I've mostly dismissed it the same way I dismiss "I could build Uber in a weekend," but with every other tech giant laying people off, maybe I…

The problem with having a lot of money sloshing around is that you can no longer say “no”. You estimate a new feature will take your team a year. But someone wants it in 6 months. “Why aren’t you hiring to meet the deadline?”

Re: Why Twitter didn’t go down: From a real Twitter SRE

#352

Post by robswc »

jakevoytko wrote: post #141
jeffbee wrote: post #81

Earlier quoted context omitted.

A few parts clearly did go down. 2FA login was just serving error codes all day a few days ago. On Saturday people were posting entire feature films, 2 minutes per tweet, because apparently their copyright content matching wasn’t working.

At least one high-follower account on my timeline posted that their newest notifications were from October

I wouldn’t be surprised. There were definitely a few bugs. I think ppl unhappy with Elon overplayed things though and now it will look silly.

Our expectations were anchored by the idea that Twitter would cease to exist. So very real problems like 2FA going down look trivial compared to the former.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#353

Post by NineStarPoint »

taolegal wrote: post #41
philistine wrote: post #33

Earlier quoted context omitted.

And Twitter's problem are nowhere near technological. The site needed to make more money, not reengineer the whole thing while advertisers are fleeing because Trump is back on on a whim!

So should the platform be guided by advertisers? Especially one that’s apparently the de facto public square?

That depends on Musk’s goals. If his goal is to make money to pay off the massive amount of debt he gained as part of buying twitter, (as the attempted quick roll out of twitter blue would indicate), then yes he probably needs to care about what advertisers think. If he just wanted to leverage his position as one of the richest men in the world to ensure that twitter was a haven for free speech and screw the profitability, then no he wouldn’t need to be guided by advertisers. He probably can’t both want immediate returns on his investment and to quickly rock the boat though.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#354

Post by randyrand »

kaba0 wrote: post #339
randyrand wrote: post #176

Earlier quoted context omitted.

servers don’t really rot if they don’t rely on 3rd party cloud software. Linux doesn’t upgrade itself. the web is very standards based. Hacker news has pretty much been running for decades on the same software hardware stack

I’m fairly sure HN had outages as well.

outages are not usually rot.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#356

Post by lcuff »

randmeerkat wrote: post #336
ezoe wrote: post #246

Earlier quoted context omitted.

Redundancy. Now the systems are stable but human workers either be sick, leave, or die eventually. Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them. The article wrote about achieving stability by the distributed system so an unexpected death of one rack doesn't affect the service availability. The same can be done for…

> Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them. People would absolutely be more engaged and more excited about their work if they were paid more. The only reason people work is literally for money…

That might be true for you, but way back when, I took a course in programming and thought it was the most fun I'd ever had. Couldn't believe that people wanted to pay me to do it. Spent the next 40 years making a great living. The only jobs I disliked was the ones where there were "glass bowl" bosses and co-workers. Problem solving was always fun. Not saying I didn't enjoy being paid well, but it wasn't the major consideration in whether I wanted to stay in any particular work environment. Here we're talking about the potential for huge schedule and technical pressure and low morale. A whole lot more money wouldn't keep me in those circumstances.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#357

Post by kaba0 »

AtlasBarfed wrote: post #164

Twitter is a lot of drama, but ultimately this is a common private equity play: 1) take private 2) fire/layoff until things break, patch up / rehire with cheaper labor. Repeat as necessary 3) spit shine/repackage what's left with a theoretically more appealing balance sheet. 4) resell to another sucker...uh, buyer, or take public again. Fundamentally Musk bought 44 billion for 5 billion of annual revenue, and presuma…

I mean, they got an offer for roughly a 10x deal. At that point it was in the best interest of shareholders to sell. Especially that it was yet to turn a profit.

Like, I like my car, but would probably sell it for an order of magnitude its worth.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#358

Post by kaba0 »

npn wrote: post #347
AtlasBarfed wrote: post #164

Twitter is a lot of drama, but ultimately this is a common private equity play: 1) take private 2) fire/layoff until things break, patch up / rehire with cheaper labor. Repeat as necessary 3) spit shine/repackage what's left with a theoretically more appealing balance sheet. 4) resell to another sucker...uh, buyer, or take public again. Fundamentally Musk bought 44 billion for 5 billion of annual revenue, and presuma…

Chinese developers are famous for making apps serving 1.4 billion people, with at most 1/3 salary of US, and the willingness to do 996 a whole year around. I would be surprised that no company in the west take advanced of that.

1/3 the salary may well give you a better life quality in China, though.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#359

Post by astrange »

jeffbee wrote: post #81
robswc wrote: post #43

It was if I was in two realities the other day. I did experience a few bugs but twitter never went "down" for me, despite everyone claiming that 1) it would and 2) it already did. All 10 of the "top trends" for me were about how twitter was dead/dying and armchair experts on reddit said it was only a matter of hours at this point. Maybe it was close (in reference to external factors not implied in your very insightfu…

A few parts clearly did go down. 2FA login was just serving error codes all day a few days ago. On Saturday people were posting entire feature films, 2 minutes per tweet, because apparently their copyright content matching wasn’t working.

The QRT system doesn't work (if a post says it has 200 quote tweets, you can always open it and never see any), but then again, it never did seem to work.

It's still entirely believable the site will go down and remain down for a week or two at any time.

Post reply on HN