Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

191–200 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#191

Post by s1artibartfast »

MuffinFlavored wrote: post #131
swellguy wrote: post #34

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

> Twitter grew 3x on the headcount front There were multiple executives making $10m/yr+ There were board members There were shareholders Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"? These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc. How can us commenters on HackerNews sit from our arm…

Has no company ever been mismanaged? Have they they ever grossly misallocated funds? The answer is of course yes, that happens all the time. Corporate leaderships are not not infallible.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#192

Post by HyperSane »

throwawaylinux wrote: post #109
hairofadog wrote: post #105

Regardless of what’s up with the tech stack, I find it difficult to want to be a part of whatever this is: https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-buc... ) If this is the new town square, you can forward my mail to a cabin in the woods.

I think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.

It would be funny for Musk to share privately with friends. It is a deeply strange and inappropriate thing for the CEO of a billion dollar company to post publicly. Something is deeply wrong with Musk's decision making process.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#193

Post by Andrew_nenakhov »

inferiorhuman wrote: post #185
fenomas wrote: post #139

Earlier quoted context omitted.

To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...

He rolled in, sold most of the fire extinguishers, and made a big show of trying to make cherries jubilee while shoving his dick in one of the few remaining fire extinguishers. Let's be clear, it isn't just the erratic layoffs it's Musk's incessant meddling that's going to be Twitter's downfall. He literally took down SMS based 2FA because "microservices bad". He fired the payroll and tax departments (HR too?). He's…

> However failure is pretty much inevitable at this point

I'm pretty sure there will be no failure at all, and Twitter will work just fine.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#194

Post by WatchDog »

areoform wrote: post #129

There's a pattern of diverging expectations here, one is the non-technical/naïve one, - Twitter is going to go down tomorrow and it's all over. RIP. The second is, - Twitter is going to experience a failure cascade over time. The third is, - It's all going to be fine. I suspect that the real question is, how many individual wires can break before the cable holding the suspended platform snaps? I am not that good of a…

Also anecdote, but twitter has historically been pretty unreliable for me.

Without a reliable twitter systems status history pre-acquisition, the reports of failures, like the issues with the 2FA system, don't mean a whole lot.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#195

Post by lmm »

MuffinFlavored wrote: post #173
frognumber wrote: post #155

Earlier quoted context omitted.

It's not goofballs. It's generally misaligned incentives. Managing a 10,000 org leads to better job prospects than a 1000 person org, than a 100 person org, than a pizza box team. Organizations tend to bloat. Random, rapid cuts might not be the fix here, but headcount was too high.

> Organizations tend to bloat. Say I'm an engineer. And I write code all day. And I get paid by a company to do it. And it blows up in production. And I tell my boss "code tends to blow up". Or I tell my boss "code tends to take longer than estimated to deliver". I wouldn't be given lots of promotions/bonuses with that outlook. We're talking about a Twitter CEO with $30m/yr+ in total compensation. There's no way his/…

> There's no way his/her view was "shrug, organizations tend to bloat" while racking in extremely competitive + good pay as somebody whose main job is to drive a company towards maximum growth/profitability.

> There's no way the CEO was able to convince a majority of people responsible for paying him (the board), to pay award him $30m/yr in stock options, while he was also "bad at his job enough" to let the organization bloat without as much as an afterthought to it.

Why not? Why would those board members have ever called him out? The more they're paying him, the more they can pay themselves too.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#196

Post by dehrmann »

AtlasBarfed wrote: post #164

Twitter is a lot of drama, but ultimately this is a common private equity play: 1) take private 2) fire/layoff until things break, patch up / rehire with cheaper labor. Repeat as necessary 3) spit shine/repackage what's left with a theoretically more appealing balance sheet. 4) resell to another sucker...uh, buyer, or take public again. Fundamentally Musk bought 44 billion for 5 billion of annual revenue, and presuma…

There are a few key differences. PE tries not to massively overpay, it'll put more effort into targeted layoffs, and it'll maintain enough of a "business as usual" posture to not scare away advertisers. Edit: they also put in a CEO whose full-time job is being CEO of that company.

But yes, this is very bad execution of the PE playbook.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#198

Post by rickette »

wyclif wrote: post #114
didip wrote: post #93

I am mostly curious about the Mesos layer itself. Mesos is dead. So you need in-house expertise to patch it without being able to leverage community knowledge. Does Twitter retain enough people to manage Mesos?

I was surprised to hear that a cutting edge social media app uses Mesos. Why did they choose that over other options?

Twitter didn't start yesterday. There was a time when Mesos was all the hotness and k8s was this new thing that looked promising but wasn't nearly production ready.

Apple is also a big Mesos user, but also moving to k8s.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#199

Post by abakker »

jdminhbg wrote: post #152
MuffinFlavored wrote: post #131

Earlier quoted context omitted.

> Twitter grew 3x on the headcount front There were multiple executives making $10m/yr+ There were board members There were shareholders Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"? These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc. How can us commenters on HackerNews sit from our arm…

> How can us commenters on HackerNews sit from our armchair and say "ah, goofballs should've just not let headcount get so high!" The cliche HN comment on sites like Twitter (and many, many others, any time headcount comes up) has always been "why do they need so many people?" I've mostly dismissed it the same way I dismiss "I could build Uber in a weekend," but with every other tech giant laying people off, maybe I…

I think this is not too far wrong. I also think in addition that the really competitive job market of the last 2 years and the hiring they were doing inevitably resulted in some highly paid people who were not good fits or good at their jobs. I think that covid wage hikes and job competitiveness is doing a bit of reckoning now as there is finally enough data to evaluate performance relative to output.

OTOH, at non tech companies, all the research interviews and surveys I have done recently continue to show a talent shortage and fears of losing tech staff, so maybe it is just that Tech companies are saturated and have too much capital for their creativity.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#200

Post by lowbloodsugar »

emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

Riiight. But what happens when there's nobody on that oncall rotation? I've worked at FAANGs, and I'm just thinking what would happen if there was a problem with an upstream team and nobody there. Maybe it's ours now. Maybe there's a principal engineer who worked on it back in the day that's still on the payroll? Maybe we're just going to try to bring it back up with no idea what is going on? What if you have a bug that silently loses data?
Post reply on HN