Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

341–350 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#341

Post by disillusioned »

randmeerkat wrote: post #336
ezoe wrote: post #246

Earlier quoted context omitted.

Redundancy. Now the systems are stable but human workers either be sick, leave, or die eventually. Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them. The article wrote about achieving stability by the distributed system so an unexpected death of one rack doesn't affect the service availability. The same can be done for…

> Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them. People would absolutely be more engaged and more excited about their work if they were paid more. The only reason people work is literally for money…

>The only reason people work is literally for money…

Well, yes, but also that's super reductive.

The primary reason people work is literally for money. The specific _place_ people work is for things that interest them, engage them, challenge them, provide a path for continued growth in their career, or contribute to a common good they believe in.

And I would argue that it's not necessarily a given that people will be "more engaged and more excited about their work if they were paid more," but instead, may be willing to stick around in an unengaging, unexciting environment _longer_ when they're paid more.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#343

Post by kaba0 »

user_ wrote: post #79
lolsoftware wrote: post #11

I'm curious how/if the ostensible decline in active users is affecting things, and to what degree.

> Twitter added 1.6M daily active users this past week, another all-time high https://twitter.com/elonmusk/status/1594865247323660290

Catastrophe tourism. Also, contrary to Elon’s shitty graph with cut off y-axis (can we stop doing that?), that’s like a 1% change around a major American event.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#344

Post by lelanthran »

ulkesh wrote: post #27

All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…

You know, I think Musk is an ass, and would never work for him, but don't you think that someone who has managed to launch and then run many successful and complex technology projects might actually know a thing or two about launching and running simpler technology projects?

And if you're going to claim that his successes have been due to the people surrounding him who actually know what they are doing, then all that tells me is that you are acknowledging that he knows how to surround himself with people who know what they are doing.

We're not fans (I'm certainly not), but it takes a special kind of mind to look at Musk's track record of successes and conclude that his latest project is doomed.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#346

Post by dymk »

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

Any sort of cached object expiring might bring the servers down. Who knows when the Death TTL will come?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#347

Post by npn »

AtlasBarfed wrote: post #164

Twitter is a lot of drama, but ultimately this is a common private equity play: 1) take private 2) fire/layoff until things break, patch up / rehire with cheaper labor. Repeat as necessary 3) spit shine/repackage what's left with a theoretically more appealing balance sheet. 4) resell to another sucker...uh, buyer, or take public again. Fundamentally Musk bought 44 billion for 5 billion of annual revenue, and presuma…

Chinese developers are famous for making apps serving 1.4 billion people, with at most 1/3 salary of US, and the willingness to do 996 a whole year around.

I would be surprised that no company in the west take advanced of that.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#348

Post by summerlight »

swellguy wrote: post #34

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

Twitter used to experience significant downtime compared to all other major platforms and one of the reason was its lack of redundancies across everything. Headcount is one such thing and it takes manpower to automate infrastructures as discussed in the post.

Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usually have higher expectations for established companies. As always, the last 2% is a hard problem and business doesn't really want to deal with a such unreliable platform. You wanna onboard big advertisers which potentially spend $100M ARR? Then you need to assign a dedicated account manager to handle all customer escalations. PMs then triage and plan their feature requests and later engineers implement it. Which all adds up.

And they also uses your competitor's product, like Google, FB, TikTok etc etc... Twitter is a severely underdog here, so you need to support at least a minimal, essential subset of features in those products to convince them to spend their money on Twitter. That alone takes hundreds of engineers, data scientists and PM thanks to modern ad serving stacks with massive complexity.

Yeah, it ultimately boils down into a simple fact that it's really hard to take other folk's money. You need to first earn trust from them. They want to see if your product is capable of following a modern standard of digital ad serving for now and foreseeable futures. Twitter has spent lots of time for earning trusts and the original post is one evidence of such efforts. And this usually needs more man power. You might be able to do that in a more efficient manner, but I don't think that's as simple as firing 75% of your entire headcount.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#349

Post by thomasahle »

JamesBarney wrote: post #254
whatshisface wrote: post #100

Earlier quoted context omitted.

If the 3x headcount increase really did add no value, there are still about 1/3rd profitable employees there now. In fact giant layoffs tend to cut the best people first because they are the ones who feel comfortable walking. The people that are the last to go are the ones who are very entrenched in the organization and who don't estimate their chances outside of it highly, and that's the exact description of who Elo…

I've found the opposite. I almost never see low performing employees fired outside of a mass layoff. In every layoffs I've seen 10x as many people were fired as quit. So you lose a bunch of low performers involuntarily, and a few top performers both voluntarily and involuntarily and that leads to the average quality improve.

You're assuming they didn't just fire 50% of people at random. It's not like they had a lot of time to think about whom to let go.

It also seems like a lot of the layoffs were achieved by offering everyone the severence of they left.

Assuming only people on visad stayed, I wonder if that skews more or less competent.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#350

Post by Mikeb85 »

MuffinFlavored wrote: post #131
swellguy wrote: post #34

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

> Twitter grew 3x on the headcount front There were multiple executives making $10m/yr+ There were board members There were shareholders Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"? These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc. How can us commenters on HackerNews sit from our arm…

> These qualified people thought at the time it was a good idea to get up to 7.5k people. How were they all wrong?

Come on, just look at the tech industry. When rates were low and stock prices kept going up, "headcount" was used as an indicator of future growth. Grow headcount, investors are happy. After all, the promise of tech stocks was "growth". Usually you're not looking to cut costs until you think growth is over. Of course, Twitter was a dog and did nothing useful for years, no innovation, no new products, nothing. But tech investors definitely saw rising headcount as a good thing...

Post reply on HN