Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

161–170 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#161

Post by frognumber »

xyzzyz wrote: post #119
whatshisface wrote: post #100

Earlier quoted context omitted.

If the 3x headcount increase really did add no value, there are still about 1/3rd profitable employees there now. In fact giant layoffs tend to cut the best people first because they are the ones who feel comfortable walking. The people that are the last to go are the ones who are very entrenched in the organization and who don't estimate their chances outside of it highly, and that's the exact description of who Elo…

> In fact giant layoffs tend to cut the best people first because they are the ones who feel comfortable walking. This is more true when the layoffs happen because the company’s situation deteriorates. If the company cuts jobs because revenues fall and products fail, better employees are indeed more likely to move to greener pastures before mediocre ones do. If, however, the company prospects improve, rather than wor…

I'm not sure. I think I'm a good employee, but I like job stability. If I'm working somewhere with layoffs, I'll likely go somewhere without.

There also isn't a measure of employee quality, and layoffs are often not about the individual (e.g. Amazon cutting robotics). I've had no clue how likely I was to be laid off during most points in my career.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#162

Post by drdrey »

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

Another thing we noticed at Netflix was that after services didn’t get pushed for a while (weeks), performance started degrading because of things like undiscovered memory leaks, threads leaks, disks filling up. You wouldn’t notice during normal operations because of regular autoscaling and code pushes, but code freezes tended to reveal these issues.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#163

Post by aegis4244 »

emodendroket wrote: post #60
throwawaylinux wrote: post #58

Earlier quoted context omitted.

It wasn't only news outlets. A majority of my... politically-noisy tech friends on Facebook went through a recent phase of intensely posting about Twitter on fire and collapsing. Now just because they're "in tech" doesn't mean they have any idea about Twitter, but they should at least know enough to know they don't know what's going to happen, but obviously they're not actually using their brains when posting comment…

I'll put my cards on the table and say I would have been just as happy to see him fail as anyone else, but it did begin to take on the tenor of the constant, never-came-true stories about how any minute now they were going to spring the trap on Donald Trump and he'd get thrown out of office. e: I did not mean for this to be an invitation for everyone to argue about the merits or demerits of Donald Trump.

Dems didn't want to throw him out. Pence was a competent politician. Trump wasn't. They wanted him embarrassed, not out. Much like the Brits had a plan to assassinate Hitler before D day. Choose not to. They were afraid he would be replaced by someone competent.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#164

Post by AtlasBarfed »

Twitter is a lot of drama, but ultimately this is a common private equity play:

1) take private

2) fire/layoff until things break, patch up / rehire with cheaper labor. Repeat as necessary

3) spit shine/repackage what's left with a theoretically more appealing balance sheet.

4) resell to another sucker...uh, buyer, or take public again.

Fundamentally Musk bought 44 billion for 5 billion of annual revenue, and presumably 5 billion in costs. Unlikely to add revenue in twitter's model, he's cutting costs.

Honestly, at this point, is there revolutionary technology that is needed to keep the lights on at twitter? Do you need graphQL gods and SREs that would ace any Amazon raising the bar and master of silicon valley interviewing?

Nah. He'll honestly probably outsource a ton of the upkeep.

It's ugly, but the Twitter board and shareholders took their money and ran, and abandoned the product and the workforce. They could have backed out and let Elon Musk off the hook of his dumb contract with them, but they just wanted the money, and sold to someone that just wants to get his money out of it too.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#165

Post by WalterBright »

ulkesh wrote: post #27

All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…

America is such a great country that a random person can just fecklessly blunder into creating a revolutionary electric car company and cluelessly blunder into creating a rocket company that is the envy of the world.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#166

Post by subroutine »

halfmatthalfcat wrote: post #125
subroutine wrote: post #115

Earlier quoted context omitted.

A bunch of people just got axed from Twitter because covid cash dried up? Shit I thought it was because Elon took over and fired anyone who refused to work at the office instead of at home.

The question was why they were hired, not why they were fired...

The unemployment rate in Feb 2020 was 3.5%; by April 2020 it was up to 14%. It has never been lower than 3.5% since then. If there was a hiring glut due to covid it must have been a very industry-specific glut.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#167

Post by xyzwave »

porknubbins wrote: post #69
emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivat…

Reminds me of an Alan J. Perlis quote:

> Is it possible that software is not like anything else, that it is meant to be discarded: that the whole point is to see it as a soap bubble?

http://www.cs.yale.edu/homes/perlis-alan/quotes.html

Re: Why Twitter didn’t go down: From a real Twitter SRE

#168

Post by EagnaIonat »

totorovirus wrote: post #44

If Twitter survives and people shut up their mouths complaining after few months we'll see who was right.

It's already breaking down. HW/SW aside, a lot of the services had teams monitoring them, which don't exist anymore.

Some of the side effects of that:

- Trends are not working correctly.

- Copyright reporting is not working.

- Appealing flagged tweets isn't being responded to.

It's only a matter of time before these get abused with no one to fix them.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#169

Post by quickthrower2 »

fenomas wrote: post #139
emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...

My analogy.

Someone purchased some land for $1. Built a house for say $100. And now spends $100,000 a year making it the perfect place to rent, receiving $100,000 a year in rent.

Someone comes along and borrows $1m to buy that house. They feel ripped off but eventually are force to go ahead with the purchase. As a result they have to pay $100,000 a year in interest. They need this thing to be profitable!

To do this they need to cut back on the $100,000 a year spent. They decide go go in quickly and so email all the services saying "go hard or go home". So the plumbers, tradie, cleaners etc that don't like it leave.

As a side hustle also charge visitors to the house $9 to be allowed to wear their bowtie they used to wear for free.

Some of the people do maintenance jobs and improvements. They keep the termites out, fix subsidence issues, and so on.

And the house didn't fall down within 3 weeks of it being purchased.

Post reply on HN