Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

281–290 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#281

Post by csomar »

Did the OP confirm Elon theory that most of the stuff is not needed?

> For four of those years I was the sole SRE for the Cache team. There was a few before me, and the whole team I worked with, where a bunch came and went. But for four years I was the one responsible for automation, reliability and operations in the team. I designed and implemented most of the tools that are keeping it running so I think I’m qualified to talk about it. (There might be only one or two other people)

If you only need one person for the caching department (which is, as I understand, is critical as it delivers most of the data); then maybe you need a handful other dozen engineers and there you have a functional Twitter.

That or the OP is full of himself. Kinda like Musk?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#282

Post by bnjms »

AtlasBarfed wrote: post #172
mikeweiss wrote: post #135

Earlier quoted context omitted.

Wait Musk saddled Twitter with debt? Please elaborate...

Yeah, it's private equity 101. I can't believe it happens. A bunch of MBA sharks get loans to buy something with some chop shop plan or other ruthless scheme to carve up or spitshine a company. The loans are then assigned to the company they bought, rather than the core sharks ... uh investors. The "investors" then repackage/selloff/spit shine as necessary to get it resold to some sucker. The investors get THEIR mone…

Why is that possible or legal? It feels like a loan for purchase of a company necessarily should belong to the purchasers of the company and be paid by the sale of the equity they bought or profit. This way just feels… rigged.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#283

Post by npn »

Let don't kid ourselves here, even if Twitter crashes and needs a few weeks or even a few months to recover it doesn't matter. People who leaved will comeback again, after a year everything will be like nothing ever happened.

And even if Twitter lost 50% of it users and revenues, it is still more profitable than have to pay the salary of 7000 people.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#284

Post by memish »

headsoup wrote: post #42
emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

Because politics and ideologues are so prevalent and everyone's attaching 'the other' (both sides) for the own. And of course people in 'important' roles who've been laid off are going to say the company is doomed, these are the probably the worst source to go off. They're not going to say "oh yeah I didn't do much at all really, just bossed people around and spent my budget every year."

It's actually scary how many people, even engineers, put their reputation on the line saying Twitter wouldn't survive the weekend. It wasn't just Twitter employees.

It's like a mass psychosis of some kind. It comes off as a kind of desperation, as though they need Elon to fail.

Why? What's driving that response?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#285

Post by superdug »

JamesBarney wrote: post #254
whatshisface wrote: post #100

Earlier quoted context omitted.

If the 3x headcount increase really did add no value, there are still about 1/3rd profitable employees there now. In fact giant layoffs tend to cut the best people first because they are the ones who feel comfortable walking. The people that are the last to go are the ones who are very entrenched in the organization and who don't estimate their chances outside of it highly, and that's the exact description of who Elo…

I've found the opposite. I almost never see low performing employees fired outside of a mass layoff. In every layoffs I've seen 10x as many people were fired as quit. So you lose a bunch of low performers involuntarily, and a few top performers both voluntarily and involuntarily and that leads to the average quality improve.

My first job out of college had layoffs 8 months in. I was called into a conference room, connected to a speakerphone with the rest of my team, immediately told that we were all laid off and that I had some paperwork to sign in my managers office.

I was given a 33% raise, moved to a salaried position, and got a $5000 cash bonus 10 minutes later.

I had mastered the new system we were implementing and converted and supported the remainder of the customers from the legacy system. Shitty support of the product my team of now one actually drove revenue as the customers would pay higher support prices because of the shitty support I offered.

The service I offered was shitty because my job was to meet minimal contractual obligations for multi-decade contracts to persuade the customers to renegotiate the contracts for a product that hadn't been EOL'd. All the while offering onsite conversion services from the legacy product for most of my day job.

What I'm saying is that company is still alive today, they had a pivot point and literally cut 55% of their employees on a single day.

Google wants to beat facebook still, there is one of a billionity pivot points that twitter could take.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#286

Post by kaba0 »

emodendroket wrote: post #186
viraptor wrote: post #106

Earlier quoted context omitted.

> as though tons of tech sites (including big names like Amazon) don't just devolve this work to their on-call rotations Not everyone has the same knowledge and skills. Not everything is documented. Not everything in the documentation is current and correct. (especially after recent changes) It's not even that they won't stay up, but depending on who left the company, the oncall response may look radically different…

Yeah, but I think a judgment is being made that having the service be less reliable is tolerable.

With that amount of staff change, I’m not sure that service would inevitably come back up at all, in case of a serious enough problem.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#287

Post by toast0 »

AtlasBarfed wrote: post #172
mikeweiss wrote: post #135

Earlier quoted context omitted.

Wait Musk saddled Twitter with debt? Please elaborate...

Yeah, it's private equity 101. I can't believe it happens. A bunch of MBA sharks get loans to buy something with some chop shop plan or other ruthless scheme to carve up or spitshine a company. The loans are then assigned to the company they bought, rather than the core sharks ... uh investors. The "investors" then repackage/selloff/spit shine as necessary to get it resold to some sucker. The investors get THEIR mone…

> Yeah, it's private equity 101. I can't believe it happens.

Why not? It's pretty good for all the stakeholders that have a say.

The old shareholders sold for a nice price.

The new ownership didn't have to pay for the whole thing.

The lender gets to charge a pretty good interest rate because there's a good chance of default. If they're lucky, they get repaid; if not, maybe they made enough in interest to make it worthwhile; maybe when it defaults, they'll be able to make something worthwhile out of the wreckage they got at a nicer price.

Leveraged buy outs aren't great for stakeholders that don't have a say. Employees usually get new terms worse than the old ones; in this case, there's been some severance at least. Customers get a promise of a big bang bankruptcy in the near to medium term, rather than a slow fizzle. Sometimes companies with a large payment they can't make can restructure, and sometimes they shutdown with little notice. As a private company without public accounting reports, there will be a lot of guessing about revenue and debt service.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#288

Post by Izkata »

porknubbins wrote: post #69
emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivat…

> The default, naive assumption should always have been programs keep running indefinitey on their own.

...for people who understand software to some extent. I get the feeling a lot of people see it more like a hamster wheel, where once the developers are gone it immediately starts noticeably slowing down as it stops (and are confused when that doesn't happen).

Re: Why Twitter didn’t go down: From a real Twitter SRE

#289

Post by NaturalPhallacy »

EagnaIonat wrote: post #168
totorovirus wrote: post #44

If Twitter survives and people shut up their mouths complaining after few months we'll see who was right.

It's already breaking down. HW/SW aside, a lot of the services had teams monitoring them, which don't exist anymore. Some of the side effects of that: - Trends are not working correctly. - Copyright reporting is not working. - Appealing flagged tweets isn't being responded to. It's only a matter of time before these get abused with no one to fix them.

>- Trends are not working correctly.

"Trends" were subject to Twitter's "trends blacklist" before; something that leaked a few years ago. Maybe they're working correctly now that they're unencumbered. Can you describe how they're not working now?

>- Appealing flagged tweets isn't being responded to.

Mine was responded to in a handful of hours. Much faster than I was expecting.

>It's only a matter of time before these get abused with no one to fix them.

"the walls are closing in" - https://www.youtube.com/watch?v=rLEchPZm318

Post reply on HN