Why Twitter didn’t go down: From a real Twitter SRE • Hacker News Board

Re: Why Twitter didn’t go down: From a real Twitter SRE

#241

Post by midoridensha » Tue, Nov 22, 2022, 5:26 AM UTC

acdha wrote: post #16

yieldcrv wrote: post #6

I don't think it's that crazy. Private Equity and Wall Street doesn't have pressure for resilience, it has pressure for churn. As long as revenue and spending... exists... things are fine. Ideally they are numbers that grow. This doesn't lend itself to clearing out a development backlog, or engineers doing the most important things, it lends itself to rapid iteration and justifications for the iterations. So now that…

The problem is that it’s run by someone who overpaid badly and needs to come up with significantly more money to pay for the debt he saddled the company with, and then his actions seriously disrupted ad revenue. That puts him more in the PE playbook of cutting costs as deeply as possible even at the extent of long-term growth. Unlike his other companies there isn’t strong government support to drive business for Twit…

No post body was provided.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#242

Post by s1artibartfast » Tue, Nov 22, 2022, 5:27 AM UTC

HyperSane wrote: post #192

throwawaylinux wrote: post #109

Earlier quoted context omitted.

I think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.

It would be funny for Musk to share privately with friends. It is a deeply strange and inappropriate thing for the CEO of a billion dollar company to post publicly. Something is deeply wrong with Musk's decision making process.

Is it strange and inappropriate if most people think it is a funny joke and laugh it off? Who goes out of their way to look at Elon posts just to be offended?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#243

Post by hairofadog » Tue, Nov 22, 2022, 5:28 AM UTC

throwawaylinux wrote: post #188

hairofadog wrote: post #149

Earlier quoted context omitted.

It doesn’t bother me because it’s lascivious, it bothers me because it’s the comedy equivalent of fart noises. It was one thing when he was just a user of Twitter, but now he’s in charge, and it seems like a tone is being set that’s not really the vibe I’m into. Whatever floats your boat, I guess.

I would say fart noises are the comedy equivalent of fart noises, but in any case, I sure do love people getting upset by the comedy equivalent of fart noises. Speaking of which, I would actually pay Musk $8 if it meant he would post a video of himself doing the old underarm fart noise thing and I could see all the fantastic "well I never" reactions from everybody who got upset by it.

It takes all kinds!

Re: Why Twitter didn’t go down: From a real Twitter SRE

#244

Post by emodendroket » Tue, Nov 22, 2022, 5:29 AM UTC

kweingar wrote: post #211

emodendroket wrote: post #66

Earlier quoted context omitted.

I think this is kind of baked in though. Part of the thought process seems to be, at least for non-paying customers, it's not actually necessary to have five nines for Twitter, because people will just put up with it if it's less reliable.

I don’t have personal experience in this, so obviously I can’t speak with any authority. But I have heard from colleagues that tons of little factors can dramatically affect user engagement. For example, even a couple dozen milliseconds of longer load times can push a noticeable number of users away from your app.

Undoubtedly some people will be put off, but think of how often Reddit used to go down -- still got pretty big. And Twitter already has all the newsmakers people want to see. If your goal isn't necessarily user growth it makes sense

Re: Why Twitter didn’t go down: From a real Twitter SRE

#245

Post by kweingar » Tue, Nov 22, 2022, 5:29 AM UTC

hairofadog wrote: post #105

Regardless of what’s up with the tech stack, I find it difficult to want to be a part of whatever this is: https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-buc... ) If this is the new town square, you can forward my mail to a cabin in the woods.

No post body was provided.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#246

Post by ezoe » Tue, Nov 22, 2022, 5:30 AM UTC

swellguy wrote: post #34

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

Redundancy.

Now the systems are stable but human workers either be sick, leave, or die eventually.

Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them.

The article wrote about achieving stability by the distributed system so an unexpected death of one rack doesn't affect the service availability. The same can be done for the human workers unexpectedly not working anymore. Have a multiple workers doing the same things improve stability.

Sure, it's inefficient in terms of money. But alternative is one sick important employee catch a COVID-19 and die lost the knowledge of the system. Documents doesn't solve it because you want the manual operation available right now rather than a few months later when replaced workers learned from the documents.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#247

Post by mtejo » Tue, Nov 22, 2022, 5:30 AM UTC

zekrioca wrote: post #197

mtejo wrote: post #2

Hey, sorry for the new account, i just like to try my best to keep my online identity separate. this for better or for worse has my real name on it. Hope this is interesting!

Weren’t Twitter moving out of Mesos to K8s?

Yeah there was a few teams trying to make it happen

Re: Why Twitter didn’t go down: From a real Twitter SRE

#248

Post by wyclif » Tue, Nov 22, 2022, 5:31 AM UTC

A4ET8a8uTh0 wrote: post #132

emodendroket wrote: post #60

Earlier quoted context omitted.

I'll put my cards on the table and say I would have been just as happy to see him fail as anyone else, but it did begin to take on the tenor of the constant, never-came-true stories about how any minute now they were going to spring the trap on Donald Trump and he'd get thrown out of office. e: I did not mean for this to be an invitation for everyone to argue about the merits or demerits of Donald Trump.

I personally have somewhat mixed feelings. I do enjoy a good dumpster fire like any good internet denizen ( and I won't deny that there is an odd level of.. excitement maybe.. that one major social platform may go down ). I have no love for social media as some of my posts may indicate and for a variety of reasons including the amount of oxygen this creation takes and the amount of confusion it causes in general popu…

It's because even though Elon Musk is no right-winger, he voted straight Democrat his entire life until very recently but then he voted for a Republican. The haters will brook no deviation from their accepted political narrative, and when Musk did it the hordes of people with pronouns in their bios turned on him.

For those people it has nothing to do with technology, which is why their takes were so hysterical and wrong. It's 100% about politics every day, 24/7.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#249

Post by mtejo » Tue, Nov 22, 2022, 5:31 AM UTC

ilaksh wrote: post #121

mtejo wrote: post #2

Hey, sorry for the new account, i just like to try my best to keep my online identity separate. this for better or for worse has my real name on it. Hope this is interesting!

So you still work there? Or did you quit recently?

I left over the summer

Re: Why Twitter didn’t go down: From a real Twitter SRE

#250

Post by polio » Tue, Nov 22, 2022, 5:31 AM UTC

drdrey wrote: post #162

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

Another thing we noticed at Netflix was that after services didn’t get pushed for a while (weeks), performance started degrading because of things like undiscovered memory leaks, threads leaks, disks filling up. You wouldn’t notice during normal operations because of regular autoscaling and code pushes, but code freezes tended to reveal these issues.

If resource leaks became a serious issue I imagine they could buy time by restarting. I'm curious what the causes were for code freezes. At Meta they would freeze around Thanksgiving and NYE because of unusually high traffic.