Why Twitter didn’t go down: From a real Twitter SRE • Hacker News Board

Re: Why Twitter didn’t go down: From a real Twitter SRE

#621

Post by hnbad » Tue, Nov 22, 2022, 9:56 AM UTC

_3u10 wrote: post #601

andreyk wrote: post #577

Earlier quoted context omitted.

Yep, agree. Twitter's revenue was primarily from ads, and now I'd bet their ads revenue has dropped a huge amount. Given how Musk is behaving now (reinstating Trump etc.), and losing many pivotal sales people, and the firing of a ton of people in charge of dealing with hate speech and such, it seems unlikely advertisers will return now.

I don’t think given their ad sales that the sales people were that pivotal.

As I understand it, most of the ad slots were sold at a yearly sales event. This time most advertisers were cautious because the company couldn't respond to questions of how Elon Musk's takeover would affect the company (and specifically the type of content alongside which the ads would appear) as this was happening before he took over but after the sale was practically a done deal. This left them with only a fraction of the ad slots "pre-sold" compared to a normal year.

Additionally it seems a lot of advertisers have cautioned their ad departments to halt spending on Twitter ads for now because they're still unclear about wtf Elon Musk is up to (not to mention the mass impersonation fiasco that allegedly resulted in share prices of several companies being affected negatively).

It's not just sales people, Twitter seems to have gotten rid of most of its external communications people to the point where journalists can't even find anyone to ask for statements other than Elon Musk himself (which may be intentional given that he likes to present himself as the face of his companies).

But if Twitter's ad sales really did rely so much on these sales events, I'd wager that the sales people largely existed to communicate directly with long-standing big-spend advertising partners. It's hard to overstate how much these types of deals hinge on "social glue" rather than cold hard numbers (which can be interpreted/reframed to be interpreted however you want as long as both sides are happy).

Re: Why Twitter didn’t go down: From a real Twitter SRE

#622

Post by yywwbbn » Tue, Nov 22, 2022, 9:57 AM UTC

drcross wrote: post #602

FranzFerdiNaN wrote: post #480

Earlier quoted context omitted.

Everything that is happening with Twitter proves Musk is yet another wealthy idiot who doenst know shit about shit, except how to blow his own horn. Musk is simlpy lucky that Twitter used to have such excellent engineers like this SRE so that the site isnt yet on fire.

Musk lead teams at the only company who can land commercial rockets and the first company to profit from making electric cars. To say he is a wealthy idiot says more about you than it does about Musk.

So Musk hired the right people. I mean he made some brilliant decisions over the years. Does that mean that everything he does is a guaranteed success? There are plenty examples of ultra successful people in history who were blinded by their own success and became a little bit unhinged over time..

At this point Musk is behaving just like a twelve year old’s idea of a mega billionaire. Hyperloop, flamethrowers, self driving cars now Twitter... It’s great that he’s having fun I don’t think there is much to it than that.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#623

Post by inawarminister » Tue, Nov 22, 2022, 9:57 AM UTC

zxcvbn4038 wrote: post #150

Now that Elon has cleaned house, when is Twitter hiring again? It might be a good opportunity to get in on the new ground floor.

I've sent Elon an email asking for internship, though sadly I can't move to Sanfran and live in the office floor for 12 weeks like @geohot can. Still...

Re: Why Twitter didn’t go down: From a real Twitter SRE

#624

Post by jansan » Tue, Nov 22, 2022, 9:58 AM UTC

bearmode wrote: post #586

jansan wrote: post #320

Earlier quoted context omitted.

Did't you see the "Day in my life at the Twitter office video!"? https://www.tiktok.com/@realpankhilpatel/video/7159187292631... Normal people don't have vacations like that.

There is nothing here that other big tech companies don't have. To attract the best, they spend a shitload on perks and benefits.

Looks like they are not only attracting the best, but also a bunch of freeloaders.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#625

Post by TMWNN » Tue, Nov 22, 2022, 10:00 AM UTC

hairofadog wrote: post #149

throwawaylinux wrote: post #109

Earlier quoted context omitted.

I think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.

It doesn’t bother me because it’s lascivious, it bothers me because it’s the comedy equivalent of fart noises. It was one thing when he was just a user of Twitter, but now he’s in charge, and it seems like a tone is being set that’s not really the vibe I’m into. Whatever floats your boat, I guess.

>It was one thing when he was just a user of Twitter, but now he’s in charge, and it seems like a tone is being set that’s not really the vibe I’m into.

What's wrong with the new owner of Twitter using Twitter in exactly the same way that a lot of other Twitter users use it for, memes and jokes?

I find that rather refreshing, actually. It's a variant of eating one's own dogfood.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#626

Post by logistark » Tue, Nov 22, 2022, 10:00 AM UTC

mirzap wrote: post #402

poulsbohemian wrote: post #87

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…

Quoted post unavailable.

Actually there are some claims in recent code reviews at Twitter that he even doesn't know how to use a for loop. https://twitter.com/JadeMasterMath/status/159280343165136896...

Re: Why Twitter didn’t go down: From a real Twitter SRE

#627

Post by bartvk » Tue, Nov 22, 2022, 10:01 AM UTC

kortilla wrote: post #555

bartvk wrote: post #471

Earlier quoted context omitted.

> this reminds me of old German industry where people retire in the place they started What is wrong with that?

Depends on if you want innovation or stability. For innovation, people building fiefdoms over decades of political maneuvering is terribly destructive to change. Very few industries require constant learning for the business to compete so a highly tenured employee likely hasn’t learned anything new beyond minute process changes for 10+ years. Once people are ossified into a role like that, they will do anything and e…

That is actually not my experience. I worked at a scientific institute on space cameras, and since projects easily last 10 years from design to launch, we have a lot of older people.

Some were resisting change. Some were embracing it. All in all, I had a great time and gained a lot of respect. I love working in teams with all ages.

Your sentence "once people are ossified..." feels like a generalization to me.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#628

Post by whizzter » Tue, Nov 22, 2022, 10:01 AM UTC

dfghbbjiiugb wrote: post #503

mjr00 wrote: post #451

Earlier quoted context omitted.

It's crazy to think about, but many people who use and build software today, including HN readers/commenters, are young enough to have only been exposed to the SaaS, cloud-first era, where software built with microservices deployed from CI/CD systems multiple times per day is just the way things are done. You're totally right; if you don't make changes to the software, it's unlikely to spontaneously stop working, esp…

Your broad point is obviously correct (most outages are caused by code or config changes) but there are still classes of failures that can happen without any real changes, like various performance degradations (maybe your table grows too large) or occasional catastrophic failures from things like disk space or id overflow or something. There's also the stability of third party systems: forced deprecations, security E…

A few of us remembers the time Slashdots 24-bit comment table ID keys overflowed, was a fun couple of days.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#629

Post by shapefrog » Tue, Nov 22, 2022, 10:02 AM UTC

mjr00 wrote: post #451

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

It's crazy to think about, but many people who use and build software today, including HN readers/commenters, are young enough to have only been exposed to the SaaS, cloud-first era, where software built with microservices deployed from CI/CD systems multiple times per day is just the way things are done. You're totally right; if you don't make changes to the software, it's unlikely to spontaneously stop working, esp…

> The cloud/SaaS approach

You dont have to push constant updates for a cloud / SaaS product - many chose to - but ultimately you dont have to.

A year of 'no new features' should be something customers and vendor alike benefit from.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#630

Post by Dave3of5 » Tue, Nov 22, 2022, 10:02 AM UTC

larsnystrom wrote: post #578

Dave3of5 wrote: post #571

I don't think it will fail on a technical level. As this article says lots of engineering has gone in to make the thing pretty resilient. I would also say that there are still enough engineers who work there who can figure out what's gone wrong and "turn it on and off again" or w/e makes it splutter back to life. In terms of changes to the Platform ditto. It's not difficult to make these changes that a team of 100's…

Are you saying Twitter is paying ~$1.5B in interest rates every year?

Yes that's the approximate amount they have to pay due to the way he's done the financing of the deal. Keep in mind that the current interest is at a similar amount to there revenues from last year.

That's the reason to deeply cut expenses and to try and make more money. He could probably have serviced that debt if everything just kept going as normal (no big ad spend cut) and he had fired everyone but that's an impossible scenario.