Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

291–300 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#291
post #95
post #79

Earlier quoted context omitted.

> Twitter added 1.6M daily active users this past week, another all-time high https://twitter.com/elonmusk/status/1594865247323660290

Want to bet that’s not unconnected to sacking the people who tried to block bots?

Sure. Who do you have on the inside who can publicly verify that you're correct?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#292
post #259

Earlier quoted context omitted.

I did see a lot of politically-motivated salivating which I thought was woefully premature. That said, I see 2 outcomes now: 1. Generally, with large, complex systems like this, everything works, until it doesn't. All the big boys have major outages periodically. I just can't fathom how Twitter is going to handle the eventual certainty of a major outage when, as the author notes, in some cases there are teams that ha…

Generally agreed. However keep in mind that bankruptcy is a purely financial event and doesn't necessarily have to have any impact on operations. Airlines are notoriously for going bankrupt regularly.

Depends.

"Chapter 7 of Title 11 of the United States Code (Bankruptcy Code) governs the process of liquidation under the bankruptcy laws of the United States, in contrast to Chapters 11 and 13, which govern the process of reorganization of a debtor. Chapter 7 is the most common form of bankruptcy in the United States."

https://en.wikipedia.org/wiki/Chapter_7,_Title_11,_United_St...

Re: Why Twitter didn’t go down: From a real Twitter SRE

#294
post #16

Earlier quoted context omitted.

The problem is that it’s run by someone who overpaid badly and needs to come up with significantly more money to pay for the debt he saddled the company with, and then his actions seriously disrupted ad revenue. That puts him more in the PE playbook of cutting costs as deeply as possible even at the extent of long-term growth. Unlike his other companies there isn’t strong government support to drive business for Twit…

Quoted post unavailable.

Ridiculous. Get a hold of yourself.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#295
post #283

Let don't kid ourselves here, even if Twitter crashes and needs a few weeks or even a few months to recover it doesn't matter. People who leaved will comeback again, after a year everything will be like nothing ever happened. And even if Twitter lost 50% of it users and revenues, it is still more profitable than have to pay the salary of 7000 people.

More profitable than what? It was and is losing money, plus now it also has some serious debt on top. With staff that actually know their shit (and a sane CEO), it may be possible to turn it profitable, but I don’t think it will happen as things currently stand.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#296

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

Tesla has 99k employees

Re: Why Twitter didn’t go down: From a real Twitter SRE

#297

I know most of my comment is going to be redundant, but I need to say it in order to point to it an year or two from now as an "I told you so!" Twitter is going to be here for a long long time. It's not going to suddenly shut down, it's not going to slowly decline, there is not going to be mass abandonment. Elon might be cocky, but at the end of the day he is a successful businessman. He didn't just sent out that loy…

What kind of woke ideology exactly? Sorry not familiar the current climate in USA.

It's a form of progressive ideology that is largely based on symbolic actions and relies on mechanisms like shame and shunning, struggle sessions, etc.

So instead of actually doing any sort of real tangible action to solve problems, you mostly spend your time denouncing your enemies, including the ones in your own ranks that are not pure enough.

It's very much associated with the left-wing parties in many Western countries, who have for the most part traded their historical defense of working classes for this new brand of performative progressive politics.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#298
post #281

Did the OP confirm Elon theory that most of the stuff is not needed? > For four of those years I was the sole SRE for the Cache team. There was a few before me, and the whole team I worked with, where a bunch came and went. But for four years I was the one responsible for automation, reliability and operations in the team. I designed and implemented most of the tools that are keeping it running so I think I’m qualifi…

One SRE, many SWE. Also have fun asking someone to be permanently oncall with one person on the team.

The cache clusters size are also described here for anyone who wants a good technical read over speculation. https://www.usenix.org/system/files/osdi20-yang.pdf

Re: Why Twitter didn’t go down: From a real Twitter SRE

#299

Earlier quoted context omitted.

To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...

Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to. So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building . But eventually people are going to start turning on stoves again… an…

I was just about to make the same analogy. The chance that anything will break when there are no new commits is way lower

Re: Why Twitter didn’t go down: From a real Twitter SRE

#300

Earlier quoted context omitted.

Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to. So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building . But eventually people are going to start turning on stoves again… an…

This is correct, but also: a sudden decline in maintenance is a kind of change in its own right. Even automated processes have humans in the loop and manual sign-offs; there's always some cronjob or short-lived certificate somewhere that a human was dutifully maintaining. Those things aren't going to fail any sooner than they would have anyways, but they're going to fail a lot harder due to the loss of institutional…

Welp, their TLS cert expires second week of the new year. I really hope for them that’s an automated process.
Post reply on HN